Events

Time	Namespace	Component	RelatedObject	Reason	Message
06:16:23	openshift-ingress		payload-pre-processing-7b98bb678c-8b96t	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-8b96t to ip-10-0-130-237.ec2.internal
06:14:56	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-vg7xv	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vg7xv to ip-10-0-134-97.ec2.internal
06:14:09	opendatahub		maas-controller-786557dc8d-dzrd7	Scheduled	Successfully assigned opendatahub/maas-controller-786557dc8d-dzrd7 to ip-10-0-134-97.ec2.internal
06:02:48	opendatahub		llmisvc-controller-manager-869d5dcc5d-x67dq	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-x67dq to ip-10-0-133-6.ec2.internal
06:15:44	opendatahub		llmisvc-controller-manager-6466c554d8-tzvrh	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-tzvrh to ip-10-0-133-6.ec2.internal
06:01:52	opendatahub		kserve-module-controller-manager-7566cf5fc8-p64k4	Scheduled	Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-p64k4 to ip-10-0-134-97.ec2.internal
06:02:48	opendatahub		kserve-controller-manager-776cc66d47-2qkl8	Scheduled	Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-2qkl8 to ip-10-0-133-6.ec2.internal
06:01:33	cert-manager		cert-manager-cainjector-5c57bbd4fc-89sg5	Scheduled	Successfully assigned cert-manager/cert-manager-cainjector-5c57bbd4fc-89sg5 to ip-10-0-133-6.ec2.internal
06:02:48	opendatahub		model-serving-api-75bd89fdfc-n84rd	Scheduled	Successfully assigned opendatahub/model-serving-api-75bd89fdfc-n84rd to ip-10-0-130-237.ec2.internal
06:16:20	opendatahub		ai-gateway-controller-7d85675f98-htkmw	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-7d85675f98-htkmw to ip-10-0-130-237.ec2.internal
06:02:48	opendatahub		odh-model-controller-769875db56-f9gvd	Scheduled	Successfully assigned opendatahub/odh-model-controller-769875db56-f9gvd to ip-10-0-130-237.ec2.internal
06:16:33	opendatahub		ai-gateway-controller-77744b856d-vd4z7	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-77744b856d-vd4z7 to ip-10-0-130-237.ec2.internal
06:19:47	opendatahub		test-curl-13239-2cb656	Scheduled	Successfully assigned opendatahub/test-curl-13239-2cb656 to ip-10-0-134-97.ec2.internal
06:01:21	openshift-jobset-operator		jobset-operator-674cd59dc5-7vs9t	Scheduled	Successfully assigned openshift-jobset-operator/jobset-operator-674cd59dc5-7vs9t to ip-10-0-130-237.ec2.internal
06:19:52	opendatahub		test-curl-13239-30c00f	Scheduled	Successfully assigned opendatahub/test-curl-13239-30c00f to ip-10-0-134-97.ec2.internal
06:16:58	llm		e2e-distinct-simulated-kserve-7b998b79d9-jtfk9	Scheduled	Successfully assigned llm/e2e-distinct-simulated-kserve-7b998b79d9-jtfk9 to ip-10-0-130-237.ec2.internal
06:20:03	openshift-ingress		payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm to ip-10-0-134-97.ec2.internal
06:19:54	opendatahub		test-curl-13239-bc6e7b	Scheduled	Successfully assigned opendatahub/test-curl-13239-bc6e7b to ip-10-0-134-97.ec2.internal
06:21:15	opendatahub		test-curl-13239-c2c87d	Scheduled	Successfully assigned opendatahub/test-curl-13239-c2c87d to ip-10-0-134-97.ec2.internal
06:14:14	opendatahub		usage-logs-tenancy-proxy-9c5d49bcd-np5fd	Scheduled	Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-np5fd to ip-10-0-134-97.ec2.internal
06:02:13	openshift-jobset-operator		jobset-controller-manager-64fcbccbf4-4g7p7	Scheduled	Successfully assigned openshift-jobset-operator/jobset-controller-manager-64fcbccbf4-4g7p7 to ip-10-0-130-237.ec2.internal
06:17:13	llm		premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w	Scheduled	Successfully assigned llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w to ip-10-0-134-97.ec2.internal
05:50:50	openshift-ingress		router-default-575669d575-qgfnp	FailedScheduling	no nodes available to schedule pods
06:01:15	opendatahub-operator-system		opendatahub-operator-controller-manager-bc995478c-4nktx	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-bc995478c-4nktx to ip-10-0-130-237.ec2.internal
06:01:48	openshift-ingress		data-science-gateway-data-science-gateway-class-6644c85865kxz26	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865kxz26 to ip-10-0-133-6.ec2.internal
06:20:23	openshift-ingress		payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2 to ip-10-0-134-97.ec2.internal
06:17:02	llm		e2e-trlp-test-simulated-kserve-69b588897d-m7vr7	Scheduled	Successfully assigned llm/e2e-trlp-test-simulated-kserve-69b588897d-m7vr7 to ip-10-0-134-97.ec2.internal
06:20:03	openshift-ingress		payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7 to ip-10-0-134-97.ec2.internal
06:12:28	kuadrant-system		authorino-56c6c9ddd5-smszd	Scheduled	Successfully assigned kuadrant-system/authorino-56c6c9ddd5-smszd to ip-10-0-134-97.ec2.internal
06:16:23	openshift-ingress		payload-processing-5f78687d5b-g9mbb	Scheduled	Successfully assigned openshift-ingress/payload-processing-5f78687d5b-g9mbb to ip-10-0-130-237.ec2.internal
06:20:23	openshift-ingress		payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4 to ip-10-0-134-97.ec2.internal
06:17:07	llm		echo-simulator-kserve-64b6f8f96c-zrsnf	Scheduled	Successfully assigned llm/echo-simulator-kserve-64b6f8f96c-zrsnf to ip-10-0-134-97.ec2.internal
06:16:59	llm		e2e-embedding-simulated-kserve-67bf594cd6-bdjld	Scheduled	Successfully assigned llm/e2e-embedding-simulated-kserve-67bf594cd6-bdjld to ip-10-0-134-97.ec2.internal
06:00:39	openshift-ingress		openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md to ip-10-0-133-6.ec2.internal
06:14:41	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-wxs2l	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-wxs2l to ip-10-0-130-237.ec2.internal
06:17:05	llm		e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw	Scheduled	Successfully assigned llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw to ip-10-0-130-237.ec2.internal
06:01:15	opendatahub-operator-system		opendatahub-operator-controller-manager-bc995478c-g7v9f	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-bc995478c-g7v9f to ip-10-0-133-6.ec2.internal
06:12:27	kuadrant-system		authorino-57f7f8659f-d9vrt	Scheduled	Successfully assigned kuadrant-system/authorino-57f7f8659f-d9vrt to ip-10-0-130-237.ec2.internal
06:01:43	cert-manager		cert-manager-webhook-5dbbbf75cc-4xdrk	Scheduled	Successfully assigned cert-manager/cert-manager-webhook-5dbbbf75cc-4xdrk to ip-10-0-130-237.ec2.internal
06:14:09	opendatahub		maas-controller-5d8f8869ff-zz8zz	Scheduled	Successfully assigned opendatahub/maas-controller-5d8f8869ff-zz8zz to ip-10-0-133-6.ec2.internal
06:12:26	kuadrant-system		kuadrant-console-plugin-66c4fb575d-jgv6q	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-66c4fb575d-jgv6q to ip-10-0-133-6.ec2.internal
06:12:26	kuadrant-system		limitador-limitador-74dfcfb894-65kg5	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-74dfcfb894-65kg5 to ip-10-0-134-97.ec2.internal
06:15:11	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-tvnpg	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tvnpg to ip-10-0-133-6.ec2.internal
06:17:09	llm		facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc	Scheduled	Successfully assigned llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc to ip-10-0-130-237.ec2.internal
06:18:10	llm		grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	Scheduled	Successfully assigned llm/grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6 to ip-10-0-133-6.ec2.internal
06:16:54	llm		e2e-distinct-2-simulated-kserve-cbd4f5586-b979m	Scheduled	Successfully assigned llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b979m to ip-10-0-130-237.ec2.internal
06:15:41	kuadrant-system		authorino-65d9b845b6-lw9wl	Scheduled	Successfully assigned kuadrant-system/authorino-65d9b845b6-lw9wl to ip-10-0-134-97.ec2.internal
06:14:41	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-ktwxr	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ktwxr to ip-10-0-134-97.ec2.internal
06:01:47	kuadrant-system		limitador-operator-controller-manager-5778bb4449-tmlk9	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-5778bb4449-tmlk9 to ip-10-0-133-6.ec2.internal
06:01:15	opendatahub-operator-system		opendatahub-operator-controller-manager-bc995478c-hmhqm	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-bc995478c-hmhqm to ip-10-0-134-97.ec2.internal
06:01:51	cert-manager		cert-manager-597b95cb84-w5ldr	Scheduled	Successfully assigned cert-manager/cert-manager-597b95cb84-w5ldr to ip-10-0-130-237.ec2.internal
06:14:56	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-hqwv6	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-hqwv6 to ip-10-0-130-237.ec2.internal
06:14:56	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-hfcrg	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-hfcrg to ip-10-0-133-6.ec2.internal
06:14:41	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-gpwmx	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-gpwmx to ip-10-0-133-6.ec2.internal
06:12:08	kuadrant-system		kuadrant-operator-controller-manager-678d87766f-mffvf	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mffvf to ip-10-0-133-6.ec2.internal
06:14:57	kuadrant-system		authorino-6889cbd8c4-gv47j	Scheduled	Successfully assigned kuadrant-system/authorino-6889cbd8c4-gv47j to ip-10-0-134-97.ec2.internal
06:15:11	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-dztcv	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-dztcv to ip-10-0-134-97.ec2.internal
06:14:56	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-dx86v	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-dx86v to ip-10-0-130-237.ec2.internal
06:14:57	kuadrant-system		authorino-6f6944bcdd-sx68x	Scheduled	Successfully assigned kuadrant-system/authorino-6f6944bcdd-sx68x to ip-10-0-134-97.ec2.internal
06:01:44	kuadrant-system		dns-operator-controller-manager-8557cbdb84-52tg6	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-8557cbdb84-52tg6 to ip-10-0-133-6.ec2.internal
06:22:34	llm		grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh	Scheduled	Successfully assigned llm/grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh to ip-10-0-133-6.ec2.internal
06:00:39	openshift-ingress		maas-default-gateway-openshift-default-596dc9545f-dsvvx	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-dsvvx to ip-10-0-133-6.ec2.internal
06:22:36	llm		grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x	Scheduled	Successfully assigned llm/grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x to ip-10-0-134-97.ec2.internal
06:01:47	openshift-ingress		kube-auth-proxy-5f7b849775-kvct7	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-5f7b849775-kvct7 to ip-10-0-130-237.ec2.internal
06:01:47	openshift-ingress		kube-auth-proxy-5f7b849775-5gcnq	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-5f7b849775-5gcnq to ip-10-0-134-97.ec2.internal
06:19:29	llm		grid-inv-abb4b23f-model-kserve-c995bc848-gztk2	Scheduled	Successfully assigned llm/grid-inv-abb4b23f-model-kserve-c995bc848-gztk2 to ip-10-0-134-97.ec2.internal
06:02:46	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-c46hh	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c46hh to ip-10-0-134-97.ec2.internal
06:17:30	kuadrant-system		authorino-75545cc4b5-k75xt	Scheduled	Successfully assigned kuadrant-system/authorino-75545cc4b5-k75xt to ip-10-0-134-97.ec2.internal
06:12:28	kuadrant-system		limitador-limitador-7c9b87d574-2nzdq	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-7c9b87d574-2nzdq to ip-10-0-133-6.ec2.internal
05:56:24	openshift-ingress		router-default-575669d575-qgfnp	Scheduled	Successfully assigned openshift-ingress/router-default-575669d575-qgfnp to ip-10-0-133-6.ec2.internal
06:00:33	openshift-ingress		istiod-openshift-gateway-7876bfcb74-thkxp	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-thkxp to ip-10-0-134-97.ec2.internal
06:19:34	llm		grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk	Scheduled	Successfully assigned llm/grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk to ip-10-0-133-6.ec2.internal
06:01:35	kuadrant-system		authorino-operator-7b7bb4bc77-ntllc	Scheduled	Successfully assigned kuadrant-system/authorino-operator-7b7bb4bc77-ntllc to ip-10-0-133-6.ec2.internal
06:18:02	llm		grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	Scheduled	Successfully assigned llm/grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd to ip-10-0-134-97.ec2.internal
06:14:57	kuadrant-system		authorino-7fd577446d-rggfh	Scheduled	Successfully assigned kuadrant-system/authorino-7fd577446d-rggfh to ip-10-0-134-97.ec2.internal
06:20:17	openshift-ingress		e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt to ip-10-0-130-237.ec2.internal
05:50:52	openshift-ingress		router-default-575669d575-qgfnp	FailedScheduling	no nodes available to schedule pods
06:01:42	kuadrant-system		kuadrant-operator-controller-manager-678d87766f-dsdlv	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dsdlv to ip-10-0-133-6.ec2.internal
06:19:57	openshift-ingress		e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n to ip-10-0-130-237.ec2.internal
05:50:21	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-575669d575 from 0 to 1
05:50:23	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
05:50:23	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
05:50:27	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
05:50:32	openshift-ingress	replicaset-controller	router-default-575669d575	FailedCreate	Error creating: pods "router-default-575669d575-" is forbidden: error fetching namespace "openshift-ingress": unable to find annotation openshift.io/sa.scc.uid-range
05:50:50	openshift-ingress	replicaset-controller	router-default-575669d575	SuccessfulCreate	Created pod: router-default-575669d575-qgfnp
05:56:05 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
05:56:32 (x5)	openshift-ingress	kubelet	router-default-575669d575-qgfnp	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
05:56:32 (x5)	openshift-ingress	kubelet	router-default-575669d575-qgfnp	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
05:56:41 (x4)	openshift-ingress	kubelet	router-default-575669d575-qgfnp	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
05:56:41 (x4)	openshift-ingress	kubelet	router-default-575669d575-qgfnp	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
05:56:45	openshift-ingress	kubelet	router-default-575669d575-qgfnp	Started	Container started
05:56:45	openshift-ingress	kubelet	router-default-575669d575-qgfnp	Created	Container created
05:56:45	openshift-ingress	kubelet	router-default-575669d575-qgfnp	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
05:56:45	openshift-ingress	multus	router-default-575669d575-qgfnp	AddedInterface	Add eth0 [10.132.0.13/23] from ovn-kubernetes
06:00:32	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
06:00:33	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-thkxp
06:00:33	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
06:00:34	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-thkxp	AddedInterface	Add eth0 [10.133.0.17/23] from ovn-kubernetes
06:00:34	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
06:00:37	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	Started	Container started
06:00:37	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	Created	Container created
06:00:37	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.08s (3.08s including waiting). Image size: 185476477 bytes.
06:00:39	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-dsvvx	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
06:00:39	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-596dc9545f from 0 to 1
06:00:39	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-dsvvx
06:00:39	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
06:00:39	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-5cb6869599 from 0 to 1
06:00:39	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-5cb6869599	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md
06:00:39 (x2)	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:00:39 (x2)	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:00:40	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
06:00:40	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
06:00:40	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
06:00:43	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
06:00:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 7.866s (7.866s including waiting). Image size: 183555333 bytes.
06:00:48	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.306s (8.306s including waiting). Image size: 183555333 bytes.
06:00:48	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md	Created	Container created
06:00:48	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cb6869599-pn2md	Started	Container started
06:00:54	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:00:54	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:00:54	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:00:54	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:01:09	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:01:09	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:01:09	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:01:09	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:01:14 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsUnknown	requirements not yet checked
06:01:14 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsNotMet	one or more requirements couldn't be found
06:01:15	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	AllRequirementsMet	all requirements found, attempting install
06:01:15	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-bc995478c	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-bc995478c-4nktx
06:01:15	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-bc995478c	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-bc995478c-g7v9f
06:01:15	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-bc995478c	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-bc995478c-hmhqm
06:01:15	opendatahub-operator-system	deployment-controller	opendatahub-operator-controller-manager	ScalingReplicaSet	Scaled up replica set opendatahub-operator-controller-manager-bc995478c from 0 to 3
06:01:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
06:01:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
06:01:16	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-bc995478c-4nktx	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
06:01:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
06:01:16	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-bc995478c-hmhqm	AddedInterface	Add eth0 [10.133.0.18/23] from ovn-kubernetes
06:01:16	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-bc995478c-g7v9f	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
06:01:17	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	Webhook install failed: conversionWebhook not ready
06:01:18 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability.
06:01:18	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsUnknown	requirements not yet checked
06:01:18 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsNotMet	one or more requirements couldn't be found
06:01:18	opendatahub-operator-system	operator-lifecycle-manager	install-xjmnm	AppliedWithWarnings	1 warning(s) generated during installation of operator "opendatahub-operator.v3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1
06:01:18 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	waiting for install components to report healthy
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Created	Container created
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.774s (2.774s including waiting). Image size: 273709574 bytes.
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Started	Container started
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Started	Container started
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Created	Container created
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.768s (2.768s including waiting). Image size: 273709574 bytes.
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.702s (2.702s including waiting). Image size: 273709574 bytes.
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Created	Container created
06:01:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Started	Container started
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 337ms (337ms including waiting). Image size: 273709574 bytes.
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Created	Container created
06:01:20	opendatahub-operator-system	opendatahub-operator-controller-manager-bc995478c-g7v9f_0b17a0af-597c-4f67-98e1-e500e578ad8b	07ed84f7.opendatahub.io	LeaderElection	opendatahub-operator-controller-manager-bc995478c-g7v9f_0b17a0af-597c-4f67-98e1-e500e578ad8b became leader
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Started	Container started
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Created	Container created
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 331ms (331ms including waiting). Image size: 273709574 bytes.
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-4nktx	Started	Container started
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 273ms (273ms including waiting). Image size: 273709574 bytes.
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-hmhqm	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Started	Container started
06:01:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-bc995478c-g7v9f	Created	Container created
06:01:21	openshift-jobset-operator	replicaset-controller	jobset-operator-674cd59dc5	SuccessfulCreate	Created pod: jobset-operator-674cd59dc5-7vs9t
06:01:21 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallWaiting	installing: waiting for deployment jobset-operator to become ready: deployment "jobset-operator" not available: Deployment does not have minimum availability.
06:01:21	openshift-jobset-operator	multus	jobset-operator-674cd59dc5-7vs9t	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
06:01:21 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	waiting for install components to report healthy
06:01:21	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	AllRequirementsMet	all requirements found, attempting install
06:01:21	openshift-jobset-operator	deployment-controller	jobset-operator	ScalingReplicaSet	Scaled up replica set jobset-operator-674cd59dc5 from 0 to 1
06:01:21	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-7vs9t	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e"
06:01:24	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-7vs9t	Started	Container started
06:01:24	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-7vs9t	Created	Container created
06:01:24	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-7vs9t	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e" in 2.485s (2.485s including waiting). Image size: 224874686 bytes.
06:01:25	openshift-jobset-operator	openshift-jobset-operator	openshift-jobset-operator-lock	LeaderElection	jobset-operator-674cd59dc5-7vs9t_22832c76-4e25-4213-a17b-d9054bfce78a became leader
06:01:25 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	install strategy completed with no errors
06:01:31	cert-manager	deployment-controller	cert-manager	ScalingReplicaSet	Scaled up replica set cert-manager-597b95cb84 from 0 to 1
06:01:31	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
06:01:31	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
06:01:32 (x8)	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	FailedCreate	Error creating: pods "cert-manager-cainjector-5c57bbd4fc-" is forbidden: error looking up service account cert-manager/cert-manager-cainjector: serviceaccount "cert-manager-cainjector" not found
06:01:32	cert-manager	deployment-controller	cert-manager-cainjector	ScalingReplicaSet	Scaled up replica set cert-manager-cainjector-5c57bbd4fc from 0 to 1
06:01:33	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	install strategy completed with no errors
06:01:33	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	SuccessfulCreate	Created pod: cert-manager-cainjector-5c57bbd4fc-89sg5
06:01:34	cert-manager	multus	cert-manager-cainjector-5c57bbd4fc-89sg5	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
06:01:34	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-89sg5	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
06:01:35	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
06:01:35	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
06:01:35	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
06:01:35	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-7b7bb4bc77 from 0 to 1
06:01:35	kuadrant-system	replicaset-controller	authorino-operator-7b7bb4bc77	SuccessfulCreate	Created pod: authorino-operator-7b7bb4bc77-ntllc
06:01:36	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
06:01:36	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
06:01:36	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-ntllc	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589"
06:01:36	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-89sg5	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 2.471s (2.471s including waiting). Image size: 309650901 bytes.
06:01:36	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-89sg5	Created	Container created
06:01:36	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-89sg5	Started	Container started
06:01:36	kuadrant-system	multus	authorino-operator-7b7bb4bc77-ntllc	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
06:01:38	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-ntllc	Started	Container started
06:01:38	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-ntllc	Created	Container created
06:01:38	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-ntllc	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589" in 2.44s (2.44s including waiting). Image size: 167738465 bytes.
06:01:39	kuadrant-system	authorino-operator-7b7bb4bc77-ntllc_2b59cf62-e937-4c96-a8d3-2f0a3d55194c	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-7b7bb4bc77-ntllc_2b59cf62-e937-4c96-a8d3-2f0a3d55194c became leader
06:01:41 (x12)	cert-manager	replicaset-controller	cert-manager-597b95cb84	FailedCreate	Error creating: pods "cert-manager-597b95cb84-" is forbidden: error looking up service account cert-manager/cert-manager: serviceaccount "cert-manager" not found
06:01:42	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
06:01:42	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-678d87766f	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-678d87766f-dsdlv
06:01:42	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-678d87766f from 0 to 1
06:01:42	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-dsdlv	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e"
06:01:42	kuadrant-system	multus	kuadrant-operator-controller-manager-678d87766f-dsdlv	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
06:01:42	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
06:01:43	cert-manager	deployment-controller	cert-manager-webhook	ScalingReplicaSet	Scaled up replica set cert-manager-webhook-5dbbbf75cc from 0 to 1
06:01:43	cert-manager	replicaset-controller	cert-manager-webhook-5dbbbf75cc	SuccessfulCreate	Created pod: cert-manager-webhook-5dbbbf75cc-4xdrk
06:01:44	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
06:01:44	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
06:01:44	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-8557cbdb84 from 0 to 1
06:01:44	kuadrant-system	replicaset-controller	dns-operator-controller-manager-8557cbdb84	SuccessfulCreate	Created pod: dns-operator-controller-manager-8557cbdb84-52tg6
06:01:44	cert-manager	multus	cert-manager-webhook-5dbbbf75cc-4xdrk	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
06:01:44	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-4xdrk	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
06:01:44	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
06:01:45	kuadrant-system	multus	dns-operator-controller-manager-8557cbdb84-52tg6	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
06:01:45	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-52tg6	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a"
06:01:45	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
06:01:45	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
06:01:46	kuadrant-system	kuadrant-operator-controller-manager-678d87766f-dsdlv_7504f5a7-c14f-4de6-b1eb-8e0ed64f9dcb	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-678d87766f-dsdlv_7504f5a7-c14f-4de6-b1eb-8e0ed64f9dcb became leader
06:01:46	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-dsdlv	Started	Container started
06:01:46	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-dsdlv	Created	Container created
06:01:46	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-dsdlv	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" in 3.662s (3.662s including waiting). Image size: 424244516 bytes.
06:01:46	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
06:01:47	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-4xdrk	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 3.104s (3.104s including waiting). Image size: 309650901 bytes.
06:01:47	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-52tg6	Started	Container started
06:01:47	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-4xdrk	Created	Container created
06:01:47	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-5778bb4449 from 0 to 1
06:01:47	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-52tg6	Created	Container created
06:01:47	openshift-ingress	replicaset-controller	kube-auth-proxy-5f7b849775	SuccessfulCreate	Created pod: kube-auth-proxy-5f7b849775-5gcnq
06:01:47	openshift-ingress	replicaset-controller	kube-auth-proxy-5f7b849775	SuccessfulCreate	Created pod: kube-auth-proxy-5f7b849775-kvct7
06:01:47	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-5f7b849775 from 0 to 2
06:01:47	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
06:01:47	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-5778bb4449	SuccessfulCreate	Created pod: limitador-operator-controller-manager-5778bb4449-tmlk9
06:01:47	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
06:01:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceAccountCreated	Created ServiceAccount/jobset-controller-manager -n openshift-jobset-operator because it was missing
06:01:47	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceCreated	Created Service/jobset-webhook-service -n openshift-jobset-operator because it was missing
06:01:47	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-4xdrk	Started	Container started
06:01:47	kuadrant-system	multus	limitador-operator-controller-manager-5778bb4449-tmlk9	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
06:01:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-manager-role because it was missing
06:01:47	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-tmlk9	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e"
06:01:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-metrics-reader because it was missing
06:01:47	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-52tg6	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a" in 1.604s (1.604s including waiting). Image size: 190804149 bytes.
06:01:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-proxy-role because it was missing
06:01:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-manager-secrets-role -n openshift-jobset-operator because it was missing
06:01:48	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-6644c85865kxz26	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
06:01:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding because it was missing
06:01:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding because it was missing
06:01:48	kuadrant-system	dns-operator-controller-manager-8557cbdb84-52tg6_5751f2c6-74e5-4f1d-a3fd-123b317fdcc9	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-8557cbdb84-52tg6_5751f2c6-74e5-4f1d-a3fd-123b317fdcc9 became leader
06:01:48	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-6644c85865	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-6644c85865kxz26
06:01:48	openshift-ingress	multus	kube-auth-proxy-5f7b849775-5gcnq	AddedInterface	Add eth0 [10.133.0.19/23] from ovn-kubernetes
06:01:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-leader-election-role -n openshift-jobset-operator because it was missing
06:01:48	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-5gcnq	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
06:01:48	openshift-ingress	multus	kube-auth-proxy-5f7b849775-kvct7	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
06:01:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6644c85865kxz26	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:01:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-manager-rolebinding because it was missing
06:01:48	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-kvct7	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
06:01:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6644c85865kxz26	Created	Container created
06:01:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6644c85865kxz26	Started	Container started
06:01:48	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-6644c85865 from 0 to 1
06:01:49	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-tmlk9	Created	Container created
06:01:49	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding -n openshift-jobset-operator because it was missing
06:01:49	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-manager-secrets-rolebinding -n openshift-jobset-operator because it was missing
06:01:49	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-tmlk9	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e" in 1.381s (1.381s including waiting). Image size: 177967631 bytes.
06:01:49 (x3)	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	IssuerCreateFailed	Failed to create Issuer.cert-manager.io/jobset-selfsigned-issuer -n openshift-jobset-operator: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
06:01:49	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-tmlk9	Started	Container started
06:01:49	kuadrant-system	limitador-operator-controller-manager-5778bb4449-tmlk9_7c4da9e9-7d15-4322-b722-52038cec7172	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-5778bb4449-tmlk9_7c4da9e9-7d15-4322-b722-52038cec7172 became leader
06:01:49	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceCreated	Created Service/jobset-controller-manager-metrics-service -n openshift-jobset-operator because it was missing
06:01:50	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
06:01:50	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-5gcnq	Created	Container created
06:01:50	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-kvct7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.255s (2.255s including waiting). Image size: 188904677 bytes.
06:01:50	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-kvct7	Created	Container created
06:01:50	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-5gcnq	Started	Container started
06:01:50	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-5gcnq	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.397s (2.397s including waiting). Image size: 188904677 bytes.
06:01:50	openshift-ingress	kubelet	kube-auth-proxy-5f7b849775-kvct7	Started	Container started
06:01:51	cert-manager	replicaset-controller	cert-manager-597b95cb84	SuccessfulCreate	Created pod: cert-manager-597b95cb84-w5ldr
06:01:52	opendatahub	multus	kserve-module-controller-manager-7566cf5fc8-p64k4	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
06:01:52	opendatahub	replicaset-controller	kserve-module-controller-manager-7566cf5fc8	SuccessfulCreate	Created pod: kserve-module-controller-manager-7566cf5fc8-p64k4
06:01:52	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-p64k4	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189"
06:01:52	cert-manager	kubelet	cert-manager-597b95cb84-w5ldr	Started	Container started
06:01:52	cert-manager	kubelet	cert-manager-597b95cb84-w5ldr	Created	Container created
06:01:52	opendatahub	deployment-controller	kserve-module-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1
06:01:52	cert-manager	kubelet	cert-manager-597b95cb84-w5ldr	Pulled	Container image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" already present on machine and can be accessed by the pod
06:01:52	cert-manager	multus	cert-manager-597b95cb84-w5ldr	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
06:01:54	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-p64k4	Created	Container created
06:01:54	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-p64k4	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.538s (1.538s including waiting). Image size: 195729000 bytes.
06:01:54	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-p64k4	Started	Container started
06:01:58	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
06:02:00	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
06:02:02	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:02:02	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:02:08	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	IssuerCreated	Created Issuer.cert-manager.io/jobset-selfsigned-issuer -n openshift-jobset-operator because it was missing
06:02:08	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-serving-cert -n openshift-jobset-operator because it was missing
06:02:08	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-metrics-cert -n openshift-jobset-operator because it was missing
06:02:10	opendatahub	kserve-module-controller-manager-7566cf5fc8-p64k4_67c22098-83a9-4b82-beea-79deddbe835e	kserve-module-controller-manager	LeaderElection	kserve-module-controller-manager-7566cf5fc8-p64k4_67c22098-83a9-4b82-beea-79deddbe835e became leader
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-metrics-cert	Issuing	Issuing certificate as Secret does not exist
06:02:12	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-metrics-cert	Generated	Stored new private key in temporary Secret resource "jobset-metrics-cert-bb4hk"
06:02:12	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-metrics-cert	Requested	Created new CertificateRequest resource "jobset-metrics-cert-1"
06:02:12	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-serving-cert	Requested	Created new CertificateRequest resource "jobset-serving-cert-1"
06:02:12	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-serving-cert	Generated	Stored new private key in temporary Secret resource "jobset-serving-cert-ftlff"
06:02:12	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-serving-cert	Issuing	Issuing certificate as Secret does not exist
06:02:12	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-metrics-cert	Issuing	The certificate has been successfully issued
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	CertificateIssued	Certificate fetched from issuer successfully
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-metrics-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
06:02:12	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceMonitorCreated	Created ServiceMonitor.monitoring.coreos.com/jobset-controller-manager-metrics-monitor -n openshift-jobset-operator because it was missing
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CustomResourceDefinitionCreated	Created CustomResourceDefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io because it was missing
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationUpdated	Updated ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it changed
06:02:13	openshift-jobset-operator	replicaset-controller	jobset-controller-manager-64fcbccbf4	SuccessfulCreate	Created pod: jobset-controller-manager-64fcbccbf4-4g7p7
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationCreated	Created MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it was missing
06:02:13	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-serving-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	DeploymentCreated	Created Deployment.apps/jobset-controller-manager -n openshift-jobset-operator because it was missing
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationUpdated	Updated MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it changed
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ConfigMapCreated	Created ConfigMap/jobset-manager-config -n openshift-jobset-operator because it was missing
06:02:13	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	CertificateIssued	Certificate fetched from issuer successfully
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
06:02:13	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationCreated	Created ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it was missing
06:02:13	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-serving-cert	Issuing	The certificate has been successfully issued
06:02:13	openshift-jobset-operator	deployment-controller	jobset-controller-manager	ScalingReplicaSet	Scaled up replica set jobset-controller-manager-64fcbccbf4 from 0 to 1
06:02:14	openshift-jobset-operator	kubelet	jobset-controller-manager-64fcbccbf4-4g7p7	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987"
06:02:14	openshift-jobset-operator	multus	jobset-controller-manager-64fcbccbf4-4g7p7	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
06:02:16	openshift-jobset-operator	kubelet	jobset-controller-manager-64fcbccbf4-4g7p7	Started	Container started
06:02:16	openshift-jobset-operator	kubelet	jobset-controller-manager-64fcbccbf4-4g7p7	Created	Container created
06:02:16	openshift-jobset-operator	kubelet	jobset-controller-manager-64fcbccbf4-4g7p7	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987" in 2.329s (2.329s including waiting). Image size: 182184678 bytes.
06:02:17	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:02:17	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:02:17	openshift-jobset-operator	jobset-controller-manager-64fcbccbf4-4g7p7_578e97a4-17f0-4596-95eb-ebcbf29ce06d	6d4f6a47.jobset.x-k8s.io	LeaderElection	jobset-controller-manager-64fcbccbf4-4g7p7_578e97a4-17f0-4596-95eb-ebcbf29ce06d became leader
06:02:20 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:02:20 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:02:46	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
06:02:46	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-c46hh
06:02:47	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-c46hh	Created	Container created
06:02:47	opendatahub	replicaset-controller	kserve-controller-manager-776cc66d47	SuccessfulCreate	Created pod: kserve-controller-manager-776cc66d47-2qkl8
06:02:47	opendatahub	deployment-controller	kserve-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1
06:02:47	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-c46hh	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
06:02:47	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-c46hh	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod
06:02:47	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-c46hh	Started	Container started
06:02:48	opendatahub	kubelet	model-serving-api-75bd89fdfc-n84rd	Pulling	Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8"
06:02:48	opendatahub	multus	model-serving-api-75bd89fdfc-n84rd	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
06:02:48	opendatahub	replicaset-controller	odh-model-controller-769875db56	SuccessfulCreate	Created pod: odh-model-controller-769875db56-f9gvd
06:02:48	opendatahub	kubelet	kserve-controller-manager-776cc66d47-2qkl8	Pulling	Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d"
06:02:48	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-thkxp
06:02:48	opendatahub	multus	llmisvc-controller-manager-869d5dcc5d-x67dq	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
06:02:48	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-thkxp	Killing	Stopping container discovery
06:02:48	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-x67dq	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
06:02:48	opendatahub	deployment-controller	odh-model-controller	ScalingReplicaSet	Scaled up replica set odh-model-controller-769875db56 from 0 to 1
06:02:48	opendatahub	kubelet	odh-model-controller-769875db56-f9gvd	Pulling	Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775"
06:02:48	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
06:02:48	opendatahub	multus	odh-model-controller-769875db56-f9gvd	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
06:02:48	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulCreate	Created pod: llmisvc-controller-manager-869d5dcc5d-x67dq
06:02:48	opendatahub	deployment-controller	model-serving-api	ScalingReplicaSet	Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1
06:02:48	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1
06:02:48	opendatahub	replicaset-controller	model-serving-api-75bd89fdfc	SuccessfulCreate	Created pod: model-serving-api-75bd89fdfc-n84rd
06:02:48	opendatahub	multus	kserve-controller-manager-776cc66d47-2qkl8	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
06:02:50	opendatahub	kubelet	model-serving-api-75bd89fdfc-n84rd	Created	Container created
06:02:50	opendatahub	kubelet	model-serving-api-75bd89fdfc-n84rd	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 1.864s (1.864s including waiting). Image size: 184925761 bytes.
06:02:50	opendatahub	kubelet	model-serving-api-75bd89fdfc-n84rd	Started	Container started
06:02:51	opendatahub	kubelet	kserve-controller-manager-776cc66d47-2qkl8	Created	Container created
06:02:51	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-x67dq	Created	Container created
06:02:51	opendatahub	kubelet	kserve-controller-manager-776cc66d47-2qkl8	Started	Container started
06:02:51	opendatahub	kubelet	odh-model-controller-769875db56-f9gvd	Started	Container started
06:02:51	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-x67dq	Started	Container started
06:02:51	opendatahub	odh-model-controller-769875db56-f9gvd_e0a0b5cb-cc3f-4461-89e3-be562ae4f823	odh-model-controller.opendatahub.io	LeaderElection	odh-model-controller-769875db56-f9gvd_e0a0b5cb-cc3f-4461-89e3-be562ae4f823 became leader
06:02:51	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-x67dq	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.957s (2.957s including waiting). Image size: 235100981 bytes.
06:02:51	opendatahub	kubelet	odh-model-controller-769875db56-f9gvd	Created	Container created
06:02:51	opendatahub	kubelet	odh-model-controller-769875db56-f9gvd	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 2.652s (2.652s including waiting). Image size: 236755567 bytes.
06:02:51	opendatahub	kubelet	kserve-controller-manager-776cc66d47-2qkl8	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.887s (2.887s including waiting). Image size: 247676581 bytes.
06:02:52	opendatahub	kserve-controller-manager-776cc66d47-2qkl8_0f206b00-f54f-42fd-bc71-e89c2da67145	kserve-controller-manager-leader-lock	LeaderElection	kserve-controller-manager-776cc66d47-2qkl8_0f206b00-f54f-42fd-bc71-e89c2da67145 became leader
06:02:55	opendatahub	llmisvc-controller-manager-869d5dcc5d-x67dq_0e24bb92-bc6e-4ef3-acaa-73d04e4d4ba6	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-869d5dcc5d-x67dq_0e24bb92-bc6e-4ef3-acaa-73d04e4d4ba6 became leader
06:03:04 (x5)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:03:04 (x5)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:03:19	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:03:19	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:12:08	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-dsdlv	Killing	Stopping container manager
06:12:08	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-678d87766f	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-678d87766f-mffvf
06:12:09	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-mffvf	Created	Container created
06:12:09	kuadrant-system	multus	kuadrant-operator-controller-manager-678d87766f-mffvf	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
06:12:09	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	NeedsReinstall	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
06:12:09	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-mffvf	Started	Container started
06:12:09 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
06:12:09 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
06:12:09 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
06:12:09	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-mffvf	Pulled	Container image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" already present on machine and can be accessed by the pod
06:12:09	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	ComponentUnhealthy	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
06:12:21 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
06:12:25	kuadrant-system	kuadrant-operator-controller-manager-678d87766f-mffvf_2924f1ad-715c-4585-accf-a88ee02428cd	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-678d87766f-mffvf_2924f1ad-715c-4585-accf-a88ee02428cd became leader
06:12:26	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-66c4fb575d from 0 to 1
06:12:26	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-74dfcfb894 from 0 to 1
06:12:26	kuadrant-system	replicaset-controller	kuadrant-console-plugin-66c4fb575d	SuccessfulCreate	Created pod: kuadrant-console-plugin-66c4fb575d-jgv6q
06:12:26	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulCreate	Created pod: limitador-limitador-74dfcfb894-65kg5
06:12:27	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-65kg5	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
06:12:27	kuadrant-system	multus	limitador-limitador-74dfcfb894-65kg5	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
06:12:27	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulCreate	Created pod: authorino-57f7f8659f-d9vrt
06:12:27	kuadrant-system	multus	kuadrant-console-plugin-66c4fb575d-jgv6q	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
06:12:27	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-jgv6q	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044"
06:12:27	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-57f7f8659f from 0 to 1
06:12:28	kuadrant-system	multus	authorino-57f7f8659f-d9vrt	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
06:12:28	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulCreate	Created pod: authorino-56c6c9ddd5-smszd
06:12:28	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-2nzdq	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
06:12:28	kuadrant-system	kubelet	authorino-56c6c9ddd5-smszd	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
06:12:28	kuadrant-system	multus	authorino-56c6c9ddd5-smszd	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
06:12:28	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-56c6c9ddd5 from 0 to 1
06:12:28	kuadrant-system	multus	limitador-limitador-7c9b87d574-2nzdq	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
06:12:28	kuadrant-system	replicaset-controller	limitador-limitador-7c9b87d574	SuccessfulCreate	Created pod: limitador-limitador-7c9b87d574-2nzdq
06:12:28	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-7c9b87d574 from 0 to 1
06:12:28	kuadrant-system	kubelet	authorino-57f7f8659f-d9vrt	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
06:12:29	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-2nzdq	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 954ms (954ms including waiting). Image size: 133379307 bytes.
06:12:29	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-2nzdq	Created	Container created
06:12:29	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-2nzdq	Started	Container started
06:12:30	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-65kg5	Started	Container started
06:12:30	kuadrant-system	kubelet	authorino-57f7f8659f-d9vrt	Started	Container started
06:12:30	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-65kg5	Created	Container created
06:12:30	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-65kg5	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 2.567s (2.567s including waiting). Image size: 133379307 bytes.
06:12:30	kuadrant-system	kubelet	authorino-57f7f8659f-d9vrt	Created	Container created
06:12:30	kuadrant-system	kubelet	authorino-57f7f8659f-d9vrt	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 2.497s (2.497s including waiting). Image size: 204096810 bytes.
06:12:30	kuadrant-system	kubelet	authorino-56c6c9ddd5-smszd	Started	Container started
06:12:30	kuadrant-system	kubelet	authorino-56c6c9ddd5-smszd	Created	Container created
06:12:30	kuadrant-system	kubelet	authorino-56c6c9ddd5-smszd	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.715s (1.715s including waiting). Image size: 204096810 bytes.
06:12:31	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-57f7f8659f from 1 to 0
06:12:31	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulDelete	Deleted pod: authorino-57f7f8659f-d9vrt
06:12:32	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-jgv6q	Started	Container started
06:12:32	kuadrant-system	kubelet	authorino-57f7f8659f-d9vrt	Killing	Stopping container authorino
06:12:32	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-jgv6q	Created	Container created
06:12:32	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-jgv6q	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044" in 4.765s (4.765s including waiting). Image size: 342538866 bytes.
06:12:41	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-74dfcfb894 from 1 to 0
06:12:41	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulDelete	Deleted pod: limitador-limitador-74dfcfb894-65kg5
06:12:41	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-65kg5	Killing	Stopping container limitador
06:14:09	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-786557dc8d from 0 to 1
06:14:09	opendatahub	replicaset-controller	maas-controller-786557dc8d	SuccessfulCreate	Created pod: maas-controller-786557dc8d-dzrd7
06:14:09	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-5d8f8869ff from 0 to 1
06:14:09	opendatahub	multus	maas-controller-786557dc8d-dzrd7	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
06:14:09	opendatahub	replicaset-controller	maas-controller-5d8f8869ff	SuccessfulCreate	Created pod: maas-controller-5d8f8869ff-zz8zz
06:14:09	opendatahub	kubelet	maas-controller-786557dc8d-dzrd7	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:cdee9d2dbe1dd7d41085ef84a6c161da9a59aea946409daa8decfd915e6f47cd"
06:14:10	opendatahub	multus	maas-controller-5d8f8869ff-zz8zz	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
06:14:10	opendatahub	kubelet	maas-controller-5d8f8869ff-zz8zz	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:cdee9d2dbe1dd7d41085ef84a6c161da9a59aea946409daa8decfd915e6f47cd"
06:14:12	opendatahub	kubelet	maas-controller-5d8f8869ff-zz8zz	Created	Container created
06:14:12	opendatahub	kubelet	maas-controller-5d8f8869ff-zz8zz	Started	Container started
06:14:12	opendatahub	kubelet	maas-controller-786557dc8d-dzrd7	Created	Container created
06:14:12	opendatahub	kubelet	maas-controller-5d8f8869ff-zz8zz	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:cdee9d2dbe1dd7d41085ef84a6c161da9a59aea946409daa8decfd915e6f47cd" in 2.385s (2.385s including waiting). Image size: 200069630 bytes.
06:14:12	opendatahub	kubelet	maas-controller-786557dc8d-dzrd7	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:cdee9d2dbe1dd7d41085ef84a6c161da9a59aea946409daa8decfd915e6f47cd" in 2.273s (2.273s including waiting). Image size: 200069630 bytes.
06:14:12	opendatahub	kubelet	maas-controller-786557dc8d-dzrd7	Started	Container started
06:14:13	opendatahub	maas-controller-786557dc8d-dzrd7_faa74117-a6c6-4555-a6fb-b782344d95fd	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-786557dc8d-dzrd7_faa74117-a6c6-4555-a6fb-b782344d95fd became leader
06:14:14	opendatahub	replicaset-controller	usage-logs-tenancy-proxy-9c5d49bcd	SuccessfulCreate	Created pod: usage-logs-tenancy-proxy-9c5d49bcd-np5fd
06:14:14	opendatahub	multus	usage-logs-tenancy-proxy-9c5d49bcd-np5fd	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
06:14:14	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-np5fd	Pulling	Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172"
06:14:14	opendatahub	deployment-controller	usage-logs-tenancy-proxy	ScalingReplicaSet	Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1
06:14:24 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Created	Container created
06:14:24 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Started	Container started
06:14:24	opendatahub	kubelet	maas-controller-786557dc8d-dzrd7	Killing	Stopping container manager
06:14:24	opendatahub	replicaset-controller	maas-controller-786557dc8d	SuccessfulDelete	Deleted pod: maas-controller-786557dc8d-dzrd7
06:14:24	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled down replica set maas-controller-786557dc8d from 1 to 0
06:14:24	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:14:25	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-np5fd	Created	Container created
06:14:25	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-np5fd	Pulled	Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.107s (10.107s including waiting). Image size: 1089502687 bytes.
06:14:25	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-np5fd	Started	Container started
06:14:29 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	ProbeError	Startup probe error: Get "http://10.132.0.29:15021/healthz/ready": dial tcp 10.132.0.29:15021: connect: connection refused body:
06:14:29 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Unhealthy	Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": dial tcp 10.132.0.29:15021: connect: connection refused
06:14:41	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-wxs2l
06:14:41	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-gpwmx
06:14:41	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
06:14:41	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-596dc9545f from 1 to 4
06:14:41	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-ktwxr
06:14:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:14:42	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-wxs2l	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
06:14:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
06:14:42	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-ktwxr	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
06:14:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
06:14:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Started	Container started
06:14:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Created	Container created
06:14:42	opendatahub	maas-controller-5d8f8869ff-zz8zz_4f608d99-b636-4daf-8b85-9b675b9607a8	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-5d8f8869ff-zz8zz_4f608d99-b636-4daf-8b85-9b675b9607a8 became leader
06:14:42	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-gpwmx	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
06:14:45	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Created	Container created
06:14:45	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.868s (2.868s including waiting). Image size: 183555333 bytes.
06:14:45	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.616s (2.616s including waiting). Image size: 183555333 bytes.
06:14:45	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Started	Container started
06:14:45	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Created	Container created
06:14:45	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Started	Container started
06:14:48 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Unhealthy	Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused
06:14:48 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	ProbeError	Startup probe error: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused body:
06:14:50 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Unhealthy	Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused
06:14:50 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	ProbeError	Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused body:
06:14:51 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	ProbeError	Startup probe error: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused body:
06:14:51 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Unhealthy	Startup probe failed: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused
06:14:56	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 8; reason:
06:14:56	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-hqwv6
06:14:56	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-vg7xv
06:14:56	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-hfcrg
06:14:56	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-dx86v
06:14:56	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-596dc9545f from 4 to 8
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	Started	Container started
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Started	Container started
06:14:57	kuadrant-system	replicaset-controller	authorino-6889cbd8c4	SuccessfulCreate	Created pod: authorino-6889cbd8c4-gv47j
06:14:57	kuadrant-system	replicaset-controller	authorino-6889cbd8c4	SuccessfulDelete	Deleted pod: authorino-6889cbd8c4-gv47j
06:14:57	kuadrant-system	multus	authorino-6f6944bcdd-sx68x	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
06:14:57	kuadrant-system	kubelet	authorino-6f6944bcdd-sx68x	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Created	Container created
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:14:57	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-hfcrg	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
06:14:57	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-hqwv6	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
06:14:57	kuadrant-system	replicaset-controller	authorino-6f6944bcdd	SuccessfulCreate	Created pod: authorino-6f6944bcdd-sx68x
06:14:57	kuadrant-system	replicaset-controller	authorino-6f6944bcdd	SuccessfulDelete	Deleted pod: authorino-6f6944bcdd-sx68x
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Created	Container created
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Started	Container started
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Created	Container created
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:14:57	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-vg7xv	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
06:14:57	kuadrant-system	replicaset-controller	authorino-7fd577446d	SuccessfulCreate	Created pod: authorino-7fd577446d-rggfh
06:14:57	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6f6944bcdd from 0 to 1
06:14:57	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6f6944bcdd from 1 to 0
06:14:57	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6889cbd8c4 from 0 to 1
06:14:57	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6889cbd8c4 from 1 to 0
06:14:57	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-7fd577446d from 0 to 1
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Started	Container started
06:14:57	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-dx86v	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:14:57	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	Created	Container created
06:14:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-56c6c9ddd5 from 1 to 0
06:14:58	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulDelete	Deleted pod: authorino-56c6c9ddd5-smszd
06:14:58	kuadrant-system	kubelet	authorino-56c6c9ddd5-smszd	Killing	Stopping container authorino
06:14:58	kuadrant-system	kubelet	authorino-7fd577446d-rggfh	Started	Container started
06:14:58	kuadrant-system	kubelet	authorino-7fd577446d-rggfh	Created	Container created
06:14:58	kuadrant-system	kubelet	authorino-7fd577446d-rggfh	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 368ms (368ms including waiting). Image size: 204096810 bytes.
06:14:58	kuadrant-system	kubelet	authorino-7fd577446d-rggfh	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
06:14:58	kuadrant-system	multus	authorino-7fd577446d-rggfh	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
06:14:58	kuadrant-system	kubelet	authorino-6f6944bcdd-sx68x	Killing	Stopping container authorino
06:14:58	kuadrant-system	kubelet	authorino-6f6944bcdd-sx68x	Started	Container started
06:14:58	kuadrant-system	kubelet	authorino-6f6944bcdd-sx68x	Created	Container created
06:14:58	kuadrant-system	kubelet	authorino-6f6944bcdd-sx68x	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 782ms (782ms including waiting). Image size: 204096810 bytes.
06:15:03 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	ProbeError	Startup probe error: Get "http://10.133.0.30:15021/healthz/ready": dial tcp 10.133.0.30:15021: connect: connection refused body:
06:15:03 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Unhealthy	Startup probe failed: Get "http://10.133.0.30:15021/healthz/ready": dial tcp 10.133.0.30:15021: connect: connection refused
06:15:03 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Unhealthy	Startup probe failed: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused
06:15:03 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Unhealthy	Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused
06:15:03 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	ProbeError	Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused body:
06:15:03 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	ProbeError	Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused body:
06:15:03 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	Unhealthy	Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused
06:15:03 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	ProbeError	Startup probe error: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused body:
06:15:11	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-tvnpg
06:15:11	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-596dc9545f-dztcv
06:15:11	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason:
06:15:11	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-596dc9545f from 8 to 10
06:15:12	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-tvnpg	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
06:15:12	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Started	Container started
06:15:12	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Created	Container created
06:15:12	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:15:12	openshift-ingress	multus	maas-default-gateway-openshift-default-596dc9545f-dztcv	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
06:15:12	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:15:12	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Created	Container created
06:15:12	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Started	Container started
06:15:18 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	ProbeError	Startup probe error: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused body:
06:15:18 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Unhealthy	Startup probe failed: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused
06:15:18 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Unhealthy	Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused
06:15:18 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	ProbeError	Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused body:
06:15:41	kuadrant-system	replicaset-controller	authorino-65d9b845b6	SuccessfulCreate	Created pod: authorino-65d9b845b6-lw9wl
06:15:42	kuadrant-system	kubelet	authorino-65d9b845b6-lw9wl	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
06:15:42	kuadrant-system	multus	authorino-65d9b845b6-lw9wl	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
06:15:43	kuadrant-system	kubelet	authorino-7fd577446d-rggfh	Killing	Stopping container authorino
06:15:43	kuadrant-system	kubelet	authorino-65d9b845b6-lw9wl	Created	Container created
06:15:43	kuadrant-system	kubelet	authorino-65d9b845b6-lw9wl	Started	Container started
06:15:43	kuadrant-system	replicaset-controller	authorino-7fd577446d	SuccessfulDelete	Deleted pod: authorino-7fd577446d-rggfh
06:15:43	kuadrant-system	kubelet	authorino-65d9b845b6-lw9wl	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 699ms (699ms including waiting). Image size: 204096810 bytes.
06:15:44	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1
06:15:44	opendatahub	replicaset-controller	llmisvc-controller-manager-6466c554d8	SuccessfulCreate	Created pod: llmisvc-controller-manager-6466c554d8-tzvrh
06:15:45	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-tzvrh	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
06:15:45	opendatahub	multus	llmisvc-controller-manager-6466c554d8-tzvrh	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
06:15:45	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-tzvrh	Started	Container started
06:15:45	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-tzvrh	Created	Container created
06:15:45	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-tzvrh	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 364ms (364ms including waiting). Image size: 235100981 bytes.
06:16:17	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulDelete	Deleted pod: llmisvc-controller-manager-869d5dcc5d-x67dq
06:16:17	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-x67dq	Killing	Stopping container manager
06:16:17	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0
06:16:20	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-7d85675f98 from 0 to 1
06:16:20	opendatahub	multus	ai-gateway-controller-7d85675f98-htkmw	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
06:16:20	opendatahub	replicaset-controller	ai-gateway-controller-7d85675f98	SuccessfulCreate	Created pod: ai-gateway-controller-7d85675f98-htkmw
06:16:20	opendatahub	kubelet	ai-gateway-controller-7d85675f98-htkmw	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
06:16:22	opendatahub	ai-gateway-controller-7d85675f98-htkmw_06154272-9ea3-471d-a966-3f397609b1c4	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-7d85675f98-htkmw_06154272-9ea3-471d-a966-3f397609b1c4 became leader
06:16:22	opendatahub	kubelet	ai-gateway-controller-7d85675f98-htkmw	Started	Container started
06:16:22	opendatahub	kubelet	ai-gateway-controller-7d85675f98-htkmw	Created	Container created
06:16:22	opendatahub	kubelet	ai-gateway-controller-7d85675f98-htkmw	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 1.512s (1.512s including waiting). Image size: 183888297 bytes.
06:16:23 (x6)	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	FailedCreate	Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
06:16:23 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
06:16:23	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5f78687d5b from 0 to 1
06:16:23	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	SuccessfulCreate	Created pod: payload-processing-5f78687d5b-g9mbb
06:16:23	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-8b96t
06:16:23	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
06:16:24	openshift-ingress	kubelet	payload-processing-5f78687d5b-g9mbb	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
06:16:24	openshift-ingress	multus	payload-processing-5f78687d5b-g9mbb	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
06:16:24	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-8b96t	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
06:16:24	openshift-ingress	multus	payload-pre-processing-7b98bb678c-8b96t	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
06:16:26	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-8b96t	Created	Container created
06:16:26	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-8b96t	Started	Container started
06:16:26	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-8b96t	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.316s (2.316s including waiting). Image size: 125317101 bytes.
06:16:26	openshift-ingress	kubelet	payload-processing-5f78687d5b-g9mbb	Created	Container created
06:16:26	openshift-ingress	kubelet	payload-processing-5f78687d5b-g9mbb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.28s (2.28s including waiting). Image size: 125317101 bytes.
06:16:26	openshift-ingress	kubelet	payload-processing-5f78687d5b-g9mbb	Started	Container started
06:16:33	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-77744b856d from 0 to 1
06:16:33	opendatahub	replicaset-controller	ai-gateway-controller-77744b856d	SuccessfulCreate	Created pod: ai-gateway-controller-77744b856d-vd4z7
06:16:34	opendatahub	llmisvc-controller-manager-6466c554d8-tzvrh_17dd1de8-4896-48c7-89d8-46c8718bc353	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-6466c554d8-tzvrh_17dd1de8-4896-48c7-89d8-46c8718bc353 became leader
06:16:34	opendatahub	multus	ai-gateway-controller-77744b856d-vd4z7	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
06:16:34	opendatahub	kubelet	ai-gateway-controller-77744b856d-vd4z7	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
06:16:34	opendatahub	kubelet	ai-gateway-controller-77744b856d-vd4z7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 396ms (396ms including waiting). Image size: 183888297 bytes.
06:16:34	opendatahub	kubelet	ai-gateway-controller-77744b856d-vd4z7	Created	Container created
06:16:34	opendatahub	kubelet	ai-gateway-controller-77744b856d-vd4z7	Started	Container started
06:16:46	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled down replica set ai-gateway-controller-7d85675f98 from 1 to 0
06:16:46	opendatahub	replicaset-controller	ai-gateway-controller-7d85675f98	SuccessfulDelete	Deleted pod: ai-gateway-controller-7d85675f98-htkmw
06:16:46	opendatahub	kubelet	ai-gateway-controller-7d85675f98-htkmw	Killing	Stopping container manager
06:16:54	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Deployment llm/e2e-distinct-2-simulated-kserve
06:16:54	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-2-simulated-kserve-workload-svc
06:16:54	llm	replicaset-controller	e2e-distinct-2-simulated-kserve-cbd4f5586	SuccessfulCreate	Created pod: e2e-distinct-2-simulated-kserve-cbd4f5586-b979m
06:16:54	llm	deployment-controller	e2e-distinct-2-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-2-simulated-kserve-cbd4f5586 from 0 to 1
06:16:54	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
06:16:54	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Service llm/e2e-distinct-2-simulated-kserve-workload-svc
06:16:54	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-2-simulated-kserve-route
06:16:55	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine-default
06:16:55	llm	multus	e2e-distinct-2-simulated-kserve-cbd4f5586-b979m	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
06:16:55	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-b979m	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:16:55	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-2-simulated-prometheus-scraping
06:16:55	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine
06:16:56	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-b979m	Started	Container started
06:16:56	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-b979m	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.496s (1.496s including waiting). Image size: 98346788 bytes.
06:16:56	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-b979m	Created	Container created
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-simulated-kserve-route
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-simulated-kserve-workload-svc
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Service llm/e2e-distinct-simulated-kserve-workload-svc
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Deployment llm/e2e-distinct-simulated-kserve
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
06:16:58	llm	replicaset-controller	e2e-distinct-simulated-kserve-7b998b79d9	SuccessfulCreate	Created pod: e2e-distinct-simulated-kserve-7b998b79d9-jtfk9
06:16:58	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-jtfk9	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:16:58	llm	deployment-controller	e2e-distinct-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-simulated-kserve-7b998b79d9 from 0 to 1
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine-default
06:16:58	llm	multus	e2e-distinct-simulated-kserve-7b998b79d9-jtfk9	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
06:16:58	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-simulated-prometheus-scraping
06:16:59	llm	replicaset-controller	e2e-embedding-simulated-kserve-67bf594cd6	SuccessfulCreate	Created pod: e2e-embedding-simulated-kserve-67bf594cd6-bdjld
06:16:59	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-jtfk9	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 212ms (212ms including waiting). Image size: 98346788 bytes.
06:16:59	llm	deployment-controller	e2e-embedding-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-embedding-simulated-kserve-67bf594cd6 from 0 to 1
06:16:59	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
06:16:59	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Deployment llm/e2e-embedding-simulated-kserve
06:16:59	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Service llm/e2e-embedding-simulated-kserve-workload-svc
06:16:59	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-jtfk9	Created	Container created
06:16:59	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-jtfk9	Started	Container started
06:16:59	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.HTTPRoute llm/e2e-embedding-simulated-kserve-route
06:16:59	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.DestinationRule llm/e2e-embedding-simulated-kserve-workload-svc
06:17:00	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine
06:17:00	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine-default
06:17:00	llm	multus	e2e-embedding-simulated-kserve-67bf594cd6-bdjld	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
06:17:00	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-bdjld	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:17:00	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.NetworkPolicy llm/e2e-embedding-simulated-prometheus-scraping
06:17:01	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Deployment llm/e2e-trlp-test-simulated-kserve
06:17:01	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
06:17:01	llm	deployment-controller	e2e-trlp-test-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-trlp-test-simulated-kserve-69b588897d from 0 to 1
06:17:01	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-bdjld	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.067s (1.067s including waiting). Image size: 98346788 bytes.
06:17:01	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-bdjld	Created	Container created
06:17:01	llm	replicaset-controller	e2e-trlp-test-simulated-kserve-69b588897d	SuccessfulCreate	Created pod: e2e-trlp-test-simulated-kserve-69b588897d-m7vr7
06:17:01	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-bdjld	Started	Container started
06:17:02	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine-default
06:17:02	llm	multus	e2e-trlp-test-simulated-kserve-69b588897d-m7vr7	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
06:17:02	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.NetworkPolicy llm/e2e-trlp-test-simulated-prometheus-scraping
06:17:02	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine
06:17:02	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-m7vr7	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 263ms (263ms including waiting). Image size: 98346788 bytes.
06:17:02	opendatahub	ai-gateway-controller-77744b856d-vd4z7_37a6ffc7-f051-4a1f-9c75-9b5cc057df79	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-77744b856d-vd4z7_37a6ffc7-f051-4a1f-9c75-9b5cc057df79 became leader
06:17:02	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.DestinationRule llm/e2e-trlp-test-simulated-kserve-workload-svc
06:17:02	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.HTTPRoute llm/e2e-trlp-test-simulated-kserve-route
06:17:02	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Service llm/e2e-trlp-test-simulated-kserve-workload-svc
06:17:02	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-m7vr7	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:17:03	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-m7vr7	Created	Container created
06:17:03	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-m7vr7	Started	Container started
06:17:05	llm	deployment-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc from 0 to 1
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
06:17:05	llm	replicaset-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc	SuccessfulCreate	Created pod: e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/e2e-unconab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/eab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine-default
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-route
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Service llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Deployment llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve
06:17:05	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-prometheus-scraping
06:17:06	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Deployment llm/echo-simulator-kserve
06:17:06	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw	Started	Container started
06:17:06	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw	Created	Container created
06:17:06	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 208ms (208ms including waiting). Image size: 98346788 bytes.
06:17:06	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Secret llm/echo-simulator-kserve-self-signed-certs
06:17:06	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:17:06	llm	multus	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tdqnw	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
06:17:06	llm	deployment-controller	echo-simulator-kserve	ScalingReplicaSet	Scaled up replica set echo-simulator-kserve-64b6f8f96c from 0 to 1
06:17:07	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Service llm/echo-simulator-kserve-workload-svc
06:17:07	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine
06:17:07	llm	multus	echo-simulator-kserve-64b6f8f96c-zrsnf	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
06:17:07	llm	kubelet	echo-simulator-kserve-64b6f8f96c-zrsnf	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0"
06:17:07	llm	replicaset-controller	echo-simulator-kserve-64b6f8f96c	SuccessfulCreate	Created pod: echo-simulator-kserve-64b6f8f96c-zrsnf
06:17:07	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.HTTPRoute llm/echo-simulator-kserve-route
06:17:07	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.DestinationRule llm/echo-simulator-kserve-workload-svc
06:17:07	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine-default
06:17:07	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.NetworkPolicy llm/echo-simulator-prometheus-scraping
06:17:09	llm	deployment-controller	facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set facebook-opt-125m-simulated-kserve-84b4ff656c from 0 to 1
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine-default
06:17:09	llm	multus	facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
06:17:09	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/facebook-opt-125m-simulated-prometheus-scraping
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine
06:17:09	llm	replicaset-controller	facebook-opt-125m-simulated-kserve-84b4ff656c	SuccessfulCreate	Created pod: facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/facebook-opt-125m-simulated-kserve-workload-svc
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Deployment llm/facebook-opt-125m-simulated-kserve
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Service llm/facebook-opt-125m-simulated-kserve-workload-svc
06:17:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/facebook-opt-125m-simulated-kserve-route
06:17:10	llm	kubelet	echo-simulator-kserve-64b6f8f96c-zrsnf	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0" in 2.3s (2.3s including waiting). Image size: 78208878 bytes.
06:17:10	llm	kubelet	echo-simulator-kserve-64b6f8f96c-zrsnf	Created	Container created
06:17:10	llm	kubelet	echo-simulator-kserve-64b6f8f96c-zrsnf	Started	Container started
06:17:10	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 208ms (208ms including waiting). Image size: 98346788 bytes.
06:17:10	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc	Created	Container created
06:17:10	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-4h4lc	Started	Container started
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.HTTPRoute llm/premium-simulated-simulated-premium-kserve-route
06:17:13	llm	replicaset-controller	premium-simulated-simulated-premium-kserve-5f5c4b459f	SuccessfulCreate	Created pod: premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w
06:17:13	llm	multus	premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
06:17:13	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Service llm/premium-simulated-simulated-premium-kserve-workload-svc
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.DestinationRule llm/premium-simulated-simulated-premium-kserve-workload-svc
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/p75d89895ad19b4ac62612d08bfe7f574-kserve-llmisvc-engine-default
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Deployment llm/premium-simulated-simulated-premium-kserve
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.NetworkPolicy llm/premium-simulated-simulated-premium-prometheus-scraping
06:17:13	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/premium-simulated-simulated-premium-kserve-llmisvc-engine
06:17:13	llm	deployment-controller	premium-simulated-simulated-premium-kserve	ScalingReplicaSet	Scaled up replica set premium-simulated-simulated-premium-kserve-5f5c4b459f from 0 to 1
06:17:14	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w	Started	Container started
06:17:14	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w	Created	Container created
06:17:14	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-br69w	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 190ms (190ms including waiting). Image size: 98346788 bytes.
06:17:14 (x2)	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Updated	Updated v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
06:17:15	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-2-simulated] is Ready
06:17:18	llm	LLMInferenceServiceController	e2e-distinct-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-simulated] is Ready
06:17:18	llm	LLMInferenceServiceController	e2e-distinct-simulated	Updated	Updated v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
06:17:20	llm	LLMInferenceServiceController	e2e-embedding-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-embedding-simulated] is Ready
06:17:20	llm	LLMInferenceServiceController	e2e-embedding-simulated	Updated	Updated v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
06:17:21	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Updated	Updated v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
06:17:22	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-trlp-test-simulated] is Ready
06:17:23	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
06:17:23	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-unconfigured-facebook-opt-125m-simulated] is Ready
06:17:25	llm	LLMInferenceServiceController	echo-simulator	LLMInferenceServiceReady	LLMInferenceService [echo-simulator] is Ready
06:17:25	llm	LLMInferenceServiceController	echo-simulator	Updated	Updated v1.Secret llm/echo-simulator-kserve-self-signed-certs
06:17:27	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
06:17:28	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [facebook-opt-125m-simulated] is Ready
06:17:28	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Updated	Updated v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
06:17:29	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	LLMInferenceServiceReady	LLMInferenceService [premium-simulated-simulated-premium] is Ready
06:17:30	kuadrant-system	replicaset-controller	authorino-75545cc4b5	SuccessfulCreate	Created pod: authorino-75545cc4b5-k75xt
06:17:31	kuadrant-system	kubelet	authorino-75545cc4b5-k75xt	Started	Container started
06:17:31	kuadrant-system	multus	authorino-75545cc4b5-k75xt	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
06:17:31	kuadrant-system	kubelet	authorino-75545cc4b5-k75xt	Created	Container created
06:17:31	kuadrant-system	kubelet	authorino-75545cc4b5-k75xt	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
06:17:31	kuadrant-system	kubelet	authorino-75545cc4b5-k75xt	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 670ms (670ms including waiting). Image size: 204096810 bytes.
06:17:32	kuadrant-system	replicaset-controller	authorino-65d9b845b6	SuccessfulDelete	Deleted pod: authorino-65d9b845b6-lw9wl
06:17:32	kuadrant-system	kubelet	authorino-65d9b845b6-lw9wl	Killing	Stopping container authorino
06:17:32 (x4)	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	(combined from similar events): Scaled down replica set authorino-65d9b845b6 from 1 to 0
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.HTTPRoute llm/grid-inv-d76247ba-model-kserve-route
06:18:02	llm	multus	grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
06:18:02	llm	kubelet	grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.NetworkPolicy llm/grid-inv-d76247ba-model-prometheus-scraping
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.DestinationRule llm/grid-inv-d76247ba-model-kserve-workload-svc
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.PodMonitor llm/grid-inv-d76247ba-model-kserve-llmisvc-engine
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.Service llm/grid-inv-d76247ba-model-kserve-workload-svc
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.Deployment llm/grid-inv-d76247ba-model-kserve
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.Secret llm/grid-inv-d76247ba-model-kserve-self-signed-certs
06:18:02	llm	kubelet	grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	Created	Container created
06:18:02	llm	kubelet	grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	Started	Container started
06:18:02	llm	deployment-controller	grid-inv-d76247ba-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-d76247ba-model-kserve-78bd8c444c from 0 to 1
06:18:02	llm	replicaset-controller	grid-inv-d76247ba-model-kserve-78bd8c444c	SuccessfulCreate	Created pod: grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd
06:18:02	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Created	Created v1.PodMonitor llm/grid-inv-d76247ba-model-kserve-llmisvc-engine-default
06:18:06	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	Updated	Updated v1.Secret llm/grid-inv-d76247ba-model-kserve-self-signed-certs
06:18:10	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.Deployment llm/grid-inv-d76247ba-unmatched-kserve
06:18:10	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.Secret llm/grid-inv-d76247ba-unmatched-kserve-self-signed-certs
06:18:10	llm	replicaset-controller	grid-inv-d76247ba-unmatched-kserve-c65b8f67c	SuccessfulCreate	Created pod: grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6
06:18:10	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.Service llm/grid-inv-d76247ba-unmatched-kserve-workload-svc
06:18:10	llm	deployment-controller	grid-inv-d76247ba-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-d76247ba-unmatched-kserve-c65b8f67c from 0 to 1
06:18:11	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.DestinationRule llm/grid-inv-d76247ba-unmatched-kserve-workload-svc
06:18:11	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-d76247ba-unmatched-kserve-route
06:18:11	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-d76247ba-unmatched-prometheus-scraping
06:18:11	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.PodMonitor llm/grid-inv-d76247ba-unmatched-kserve-llmisvc-engine
06:18:11	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Created	Created v1.PodMonitor llm/grid-inv-d76247ba-unmatched-kserve-llmisvc-engine-default
06:18:11	llm	kubelet	grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1"
06:18:11	llm	multus	grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
06:18:12	llm	kubelet	grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" in 1.113s (1.113s including waiting). Image size: 98346788 bytes.
06:18:12	llm	kubelet	grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	Started	Container started
06:18:12	llm	kubelet	grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	Created	Container created
06:18:19	llm	LLMInferenceServiceController	grid-inv-d76247ba-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-d76247ba-model] is Ready
06:18:21 (x2)	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	Updated	Updated v1.Secret llm/grid-inv-d76247ba-unmatched-kserve-self-signed-certs
06:18:23	llm	LLMInferenceServiceController	grid-inv-d76247ba-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-d76247ba-unmatched] is Ready
06:19:20	llm	kubelet	grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	Killing	Stopping container main
06:19:22	llm	kubelet	grid-inv-d76247ba-unmatched-kserve-c65b8f67c-82vg6	Killing	Stopping container main
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.Deployment llm/grid-inv-abb4b23f-model-kserve
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.NetworkPolicy llm/grid-inv-abb4b23f-model-prometheus-scraping
06:19:29	llm	kubelet	grid-inv-abb4b23f-model-kserve-c995bc848-gztk2	Created	Container created
06:19:29	llm	kubelet	grid-inv-abb4b23f-model-kserve-c995bc848-gztk2	Started	Container started
06:19:29	llm	multus	grid-inv-abb4b23f-model-kserve-c995bc848-gztk2	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
06:19:29	llm	replicaset-controller	grid-inv-abb4b23f-model-kserve-c995bc848	SuccessfulCreate	Created pod: grid-inv-abb4b23f-model-kserve-c995bc848-gztk2
06:19:29	llm	deployment-controller	grid-inv-abb4b23f-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-abb4b23f-model-kserve-c995bc848 from 0 to 1
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.Secret llm/grid-inv-abb4b23f-model-kserve-self-signed-certs
06:19:29	llm	kubelet	grid-inv-abb4b23f-model-kserve-c995bc848-gztk2	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.Service llm/grid-inv-abb4b23f-model-kserve-workload-svc
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.HTTPRoute llm/grid-inv-abb4b23f-model-kserve-route
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.DestinationRule llm/grid-inv-abb4b23f-model-kserve-workload-svc
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.PodMonitor llm/grid-inv-abb4b23f-model-kserve-llmisvc-engine-default
06:19:29	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Created	Created v1.PodMonitor llm/grid-inv-abb4b23f-model-kserve-llmisvc-engine
06:19:33	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.Secret llm/grid-inv-abb4b23f-unmatched-kserve-self-signed-certs
06:19:33	llm	deployment-controller	grid-inv-abb4b23f-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-abb4b23f-unmatched-kserve-5584575686 from 0 to 1
06:19:33	llm	replicaset-controller	grid-inv-abb4b23f-unmatched-kserve-5584575686	SuccessfulCreate	Created pod: grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk
06:19:33	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.Deployment llm/grid-inv-abb4b23f-unmatched-kserve
06:19:34	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.Service llm/grid-inv-abb4b23f-unmatched-kserve-workload-svc
06:19:34	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.DestinationRule llm/grid-inv-abb4b23f-unmatched-kserve-workload-svc
06:19:34	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.PodMonitor llm/grid-inv-abb4b23f-unmatched-kserve-llmisvc-engine-default
06:19:34	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.PodMonitor llm/grid-inv-abb4b23f-unmatched-kserve-llmisvc-engine
06:19:34	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-abb4b23f-unmatched-prometheus-scraping
06:19:34	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-abb4b23f-unmatched-kserve-route
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	ProbeError	Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Unhealthy	Readiness probe failed: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Unhealthy	Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	ProbeError	Readiness probe error: Get "http://10.133.0.30:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Unhealthy	Readiness probe failed: Get "http://10.132.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Unhealthy	Readiness probe failed: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	ProbeError	Readiness probe error: Get "http://10.132.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	llm	kubelet	grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk	Created	Container created
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Unhealthy	Readiness probe failed: Get "http://10.133.0.30:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	ProbeError	Readiness probe error: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	ProbeError	Readiness probe error: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	llm	kubelet	grid-inv-d76247ba-model-kserve-78bd8c444c-gwwbd	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8000/ready": dial tcp 10.133.0.41:8000: connect: connection refused
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	ProbeError	Readiness probe error: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dx86v	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Unhealthy	Readiness probe failed: Get "http://10.133.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:19:35	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	ProbeError	Readiness probe error: Get "http://10.133.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:19:35	llm	kubelet	grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
06:19:35	llm	multus	grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
06:19:36	llm	kubelet	grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk	Started	Container started
06:19:36	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	Updated	Updated v1.Secret llm/grid-inv-abb4b23f-model-kserve-self-signed-certs
06:19:42	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	Updated	Updated v1.Secret llm/grid-inv-abb4b23f-unmatched-kserve-self-signed-certs
06:19:43	llm	LLMInferenceServiceController	grid-inv-abb4b23f-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-abb4b23f-model] is Ready
06:19:45	llm	LLMInferenceServiceController	grid-inv-abb4b23f-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-abb4b23f-unmatched] is Ready
06:19:48	opendatahub	kubelet	test-curl-13239-2cb656	Pulling	Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5"
06:19:48	opendatahub	multus	test-curl-13239-2cb656	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
06:19:50	opendatahub	kubelet	test-curl-13239-2cb656	Started	Container started
06:19:50	opendatahub	kubelet	test-curl-13239-2cb656	Created	Container created
06:19:50	opendatahub	kubelet	test-curl-13239-2cb656	Pulled	Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.358s (2.358s including waiting). Image size: 106100986 bytes.
06:19:53	opendatahub	kubelet	test-curl-13239-30c00f	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
06:19:53	opendatahub	multus	test-curl-13239-30c00f	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
06:19:53	opendatahub	kubelet	test-curl-13239-30c00f	Started	Container started
06:19:53	opendatahub	kubelet	test-curl-13239-2cb656	Killing	Stopping container test-curl-13239-2cb656
06:19:53	opendatahub	kubelet	test-curl-13239-30c00f	Created	Container created
06:19:55	opendatahub	kubelet	test-curl-13239-bc6e7b	Created	Container created
06:19:55	opendatahub	multus	test-curl-13239-bc6e7b	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
06:19:55	opendatahub	kubelet	test-curl-13239-bc6e7b	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
06:19:55	opendatahub	kubelet	test-curl-13239-30c00f	Killing	Stopping container test-curl-13239-30c00f
06:19:55	opendatahub	kubelet	test-curl-13239-bc6e7b	Started	Container started
06:19:57	opendatahub	kubelet	test-curl-13239-bc6e7b	Killing	Stopping container test-curl-13239-bc6e7b
06:19:57	openshift-ingress	multus	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
06:19:57	openshift-ingress	replicaset-controller	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c	SuccessfulCreate	Created pod: e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n
06:19:57	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-d5cb1c	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-d5cb1c
06:19:57	openshift-ingress	deployment-controller	e2e-shared-a-main-d5cb1c-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c from 0 to 1
06:20:02	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-d5cb1c	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ff from 0 to 1
06:20:02	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-d5cb1c	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-d5cb1c-dd495866d from 0 to 1
06:20:03	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
06:20:03	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7
06:20:03 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-d5cb1c: serviceaccount "payload-processing-e2e-shared-a-main-d5cb1c" not found
06:20:03	openshift-ingress	multus	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
06:20:03	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
06:20:03	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
06:20:03 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ff	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-d5cb1c: serviceaccount "payload-processing-e2e-shared-a-main-d5cb1c" not found
06:20:03	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ff	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm
06:20:06	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.27s (2.27s including waiting). Image size: 125317101 bytes.
06:20:06	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	Started	Container started
06:20:06	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.289s (2.289s including waiting). Image size: 125317101 bytes.
06:20:06	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	Created	Container created
06:20:06	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	Created	Container created
06:20:06	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	Started	Container started
06:20:15	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	ProbeError	Readiness probe error: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:20:15	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Unhealthy	Readiness probe failed: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:20:17 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Created	Container created
06:20:17 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Started	Container started
06:20:17 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:20:17	openshift-ingress	multus	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
06:20:17	openshift-ingress	replicaset-controller	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc	SuccessfulCreate	Created pod: e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt
06:20:17	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-ca863e	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-ca863e
06:20:17	openshift-ingress	deployment-controller	e2e-shared-b-main-ca863e-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc from 0 to 1
06:20:23 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Unhealthy	Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused
06:20:23 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	ProbeError	Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused body:
06:20:23	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-ca863e	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595 from 0 to 1
06:20:23	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-ca863e	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-ca863e-6bb4985b66 from 0 to 1
06:20:23	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2
06:20:23 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-ca863e: serviceaccount "payload-processing-e2e-shared-b-main-ca863e" not found
06:20:23	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
06:20:23	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
06:20:23	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4
06:20:23 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-ca863e: serviceaccount "payload-processing-e2e-shared-b-main-ca863e" not found
06:20:24	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	Created	Container created
06:20:24	openshift-ingress	multus	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
06:20:24	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	Started	Container started
06:20:24	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	Created	Container created
06:20:24	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	Started	Container started
06:20:24	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
06:20:27 (x2)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-d5cb1c-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:20:27 (x2)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-d5cb1c-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:20:32	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-ca863e-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:20:32	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-ca863e-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
06:20:40 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Created	Container created
06:20:40 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
06:20:40 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Started	Container started
06:20:42	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-d5cb1c-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:20:42	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-d5cb1c-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:20:45 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	ProbeError	Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused body:
06:20:45 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Unhealthy	Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused
06:21:16	opendatahub	kubelet	test-curl-13239-c2c87d	Created	Container created
06:21:16	opendatahub	kubelet	test-curl-13239-c2c87d	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
06:21:16	opendatahub	kubelet	test-curl-13239-c2c87d	Started	Container started
06:21:16	opendatahub	multus	test-curl-13239-c2c87d	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
06:21:17 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-ca863e-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:21:17 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-ca863e-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
06:21:18	opendatahub	kubelet	test-curl-13239-c2c87d	Killing	Stopping container test-curl-13239-c2c87d
06:22:26	llm	kubelet	grid-inv-abb4b23f-model-kserve-c995bc848-gztk2	Killing	Stopping container main
06:22:27	llm	kubelet	grid-inv-abb4b23f-unmatched-kserve-5584575686-w8sfk	Killing	Stopping container main
06:22:31 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Unhealthy	Readiness probe failed: Get "http://10.132.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:22:31 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	ProbeError	Readiness probe error: Get "http://10.132.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.Service llm/grid-inv-2935b643-model-kserve-workload-svc
06:22:34	llm	deployment-controller	grid-inv-2935b643-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-2935b643-model-kserve-69b7bfcf9b from 0 to 1
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.DestinationRule llm/grid-inv-2935b643-model-kserve-workload-svc
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.HTTPRoute llm/grid-inv-2935b643-model-kserve-route
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.PodMonitor llm/grid-inv-2935b643-model-kserve-llmisvc-engine
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.Deployment llm/grid-inv-2935b643-model-kserve
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.Secret llm/grid-inv-2935b643-model-kserve-self-signed-certs
06:22:34	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.PodMonitor llm/grid-inv-2935b643-model-kserve-llmisvc-engine-default
06:22:34	llm	replicaset-controller	grid-inv-2935b643-model-kserve-69b7bfcf9b	SuccessfulCreate	Created pod: grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh
06:22:35	llm	kubelet	grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh	Started	Container started
06:22:35	llm	kubelet	grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh	Created	Container created
06:22:35	llm	kubelet	grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
06:22:35	llm	multus	grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
06:22:35	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Created	Created v1.NetworkPolicy llm/grid-inv-2935b643-model-prometheus-scraping
06:22:36	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.Service llm/grid-inv-2935b643-unmatched-kserve-workload-svc
06:22:36	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-2935b643-unmatched-kserve-route
06:22:36	llm	replicaset-controller	grid-inv-2935b643-unmatched-kserve-667b78ffb6	SuccessfulCreate	Created pod: grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x
06:22:36	llm	deployment-controller	grid-inv-2935b643-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-2935b643-unmatched-kserve-667b78ffb6 from 0 to 1
06:22:36	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.Secret llm/grid-inv-2935b643-unmatched-kserve-self-signed-certs
06:22:36	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.Deployment llm/grid-inv-2935b643-unmatched-kserve
06:22:36	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.PodMonitor llm/grid-inv-2935b643-unmatched-kserve-llmisvc-engine-default
06:22:36	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.DestinationRule llm/grid-inv-2935b643-unmatched-kserve-workload-svc
06:22:37	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.PodMonitor llm/grid-inv-2935b643-unmatched-kserve-llmisvc-engine
06:22:37	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-2935b643-unmatched-prometheus-scraping
06:22:37	llm	kubelet	grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x	Started	Container started
06:22:37	llm	kubelet	grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x	Created	Container created
06:22:37	llm	kubelet	grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
06:22:37	llm	multus	grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
06:22:38	llm	LLMInferenceServiceController	grid-inv-2935b643-model	Updated	Updated v1.Secret llm/grid-inv-2935b643-model-kserve-self-signed-certs
06:22:44	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	Updated	Updated v1.Secret llm/grid-inv-2935b643-unmatched-kserve-self-signed-certs
06:22:46	llm	LLMInferenceServiceController	grid-inv-2935b643-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-2935b643-model] is Ready
06:22:48	llm	LLMInferenceServiceController	grid-inv-2935b643-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-2935b643-unmatched] is Ready
06:23:53 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	ProbeError	Readiness probe error: Get "http://10.133.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:23:53 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Unhealthy	Readiness probe failed: Get "http://10.133.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:24:12	llm	kubelet	grid-inv-2935b643-model-kserve-69b7bfcf9b-mkgjh	Killing	Stopping container main
06:24:17	llm	kubelet	grid-inv-2935b643-unmatched-kserve-667b78ffb6-blf7x	Killing	Stopping container main
06:24:22 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	ProbeError	Readiness probe error: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
06:24:22 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Unhealthy	Readiness probe failed: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
06:24:29	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-d5cb1c-dd495866d-xjzj7	Killing	Stopping container payload-processing
06:24:29	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-d5cb1c-5bf88b86ffrqlqm	Killing	Stopping container payload-pre-processing
06:24:41	openshift-ingress	kubelet	e2e-shared-a-main-d5cb1c-openshift-default-77db498d8c-rd47n	Killing	Stopping container istio-proxy
06:24:52	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	Killing	Stopping container payload-pre-processing
06:24:52	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	Killing	Stopping container payload-processing
06:25:01	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:25:01	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	Killing	Stopping container istio-proxy
06:25:01	openshift-ingress	kubelet	e2e-shared-b-main-ca863e-openshift-default-6f44d77dfc-s7vdt	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:25:15 (x2)	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ca863e-6bb4985b66-gsmq2	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
06:25:15 (x2)	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ca863e-5585dd6595hlnb4	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
06:29:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Killing	Stopping container istio-proxy
06:29:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Killing	Stopping container istio-proxy
06:29:59	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-tvnpg
06:29:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Killing	Stopping container istio-proxy
06:29:59	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-gpwmx
06:29:59	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: All metrics below target
06:29:59	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-wxs2l
06:29:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Killing	Stopping container istio-proxy
06:29:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-ktwxr	Killing	Stopping container istio-proxy
06:29:59	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-dsvvx
06:29:59	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-hfcrg
06:29:59	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-596dc9545f from 10 to 4
06:29:59	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-ktwxr
06:29:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-wxs2l	Killing	Stopping container istio-proxy
06:30:00	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:30:00	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dsvvx	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:30:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:30:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:30:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	ProbeError	Readiness probe error: Get "http://10.132.0.51:15021/healthz/ready": read tcp 10.132.0.2:52294->10.132.0.51:15021: read: connection reset by peer body:
06:30:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-tvnpg	Unhealthy	Readiness probe failed: Get "http://10.132.0.51:15021/healthz/ready": read tcp 10.132.0.2:52294->10.132.0.51:15021: read: connection reset by peer
06:30:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-gpwmx	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:30:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hfcrg	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:30:14	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 2; reason: All metrics below target
06:30:14	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Killing	Stopping container istio-proxy
06:30:14	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Killing	Stopping container istio-proxy
06:30:14	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-dztcv
06:30:14	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-hqwv6
06:30:14	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-596dc9545f from 4 to 2
06:30:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:30:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:30:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-hqwv6	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
06:30:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-dztcv	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
06:30:29	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-596dc9545f	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-596dc9545f-vg7xv
06:30:29	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 1; reason: All metrics below target
06:30:29	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-596dc9545f from 2 to 1
06:30:29	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Killing	Stopping container istio-proxy
06:30:34	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	Unhealthy	Readiness probe failed: Get "http://10.133.0.30:15021/healthz/ready": dial tcp 10.133.0.30:15021: connect: connection refused
06:30:34	openshift-ingress	kubelet	maas-default-gateway-openshift-default-596dc9545f-vg7xv	ProbeError	Readiness probe error: Get "http://10.133.0.30:15021/healthz/ready": dial tcp 10.133.0.30:15021: connect: connection refused body: