Events

Time	Namespace	Component	RelatedObject	Reason	Message
15:00:24	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp to ip-10-0-128-169.ec2.internal
14:33:49	kuadrant-system		authorino-operator-7587b89b76-649lf	Scheduled	Successfully assigned kuadrant-system/authorino-operator-7587b89b76-649lf to ip-10-0-136-132.ec2.internal
14:40:13	kserve-ci-e2e-test		llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz to ip-10-0-128-169.ec2.internal
14:40:49	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-75469f89c9-dfbfc to ip-10-0-136-132.ec2.internal
14:40:49	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-75469f89c9-xmbsz to ip-10-0-138-116.ec2.internal
14:40:50	kserve-ci-e2e-test		precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Scheduled	Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q to ip-10-0-136-132.ec2.internal
14:37:12	kserve-ci-e2e-test		router-gateway-1-openshift-default-6c59fbf55c-74r8s	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-74r8s to ip-10-0-128-169.ec2.internal
15:00:07	kserve-ci-e2e-test		router-gateway-2-openshift-default-6866b85949-nt4x9	Scheduled	Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-nt4x9 to ip-10-0-138-116.ec2.internal
15:00:24	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-76696559d6-c5f2b	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-76696559d6-c5f2b to ip-10-0-128-169.ec2.internal
15:00:24	kserve-ci-e2e-test		router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd to ip-10-0-136-132.ec2.internal
14:41:37	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h to ip-10-0-136-132.ec2.internal
14:44:44	kserve-ci-e2e-test		custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-6cbfb55898-jn9bs to ip-10-0-128-169.ec2.internal
14:34:38	kuadrant-system		authorino-674b59b84c-85wgl	Scheduled	Successfully assigned kuadrant-system/authorino-674b59b84c-85wgl to ip-10-0-136-132.ec2.internal
14:40:13	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz to ip-10-0-128-169.ec2.internal
14:51:45	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 to ip-10-0-128-169.ec2.internal
14:26:18	openshift-ingress		router-default-5d8799dfd8-qtxtc	Scheduled	Successfully assigned openshift-ingress/router-default-5d8799dfd8-qtxtc to ip-10-0-138-116.ec2.internal
14:54:35	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z to ip-10-0-136-132.ec2.internal
14:35:12	openshift-ingress		istiod-openshift-gateway-55ff986f96-6d7fr	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-55ff986f96-6d7fr to ip-10-0-138-116.ec2.internal
14:48:17	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x to ip-10-0-136-132.ec2.internal
14:33:41	kuadrant-system		limitador-operator-controller-manager-c7fb4c8d5-s65gz	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-s65gz to ip-10-0-128-169.ec2.internal
15:01:51	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg to ip-10-0-136-132.ec2.internal
14:33:47	kuadrant-system		dns-operator-controller-manager-844548ff4c-qzx86	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-844548ff4c-qzx86 to ip-10-0-136-132.ec2.internal
14:37:20	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw to ip-10-0-128-169.ec2.internal
14:34:37	kuadrant-system		limitador-limitador-67566c68b4-lx5zw	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-67566c68b4-lx5zw to ip-10-0-128-169.ec2.internal
14:51:45	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 to ip-10-0-136-132.ec2.internal
15:01:51	kserve-ci-e2e-test		llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt to ip-10-0-136-132.ec2.internal
14:34:37	kuadrant-system		limitador-limitador-64c8f475fb-tkn5h	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-64c8f475fb-tkn5h to ip-10-0-138-116.ec2.internal
14:33:09	openshift-ingress		openshift-ai-inference-openshift-default-7c5447bb76-hndv9	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-hndv9 to ip-10-0-128-169.ec2.internal
14:39:28	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7 to ip-10-0-136-132.ec2.internal
14:54:36	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn to ip-10-0-128-169.ec2.internal
14:39:28	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9 to ip-10-0-128-169.ec2.internal
14:39:28	kserve-ci-e2e-test		scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9d798b947-brwzq to ip-10-0-136-132.ec2.internal
14:37:31	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s to ip-10-0-128-169.ec2.internal
14:44:44	kserve-ci-e2e-test		custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns to ip-10-0-128-169.ec2.internal
14:34:38	kuadrant-system		authorino-79cbc94b89-dtgl2	Scheduled	Successfully assigned kuadrant-system/authorino-79cbc94b89-dtgl2 to ip-10-0-136-132.ec2.internal
14:44:17	kserve-ci-e2e-test		stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m to ip-10-0-136-132.ec2.internal
14:37:30	kserve-ci-e2e-test		scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf to ip-10-0-136-132.ec2.internal
14:41:37	kserve-ci-e2e-test		stop-feature-test-kserve-88fb48646-cq9c6	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-88fb48646-cq9c6 to ip-10-0-136-132.ec2.internal
14:46:41	kserve-ci-e2e-test		router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw to ip-10-0-128-169.ec2.internal
14:51:45	kserve-ci-e2e-test		llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d to ip-10-0-136-132.ec2.internal
14:35:02	kuadrant-system		authorino-68bd676465-8cn2c	Scheduled	Successfully assigned kuadrant-system/authorino-68bd676465-8cn2c to ip-10-0-136-132.ec2.internal
14:33:45	kuadrant-system		kuadrant-operator-controller-manager-6ddf9554fc-f27mm	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f27mm to ip-10-0-128-169.ec2.internal
14:33:54	kuadrant-system		kuadrant-console-plugin-6c886788f8-ghmw2	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-6c886788f8-ghmw2 to ip-10-0-138-116.ec2.internal
15:01:52	kserve-ci-e2e-test		llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Scheduled	Successfully assigned kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn to ip-10-0-128-169.ec2.internal
14:46:40	kserve-ci-e2e-test		router-with-refs-test-kserve-5cfbdd5558-h5cjz	Scheduled	Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-5cfbdd5558-h5cjz to ip-10-0-128-169.ec2.internal
14:54:35	kserve-ci-e2e-test		custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Scheduled	Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9 to ip-10-0-136-132.ec2.internal
14:33:01	openshift-ingress		istiod-openshift-gateway-7cd77c7ffd-bklns	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-bklns to ip-10-0-128-169.ec2.internal
14:53:53	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-65579b6fcf-5z89v to ip-10-0-136-132.ec2.internal
14:53:53	kserve-ci-e2e-test		scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Scheduled	Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc to ip-10-0-136-132.ec2.internal
14:44:17	kserve-ci-e2e-test		stop-feature-test-kserve-88fb48646-7mxhg	Scheduled	Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-88fb48646-7mxhg to ip-10-0-136-132.ec2.internal
14:20:18	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-5d8799dfd8 from 0 to 1
14:20:19 (x9)	openshift-ingress	replicaset-controller	router-default-5d8799dfd8	FailedCreate	Error creating: pods "router-default-5d8799dfd8-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
14:20:54	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
14:20:54	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
14:20:59	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
14:23:34 (x16)	openshift-ingress	replicaset-controller	router-default-5d8799dfd8	FailedCreate	Error creating: pods "router-default-5d8799dfd8-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
14:24:14 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
14:26:18	openshift-ingress	replicaset-controller	router-default-5d8799dfd8	SuccessfulCreate	Created pod: router-default-5d8799dfd8-qtxtc
14:26:25 (x5)	openshift-ingress	kubelet	router-default-5d8799dfd8-qtxtc	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
14:26:25 (x5)	openshift-ingress	kubelet	router-default-5d8799dfd8-qtxtc	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
14:26:34	openshift-ingress	multus	router-default-5d8799dfd8-qtxtc	AddedInterface	Add eth0 [10.134.0.9/23] from ovn-kubernetes
14:26:34	openshift-ingress	kubelet	router-default-5d8799dfd8-qtxtc	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:294af5c64228434d1ed6ee8ea3ac802e3c999aa847223e3b2efa18425a9fe421" already present on machine
14:26:34	openshift-ingress	kubelet	router-default-5d8799dfd8-qtxtc	Created	Created container: router
14:26:34	openshift-ingress	kubelet	router-default-5d8799dfd8-qtxtc	Started	Started container router
14:33:01	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7cd77c7ffd from 0 to 1
14:33:01	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7cd77c7ffd	SuccessfulCreate	Created pod: istiod-openshift-gateway-7cd77c7ffd-bklns
14:33:02	openshift-ingress	multus	istiod-openshift-gateway-7cd77c7ffd-bklns	AddedInterface	Add eth0 [10.132.0.16/23] from ovn-kubernetes
14:33:02	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8"
14:33:05	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8" in 3.26s (3.26s including waiting). Image size: 179322965 bytes.
14:33:05	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	Created	Created container: discovery
14:33:05	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	Started	Started container discovery
14:33:06	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:33:06	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
14:33:09	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-default	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-default-7c5447bb76 from 0 to 1
14:33:09	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-default-7c5447bb76	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-default-7c5447bb76-hndv9
14:33:09	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
14:33:09	openshift-ingress	gateway_labeler_controller	openshift-ai-inference	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference
14:33:09	openshift-ingress	multus	openshift-ai-inference-openshift-default-7c5447bb76-hndv9	AddedInterface	Add eth0 [10.132.0.17/23] from ovn-kubernetes
14:33:09	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-7c5447bb76-hndv9	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe"
14:33:12	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-7c5447bb76-hndv9	Started	Started container istio-proxy
14:33:12	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-7c5447bb76-hndv9	Created	Created container: istio-proxy
14:33:12	openshift-ingress	kubelet	openshift-ai-inference-openshift-default-7c5447bb76-hndv9	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" in 2.81s (2.81s including waiting). Image size: 163191192 bytes.
14:33:13	openshift-ingress	service-controller	openshift-ai-inference-openshift-default	EnsuredLoadBalancer	Ensured load balancer
14:33:16	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:33:16	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:33:38	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.3.0	RequirementsUnknown	requirements not yet checked
14:33:38	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.3.0	RequirementsNotMet	one or more requirements couldn't be found
14:33:41	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.3.2	RequirementsNotMet	one or more requirements couldn't be found
14:33:41	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.3.2	RequirementsUnknown	requirements not yet checked
14:33:41	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-c7fb4c8d5 from 0 to 1
14:33:41	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-c7fb4c8d5	SuccessfulCreate	Created pod: limitador-operator-controller-manager-c7fb4c8d5-s65gz
14:33:41	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.3.0	AllRequirementsMet	all requirements found, attempting install
14:33:41	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.3.0	InstallSucceeded	waiting for install components to report healthy
14:33:41	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.3.0	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
14:33:42	kuadrant-system	multus	limitador-operator-controller-manager-c7fb4c8d5-s65gz	AddedInterface	Add eth0 [10.132.0.18/23] from ovn-kubernetes
14:33:42	kuadrant-system	kubelet	limitador-operator-controller-manager-c7fb4c8d5-s65gz	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:9a3ccb54bc904521277b87e1d178d8134cc6ef744eb4ec5b64177355fd5a9e66"
14:33:44	kuadrant-system	limitador-operator-controller-manager-c7fb4c8d5-s65gz_bd61bb94-8fef-47f4-b9c6-2e8e5744df45	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-c7fb4c8d5-s65gz_bd61bb94-8fef-47f4-b9c6-2e8e5744df45 became leader
14:33:44	kuadrant-system	kubelet	limitador-operator-controller-manager-c7fb4c8d5-s65gz	Started	Started container manager
14:33:44	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.3.2	AllRequirementsMet	all requirements found, attempting install
14:33:44	kuadrant-system	kubelet	limitador-operator-controller-manager-c7fb4c8d5-s65gz	Created	Created container: manager
14:33:44	kuadrant-system	kubelet	limitador-operator-controller-manager-c7fb4c8d5-s65gz	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:9a3ccb54bc904521277b87e1d178d8134cc6ef744eb4ec5b64177355fd5a9e66" in 2.648s (2.648s including waiting). Image size: 175506011 bytes.
14:33:45	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.3.2	InstallSucceeded	waiting for install components to report healthy
14:33:45	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.3.0	RequirementsNotMet	one or more requirements couldn't be found
14:33:45	kuadrant-system	kubelet	kuadrant-operator-controller-manager-6ddf9554fc-f27mm	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:7aa6d6dbbde488260789adb94fa232a06d1e635cc2aaf86af24f8cf14a17eef8"
14:33:45	kuadrant-system	multus	kuadrant-operator-controller-manager-6ddf9554fc-f27mm	AddedInterface	Add eth0 [10.132.0.19/23] from ovn-kubernetes
14:33:45	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.3.0	RequirementsUnknown	requirements not yet checked
14:33:45	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.3.2	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
14:33:45	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-6ddf9554fc	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-6ddf9554fc-f27mm
14:33:45	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-6ddf9554fc from 0 to 1
14:33:47	kuadrant-system	multus	dns-operator-controller-manager-844548ff4c-qzx86	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
14:33:47	kuadrant-system	replicaset-controller	dns-operator-controller-manager-844548ff4c	SuccessfulCreate	Created pod: dns-operator-controller-manager-844548ff4c-qzx86
14:33:47	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.3.0	AllRequirementsMet	all requirements found, attempting install
14:33:47	kuadrant-system	kubelet	dns-operator-controller-manager-844548ff4c-qzx86	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:b4e7ba67509320ca9ac5d63cc4add987fad05b098c4a7cd8dd91f264731177cf"
14:33:47	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.3.0	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
14:33:47	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.3.0	InstallSucceeded	waiting for install components to report healthy
14:33:47	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-844548ff4c from 0 to 1
14:33:47	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.3.0	RequirementsNotMet	one or more requirements couldn't be found
14:33:47	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.3.0	RequirementsUnknown	requirements not yet checked
14:33:49	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.3.0	AllRequirementsMet	all requirements found, attempting install
14:33:49	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.3.0	InstallSucceeded	waiting for install components to report healthy
14:33:49	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-7587b89b76 from 0 to 1
14:33:49	kuadrant-system	replicaset-controller	authorino-operator-7587b89b76	SuccessfulCreate	Created pod: authorino-operator-7587b89b76-649lf
14:33:50	kuadrant-system	kubelet	authorino-operator-7587b89b76-649lf	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:4e581c0172549df72c6ef8978e96d771645bc36298d2ccf9240af4e71c088733"
14:33:50	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.3.0	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
14:33:50	kuadrant-system	multus	authorino-operator-7587b89b76-649lf	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
14:33:51	kuadrant-system	kubelet	dns-operator-controller-manager-844548ff4c-qzx86	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:b4e7ba67509320ca9ac5d63cc4add987fad05b098c4a7cd8dd91f264731177cf" in 3.815s (3.815s including waiting). Image size: 186790172 bytes.
14:33:51	kuadrant-system	kubelet	dns-operator-controller-manager-844548ff4c-qzx86	Created	Created container: manager
14:33:51	kuadrant-system	dns-operator-controller-manager-844548ff4c-qzx86_6bb12f93-44cf-44a2-ba02-4598afadfe6e	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-844548ff4c-qzx86_6bb12f93-44cf-44a2-ba02-4598afadfe6e became leader
14:33:51	kuadrant-system	kubelet	dns-operator-controller-manager-844548ff4c-qzx86	Started	Started container manager
14:33:52	kuadrant-system	authorino-operator-7587b89b76-649lf_316eecb1-d27b-4195-b925-97d269dd3c9d	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-7587b89b76-649lf_316eecb1-d27b-4195-b925-97d269dd3c9d became leader
14:33:52	kuadrant-system	kubelet	authorino-operator-7587b89b76-649lf	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:4e581c0172549df72c6ef8978e96d771645bc36298d2ccf9240af4e71c088733" in 1.858s (1.858s including waiting). Image size: 161991975 bytes.
14:33:52	kuadrant-system	kubelet	authorino-operator-7587b89b76-649lf	Created	Created container: manager
14:33:52	kuadrant-system	kubelet	kuadrant-operator-controller-manager-6ddf9554fc-f27mm	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:7aa6d6dbbde488260789adb94fa232a06d1e635cc2aaf86af24f8cf14a17eef8" in 6.845s (6.845s including waiting). Image size: 411782238 bytes.
14:33:52	kuadrant-system	kuadrant-operator-controller-manager-6ddf9554fc-f27mm_eb441c17-9146-46ae-acad-5dd38e4585ef	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-6ddf9554fc-f27mm_eb441c17-9146-46ae-acad-5dd38e4585ef became leader
14:33:52	kuadrant-system	kubelet	authorino-operator-7587b89b76-649lf	Started	Started container manager
14:33:52	kuadrant-system	kubelet	kuadrant-operator-controller-manager-6ddf9554fc-f27mm	Started	Started container manager
14:33:52	kuadrant-system	kubelet	kuadrant-operator-controller-manager-6ddf9554fc-f27mm	Created	Created container: manager
14:33:54	kuadrant-system	replicaset-controller	kuadrant-console-plugin-6c886788f8	SuccessfulCreate	Created pod: kuadrant-console-plugin-6c886788f8-ghmw2
14:33:54	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-6c886788f8 from 0 to 1
14:33:55	kuadrant-system	multus	kuadrant-console-plugin-6c886788f8-ghmw2	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
14:33:55	kuadrant-system	kubelet	kuadrant-console-plugin-6c886788f8-ghmw2	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:41755d5e16ee5409cfe8888cf5883b262820baea4eae449fcf61f06ab1756f57"
14:33:56	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.3.0	InstallSucceeded	install strategy completed with no errors
14:33:59	kuadrant-system	kubelet	kuadrant-console-plugin-6c886788f8-ghmw2	Created	Created container: kuadrant-console-plugin
14:33:59	kuadrant-system	kubelet	kuadrant-console-plugin-6c886788f8-ghmw2	Started	Started container kuadrant-console-plugin
14:33:59	kuadrant-system	kubelet	kuadrant-console-plugin-6c886788f8-ghmw2	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:41755d5e16ee5409cfe8888cf5883b262820baea4eae449fcf61f06ab1756f57" in 4.457s (4.457s including waiting). Image size: 339541674 bytes.
14:34:03	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.3.0	InstallSucceeded	install strategy completed with no errors
14:34:03	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.3.2	InstallSucceeded	install strategy completed with no errors
14:34:03	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.3.0	InstallSucceeded	install strategy completed with no errors
14:34:37	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-67566c68b4 from 0 to 1
14:34:37	kuadrant-system	multus	limitador-limitador-64c8f475fb-tkn5h	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
14:34:37	kuadrant-system	kubelet	limitador-limitador-64c8f475fb-tkn5h	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0"
14:34:37	kuadrant-system	replicaset-controller	limitador-limitador-64c8f475fb	SuccessfulCreate	Created pod: limitador-limitador-64c8f475fb-tkn5h
14:34:37	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-64c8f475fb from 0 to 1
14:34:37	kuadrant-system	replicaset-controller	limitador-limitador-67566c68b4	SuccessfulCreate	Created pod: limitador-limitador-67566c68b4-lx5zw
14:34:38	kuadrant-system	kubelet	authorino-674b59b84c-85wgl	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"
14:34:38	kuadrant-system	kubelet	limitador-limitador-67566c68b4-lx5zw	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0"
14:34:38	kuadrant-system	kubelet	authorino-79cbc94b89-dtgl2	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"
14:34:38	kuadrant-system	multus	limitador-limitador-67566c68b4-lx5zw	AddedInterface	Add eth0 [10.132.0.20/23] from ovn-kubernetes
14:34:38	kuadrant-system	multus	authorino-674b59b84c-85wgl	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
14:34:38	kuadrant-system	replicaset-controller	authorino-79cbc94b89	SuccessfulCreate	Created pod: authorino-79cbc94b89-dtgl2
14:34:38	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-79cbc94b89 from 0 to 1
14:34:38	kuadrant-system	replicaset-controller	authorino-674b59b84c	SuccessfulCreate	Created pod: authorino-674b59b84c-85wgl
14:34:38	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-674b59b84c from 0 to 1
14:34:38	kuadrant-system	multus	authorino-79cbc94b89-dtgl2	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
14:34:39	kuadrant-system	kubelet	limitador-limitador-64c8f475fb-tkn5h	Started	Started container limitador
14:34:39	kuadrant-system	kubelet	limitador-limitador-64c8f475fb-tkn5h	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0" in 1.395s (1.395s including waiting). Image size: 249943843 bytes.
14:34:39	kuadrant-system	kubelet	limitador-limitador-64c8f475fb-tkn5h	Created	Created container: limitador
14:34:42	kuadrant-system	kubelet	authorino-79cbc94b89-dtgl2	Created	Created container: authorino
14:34:42	kuadrant-system	kubelet	authorino-79cbc94b89-dtgl2	Started	Started container authorino
14:34:42	kuadrant-system	kubelet	authorino-79cbc94b89-dtgl2	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 3.849s (3.849s including waiting). Image size: 199256829 bytes.
14:34:42	kuadrant-system	kubelet	limitador-limitador-67566c68b4-lx5zw	Created	Created container: limitador
14:34:42	kuadrant-system	kubelet	limitador-limitador-67566c68b4-lx5zw	Started	Started container limitador
14:34:42	kuadrant-system	kubelet	authorino-674b59b84c-85wgl	Started	Started container authorino
14:34:42	kuadrant-system	kubelet	authorino-674b59b84c-85wgl	Created	Created container: authorino
14:34:42	kuadrant-system	kubelet	authorino-674b59b84c-85wgl	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 3.924s (3.924s including waiting). Image size: 199256829 bytes.
14:34:42	kuadrant-system	kubelet	limitador-limitador-67566c68b4-lx5zw	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0" in 3.855s (3.855s including waiting). Image size: 249943843 bytes.
14:34:43	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-674b59b84c from 1 to 0
14:34:43	kuadrant-system	replicaset-controller	authorino-674b59b84c	SuccessfulDelete	Deleted pod: authorino-674b59b84c-85wgl
14:34:45	kuadrant-system	kubelet	authorino-674b59b84c-85wgl	Killing	Stopping container authorino
14:34:53	kuadrant-system	kubelet	limitador-limitador-64c8f475fb-tkn5h	Killing	Stopping container limitador
14:34:53	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-64c8f475fb from 1 to 0
14:34:53	kuadrant-system	replicaset-controller	limitador-limitador-64c8f475fb	SuccessfulDelete	Deleted pod: limitador-limitador-64c8f475fb-tkn5h
14:35:02	kuadrant-system	replicaset-controller	authorino-68bd676465	SuccessfulCreate	Created pod: authorino-68bd676465-8cn2c
14:35:02	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-68bd676465 from 0 to 1
14:35:02	kuadrant-system	multus	authorino-68bd676465-8cn2c	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
14:35:02	kuadrant-system	kubelet	authorino-68bd676465-8cn2c	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"
14:35:03	kuadrant-system	kubelet	authorino-68bd676465-8cn2c	Created	Created container: authorino
14:35:03	kuadrant-system	kubelet	authorino-68bd676465-8cn2c	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 650ms (650ms including waiting). Image size: 199256829 bytes.
14:35:03	kuadrant-system	kubelet	authorino-68bd676465-8cn2c	Started	Started container authorino
14:35:04	kuadrant-system	replicaset-controller	authorino-79cbc94b89	SuccessfulDelete	Deleted pod: authorino-79cbc94b89-dtgl2
14:35:04	kuadrant-system	kubelet	authorino-79cbc94b89-dtgl2	Killing	Stopping container authorino
14:35:04	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-79cbc94b89 from 1 to 0
14:35:12	openshift-ingress	replicaset-controller	istiod-openshift-gateway-55ff986f96	SuccessfulCreate	Created pod: istiod-openshift-gateway-55ff986f96-6d7fr
14:35:12	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-55ff986f96 from 0 to 1
14:35:13	openshift-ingress	kubelet	istiod-openshift-gateway-55ff986f96-6d7fr	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8"
14:35:13	openshift-ingress	multus	istiod-openshift-gateway-55ff986f96-6d7fr	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
14:35:15	openshift-ingress	kubelet	istiod-openshift-gateway-55ff986f96-6d7fr	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8" in 2.42s (2.42s including waiting). Image size: 179322965 bytes.
14:35:15	openshift-ingress	kubelet	istiod-openshift-gateway-55ff986f96-6d7fr	Started	Started container discovery
14:35:15	openshift-ingress	kubelet	istiod-openshift-gateway-55ff986f96-6d7fr	Created	Created container: discovery
14:35:16	openshift-ingress	kubelet	istiod-openshift-gateway-55ff986f96-6d7fr	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
14:35:16	openshift-ingress	kubelet	istiod-openshift-gateway-55ff986f96-6d7fr	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:35:17	openshift-ingress	kubelet	istiod-openshift-gateway-7cd77c7ffd-bklns	Killing	Stopping container discovery
14:35:17	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7cd77c7ffd	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7cd77c7ffd-bklns
14:35:17	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7cd77c7ffd from 1 to 0
14:35:31 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:35:31 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:37:12	kserve-ci-e2e-test	deployment-controller	router-gateway-1-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-1-openshift-default-6c59fbf55c from 0 to 1
14:37:12	kserve-ci-e2e-test	replicaset-controller	router-gateway-1-openshift-default-6c59fbf55c	SuccessfulCreate	Created pod: router-gateway-1-openshift-default-6c59fbf55c-74r8s
14:37:12	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
14:37:13	kserve-ci-e2e-test	multus	router-gateway-1-openshift-default-6c59fbf55c-74r8s	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
14:37:13	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6c59fbf55c-74r8s	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" already present on machine
14:37:13	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6c59fbf55c-74r8s	Created	Created container: istio-proxy
14:37:13	kserve-ci-e2e-test	kubelet	router-gateway-1-openshift-default-6c59fbf55c-74r8s	Started	Started container istio-proxy
14:37:16	kserve-ci-e2e-test	service-controller	router-gateway-1-openshift-default	EnsuredLoadBalancer	Ensured load balancer
14:37:18 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-6fa1027a	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route" not found
14:37:19	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-6fa1027a	Created	Created v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs
14:37:20	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
14:37:20	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-6fa1027a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc
14:37:20	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-6fa1027a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve
14:37:20	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcf from 0 to 1
14:37:20	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-6fa1027a	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route
14:37:20	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18"
14:37:20	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-6fa1027a	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc
14:37:20	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcf	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw
14:37:23 (x13)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-configmap-ref-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found
14:37:24	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" in 3.943s (3.943s including waiting). Image size: 300957076 bytes.
14:37:24	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Created	Created container: storage-initializer
14:37:24	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Started	Started container storage-initializer
14:37:29	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
14:37:30	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.244s (1.244s including waiting). Image size: 98346788 bytes.
14:37:30	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Started	Started container main
14:37:30	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-646894c5cf	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf
14:37:30	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-646894c5cf from 0 to 1
14:37:30	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Created	Created container: main
14:37:30	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
14:37:30	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve
14:37:30	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc
14:37:30	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa
14:37:31	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Created	Created container: storage-initializer
14:37:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service
14:37:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb
14:37:31	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:37:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
14:37:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role
14:37:31 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc
14:37:31	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Started	Started container storage-initializer
14:37:31	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
14:37:31	kserve-ci-e2e-test	replicaset-controller	scheduler-configmap-ref-test-kserve-router-scheduler-74dc589989	SuccessfulCreate	Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s
14:37:31	kserve-ci-e2e-test	deployment-controller	scheduler-configmap-ref-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-74dc589989 from 0 to 1
14:37:31	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18"
14:37:31	kserve-ci-e2e-test	multus	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
14:37:31	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler
14:37:33	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"
14:37:34	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Created	Created container: storage-initializer
14:37:34	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" in 3.321s (3.321s including waiting). Image size: 300957076 bytes.
14:37:34	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Started	Started container storage-initializer
14:37:35	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.6s (1.6s including waiting). Image size: 87241021 bytes.
14:37:35	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Created	Created container: main
14:37:35	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Started	Started container main
14:37:35	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Pulling	Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"
14:37:55	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-6fa1027a	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs
14:38:00	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs
14:38:03	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Started	Started container tokenizer
14:38:03	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 28.786s (28.786s including waiting). Image size: 2907020949 bytes.
14:38:03	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Created	Created container: tokenizer
14:39:11	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
14:39:12	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.269s (1.269s including waiting). Image size: 98346788 bytes.
14:39:12	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Created	Created container: main
14:39:12	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Started	Started container main
14:39:22	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Killing	Stopping container main
14:39:22	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-router-scheduler-74dc59cd9s	Killing	Stopping container tokenizer
14:39:22	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-configmap-ref-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb
14:39:23 (x2)	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	FailedMount	MountVolume.SetUp failed for volume "tls-certs" : secret "scheduler-configmap-ref-test-kserve-self-signed-certs" not found
14:39:24	kserve-ci-e2e-test	kubelet	scheduler-configmap-ref-test-kserve-646894c5cf-pg8tf	Killing	Stopping container main
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler
14:39:28	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-9d798b947	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-9d798b947-brwzq
14:39:28	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d47767	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9
14:39:28	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
14:39:28 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-ha-replicas-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found
14:39:28	kserve-ci-e2e-test	replicaset-controller	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d47767	SuccessfulCreate	Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7
14:39:28	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-9d798b947 from 0 to 1
14:39:28	kserve-ci-e2e-test	deployment-controller	scheduler-ha-replicas-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-fc6d47767 from 0 to 2
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Started	Started container storage-initializer
14:39:29 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc
14:39:29	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:39:29	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Created	Created container: storage-initializer
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Created	Created container: storage-initializer
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Started	Started container storage-initializer
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Created	Created container: storage-initializer
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:39:29	kserve-ci-e2e-test	multus	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
14:39:29	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Started	Started container storage-initializer
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Started	Started container main
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Started	Started container tokenizer
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Created	Created container: tokenizer
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Created	Created container: main
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:39:30	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"
14:39:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Pulling	Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"
14:39:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Started	Started container main
14:39:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Created	Created container: main
14:39:31	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.021s (1.021s including waiting). Image size: 87241021 bytes.
14:39:36	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
14:39:59	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Created	Created container: tokenizer
14:39:59	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-7946664fcfhhhfw	Killing	Stopping container main
14:39:59	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Started	Started container tokenizer
14:39:59	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 28.572s (28.572s including waiting). Image size: 2907020949 bytes.
14:40:10 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-55f7ae4a	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route" not found
14:40:13	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-role
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler
14:40:13	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
14:40:13	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Created	Created container: storage-initializer
14:40:13	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Started	Started container storage-initializer
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode
14:40:13	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Created	Created container: storage-initializer
14:40:13	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:40:13	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd from 0 to 1
14:40:13	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Started	Started container storage-initializer
14:40:13	kserve-ci-e2e-test	replicaset-controller	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-56996cc696	SuccessfulCreate	Created pod: llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-service
14:40:13	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz
14:40:13	kserve-ci-e2e-test	deployment-controller	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-56996cc696 from 0 to 1
14:40:13	kserve-ci-e2e-test	multus	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-workload-svc
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs
14:40:13	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-rb
14:40:14 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-shadow-svc
14:40:15	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Created	Created container: main
14:40:15	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:40:15	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:40:15	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Started	Started container main
14:40:15	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Created	Created container: tokenizer
14:40:15	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Started	Started container tokenizer
14:40:24	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Started	Started container main
14:40:24	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Created	Created container: main
14:40:24	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:40:33	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs
14:40:44 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	precise-prefix-cache-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found
14:40:49	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-75469f89c9	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-75469f89c9-xmbsz
14:40:49	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa
14:40:49	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc
14:40:49	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
14:40:49	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve
14:40:49	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
14:40:49	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-75469f89c9	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-75469f89c9-dfbfc
14:40:49	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-75469f89c9 from 0 to 2
14:40:50 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Started	Started container storage-initializer
14:40:50	kserve-ci-e2e-test	deployment-controller	precise-prefix-cache-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-667fbc78df from 0 to 1
14:40:50	kserve-ci-e2e-test	replicaset-controller	precise-prefix-cache-test-kserve-router-scheduler-667fbc78df	SuccessfulCreate	Created pod: precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q
14:40:50	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:40:50	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb
14:40:50	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
14:40:50	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Created	Created container: storage-initializer
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Created	Created container: storage-initializer
14:40:50	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service
14:40:50	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Created	Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18"
14:40:50	kserve-ci-e2e-test	multus	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
14:40:50	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Started	Started container storage-initializer
14:40:51	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Killing	Stopping container main
14:40:51	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-9d798b947-brwzq	Killing	Stopping container main
14:40:51	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d4779jjd9	Killing	Stopping container tokenizer
14:40:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:40:51	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-ha-replicas-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb
14:40:51	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Killing	Stopping container main
14:40:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Created	Created container: tokenizer
14:40:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:40:51 (x10)	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
14:40:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Started	Started container tokenizer
14:40:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Started	Started container main
14:40:51	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Created	Created container: main
14:40:51	kserve-ci-e2e-test	kubelet	scheduler-ha-replicas-test-kserve-router-scheduler-fc6d477mptc7	Killing	Stopping container tokenizer
14:40:53	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Created	Created container: storage-initializer
14:40:53	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" in 3.388s (3.388s including waiting). Image size: 300957076 bytes.
14:40:53	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Started	Started container storage-initializer
14:40:58	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
14:40:59	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Created	Created container: main
14:40:59	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Started	Started container main
14:40:59	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.171s (1.171s including waiting). Image size: 98346788 bytes.
14:41:01 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Updated	Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
14:41:11 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Unhealthy	Liveness probe failed: timeout: failed to connect service "10.133.0.52:9003" within 1s: context deadline exceeded
14:41:20	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Started	Started container main
14:41:20	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Created	Created container: main
14:41:20	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:41:31	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Killing	Stopping container main
14:41:31	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-router-scheduler-667fbc78lr57q	Killing	Stopping container tokenizer
14:41:31	kserve-ci-e2e-test	LLMInferenceServiceController	precise-prefix-cache-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb
14:41:31	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-xmbsz	Killing	Stopping container main
14:41:32 (x2)	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	FailedMount	MountVolume.SetUp failed for volume "tls-certs" : secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
14:41:32	kserve-ci-e2e-test	kubelet	precise-prefix-cache-test-kserve-75469f89c9-dfbfc	Killing	Stopping container main
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
14:41:37	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-7cb4fb9c5d from 0 to 1
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
14:41:37	kserve-ci-e2e-test	multus	stop-feature-test-kserve-88fb48646-cq9c6	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
14:41:37	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:41:37	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Created	Created container: storage-initializer
14:41:37	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
14:41:37	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
14:41:37	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-88fb48646	SuccessfulCreate	Created pod: stop-feature-test-kserve-88fb48646-cq9c6
14:41:37	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-88fb48646 from 0 to 1
14:41:37	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Started	Started container storage-initializer
14:41:37	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
14:41:37	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb
14:41:38	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Created	Created container: storage-initializer
14:41:38	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Started	Started container storage-initializer
14:41:38 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc
14:41:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Started	Started container tokenizer
14:41:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:41:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Created	Created container: main
14:41:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Started	Started container main
14:41:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Created	Created container: tokenizer
14:41:39	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:41:42	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1"
14:41:47	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Unhealthy	Startup probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused
14:41:53	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Updated	Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
14:42:04	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Pulling	Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1"
14:42:24	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" in 41.503s (41.503s including waiting). Image size: 3458932386 bytes.
14:42:24	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Created	Created container: main
14:42:24	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Started	Started container main
14:42:49	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Pulled	Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" in 45.04s (45.04s including waiting). Image size: 3458932386 bytes.
14:42:49	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Created	Created container: main
14:42:49	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Started	Started container main
14:43:47 (x9)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Unhealthy	Startup probe failed: Get "https://10.133.0.53:8000/health": dial tcp 10.133.0.53:8000: connect: connection refused
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1alpha2.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool
14:43:59	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Killing	Stopping container main
14:43:59	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-vb62h	Killing	Stopping container tokenizer
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
14:43:59	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve
14:43:59	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-cq9c6	Killing	Stopping container main
14:44:13 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Unhealthy	Startup probe failed: Get "https://10.132.0.32:8000/health": dial tcp 10.132.0.32:8000: connect: connection refused
14:44:17	kserve-ci-e2e-test	multus	stop-feature-test-kserve-88fb48646-7mxhg	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
14:44:17	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-88fb48646 from 0 to 1
14:44:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Started	Started container storage-initializer
14:44:17	kserve-ci-e2e-test	deployment-controller	stop-feature-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set stop-feature-test-kserve-router-scheduler-7cb4fb9c5d from 0 to 1
14:44:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:44:17	kserve-ci-e2e-test	multus	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
14:44:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Created	Created container: storage-initializer
14:44:17 (x23)	kserve-ci-e2e-test	OpenDataHubModelController	stop-feature-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found
14:44:17	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d	SuccessfulCreate	Created pod: stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m
14:44:17	kserve-ci-e2e-test	replicaset-controller	stop-feature-test-kserve-88fb48646	SuccessfulCreate	Created pod: stop-feature-test-kserve-88fb48646-7mxhg
14:44:17	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:44:18	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Created	Created container: storage-initializer
14:44:18	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Started	Started container storage-initializer
14:44:19	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:44:19	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:44:19	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Started	Started container main
14:44:19	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Started	Started container tokenizer
14:44:19	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Created	Created container: tokenizer
14:44:19	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Created	Created container: main
14:44:37	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Killing	Stopping container tokenizer
14:44:37	kserve-ci-e2e-test	kubelet	llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekbxkz	Killing	Stopping container main
14:44:37	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-fbc44c7dd-k98sz	Killing	Stopping container main
14:44:37	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-55f7ae4a	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode
14:44:42 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found
14:44:43	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
14:44:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Started	Started container storage-initializer
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
14:44:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler
14:44:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Created	Created container: storage-initializer
14:44:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role
14:44:44	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
14:44:44	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-router-scheduler-79f9c87c88	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns
14:44:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Started	Started container storage-initializer
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc
14:44:44	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-test-kserve-6cbfb55898	SuccessfulCreate	Created pod: custom-route-timeout-test-kserve-6cbfb55898-jn9bs
14:44:44	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Created	Created container: storage-initializer
14:44:44	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve
14:44:44	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-79f9c87c88 from 0 to 1
14:44:44	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-test-kserve-6cbfb55898 from 0 to 1
14:44:44	kserve-ci-e2e-test	multus	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
14:44:45 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc
14:44:45	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:44:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:44:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Started	Started container tokenizer
14:44:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Created	Created container: main
14:44:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Started	Started container main
14:44:46	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Created	Created container: tokenizer
14:44:49	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Started	Started container main
14:44:49	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Created	Created container: main
14:44:49	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:44:55	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
14:45:25	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:45:25	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Created	Created container: main
14:45:25	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Started	Started container main
14:46:14 (x9)	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Unhealthy	Startup probe failed: Get "https://10.132.0.34:8000/health": dial tcp 10.132.0.34:8000: connect: connection refused
14:46:29	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Killing	Stopping container tokenizer
14:46:29 (x2)	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-1
14:46:29	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-router-scheduler-79f9c87cndzns	Killing	Stopping container main
14:46:29	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb
14:46:29	kserve-ci-e2e-test	kubelet	custom-route-timeout-test-kserve-6cbfb55898-jn9bs	Killing	Stopping container main
14:46:40	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-5cfbdd5558 from 0 to 1
14:46:40	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-5cfbdd5558	SuccessfulCreate	Created pod: router-with-refs-test-kserve-5cfbdd5558-h5cjz
14:46:40	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa
14:46:40	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc
14:46:40	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve
14:46:40	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
14:46:41	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-5cfbdd5558-h5cjz	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
14:46:41	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cfbdd5558-h5cjz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:46:41	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cfbdd5558-h5cjz	Created	Created container: storage-initializer
14:46:41	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:46:41	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service
14:46:41	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler
14:46:41	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb
14:46:41	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role
14:46:41	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
14:46:41	kserve-ci-e2e-test	deployment-controller	router-with-refs-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-test-kserve-router-scheduler-55ddf8c976 from 0 to 1
14:46:41	kserve-ci-e2e-test	replicaset-controller	router-with-refs-test-kserve-router-scheduler-55ddf8c976	SuccessfulCreate	Created pod: router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw
14:46:41	kserve-ci-e2e-test	multus	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
14:46:41	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cfbdd5558-h5cjz	Started	Started container storage-initializer
14:46:41	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Created	Created container: storage-initializer
14:46:41	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Started	Started container storage-initializer
14:46:42 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc
14:46:43	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:46:43	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Started	Started container main
14:46:43	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Created	Created container: main
14:46:43	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:46:43	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Created	Created container: tokenizer
14:46:43	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Started	Started container tokenizer
14:46:47 (x9)	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Unhealthy	Startup probe failed: Get "https://10.133.0.55:8000/health": dial tcp 10.133.0.55:8000: connect: connection refused
14:47:11	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-88fb48646-7mxhg	Killing	Stopping container main
14:47:11	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Killing	Stopping container main
14:47:11	kserve-ci-e2e-test	kubelet	stop-feature-test-kserve-router-scheduler-7cb4fb9c5d-4m59m	Killing	Stopping container tokenizer
14:47:11 (x4)	kserve-ci-e2e-test	LLMInferenceServiceController	stop-feature-test	Deleted	(combined from similar events): Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb
14:47:11 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
14:48:13 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-02ac88e7	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route" not found
14:48:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-02ac88e7	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve
14:48:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-02ac88e7	Created	Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc
14:48:17	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4d	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x
14:48:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-02ac88e7	Created	Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route
14:48:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-02ac88e7	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc
14:48:17	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4d from 0 to 1
14:48:17	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-02ac88e7	Created	Created v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs
14:48:18	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Started	Started container storage-initializer
14:48:18	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Created	Created container: storage-initializer
14:48:18	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:48:18	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
14:48:23	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-02ac88e7	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs
14:49:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Created	Created container: main
14:49:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:49:54	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Started	Started container main
14:51:18 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Unhealthy	Startup probe failed: Get "https://10.133.0.57:8000/health": dial tcp 10.133.0.57:8000: connect: connection refused
14:51:32	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7969569c4dsmh7x	Killing	Stopping container main
14:51:42 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-f312f5ec	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route" not found
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-scc
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa
14:51:45	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn successful
14:51:45 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn	GroupsProgressing	Replicas are progressing, with 0 groups ready of total 1 groups
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs
14:51:45	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn	CreatingRevision	Creating revision with key 7cfc47b46 for a newly created LeaderWorkerSet
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-workload-svc
14:51:45	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn	GroupsProgressing	Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-role
14:51:45	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-rb
14:51:45	kserve-ci-e2e-test	replicaset-controller	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-678bcc58b6	SuccessfulCreate	Created pod: llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d
14:51:45	kserve-ci-e2e-test	statefulset-controller	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	SuccessfulCreate	create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 successful
14:51:45	kserve-ci-e2e-test	deployment-controller	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-678bcc58b6 from 0 to 1
14:51:45	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn	GroupsProgressing	Created leader statefulset llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Created	Created container: storage-initializer
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Started	Started container storage-initializer
14:51:46	kserve-ci-e2e-test	multus	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Created	Created container: storage-initializer
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Started	Started container storage-initializer
14:51:46	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:51:46 (x8)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-shadow-svc
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Started	Started container storage-initializer
14:51:46	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:51:46	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Created	Created container: storage-initializer
14:51:47	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Created	Created container: main
14:51:47	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Started	Started container tokenizer
14:51:47	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:51:47	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Started	Started container main
14:51:47	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:51:47	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Created	Created container: tokenizer
14:51:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Started	Started container main
14:51:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Created	Created container: main
14:51:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:52:02	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs
14:52:40	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Created	Created container: main
14:52:40	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Started	Started container main
14:52:40	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:53:16 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Unhealthy	Startup probe failed: Get "https://10.133.0.58:8000/health": dial tcp 10.133.0.58:8000: connect: connection refused
14:53:26 (x2)	kserve-ci-e2e-test	leaderworkerset	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn	AllGroupsReady	All replicas are ready, with 1 groups ready of total 1 groups
14:53:33	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0	Killing	Stopping container main
14:53:33	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Killing	Stopping container main
14:53:33	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-f312f5ec	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode
14:53:33	kserve-ci-e2e-test	kubelet	llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sches594d	Killing	Stopping container tokenizer
14:53:34	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1	Killing	Stopping container main
14:53:44 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	scheduler-inline-config-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found
14:53:53	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-65579b6fcf from 0 to 1
14:53:53	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-65579b6fcf	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-65579b6fcf-5z89v
14:53:53	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
14:53:53	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:53:53	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Created	Created container: storage-initializer
14:53:53	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Started	Started container storage-initializer
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve
14:53:53	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
14:53:53	kserve-ci-e2e-test	deployment-controller	scheduler-inline-config-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-55cc9cb695 from 0 to 1
14:53:53	kserve-ci-e2e-test	replicaset-controller	scheduler-inline-config-test-kserve-router-scheduler-55cc9cb695	SuccessfulCreate	Created pod: scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc
14:53:54	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Created	Created container: storage-initializer
14:53:54	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:53:54	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Started	Started container storage-initializer
14:53:54	kserve-ci-e2e-test	multus	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	AddedInterface	Add eth0 [10.133.0.61/23] from ovn-kubernetes
14:53:54 (x6)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc
14:53:55	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Started	Started container tokenizer
14:53:55	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:53:55	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Created	Created container: main
14:53:55	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Started	Started container main
14:53:55	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:53:55	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Created	Created container: tokenizer
14:54:03	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine
14:54:03	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Created	Created container: main
14:54:03	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Started	Started container main
14:54:16 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Updated	Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
14:54:26	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Killing	Stopping container tokenizer
14:54:26	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-router-scheduler-55cc9h69nc	Killing	Stopping container main
14:54:26	kserve-ci-e2e-test	LLMInferenceServiceController	scheduler-inline-config-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
14:54:26	kserve-ci-e2e-test	kubelet	scheduler-inline-config-test-kserve-65579b6fcf-5z89v	Killing	Stopping container main
14:54:31 (x11)	kserve-ci-e2e-test	OpenDataHubModelController	custom-route-timeout-pd-test	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found
14:54:35	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-578d77b467 from 0 to 1
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb
14:54:35	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-prefill-578d77b467	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve
14:54:35	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-7cbf8d89d from 0 to 1
14:54:35	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-7cbf8d89d	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill
14:54:35	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Created	Created container: storage-initializer
14:54:36	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Pulling	Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"
14:54:36	kserve-ci-e2e-test	deployment-controller	custom-route-timeout-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-647b5ccccc from 0 to 1
14:54:36 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Started	Started container storage-initializer
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Created	Created container: storage-initializer
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:54:36	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
14:54:36	kserve-ci-e2e-test	replicaset-controller	custom-route-timeout-pd-test-kserve-router-scheduler-647b5ccccc	SuccessfulCreate	Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn
14:54:36	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Started	Started container storage-initializer
14:54:36	kserve-ci-e2e-test	multus	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
14:54:37	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Started	Started container llm-d-routing-sidecar
14:54:37	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Created	Created container: llm-d-routing-sidecar
14:54:37	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
14:54:37	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 811ms (811ms including waiting). Image size: 67767940 bytes.
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Created	Created container: tokenizer
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Started	Started container tokenizer
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Created	Created container: main
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Created	Created container: storage-initializer
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Started	Started container storage-initializer
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:54:38	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Started	Started container main
14:54:43	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:54:43	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Created	Created container: main
14:54:43	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Started	Started container main
14:54:50 (x2)	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
14:56:06 (x9)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Unhealthy	Startup probe failed: Get "https://10.133.0.62:8001/health": dial tcp 10.133.0.62:8001: connect: connection refused
14:58:18	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Started	Started container main
14:58:18	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
14:58:18	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Created	Created container: main
14:59:46 (x9)	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Unhealthy	Startup probe failed: Get "https://10.133.0.63:8000/health": dial tcp 10.133.0.63:8000: connect: connection refused
15:00:07	kserve-ci-e2e-test	deployment-controller	router-gateway-2-openshift-default	ScalingReplicaSet	Scaled up replica set router-gateway-2-openshift-default-6866b85949 from 0 to 1
15:00:07	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuringLoadBalancer	Ensuring load balancer
15:00:07	kserve-ci-e2e-test	replicaset-controller	router-gateway-2-openshift-default-6866b85949	SuccessfulCreate	Created pod: router-gateway-2-openshift-default-6866b85949-nt4x9
15:00:07	kserve-ci-e2e-test	gateway_labeler_controller	router-gateway-2	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway router-gateway-2
15:00:07	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Killing	Stopping container tokenizer
15:00:07	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Killing	Stopping container main
15:00:07	kserve-ci-e2e-test	LLMInferenceServiceController	custom-route-timeout-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb
15:00:07	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-prefill-578d77b467-zfd6z	Killing	Stopping container main
15:00:07	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Killing	Stopping container main
15:00:07	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-7cbf8d89d-8j8f9	Killing	Stopping container llm-d-routing-sidecar
15:00:08	kserve-ci-e2e-test	multus	router-gateway-2-openshift-default-6866b85949-nt4x9	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
15:00:08	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-6866b85949-nt4x9	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe"
15:00:09	kserve-ci-e2e-test	kubelet	custom-route-timeout-pd-test-kserve-router-scheduler-647b5sprvn	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.132.0.39:9003" within 1s: context deadline exceeded
15:00:11	kserve-ci-e2e-test	service-controller	router-gateway-2-openshift-default	EnsuredLoadBalancer	Ensured load balancer
15:00:14	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-6866b85949-nt4x9	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" in 5.862s (5.862s including waiting). Image size: 163191192 bytes.
15:00:14	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-6866b85949-nt4x9	Started	Started container istio-proxy
15:00:14	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-6866b85949-nt4x9	Created	Created container: istio-proxy
15:00:17 (x3)	kserve-ci-e2e-test	kubelet	router-gateway-2-openshift-default-6866b85949-nt4x9	Unhealthy	Startup probe failed: Get "http://10.134.0.29:15021/healthz/ready": dial tcp 10.134.0.29:15021: connect: connection refused
15:00:24	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-76696559d6 from 0 to 1
15:00:24	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-76696559d6-c5f2b	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve
15:00:24	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
15:00:24	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9c from 0 to 1
15:00:24	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9c	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp
15:00:24	kserve-ci-e2e-test	deployment-controller	router-with-refs-pd-test-kserve-prefill	ScalingReplicaSet	Scaled up replica set router-with-refs-pd-test-kserve-prefill-5c8777dc from 0 to 1
15:00:24	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-prefill-5c8777dc	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd
15:00:24	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Started	Started container storage-initializer
15:00:24	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Created	Created container: storage-initializer
15:00:24	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:00:24	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	AddedInterface	Add eth0 [10.133.0.64/23] from ovn-kubernetes
15:00:24	kserve-ci-e2e-test	replicaset-controller	router-with-refs-pd-test-kserve-76696559d6	SuccessfulCreate	Created pod: router-with-refs-pd-test-kserve-76696559d6-c5f2b
15:00:24	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Pulling	Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"
15:00:25	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:00:25	kserve-ci-e2e-test	multus	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
15:00:25	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Created	Created container: storage-initializer
15:00:25	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Started	Started container storage-initializer
15:00:25 (x9)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc
15:00:26	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Started	Started container tokenizer
15:00:26	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
15:00:26	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Created	Created container: main
15:00:26	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Started	Started container main
15:00:26	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
15:00:26	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Created	Created container: tokenizer
15:00:55	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Started	Started container main
15:00:55	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
15:00:55	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Created	Created container: main
15:00:57	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Started	Started container llm-d-routing-sidecar
15:00:57	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Created	Created container: llm-d-routing-sidecar
15:00:57	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 32.504s (32.504s including waiting). Image size: 67767940 bytes.
15:00:58	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Started	Started container storage-initializer
15:00:58	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Created	Created container: storage-initializer
15:00:58	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:01:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
15:01:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Started	Started container main
15:01:03	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Created	Created container: main
15:01:04 (x5)	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Updated	Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
15:01:33	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Killing	Stopping container main
15:01:33	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
15:01:33	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-router-scheduler-55ddf8c976-qb7gw	Killing	Stopping container tokenizer
15:01:34	kserve-ci-e2e-test	kubelet	router-with-refs-test-kserve-5cfbdd5558-h5cjz	Killing	Stopping container storage-initializer
15:01:44 (x12)	kserve-ci-e2e-test	OpenDataHubModelController	llmisvc-model-fb-opt-125m-route-05aa9bba	ReconcileError	Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route" not found
15:01:51	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7fff48ccc8 from 0 to 1
15:01:51	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7fff48ccc8	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-workload-svc
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve
15:01:51	kserve-ci-e2e-test	deployment-controller	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve	ScalingReplicaSet	Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59f from 0 to 1
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-rb
15:01:51	kserve-ci-e2e-test	replicaset-controller	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59f	SuccessfulCreate	Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve
15:01:51	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-role
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Created	Created container: storage-initializer
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Pulled	Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine
15:01:52	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	AddedInterface	Add eth0 [10.133.0.66/23] from ovn-kubernetes
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Started	Started container storage-initializer
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Created	Created container: storage-initializer
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Started	Started container llm-d-routing-sidecar
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Created	Created container: llm-d-routing-sidecar
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Started	Started container storage-initializer
15:01:52	kserve-ci-e2e-test	multus	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	AddedInterface	Add eth0 [10.133.0.65/23] from ovn-kubernetes
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Created	Created container: storage-initializer
15:01:52	kserve-ci-e2e-test	replicaset-controller	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-7454c969c4	SuccessfulCreate	Created pod: llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn
15:01:52	kserve-ci-e2e-test	deployment-controller	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler	ScalingReplicaSet	Scaled up replica set llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-7454c969c4 from 0 to 1
15:01:52	kserve-ci-e2e-test	multus	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:01:52	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Started	Started container storage-initializer
15:01:52 (x10)	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Created	(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-shadow-svc
15:01:53	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Pulled	Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
15:01:53	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Created	Created container: main
15:01:53	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine
15:01:53	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Created	Created container: tokenizer
15:01:53	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Started	Started container tokenizer
15:01:53	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Started	Started container main
15:01:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Started	Started container main
15:01:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Created	Created container: main
15:01:57	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
15:02:07	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Updated	Updated v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs
15:02:24 (x9)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Unhealthy	Startup probe failed: Get "https://10.133.0.64:8000/health": dial tcp 10.133.0.64:8000: connect: connection refused
15:02:24 (x9)	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Unhealthy	Startup probe failed: Get "https://10.132.0.40:8001/health": dial tcp 10.132.0.40:8001: connect: connection refused
15:02:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Killing	Stopping container main
15:02:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Killing	Stopping container main
15:02:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Killing	Stopping container tokenizer
15:02:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-prefill-5c8777dc-txbgd	Killing	Stopping container main
15:02:45	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-76696559d6-c5f2b	Killing	Stopping container llm-d-routing-sidecar
15:02:45	kserve-ci-e2e-test	LLMInferenceServiceController	router-with-refs-pd-test	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
15:02:47	kserve-ci-e2e-test	kubelet	router-with-refs-pd-test-kserve-router-scheduler-5f9bf4f9csrhbp	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.132.0.41:9003" within 1s: context deadline exceeded
15:02:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Started	Started container main
15:02:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Created	Created container: main
15:02:52	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Pulled	Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
15:03:22 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Unhealthy	Startup probe failed: Get "https://10.133.0.65:8001/health": dial tcp 10.133.0.65:8001: connect: connection refused
15:04:22 (x9)	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Unhealthy	Startup probe failed: Get "https://10.133.0.66:8000/health": dial tcp 10.133.0.66:8000: connect: connection refused
15:04:43	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7f2szqt	Killing	Stopping container main
15:04:43	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Killing	Stopping container main
15:04:43	kserve-ci-e2e-test	kubelet	llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schegn5bn	Killing	Stopping container tokenizer
15:04:43	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Killing	Stopping container llm-d-routing-sidecar
15:04:43	kserve-ci-e2e-test	kubelet	llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-568864c59fhvjqg	Killing	Stopping container main
15:04:43	kserve-ci-e2e-test	LLMInferenceServiceController	llmisvc-model-fb-opt-125m-route-05aa9bba	Deleted	Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode