Events

Time	Namespace	Component	RelatedObject	Reason	Message
07:39:29	e2e-gw0-5c1f590758		isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Scheduled	Successfully assigned e2e-gw0-5c1f590758/isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs to ip-10-0-136-240.us-east-2.compute.internal
07:51:29	e2e-gw0-5c1f590758		success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Scheduled	Successfully assigned e2e-gw0-5c1f590758/success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk to ip-10-0-134-26.us-east-2.compute.internal
07:52:39	e2e-gw0-5c1f590758		sequence-graph-b7c3b-75dbd6c6f4-hqpx5	Scheduled	Successfully assigned e2e-gw0-5c1f590758/sequence-graph-b7c3b-75dbd6c6f4-hqpx5 to ip-10-0-134-26.us-east-2.compute.internal
08:02:27	e2e-gw0-5c1f590758		success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Scheduled	Successfully assigned e2e-gw0-5c1f590758/success-200-isvc-8ae00-predictor-5cd6564857-gskx9 to ip-10-0-134-26.us-east-2.compute.internal
07:52:56	e2e-gw0-5c1f590758		success-200-isvc-45944-predictor-7859fc774f-rbsfn	Scheduled	Successfully assigned e2e-gw0-5c1f590758/success-200-isvc-45944-predictor-7859fc774f-rbsfn to ip-10-0-134-26.us-east-2.compute.internal
07:41:57	e2e-gw0-5c1f590758		error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Scheduled	Successfully assigned e2e-gw0-5c1f590758/error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv to ip-10-0-130-48.us-east-2.compute.internal
07:39:28	e2e-gw0-5c1f590758		isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Scheduled	Successfully assigned e2e-gw0-5c1f590758/isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp to ip-10-0-136-240.us-east-2.compute.internal
08:02:27	e2e-gw0-5c1f590758		error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Scheduled	Successfully assigned e2e-gw0-5c1f590758/error-404-isvc-8ae00-predictor-7f849b78d-rjwg7 to ip-10-0-130-48.us-east-2.compute.internal
07:43:07	e2e-gw0-5c1f590758		sequence-graph-48632-ff6f99496-xjtvw	Scheduled	Successfully assigned e2e-gw0-5c1f590758/sequence-graph-48632-ff6f99496-xjtvw to ip-10-0-134-26.us-east-2.compute.internal
07:41:57	e2e-gw0-5c1f590758		success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Scheduled	Successfully assigned e2e-gw0-5c1f590758/success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs to ip-10-0-134-26.us-east-2.compute.internal
07:39:28	e2e-gw0-5c1f590758		isvc-xgboost-graph-predictor-698b9676f-njgxs	Scheduled	Successfully assigned e2e-gw0-5c1f590758/isvc-xgboost-graph-predictor-698b9676f-njgxs to ip-10-0-134-26.us-east-2.compute.internal
08:03:37	e2e-gw0-5c1f590758		switch-graph-8ae00-86769987cd-rhthk	Scheduled	Successfully assigned e2e-gw0-5c1f590758/switch-graph-8ae00-86769987cd-rhthk to ip-10-0-130-48.us-east-2.compute.internal
07:52:56	e2e-gw0-5c1f590758		error-404-isvc-45944-predictor-7f69cd48-45twp	Scheduled	Successfully assigned e2e-gw0-5c1f590758/error-404-isvc-45944-predictor-7f69cd48-45twp to ip-10-0-130-48.us-east-2.compute.internal
07:54:06	e2e-gw0-5c1f590758		sequence-graph-45944-6dd4d9b5bd-rxbc2	Scheduled	Successfully assigned e2e-gw0-5c1f590758/sequence-graph-45944-6dd4d9b5bd-rxbc2 to ip-10-0-134-26.us-east-2.compute.internal
07:51:29	e2e-gw0-5c1f590758		error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Scheduled	Successfully assigned e2e-gw0-5c1f590758/error-404-isvc-b7c3b-predictor-54b5879974-6vrkz to ip-10-0-130-48.us-east-2.compute.internal
07:41:38	e2e-gw0-5c1f590758		model-chainer-6b979c876d-j6srw	Scheduled	Successfully assigned e2e-gw0-5c1f590758/model-chainer-6b979c876d-j6srw to ip-10-0-130-48.us-east-2.compute.internal
07:39:28	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-sklearn-graph-1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again
07:39:28	e2e-gw0-5c1f590758	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-698b9676f from 0 to 1
07:39:28	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-sklearn-graph-1	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again
07:39:28	e2e-gw0-5c1f590758	replicaset-controller	isvc-xgboost-graph-predictor-698b9676f	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-698b9676f-njgxs
07:39:28	e2e-gw0-5c1f590758	replicaset-controller	isvc-sklearn-graph-1-predictor-6ffd4655dd	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp
07:39:28	e2e-gw0-5c1f590758	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-6ffd4655dd from 0 to 1
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937" already present on machine and can be accessed by the pod
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937" already present on machine and can be accessed by the pod
07:39:29	e2e-gw0-5c1f590758	replicaset-controller	isvc-sklearn-graph-2-predictor-ddd6c579c	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs
07:39:29	e2e-gw0-5c1f590758	multus	isvc-xgboost-graph-predictor-698b9676f-njgxs	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937"
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Started	Container started
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Created	Container created
07:39:29	e2e-gw0-5c1f590758	multus	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	AddedInterface	Add eth0 [10.133.0.18/23] from ovn-kubernetes
07:39:29	e2e-gw0-5c1f590758	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-ddd6c579c from 0 to 1
07:39:29	e2e-gw0-5c1f590758	multus	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	AddedInterface	Add eth0 [10.133.0.17/23] from ovn-kubernetes
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Started	Container started
07:39:29	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Created	Container created
07:39:33	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2157"
07:39:34	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2157"
07:39:35	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Started	Container started
07:39:35	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Created	Container created
07:39:35	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937" in 5.902s (5.902s including waiting). Image size: 330721193 bytes.
07:39:40	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Pulling	Pulling image "kserve/xgbserver:latest"
07:39:51	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2157" in 18.649s (18.649s including waiting). Image size: 1629727116 bytes.
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Started	Container started
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Started	Container started
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Pulling	Pulling image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f"
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Created	Container created
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Created	Container created
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2157" in 18.381s (18.381s including waiting). Image size: 1629727116 bytes.
07:39:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Pulling	Pulling image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f"
07:39:54	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" in 2.541s (2.541s including waiting). Image size: 184747123 bytes.
07:39:54	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Created	Container created
07:39:54	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" in 2.091s (2.091s including waiting). Image size: 184747123 bytes.
07:39:54	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Created	Container created
07:39:54	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Started	Container started
07:39:54	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Started	Container started
07:40:01	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Created	Container created
07:40:01	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Started	Container started
07:40:01	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 21.032s (21.032s including waiting). Image size: 1460333318 bytes.
07:40:01	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:40:01	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Started	Container started
07:40:01	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Created	Container created
07:40:28 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:40:28 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:40:29 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:40:29 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:40:43 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:40:43 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:40:59 (x8)	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused
07:41:07 (x10)	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Unhealthy	Readiness probe failed: dial tcp 10.133.0.18:8080: connect: connection refused
07:41:09 (x9)	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:41:09	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
07:41:11 (x9)	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:41:11	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
07:41:17	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
07:41:17 (x9)	e2e-gw0-5c1f590758	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:41:28 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:41:28 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:41:38	e2e-gw0-5c1f590758	kubelet	model-chainer-6b979c876d-j6srw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-serving-cert" not found
07:41:38	e2e-gw0-5c1f590758	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-6b979c876d from 0 to 1
07:41:38	e2e-gw0-5c1f590758	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
07:41:38	e2e-gw0-5c1f590758	replicaset-controller	model-chainer-6b979c876d	SuccessfulCreate	Created pod: model-chainer-6b979c876d-j6srw
07:41:39	e2e-gw0-5c1f590758	kubelet	model-chainer-6b979c876d-j6srw	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7" already present on machine and can be accessed by the pod
07:41:39	e2e-gw0-5c1f590758	multus	model-chainer-6b979c876d-j6srw	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
07:41:39	e2e-gw0-5c1f590758	kubelet	model-chainer-6b979c876d-j6srw	Created	Container created
07:41:39	e2e-gw0-5c1f590758	kubelet	model-chainer-6b979c876d-j6srw	Started	Container started
07:41:43 (x5)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:41:43 (x5)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:41:44 (x5)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:41:44 (x5)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:41:50	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Killing	Stopping container kserve-container
07:41:50	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Killing	Stopping container kube-rbac-proxy
07:41:50	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Killing	Stopping container kserve-container
07:41:50	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Killing	Stopping container kserve-container
07:41:50	e2e-gw0-5c1f590758	kubelet	isvc-xgboost-graph-predictor-698b9676f-njgxs	Killing	Stopping container kube-rbac-proxy
07:41:50	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Killing	Stopping container kube-rbac-proxy
07:41:51	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Unhealthy	Readiness probe failed: Get "https://10.133.0.17:8643/healthz": dial tcp 10.133.0.17:8643: connect: connection refused
07:41:51 (x10)	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-1-predictor-6ffd4655dd-hhlbp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.17:8080: connect: connection refused
07:41:52	e2e-gw0-5c1f590758	kubelet	isvc-sklearn-graph-2-predictor-ddd6c579c-t4sjs	Unhealthy	Readiness probe failed: Get "https://10.133.0.18:8643/healthz": dial tcp 10.133.0.18:8643: connect: connection refused
07:41:57	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-48632	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-48632": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-48632": the object has been modified; please apply your changes to the latest version and try again
07:41:57	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-48632	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-48632": the object has been modified; please apply your changes to the latest version and try again
07:41:57	e2e-gw0-5c1f590758	replicaset-controller	success-200-isvc-48632-predictor-5c6dd99dc9	SuccessfulCreate	Created pod: success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs
07:41:57	e2e-gw0-5c1f590758	deployment-controller	success-200-isvc-48632-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-48632-predictor-5c6dd99dc9 from 0 to 1
07:41:57	e2e-gw0-5c1f590758	replicaset-controller	error-404-isvc-48632-predictor-7c7bc8bf7d	SuccessfulCreate	Created pod: error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv
07:41:57	e2e-gw0-5c1f590758	deployment-controller	error-404-isvc-48632-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-48632-predictor-7c7bc8bf7d from 0 to 1
07:41:58	e2e-gw0-5c1f590758	multus	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
07:41:58	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Started	Container started
07:41:58	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
07:41:58	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Created	Container created
07:41:58	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Started	Container started
07:41:58	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:41:58	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Created	Container created
07:41:58	e2e-gw0-5c1f590758	multus	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
07:41:58	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Started	Container started
07:41:58	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Created	Container created
07:41:58	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:41:58	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Started	Container started
07:41:58	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Created	Container created
07:41:58	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
07:42:23 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	model-chainer	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:42:23 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	model-chainer	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:42:42 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-48632-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:42:42 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-48632-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:42:42 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-48632-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:42:42 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-48632-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:42:55 (x5)	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-48632	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:42:55	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-48632	InferenceServiceReady	InferenceService [error-404-isvc-48632] is Ready
07:42:56 (x5)	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-48632	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:42:56	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-48632	InferenceServiceReady	InferenceService [success-200-isvc-48632] is Ready
07:43:07	e2e-gw0-5c1f590758	InferenceGraphController	sequence-graph-48632	InferenceGraphReady	InferenceGraph [sequence-graph-48632] is Ready
07:43:07	e2e-gw0-5c1f590758	replicaset-controller	sequence-graph-48632-ff6f99496	SuccessfulCreate	Created pod: sequence-graph-48632-ff6f99496-xjtvw
07:43:07	e2e-gw0-5c1f590758	deployment-controller	sequence-graph-48632	ScalingReplicaSet	Scaled up replica set sequence-graph-48632-ff6f99496 from 0 to 1
07:43:08	e2e-gw0-5c1f590758	multus	sequence-graph-48632-ff6f99496-xjtvw	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
07:43:08	e2e-gw0-5c1f590758	kubelet	sequence-graph-48632-ff6f99496-xjtvw	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7"
07:43:10	e2e-gw0-5c1f590758	kubelet	sequence-graph-48632-ff6f99496-xjtvw	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7" in 1.592s (1.592s including waiting). Image size: 216428108 bytes.
07:43:10	e2e-gw0-5c1f590758	kubelet	sequence-graph-48632-ff6f99496-xjtvw	Created	Container created
07:43:10	e2e-gw0-5c1f590758	kubelet	sequence-graph-48632-ff6f99496-xjtvw	Started	Container started
07:43:27 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-48632-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:43:27 (x4)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-48632-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:43:28 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-48632-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:43:28 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-48632-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:43:37 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-48632	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:43:37 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-48632	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:51:24	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Killing	Stopping container kserve-container
07:51:24	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Killing	Stopping container kserve-container
07:51:24	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Killing	Stopping container kube-rbac-proxy
07:51:24	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Killing	Stopping container kube-rbac-proxy
07:51:25 (x8)	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
07:51:25	e2e-gw0-5c1f590758	kubelet	error-404-isvc-48632-predictor-7c7bc8bf7d-xj4tv	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused
07:51:26 (x8)	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
07:51:26	e2e-gw0-5c1f590758	kubelet	success-200-isvc-48632-predictor-5c6dd99dc9-xlsjs	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
07:51:29	e2e-gw0-5c1f590758	replicaset-controller	success-200-isvc-b7c3b-predictor-7f789bb5cf	SuccessfulCreate	Created pod: success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk
07:51:29	e2e-gw0-5c1f590758	replicaset-controller	error-404-isvc-b7c3b-predictor-54b5879974	SuccessfulCreate	Created pod: error-404-isvc-b7c3b-predictor-54b5879974-6vrkz
07:51:29	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-b7c3b	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-b7c3b": the object has been modified; please apply your changes to the latest version and try again
07:51:29	e2e-gw0-5c1f590758	deployment-controller	success-200-isvc-b7c3b-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-b7c3b-predictor-7f789bb5cf from 0 to 1
07:51:29	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-b7c3b	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-b7c3b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-b7c3b": the object has been modified; please apply your changes to the latest version and try again
07:51:29	e2e-gw0-5c1f590758	deployment-controller	error-404-isvc-b7c3b-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-b7c3b-predictor-54b5879974 from 0 to 1
07:51:30	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
07:51:30	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:51:30	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Started	Container started
07:51:30	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Created	Container created
07:51:30	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Created	Container created
07:51:30	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Started	Container started
07:51:30	e2e-gw0-5c1f590758	multus	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
07:51:30	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:51:30	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Started	Container started
07:51:30	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Created	Container created
07:51:30	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Created	Container created
07:51:30	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Started	Container started
07:51:30	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
07:51:30	e2e-gw0-5c1f590758	multus	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
07:52:07 (x6)	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused
07:52:14 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-b7c3b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:52:14 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-b7c3b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:52:14 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-b7c3b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:52:14 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-b7c3b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:52:17 (x6)	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-b7c3b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:52:17 (x7)	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
07:52:17	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-b7c3b	InferenceServiceReady	InferenceService [error-404-isvc-b7c3b] is Ready
07:52:27 (x5)	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-b7c3b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:52:27	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-b7c3b	InferenceServiceReady	InferenceService [success-200-isvc-b7c3b] is Ready
07:52:39	e2e-gw0-5c1f590758	InferenceGraphController	sequence-graph-b7c3b	InferenceGraphReady	InferenceGraph [sequence-graph-b7c3b] is Ready
07:52:39	e2e-gw0-5c1f590758	deployment-controller	sequence-graph-b7c3b	ScalingReplicaSet	Scaled up replica set sequence-graph-b7c3b-75dbd6c6f4 from 0 to 1
07:52:39	e2e-gw0-5c1f590758	replicaset-controller	sequence-graph-b7c3b-75dbd6c6f4	SuccessfulCreate	Created pod: sequence-graph-b7c3b-75dbd6c6f4-hqpx5
07:52:40	e2e-gw0-5c1f590758	kubelet	sequence-graph-b7c3b-75dbd6c6f4-hqpx5	Created	Container created
07:52:40	e2e-gw0-5c1f590758	multus	sequence-graph-b7c3b-75dbd6c6f4-hqpx5	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
07:52:40	e2e-gw0-5c1f590758	kubelet	sequence-graph-b7c3b-75dbd6c6f4-hqpx5	Started	Container started
07:52:40	e2e-gw0-5c1f590758	kubelet	sequence-graph-b7c3b-75dbd6c6f4-hqpx5	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7" already present on machine and can be accessed by the pod
07:52:44 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-b7c3b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:52:44 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-b7c3b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:52:44 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-b7c3b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:52:44 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-b7c3b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:52:50	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Killing	Stopping container kube-rbac-proxy
07:52:50	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Killing	Stopping container kserve-container
07:52:50	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Killing	Stopping container kube-rbac-proxy
07:52:50	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Killing	Stopping container kserve-container
07:52:52	e2e-gw0-5c1f590758	kubelet	error-404-isvc-b7c3b-predictor-54b5879974-6vrkz	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
07:52:52	e2e-gw0-5c1f590758	kubelet	success-200-isvc-b7c3b-predictor-7f789bb5cf-dqrhk	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
07:52:56	e2e-gw0-5c1f590758	deployment-controller	error-404-isvc-45944-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-45944-predictor-7f69cd48 from 0 to 1
07:52:56	e2e-gw0-5c1f590758	replicaset-controller	error-404-isvc-45944-predictor-7f69cd48	SuccessfulCreate	Created pod: error-404-isvc-45944-predictor-7f69cd48-45twp
07:52:56	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-45944	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-45944": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-45944": the object has been modified; please apply your changes to the latest version and try again
07:52:56	e2e-gw0-5c1f590758	deployment-controller	success-200-isvc-45944-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-45944-predictor-7859fc774f from 0 to 1
07:52:56	e2e-gw0-5c1f590758	replicaset-controller	success-200-isvc-45944-predictor-7859fc774f	SuccessfulCreate	Created pod: success-200-isvc-45944-predictor-7859fc774f-rbsfn
07:52:56	e2e-gw0-5c1f590758	multus	success-200-isvc-45944-predictor-7859fc774f-rbsfn	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
07:52:56	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
07:52:56	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Created	Container created
07:52:56	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Started	Container started
07:52:56	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:52:56	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-45944	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-45944": the object has been modified; please apply your changes to the latest version and try again
07:52:57	e2e-gw0-5c1f590758	multus	error-404-isvc-45944-predictor-7f69cd48-45twp	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
07:52:57	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Created	Container created
07:52:57	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
07:52:57	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Started	Container started
07:52:57	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Created	Container created
07:52:57	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Started	Container started
07:52:57	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Created	Container created
07:52:57	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Started	Container started
07:52:57	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
07:53:09 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-b7c3b	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:53:09 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-b7c3b	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:53:26 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-45944-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:53:26 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-45944-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:53:26 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-45944-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:53:26 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-45944-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:53:53	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-45944	InferenceServiceReady	InferenceService [error-404-isvc-45944] is Ready
07:53:53 (x5)	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-45944	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:53:53	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-45944	InferenceServiceReady	InferenceService [success-200-isvc-45944] is Ready
07:53:54 (x5)	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-45944	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
07:54:06	e2e-gw0-5c1f590758	replicaset-controller	sequence-graph-45944-6dd4d9b5bd	SuccessfulCreate	Created pod: sequence-graph-45944-6dd4d9b5bd-rxbc2
07:54:06	e2e-gw0-5c1f590758	deployment-controller	sequence-graph-45944	ScalingReplicaSet	Scaled up replica set sequence-graph-45944-6dd4d9b5bd from 0 to 1
07:54:06	e2e-gw0-5c1f590758	InferenceGraphController	sequence-graph-45944	InferenceGraphReady	InferenceGraph [sequence-graph-45944] is Ready
07:54:07	e2e-gw0-5c1f590758	kubelet	sequence-graph-45944-6dd4d9b5bd-rxbc2	Created	Container created
07:54:07	e2e-gw0-5c1f590758	kubelet	sequence-graph-45944-6dd4d9b5bd-rxbc2	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7" already present on machine and can be accessed by the pod
07:54:07	e2e-gw0-5c1f590758	kubelet	sequence-graph-45944-6dd4d9b5bd-rxbc2	Started	Container started
07:54:07	e2e-gw0-5c1f590758	multus	sequence-graph-45944-6dd4d9b5bd-rxbc2	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
07:54:11 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-45944-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:54:11 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-45944-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:54:21	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-45944	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:54:21	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-45944	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
07:54:41 (x5)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-45944-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
07:54:41 (x5)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-45944-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:02:22	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Killing	Stopping container kube-rbac-proxy
08:02:22	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Killing	Stopping container kube-rbac-proxy
08:02:22	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Killing	Stopping container kserve-container
08:02:22	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Killing	Stopping container kserve-container
08:02:23	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Unhealthy	Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused
08:02:23 (x8)	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
08:02:23	e2e-gw0-5c1f590758	kubelet	success-200-isvc-45944-predictor-7859fc774f-rbsfn	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
08:02:23 (x8)	e2e-gw0-5c1f590758	kubelet	error-404-isvc-45944-predictor-7f69cd48-45twp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
08:02:27	e2e-gw0-5c1f590758	deployment-controller	success-200-isvc-8ae00-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-8ae00-predictor-5cd6564857 from 0 to 1
08:02:27	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-8ae00	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-8ae00": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-8ae00": the object has been modified; please apply your changes to the latest version and try again
08:02:27	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-8ae00	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-8ae00": the object has been modified; please apply your changes to the latest version and try again
08:02:27	e2e-gw0-5c1f590758	replicaset-controller	success-200-isvc-8ae00-predictor-5cd6564857	SuccessfulCreate	Created pod: success-200-isvc-8ae00-predictor-5cd6564857-gskx9
08:02:27	e2e-gw0-5c1f590758	replicaset-controller	error-404-isvc-8ae00-predictor-7f849b78d	SuccessfulCreate	Created pod: error-404-isvc-8ae00-predictor-7f849b78d-rjwg7
08:02:27	e2e-gw0-5c1f590758	deployment-controller	error-404-isvc-8ae00-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-8ae00-predictor-7f849b78d from 0 to 1
08:02:28	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Created	Container created
08:02:28	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Started	Container started
08:02:28	e2e-gw0-5c1f590758	multus	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
08:02:28	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Created	Container created
08:02:28	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Created	Container created
08:02:28	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Started	Container started
08:02:28	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Started	Container started
08:02:28	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
08:02:28	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Started	Container started
08:02:28	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Created	Container created
08:02:28	e2e-gw0-5c1f590758	multus	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
08:02:28	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
08:02:28	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
08:02:28	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2157" already present on machine and can be accessed by the pod
08:03:06 (x6)	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused
08:03:12 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-8ae00-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:03:12 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-8ae00-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:03:12 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-8ae00-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:03:12 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-8ae00-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:03:14 (x7)	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
08:03:16	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-8ae00	InferenceServiceReady	InferenceService [error-404-isvc-8ae00] is Ready
08:03:16 (x6)	e2e-gw0-5c1f590758	v1beta1Controllers	error-404-isvc-8ae00	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
08:03:24	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-8ae00	InferenceServiceReady	InferenceService [success-200-isvc-8ae00] is Ready
08:03:24 (x5)	e2e-gw0-5c1f590758	v1beta1Controllers	success-200-isvc-8ae00	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
08:03:37	e2e-gw0-5c1f590758	InferenceGraphController	switch-graph-8ae00	InferenceGraphReady	InferenceGraph [switch-graph-8ae00] is Ready
08:03:37	e2e-gw0-5c1f590758	deployment-controller	switch-graph-8ae00	ScalingReplicaSet	Scaled up replica set switch-graph-8ae00-86769987cd from 0 to 1
08:03:37	e2e-gw0-5c1f590758	replicaset-controller	switch-graph-8ae00-86769987cd	SuccessfulCreate	Created pod: switch-graph-8ae00-86769987cd-rhthk
08:03:38	e2e-gw0-5c1f590758	kubelet	switch-graph-8ae00-86769987cd-rhthk	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7" already present on machine and can be accessed by the pod
08:03:38	e2e-gw0-5c1f590758	multus	switch-graph-8ae00-86769987cd-rhthk	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
08:03:38	e2e-gw0-5c1f590758	kubelet	switch-graph-8ae00-86769987cd-rhthk	Started	Container started
08:03:38	e2e-gw0-5c1f590758	kubelet	switch-graph-8ae00-86769987cd-rhthk	Created	Container created
08:03:42 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-8ae00-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:03:42 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-8ae00-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:03:42 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	error-404-isvc-8ae00-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:03:42 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	success-200-isvc-8ae00-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:04:07 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	switch-graph-8ae00	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:04:07 (x2)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	switch-graph-8ae00	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:19:59	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Killing	Stopping container kserve-container
08:19:59	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Killing	Stopping container kserve-container
08:19:59	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
08:19:59	e2e-gw0-5c1f590758	kubelet	success-200-isvc-8ae00-predictor-5cd6564857-gskx9	Killing	Stopping container kube-rbac-proxy
08:19:59	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Killing	Stopping container kube-rbac-proxy
08:20:01	e2e-gw0-5c1f590758	kubelet	error-404-isvc-8ae00-predictor-7f849b78d-rjwg7	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
08:20:37 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-45944	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:20:37 (x3)	e2e-gw0-5c1f590758	horizontal-pod-autoscaler	sequence-graph-45944	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)