Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:48:34	kserve-ci-e2e-test		isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf to ip-10-0-135-205.ec2.internal
19:45:18	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk to ip-10-0-131-221.ec2.internal
19:29:12	kserve-ci-e2e-test		message-dumper-predictor-796c8b7645-z8kn6	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6 to ip-10-0-135-205.ec2.internal
19:56:09	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5 to ip-10-0-135-205.ec2.internal
20:05:48	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t to ip-10-0-131-221.ec2.internal
19:57:14	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb to ip-10-0-135-205.ec2.internal
19:52:42	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc to ip-10-0-135-205.ec2.internal
19:54:33	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps to ip-10-0-135-205.ec2.internal
19:58:48	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc to ip-10-0-131-221.ec2.internal
19:36:50	kserve-ci-e2e-test		isvc-sklearn-predictor-7857ffc886-cghqq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq to ip-10-0-131-221.ec2.internal
20:06:11	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd to ip-10-0-135-205.ec2.internal
19:42:01	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26 to ip-10-0-131-221.ec2.internal
20:04:14	kserve-ci-e2e-test		isvc-pmml-predictor-57685d4d84-bx4tr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr to ip-10-0-135-205.ec2.internal
19:58:50	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46 to ip-10-0-135-205.ec2.internal
20:03:54	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7 to ip-10-0-131-221.ec2.internal
20:02:13	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z to ip-10-0-135-205.ec2.internal
20:02:02	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q to ip-10-0-131-221.ec2.internal
20:00:31	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t to ip-10-0-135-205.ec2.internal
20:05:37	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d to ip-10-0-131-221.ec2.internal
20:04:05	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2 to ip-10-0-135-205.ec2.internal
19:45:20	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s to ip-10-0-135-205.ec2.internal
19:46:55	kserve-ci-e2e-test		isvc-tensorflow-predictor-6dfddc978b-9jtks	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks to ip-10-0-135-205.ec2.internal
19:53:38	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g to ip-10-0-135-205.ec2.internal
19:50:47	kserve-ci-e2e-test		isvc-xgboost-predictor-9fb66b6c4-cwkm2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2 to ip-10-0-135-205.ec2.internal
19:29:32	kserve-ci-e2e-test		isvc-logger-predictor-6b98ddf9c7-hkjkq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq to ip-10-0-131-221.ec2.internal
19:33:59	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr to ip-10-0-131-221.ec2.internal
19:54:16	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp to ip-10-0-135-205.ec2.internal
19:41:53	kserve-ci-e2e-test		isvc-canary-create-predictor-7b568f75-7j6hq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq to ip-10-0-135-205.ec2.internal
19:41:53	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7 to ip-10-0-131-221.ec2.internal
19:25:37	kserve-ci-e2e-test		isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4 to ip-10-0-135-205.ec2.internal
19:46:54	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz to ip-10-0-131-221.ec2.internal
20:02:57	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt to ip-10-0-135-205.ec2.internal
19:48:29	kserve-ci-e2e-test		isvc-triton-predictor-7d578d8b99-5f5ng	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng to ip-10-0-135-205.ec2.internal
19:43:38	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s to ip-10-0-135-205.ec2.internal
19:31:09	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k to ip-10-0-135-205.ec2.internal
19:25:52	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn to ip-10-0-135-205.ec2.internal
19:52:19	kserve-ci-e2e-test		isvc-lightgbm-predictor-59b9d969c9-cv995	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995 to ip-10-0-131-221.ec2.internal
19:27:36	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh to ip-10-0-131-221.ec2.internal
19:41:53	kserve-ci-e2e-test		isvc-canary-create-predictor-7b568f75-bzxwz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz to ip-10-0-131-221.ec2.internal
19:57:17	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph to ip-10-0-135-205.ec2.internal
19:47:42	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t to ip-10-0-131-221.ec2.internal
19:38:56	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m to ip-10-0-135-205.ec2.internal
19:41:12	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv to ip-10-0-131-221.ec2.internal
20:09:43	kserve-ci-e2e-test		isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr to ip-10-0-135-205.ec2.internal
19:36:21	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m to ip-10-0-131-221.ec2.internal
19:31:08	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7 to ip-10-0-131-221.ec2.internal
19:56:22	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx to ip-10-0-135-205.ec2.internal
19:29:34	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc to ip-10-0-135-205.ec2.internal
19:40:25	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w to ip-10-0-131-221.ec2.internal
19:50:29	kserve-ci-e2e-test		isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8 to ip-10-0-131-221.ec2.internal
19:50:30	kserve-ci-e2e-test		isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr to ip-10-0-135-205.ec2.internal
19:50:29	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn to ip-10-0-131-221.ec2.internal
19:32:55	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8 to ip-10-0-135-205.ec2.internal
19:43:44	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-587f57f874-krptn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn to ip-10-0-131-221.ec2.internal
20:01:40	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw to ip-10-0-135-205.ec2.internal
19:27:48	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr to ip-10-0-135-205.ec2.internal
19:42:38	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8 to ip-10-0-135-205.ec2.internal
20:00:13	kserve-ci-e2e-test		isvc-paddle-predictor-69579d8675-rc888	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888 to ip-10-0-135-205.ec2.internal
19:57:53	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh to ip-10-0-135-205.ec2.internal
20:07:58	kserve-ci-e2e-test		isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62 to ip-10-0-135-205.ec2.internal
19:46:54	kserve-ci-e2e-test		isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq to ip-10-0-135-205.ec2.internal
19:25:37	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8 to ip-10-0-131-221.ec2.internal
19:43:38	kserve-ci-e2e-test		isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9 to ip-10-0-135-205.ec2.internal
19:34:30	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4 to ip-10-0-135-205.ec2.internal
19:43:38	kserve-ci-e2e-test		isvc-canary-promote-predictor-75f64ff6c9-5rffs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs to ip-10-0-131-221.ec2.internal
19:46:54	kserve-ci-e2e-test		isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7 to ip-10-0-131-221.ec2.internal
19:38:23	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f to ip-10-0-131-221.ec2.internal
19:25:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-9735d5	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-9735d5": the object has been modified; please apply your changes to the latest version and try again
19:25:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-7b5b55bfb6	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8
19:25:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-9735d5	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-9735d5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-9735d5": the object has been modified; please apply your changes to the latest version and try again
19:25:37	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-9735d5-predictor-cd8fc98b4	SuccessfulCreate	Created pod: isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4
19:25:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-7b5b55bfb6 from 0 to 1
19:25:37	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-9735d5-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-9735d5-predictor-cd8fc98b4 from 0 to 1
19:25:38	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
19:25:38	kserve-ci-e2e-test	multus	isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
19:25:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3"
19:25:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" in 3.29s (3.29s including waiting). Image size: 308177839 bytes.
19:25:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Started	Started container storage-initializer
19:25:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Created	Created container: storage-initializer
19:25:42 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4	Started	Started container storage-initializer
19:25:42 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4	Created	Created container: storage-initializer
19:25:42 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:25:44	kserve-ci-e2e-test	kubelet	isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4	BackOff	Back-off restarting failed container storage-initializer in pod isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_kserve-ci-e2e-test(a54250f6-fc33-49a0-a396-f1345f03f120)
19:25:44 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-9735d5	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970"
19:25:52	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-6dc55b9cf4	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn
19:25:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:25:52	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-6dc55b9cf4 from 0 to 1
19:25:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:25:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Started	Started container storage-initializer
19:25:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:25:53	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
19:25:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Created	Created container: storage-initializer
19:25:57	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Pulling	Pulling image "kserve/predictiveserver:latest"
19:25:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Started	Started container kserve-container
19:25:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" in 13.667s (13.667s including waiting). Image size: 1605424527 bytes.
19:25:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Created	Created container: kserve-container
19:25:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:26:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.524s (2.524s including waiting). Image size: 211946088 bytes.
19:26:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:5c05f01febc2fcb46d2803493fb076d1e99486e8fba24136c3d6905a978cedc9"
19:26:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Created	Created container: kube-rbac-proxy
19:26:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Started	Started container kube-rbac-proxy
19:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:5c05f01febc2fcb46d2803493fb076d1e99486e8fba24136c3d6905a978cedc9" in 2.799s (2.799s including waiting). Image size: 240471122 bytes.
19:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Started	Started container agent
19:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Created	Created container: agent
19:26:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Started	Started container kserve-container
19:26:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.075s (22.075s including waiting). Image size: 2439755912 bytes.
19:26:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Created	Created container: kserve-container
19:26:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:26:22	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Started	Started container kube-rbac-proxy
19:26:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:22	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Created	Created container: kube-rbac-proxy
19:26:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:22	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.857s (2.857s including waiting). Image size: 211946088 bytes.
19:26:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
19:27:21 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:29 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused
19:27:36	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-86bd8b8bfb from 0 to 1
19:27:36	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh
19:27:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Killing	Stopping container kserve-container
19:27:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Killing	Stopping container agent
19:27:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Killing	Stopping container kube-rbac-proxy
19:27:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:27:37	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
19:27:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Started	Started container storage-initializer
19:27:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Created	Created container: storage-initializer
19:27:39 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
19:27:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:41 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused
19:27:41 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:27:42 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Started	Started container kserve-container
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Created	Created container: kube-rbac-proxy
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Started	Started container kube-rbac-proxy
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:5c05f01febc2fcb46d2803493fb076d1e99486e8fba24136c3d6905a978cedc9" already present on machine
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Created	Created container: agent
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Started	Started container agent
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Created	Created container: kserve-container
19:27:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:27:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Killing	Stopping container kube-rbac-proxy
19:27:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Killing	Stopping container kserve-container
19:27:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
19:27:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:27:48	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
19:27:48	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-6f947fc4c7 from 0 to 1
19:27:48	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-6f947fc4c7	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr
19:27:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Started	Started container storage-initializer
19:27:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Created	Created container: storage-initializer
19:27:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:27:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Started	Started container kserve-container
19:27:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Started	Started container kube-rbac-proxy
19:27:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Created	Created container: kserve-container
19:27:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:27:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Created	Created container: kube-rbac-proxy
19:28:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
19:29:06 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:06 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Killing	Stopping container agent
19:29:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Killing	Stopping container kube-rbac-proxy
19:29:12	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
19:29:12	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
19:29:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Killing	Stopping container kserve-container
19:29:12 (x2)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
19:29:12	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-796c8b7645	SuccessfulCreate	Created pod: message-dumper-predictor-796c8b7645-z8kn6
19:29:12	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-796c8b7645 from 0 to 1
19:29:13	kserve-ci-e2e-test	multus	message-dumper-predictor-796c8b7645-z8kn6	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
19:29:13	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
19:29:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Created	Created container: kserve-container
19:29:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Started	Started container kube-rbac-proxy
19:29:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.154s (1.154s including waiting). Image size: 14813193 bytes.
19:29:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Started	Started container kserve-container
19:29:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Created	Created container: kube-rbac-proxy
19:29:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
19:29:20 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:20 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:29:23	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
19:29:23 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:28	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Killing	Stopping container kube-rbac-proxy
19:29:28	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Killing	Stopping container kserve-container
19:29:30 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
19:29:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Unhealthy	Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused
19:29:30 (x11)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused
19:29:30 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh	Unhealthy	Readiness probe failed: dial tcp 10.132.0.36:5000: connect: connection refused
19:29:32	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-6b98ddf9c7	SuccessfulCreate	Created pod: isvc-logger-predictor-6b98ddf9c7-hkjkq
19:29:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
19:29:32	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-6b98ddf9c7 from 0 to 1
19:29:33	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Started	Started container storage-initializer
19:29:33	kserve-ci-e2e-test	multus	isvc-logger-predictor-6b98ddf9c7-hkjkq	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
19:29:33	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:29:33	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Created	Created container: storage-initializer
19:29:34	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
19:29:34	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-76f4864458	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-76f4864458-8vtsc
19:29:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Started	Started container storage-initializer
19:29:34	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-76f4864458 from 0 to 1
19:29:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Created	Created container: storage-initializer
19:29:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:5c05f01febc2fcb46d2803493fb076d1e99486e8fba24136c3d6905a978cedc9" already present on machine
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Created	Created container: agent
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Created	Created container: kserve-container
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Started	Started container kserve-container
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Created	Created container: kube-rbac-proxy
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Started	Started container kube-rbac-proxy
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:29:37	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Started	Started container agent
19:29:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Created	Created container: kserve-container
19:29:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Started	Started container kserve-container
19:29:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Created	Created container: kube-rbac-proxy
19:29:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Started	Started container kube-rbac-proxy
19:29:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:29:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:30:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:50 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
19:30:56 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
19:31:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
19:31:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Killing	Stopping container kserve-container
19:31:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Killing	Stopping container kube-rbac-proxy
19:31:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-76f4864458-8vtsc	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
19:31:08	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Killing	Stopping container kube-rbac-proxy
19:31:08	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-7458d6ddd from 0 to 1
19:31:08	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
19:31:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:31:08	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Killing	Stopping container kube-rbac-proxy
19:31:08	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Killing	Stopping container agent
19:31:08	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Killing	Stopping container kserve-container
19:31:08	kserve-ci-e2e-test	kubelet	message-dumper-predictor-796c8b7645-z8kn6	Killing	Stopping container kserve-container
19:31:08	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-7458d6ddd	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-7458d6ddd-6dqs7
19:31:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Created	Created container: storage-initializer
19:31:09	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-84c6985b7d from 0 to 1
19:31:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Started	Started container storage-initializer
19:31:09	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-84c6985b7d	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k
19:31:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:31:10	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
19:31:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:31:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Created	Created container: storage-initializer
19:31:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Started	Started container storage-initializer
19:31:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Started	Started container kserve-container
19:31:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Created	Created container: kube-rbac-proxy
19:31:15 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:31:15 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
19:31:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Created	Created container: kserve-container
19:31:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Started	Started container kube-rbac-proxy
19:31:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:31:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:31:25 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b98ddf9c7-hkjkq	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
19:31:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Pulling	Pulling image "kserve/autogluonserver:latest"
19:31:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:43 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:32:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
19:32:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Killing	Stopping container kube-rbac-proxy
19:32:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Killing	Stopping container kserve-container
19:32:53 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Unhealthy	Readiness probe failed: Get "http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.30:8080: connect: connection refused
19:32:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
19:32:55	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Started	Started container storage-initializer
19:32:55	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-748c464b49 from 0 to 1
19:32:55	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-748c464b49	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8
19:32:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:32:55	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
19:32:55	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:32:55	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Created	Created container: storage-initializer
19:33:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Started	Started container kube-rbac-proxy
19:33:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Created	Created container: kube-rbac-proxy
19:33:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Started	Started container kserve-container
19:33:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Created	Created container: kserve-container
19:33:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:33:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 1m55.996s (1m55.996s including waiting). Image size: 8949490981 bytes.
19:33:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Created	Created container: kserve-container
19:33:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Started	Started container kserve-container
19:33:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Created	Created container: kube-rbac-proxy
19:33:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Started	Started container kube-rbac-proxy
19:33:33 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
19:33:44 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
19:33:53 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:53 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Killing	Stopping container kserve-container
19:33:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7458d6ddd-6dqs7	Killing	Stopping container kube-rbac-proxy
19:33:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
19:33:59	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-67fdc6896b from 0 to 1
19:33:59	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-67fdc6896b	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-67fdc6896b-skxsr
19:34:00	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Started	Started container storage-initializer
19:34:00	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Created	Created container: storage-initializer
19:34:00	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:34:00	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
19:34:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:12 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Unhealthy	Readiness probe failed: Get "http://10.134.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.134.0.31:8080: connect: connection refused
19:34:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
19:34:23 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:25	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Killing	Stopping container kserve-container
19:34:25	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Killing	Stopping container kube-rbac-proxy
19:34:28	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8	Unhealthy	Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused
19:34:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Created	Created container: kserve-container
19:34:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Started	Started container kserve-container
19:34:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:34:30	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-54dcd58b96 from 0 to 1
19:34:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Created	Created container: kube-rbac-proxy
19:34:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:34:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
19:34:30	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-54dcd58b96	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4
19:34:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Started	Started container kube-rbac-proxy
19:34:31	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:34:31	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
19:34:31	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Created	Created container: storage-initializer
19:34:31	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Started	Started container storage-initializer
19:34:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:34:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Created	Created container: kserve-container
19:34:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Started	Started container kserve-container
19:34:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:34:35	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Created	Created container: kube-rbac-proxy
19:34:35	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Started	Started container kube-rbac-proxy
19:34:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:00 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:00 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:01 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:36:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
19:36:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
19:36:02 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:36:10	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Killing	Stopping container kube-rbac-proxy
19:36:10	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Killing	Stopping container kserve-container
19:36:10 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Unhealthy	Readiness probe failed: Get "http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.134.0.32:8080: connect: connection refused
19:36:11	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
19:36:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Unhealthy	Readiness probe failed: Get "http://10.132.0.39:8080/v2/models/isvc-autogluon-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:36:15 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:36:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:36:15 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-7764bb7d48	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-7764bb7d48-6plhg
19:36:15	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-7764bb7d48 from 0 to 1
19:36:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:36:15 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Killing	Stopping container kserve-container
19:36:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67fdc6896b-skxsr	Killing	Stopping container kube-rbac-proxy
19:36:21	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
19:36:21	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:36:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
19:36:21	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-9f89ddc5c	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m
19:36:21	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-9f89ddc5c from 0 to 1
19:36:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Created	Created container: storage-initializer
19:36:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Started	Started container storage-initializer
19:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Created	Created container: kserve-container
19:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Started	Started container kube-rbac-proxy
19:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Created	Created container: kube-rbac-proxy
19:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Started	Started container kserve-container
19:36:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:49	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-7857ffc886 from 0 to 1
19:36:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-7857ffc886	SuccessfulCreate	Created pod: isvc-sklearn-predictor-7857ffc886-cghqq
19:36:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:36:50 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:36:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-predictor-serving-cert" not found
19:36:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:36:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Started	Started container storage-initializer
19:36:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Created	Created container: storage-initializer
19:36:51	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-7857ffc886-cghqq	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
19:36:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:36:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Created	Created container: kserve-container
19:36:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Started	Started container kserve-container
19:36:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Started	Started container kube-rbac-proxy
19:36:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Created	Created container: kube-rbac-proxy
19:36:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:37:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:37:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:38:01 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:38:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
19:38:06 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:38:06 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:38:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Unhealthy	Readiness probe failed: Get "http://10.132.0.40:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:38:14 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:38:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
19:38:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Killing	Stopping container kserve-container
19:38:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m	Killing	Stopping container kube-rbac-proxy
19:38:23	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5cdb8fd688	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f
19:38:23	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
19:38:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:38:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Created	Created container: storage-initializer
19:38:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Started	Started container storage-initializer
19:38:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:38:23	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5cdb8fd688 from 0 to 1
19:38:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:38:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:38:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Started	Started container kube-rbac-proxy
19:38:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Created	Created container: kube-rbac-proxy
19:38:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Started	Started container kserve-container
19:38:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Created	Created container: kserve-container
19:38:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:38:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:38:50 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:38:50 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Killing	Stopping container kube-rbac-proxy
19:38:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Killing	Stopping container kserve-container
19:38:54 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
19:38:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7857ffc886-cghqq	Unhealthy	Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused
19:38:56	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:38:56	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-55fb4d8685 from 0 to 1
19:38:56	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-55fb4d8685	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m
19:38:56 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
19:38:58	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Created	Created container: storage-initializer
19:38:58	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
19:38:58	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:38:58	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Started	Started container storage-initializer
19:39:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
19:40:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
19:40:03 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:40:08 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:40:08 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:40:14	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Unhealthy	Readiness probe failed: Get "http://10.132.0.42:8080/v2/models/isvc-autogluon-v2-noslash/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:40:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Killing	Stopping container kube-rbac-proxy
19:40:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f	Killing	Stopping container kserve-container
19:40:24	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-68594f479c from 0 to 1
19:40:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Started	Started container storage-initializer
19:40:25 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:40:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:40:25	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
19:40:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:40:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Created	Created container: storage-initializer
19:40:25	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-68594f479c	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-68594f479c-52l8w
19:40:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:40:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Created	Created container: kserve-container
19:40:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:40:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Started	Started container kserve-container
19:40:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Created	Created container: kube-rbac-proxy
19:40:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Started	Started container kube-rbac-proxy
19:40:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
19:40:49 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:40:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:40:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:41:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m0.7s (2m0.7s including waiting). Image size: 10890461297 bytes.
19:41:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Started	Started container kserve-container
19:41:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Created	Created container: kube-rbac-proxy
19:41:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Created	Created container: kserve-container
19:41:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:41:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Started	Started container kube-rbac-proxy
19:41:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Killing	Stopping container kube-rbac-proxy
19:41:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Killing	Stopping container kserve-container
19:41:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
19:41:09 (x4)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-68594f479c-52l8w	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
19:41:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
19:41:12 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
19:41:12	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8 from 0 to 1
19:41:12	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv
19:41:13	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
19:41:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Started	Started container storage-initializer
19:41:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:41:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Created	Created container: storage-initializer
19:41:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Created	Created container: kserve-container
19:41:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Started	Started container kube-rbac-proxy
19:41:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Started	Started container kserve-container
19:41:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:41:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Created	Created container: kube-rbac-proxy
19:41:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:41:26 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:41:26 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:41:26 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused
19:41:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
19:41:36 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:41:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:41:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:41:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:41:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Killing	Stopping container kube-rbac-proxy
19:41:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv	Killing	Stopping container kserve-container
19:41:50	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
19:41:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:41:53 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:41:53 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:41:53	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-5c6467bc9d from 0 to 1
19:41:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-5c6467bc9d	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7
19:41:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found
19:41:53	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-7b568f75 from 0 to 2
19:41:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-7b568f75	SuccessfulCreate	Created pod: isvc-canary-create-predictor-7b568f75-7j6hq
19:41:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-7b568f75	SuccessfulCreate	Created pod: isvc-canary-create-predictor-7b568f75-bzxwz
19:41:54	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Created	Created container: storage-initializer
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Created	Created container: storage-initializer
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Created	Created container: storage-initializer
19:41:54	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-7b568f75-bzxwz	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:41:54	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-7b568f75-7j6hq	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Started	Started container storage-initializer
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Started	Started container storage-initializer
19:41:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Started	Started container storage-initializer
19:41:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:41:56	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Killing	Stopping container kserve-container
19:41:56	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Killing	Stopping container kube-rbac-proxy
19:41:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Created	Created container: kube-rbac-proxy
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Created	Created container: kserve-container
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Started	Started container kserve-container
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970"
19:41:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Started	Started container kube-rbac-proxy
19:41:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Created	Created container: kserve-container
19:41:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Started	Started container kserve-container
19:41:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:41:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:41:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Created	Created container: kube-rbac-proxy
19:41:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Started	Started container kube-rbac-proxy
19:42:00	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Unhealthy	Readiness probe failed: Get "http://10.134.0.35:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.134.0.35:8080: connect: connection refused
19:42:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
19:42:01	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-99bbb4b5d from 0 to 1
19:42:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
19:42:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
19:42:01	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-99bbb4b5d	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26
19:42:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Started	Started container storage-initializer
19:42:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Created	Created container: storage-initializer
19:42:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:42:02	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
19:42:05 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Unhealthy	Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused
19:42:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:42:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Started	Started container kube-rbac-proxy
19:42:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Created	Created container: kube-rbac-proxy
19:42:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:42:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Started	Started container kserve-container
19:42:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Created	Created container: kserve-container
19:42:11	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Unhealthy	Readiness probe failed: Get "https://10.134.0.35:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
19:42:11	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m	Unhealthy	Readiness probe failed: Get "http://10.134.0.35:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:42:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Created	Created container: kserve-container
19:42:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" in 13.392s (13.392s including waiting). Image size: 1605424527 bytes.
19:42:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Started	Started container kserve-container
19:42:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:42:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Started	Started container kube-rbac-proxy
19:42:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Created	Created container: kube-rbac-proxy
19:42:15 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Unhealthy	Readiness probe failed: dial tcp 10.132.0.47:8080: connect: connection refused
19:42:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
19:42:25 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:42:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:42:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:42:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Killing	Stopping container kserve-container
19:42:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26	Killing	Stopping container kube-rbac-proxy
19:42:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-698d8768cc from 0 to 1
19:42:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-698d8768cc	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8
19:42:39	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
19:42:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:42:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:42:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:42:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Created	Created container: storage-initializer
19:42:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Started	Started container storage-initializer
19:42:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:42:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Started	Started container kube-rbac-proxy
19:42:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Created	Created container: kserve-container
19:42:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Started	Started container kserve-container
19:42:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:42:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Created	Created container: kube-rbac-proxy
19:43:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:43:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:43:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:43:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
19:43:26	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-7b568f75 from 2 to 1
19:43:26	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Killing	Stopping container kserve-container
19:43:26	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-7b568f75	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-7b568f75-7j6hq
19:43:26	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Killing	Stopping container kube-rbac-proxy
19:43:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
19:43:27 (x19)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:43:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
19:43:31 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:43:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Killing	Stopping container kserve-container
19:43:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Killing	Stopping container kserve-container
19:43:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Killing	Stopping container kube-rbac-proxy
19:43:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Killing	Stopping container kube-rbac-proxy
19:43:35	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
19:43:35 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
19:43:35	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-bzxwz	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
19:43:36 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused
19:43:38	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-7d7f6d88c6 from 0 to 1
19:43:38	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-7d7f6d88c6	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s
19:43:38	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-75f64ff6c9	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-75f64ff6c9-sqmh9
19:43:38	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-75f64ff6c9 from 0 to 2
19:43:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
19:43:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:43:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:43:38	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-75f64ff6c9	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-75f64ff6c9-5rffs
19:43:39	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-75f64ff6c9-5rffs	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Created	Created container: storage-initializer
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Created	Created container: storage-initializer
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:43:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Killing	Stopping container kserve-container
19:43:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Killing	Stopping container kube-rbac-proxy
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Started	Started container storage-initializer
19:43:39	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Started	Started container storage-initializer
19:43:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:43:39	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
19:43:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Started	Started container storage-initializer
19:43:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Created	Created container: storage-initializer
19:43:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Unhealthy	Readiness probe failed: Get "http://10.134.0.37:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Created	Created container: kube-rbac-proxy
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Started	Started container kube-rbac-proxy
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Started	Started container kube-rbac-proxy
19:43:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Started	Started container kserve-container
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Created	Created container: kserve-container
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Created	Created container: kube-rbac-proxy
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Started	Started container kube-rbac-proxy
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:43:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Started	Started container kserve-container
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Created	Created container: kube-rbac-proxy
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Started	Started container kserve-container
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Created	Created container: kserve-container
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Created	Created container: kserve-container
19:43:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:43:44	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-587f57f874	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-587f57f874-krptn
19:43:44	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-587f57f874 from 0 to 1
19:43:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Started	Started container storage-initializer
19:43:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Created	Created container: storage-initializer
19:43:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:43:45	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-587f57f874-krptn	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
19:43:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
19:43:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Started	Started container kserve-container
19:43:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:43:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Created	Created container: kserve-container
19:43:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:43:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Started	Started container kube-rbac-proxy
19:43:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Created	Created container: kube-rbac-proxy
19:43:54 (x6)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7b568f75-7j6hq	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
19:44:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:44:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:45:01 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused
19:45:01 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
19:45:02 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Unhealthy	Readiness probe failed: dial tcp 10.132.0.49:8080: connect: connection refused
19:45:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Killing	Stopping container kserve-container
19:45:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Killing	Stopping container kube-rbac-proxy
19:45:07	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-75f64ff6c9 from 2 to 1
19:45:07	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-75f64ff6c9	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-75f64ff6c9-5rffs
19:45:08 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:45:08 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:45:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-5rffs	Unhealthy	Readiness probe failed: Get "https://10.132.0.48:8643/healthz": dial tcp 10.132.0.48:8643: connect: connection refused
19:45:12 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:45:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
19:45:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:45:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:45:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Killing	Stopping container kserve-container
19:45:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Killing	Stopping container kube-rbac-proxy
19:45:18	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Killing	Stopping container kube-rbac-proxy
19:45:18	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-7d7f6d88c6 from 1 to 2
19:45:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-587f57f874-krptn	Unhealthy	Readiness probe failed: Get "https://10.132.0.49:8643/healthz": dial tcp 10.132.0.49:8643: connect: connection refused
19:45:18	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-7d7f6d88c6	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk
19:45:18	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Killing	Stopping container kserve-container
19:45:18 (x19)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:45:19	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
19:45:19	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-f7b4585f from 0 to 1
19:45:19	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:45:19	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Created	Created container: storage-initializer
19:45:19	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Started	Started container storage-initializer
19:45:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
19:45:20	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-f7b4585f	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s
19:45:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
19:45:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Started	Started container storage-initializer
19:45:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Created	Created container: storage-initializer
19:45:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:45:21	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
19:45:22	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
19:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Started	Started container kube-rbac-proxy
19:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Created	Created container: kserve-container
19:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Started	Started container kserve-container
19:45:23 (x12)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-75f64ff6c9-sqmh9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
19:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Created	Created container: kube-rbac-proxy
19:45:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:45:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:45:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Created	Created container: kserve-container
19:45:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Started	Started container kserve-container
19:45:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Created	Created container: kube-rbac-proxy
19:45:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Started	Started container kube-rbac-proxy
19:46:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:46:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:46:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
19:46:41 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
19:46:46 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:46:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
19:46:48	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Killing	Stopping container kserve-container
19:46:48	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Killing	Stopping container kube-rbac-proxy
19:46:48	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Killing	Stopping container kserve-container
19:46:48	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Killing	Stopping container kube-rbac-proxy
19:46:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Killing	Stopping container kube-rbac-proxy
19:46:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:46:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:46:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Killing	Stopping container kserve-container
19:46:51	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
19:46:51 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.50:8080: connect: connection refused
19:46:51	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk	Unhealthy	Readiness probe failed: Get "https://10.132.0.50:8643/healthz": dial tcp 10.132.0.50:8643: connect: connection refused
19:46:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:46:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-794fd55d6c	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-794fd55d6c-mfrhq
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Started	Started container storage-initializer
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Created	Created container: storage-initializer
19:46:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-6d75476b46	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz
19:46:54	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-6d75476b46 from 0 to 1
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Started	Started container storage-initializer
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Created	Created container: storage-initializer
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:46:54	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
19:46:54	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Created	Created container: storage-initializer
19:46:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Started	Started container storage-initializer
19:46:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-794fd55d6c	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-794fd55d6c-fxhm7
19:46:54	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-794fd55d6c from 0 to 2
19:46:54	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
19:46:55	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-6dfddc978b from 0 to 1
19:46:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
19:46:55	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-6dfddc978b	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-6dfddc978b-9jtks
19:46:56	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-predictor-serving-cert" not found
19:46:57	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Started	Started container storage-initializer
19:46:57	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Created	Created container: storage-initializer
19:46:57	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:46:57	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-6dfddc978b-9jtks	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Started	Started container kube-rbac-proxy
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Created	Created container: kserve-container
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Started	Started container kserve-container
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Created	Created container: kube-rbac-proxy
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Started	Started container kube-rbac-proxy
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Created	Created container: kube-rbac-proxy
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Created	Created container: kserve-container
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Started	Started container kserve-container
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Created	Created container: kserve-container
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Created	Created container: kube-rbac-proxy
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Started	Started container kserve-container
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Started	Started container kube-rbac-proxy
19:46:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:47:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:47:06	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Started	Started container kserve-container
19:47:06	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.779s (3.779s including waiting). Image size: 425873876 bytes.
19:47:06	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Created	Created container: kserve-container
19:47:06	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:47:06	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Created	Created container: kube-rbac-proxy
19:47:06	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Started	Started container kube-rbac-proxy
19:47:12 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
19:47:22 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:47:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
19:47:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:47:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:47:37	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Killing	Stopping container kserve-container
19:47:37	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Killing	Stopping container kube-rbac-proxy
19:47:42	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-588dc9cf7d from 0 to 1
19:47:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
19:47:42	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-588dc9cf7d	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t
19:47:43	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
19:47:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Started	Started container storage-initializer
19:47:44	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
19:47:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:47:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Created	Created container: storage-initializer
19:47:49	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:47:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.8s (3.8s including waiting). Image size: 425873876 bytes.
19:47:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Created	Created container: kserve-container
19:47:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Started	Started container kserve-container
19:47:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:47:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Started	Started container kube-rbac-proxy
19:47:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Created	Created container: kube-rbac-proxy
19:47:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:47:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:48:00 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Unhealthy	Readiness probe failed: dial tcp 10.132.0.53:8080: connect: connection refused
19:48:07 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6dfddc978b-9jtks	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
19:48:10 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:48:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
19:48:12	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:48:13	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:48:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
19:48:24	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Killing	Stopping container kube-rbac-proxy
19:48:24	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Killing	Stopping container kserve-container
19:48:27	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Killing	Stopping container kube-rbac-proxy
19:48:27	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
19:48:27	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-794fd55d6c	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-794fd55d6c-mfrhq
19:48:27	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Killing	Stopping container kserve-container
19:48:27	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-794fd55d6c from 2 to 1
19:48:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:48:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
19:48:29	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-7d578d8b99 from 0 to 1
19:48:29	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-7d578d8b99	SuccessfulCreate	Created pod: isvc-triton-predictor-7d578d8b99-5f5ng
19:48:29	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-triton-predictor-serving-cert" not found
19:48:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Started	Started container storage-initializer
19:48:30	kserve-ci-e2e-test	multus	isvc-triton-predictor-7d578d8b99-5f5ng	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
19:48:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:48:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Created	Created container: storage-initializer
19:48:33	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-mfrhq	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": context deadline exceeded
19:48:34	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-794fd55d6c from 1 to 2
19:48:34	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
19:48:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:48:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Created	Created container: storage-initializer
19:48:34	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-794fd55d6c	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-794fd55d6c-pdpsf
19:48:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Started	Started container storage-initializer
19:48:34	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
19:48:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Killing	Stopping container kserve-container
19:48:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Killing	Stopping container kube-rbac-proxy
19:48:34 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:48:34 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:48:37 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.52:8080: connect: connection refused
19:48:37	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz	Unhealthy	Readiness probe failed: Get "https://10.132.0.52:8643/healthz": dial tcp 10.132.0.52:8643: connect: connection refused
19:48:38	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Created	Created container: kserve-container
19:48:38	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:48:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:48:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Started	Started container kube-rbac-proxy
19:48:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Created	Created container: kube-rbac-proxy
19:48:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Started	Started container kserve-container
19:48:50 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t	Unhealthy	Readiness probe failed: Get "https://10.132.0.53:8643/healthz": dial tcp 10.132.0.53:8643: connect: connection refused
19:50:13 (x12)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
19:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Killing	Stopping container kserve-container
19:50:24 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
19:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Killing	Stopping container kube-rbac-proxy
19:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Killing	Stopping container kserve-container
19:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Killing	Stopping container kube-rbac-proxy
19:50:25	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-pdpsf	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
19:50:26 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.51:8080: connect: connection refused
19:50:26	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-794fd55d6c-fxhm7	Unhealthy	Readiness probe failed: Get "https://10.132.0.51:8643/healthz": dial tcp 10.132.0.51:8643: connect: connection refused
19:50:29 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:50:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-6fc7d5b8c9 from 0 to 2
19:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
19:50:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-dc6dbfb86	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn
19:50:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-dc6dbfb86 from 0 to 1
19:50:29 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:50:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-6fc7d5b8c9	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8
19:50:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Started	Started container kube-rbac-proxy
19:50:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m55.306s (1m55.306s including waiting). Image size: 12907074623 bytes.
19:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Started	Started container storage-initializer
19:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Created	Created container: storage-initializer
19:50:30	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-6fc7d5b8c9	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr
19:50:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Created	Created container: kube-rbac-proxy
19:50:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Started	Started container kserve-container
19:50:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Created	Created container: kserve-container
19:50:30 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:50:30	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
19:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Started	Started container storage-initializer
19:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Created	Created container: storage-initializer
19:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:50:30	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
19:50:30 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:50:31	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
19:50:31	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:50:31	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Started	Started container storage-initializer
19:50:31	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Created	Created container: storage-initializer
19:50:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Created	Created container: kube-rbac-proxy
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Started	Started container kserve-container
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Created	Created container: kserve-container
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Started	Started container kserve-container
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Started	Started container kube-rbac-proxy
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Created	Created container: kube-rbac-proxy
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Created	Created container: kserve-container
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Started	Started container kube-rbac-proxy
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Started	Started container kube-rbac-proxy
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Created	Created container: kube-rbac-proxy
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Started	Started container kserve-container
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Created	Created container: kserve-container
19:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:50:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
19:50:37 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:50:41	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Killing	Stopping container kube-rbac-proxy
19:50:41	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Killing	Stopping container kserve-container
19:50:42	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-7d578d8b99-5f5ng	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
19:50:46	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-9fb66b6c4 from 0 to 1
19:50:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:50:47 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:50:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-predictor-serving-cert" not found
19:50:47	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-9fb66b6c4	SuccessfulCreate	Created pod: isvc-xgboost-predictor-9fb66b6c4-cwkm2
19:50:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Created	Created container: storage-initializer
19:50:48	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-9fb66b6c4-cwkm2	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
19:50:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Started	Started container storage-initializer
19:50:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:50:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Pulling	Pulling image "kserve/xgbserver:latest"
19:51:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Created	Created container: kserve-container
19:51:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 18.751s (18.751s including waiting). Image size: 1436472579 bytes.
19:51:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Started	Started container kserve-container
19:51:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:51:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Created	Created container: kube-rbac-proxy
19:51:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Started	Started container kube-rbac-proxy
19:51:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:44 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Unhealthy	Readiness probe failed: dial tcp 10.132.0.54:8080: connect: connection refused
19:51:45 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.55:8080: connect: connection refused
19:51:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-6fc7d5b8c9	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8
19:51:54	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Killing	Stopping container kube-rbac-proxy
19:51:54	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Killing	Stopping container kserve-container
19:51:54	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-6fc7d5b8c9 from 2 to 1
19:51:58	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8	Unhealthy	Readiness probe failed: Get "https://10.132.0.55:8643/healthz": dial tcp 10.132.0.55:8643: connect: connection refused
19:52:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
19:52:09	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Killing	Stopping container kserve-container
19:52:09	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Killing	Stopping container kube-rbac-proxy
19:52:09	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Killing	Stopping container kube-rbac-proxy
19:52:09	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Killing	Stopping container kserve-container
19:52:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceNotReady	InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready
19:52:09 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:52:12	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
19:52:12	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn	Unhealthy	Readiness probe failed: Get "https://10.132.0.54:8643/healthz": dial tcp 10.132.0.54:8643: connect: connection refused
19:52:12 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
19:52:19	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-59b9d969c9 from 0 to 1
19:52:19	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-59b9d969c9	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-59b9d969c9-cv995
19:52:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:52:19 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:52:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-predictor-serving-cert" not found
19:52:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:52:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Started	Started container storage-initializer
19:52:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Created	Created container: storage-initializer
19:52:21	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-59b9d969c9-cv995	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
19:52:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Pulling	Pulling image "kserve/lgbserver:latest"
19:52:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
19:52:28 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:52:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Started	Started container kube-rbac-proxy
19:52:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.444s (6.444s including waiting). Image size: 644656917 bytes.
19:52:32 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:32 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Created	Created container: kube-rbac-proxy
19:52:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:52:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Started	Started container kserve-container
19:52:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Created	Created container: kserve-container
19:52:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Killing	Stopping container kserve-container
19:52:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Killing	Stopping container kube-rbac-proxy
19:52:38 (x10)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
19:52:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-9fb66b6c4-cwkm2	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
19:52:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:52:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:52:42	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-574749c459 from 0 to 1
19:52:42	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-574749c459	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-574749c459-466bc
19:52:43	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:52:43	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Started	Started container storage-initializer
19:52:43	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
19:52:43	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Created	Created container: storage-initializer
19:52:43	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
19:52:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Created	Created container: kserve-container
19:52:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Started	Started container kube-rbac-proxy
19:52:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Started	Started container kserve-container
19:52:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Created	Created container: kube-rbac-proxy
19:52:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:52:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:52:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:52:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
19:53:26 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:53:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:53:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:53:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Killing	Stopping container kserve-container
19:53:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Killing	Stopping container kube-rbac-proxy
19:53:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
19:53:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-574749c459-466bc	Unhealthy	Readiness probe failed: Get "http://10.134.0.47:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.134.0.47:8080: connect: connection refused
19:53:38	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Created	Created container: storage-initializer
19:53:38	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:53:38	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
19:53:38	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-6d58fc45fb	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g
19:53:38	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-6d58fc45fb from 0 to 1
19:53:38	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:53:38	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Started	Started container storage-initializer
19:53:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:53:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Started	Started container kube-rbac-proxy
19:53:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Created	Created container: kube-rbac-proxy
19:53:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:53:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Started	Started container kserve-container
19:53:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Created	Created container: kserve-container
19:53:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:56 (x11)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Unhealthy	Readiness probe failed: dial tcp 10.132.0.56:8080: connect: connection refused
19:54:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:54:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
19:54:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Killing	Stopping container kube-rbac-proxy
19:54:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Killing	Stopping container kserve-container
19:54:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-59b9d969c9-cv995	Unhealthy	Readiness probe failed: Get "https://10.132.0.56:8643/healthz": dial tcp 10.132.0.56:8643: connect: connection refused
19:54:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
19:54:16	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-7b5c95bfbc	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp
19:54:16	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-7b5c95bfbc from 0 to 1
19:54:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
19:54:17	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
19:54:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:54:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Created	Created container: storage-initializer
19:54:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Started	Started container storage-initializer
19:54:21 (x10)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:54:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Pulling	Pulling image "kserve/lgbserver:latest"
19:54:21	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
19:54:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:54:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.157s (6.157s including waiting). Image size: 644656917 bytes.
19:54:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Created	Created container: kserve-container
19:54:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Started	Started container kserve-container
19:54:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Started	Started container kube-rbac-proxy
19:54:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Created	Created container: kube-rbac-proxy
19:54:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Killing	Stopping container kserve-container
19:54:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Killing	Stopping container kube-rbac-proxy
19:54:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
19:54:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g	Unhealthy	Readiness probe failed: Get "http://10.134.0.48:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.134.0.48:8080: connect: connection refused
19:54:33	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
19:54:33	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-6fdbb69fc8 from 0 to 1
19:54:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
19:54:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:54:33	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-6fdbb69fc8	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps
19:54:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Created	Created container: storage-initializer
19:54:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Started	Started container storage-initializer
19:54:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:54:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Started	Started container kube-rbac-proxy
19:54:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:54:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Created	Created container: kserve-container
19:54:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Started	Started container kserve-container
19:54:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Created	Created container: kube-rbac-proxy
19:55:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:45 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
19:55:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
19:55:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:56:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Killing	Stopping container kserve-container
19:56:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Killing	Stopping container kube-rbac-proxy
19:56:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:56:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
19:56:09 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:56:09	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-6b7777c6	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5
19:56:09	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-6b7777c6 from 0 to 1
19:56:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
19:56:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:56:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Created	Created container: storage-initializer
19:56:10 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
19:56:10	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
19:56:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:56:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Started	Started container storage-initializer
19:56:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:56:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Created	Created container: kube-rbac-proxy
19:56:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Started	Started container kube-rbac-proxy
19:56:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Started	Started container kserve-container
19:56:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Created	Created container: kserve-container
19:56:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:56:16 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:16 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Killing	Stopping container kserve-container
19:56:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Killing	Stopping container kube-rbac-proxy
19:56:18 (x13)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
19:56:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
19:56:22	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-58f9f86dd8 from 0 to 1
19:56:22	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx
19:56:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:56:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:56:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Started	Started container storage-initializer
19:56:23	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
19:56:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Created	Created container: storage-initializer
19:56:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Created	Created container: kube-rbac-proxy
19:56:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:56:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Started	Started container kserve-container
19:56:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Created	Created container: kserve-container
19:56:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:56:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Started	Started container kube-rbac-proxy
19:56:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:57:04 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:57:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
19:57:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
19:57:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:57:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Killing	Stopping container kube-rbac-proxy
19:57:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Killing	Stopping container kserve-container
19:57:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Killing	Stopping container kube-rbac-proxy
19:57:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx	Killing	Stopping container kserve-container
19:57:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Unhealthy	Readiness probe failed: Get "https://10.134.0.51:8643/healthz": dial tcp 10.134.0.51:8643: connect: connection refused
19:57:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5	Unhealthy	Readiness probe failed: Get "http://10.134.0.51:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.134.0.51:8080: connect: connection refused
19:57:14	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-5c5d6c7f44 from 0 to 1
19:57:14	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-5c5d6c7f44	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb
19:57:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:57:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Started	Started container storage-initializer
19:57:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Created	Created container: storage-initializer
19:57:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:57:15	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
19:57:17	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-544db88986 from 0 to 1
19:57:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:57:17	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-544db88986	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph
19:57:18	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
19:57:18	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:57:18	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Created	Created container: storage-initializer
19:57:18	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Started	Started container storage-initializer
19:57:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Started	Started container kserve-container
19:57:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:57:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Created	Created container: kube-rbac-proxy
19:57:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Started	Started container kube-rbac-proxy
19:57:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Created	Created container: kserve-container
19:57:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Created	Created container: kserve-container
19:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Created	Created container: kube-rbac-proxy
19:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Started	Started container kserve-container
19:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Started	Started container kube-rbac-proxy
19:57:31 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
19:57:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:57:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
19:57:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Killing	Stopping container kserve-container
19:57:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph	Killing	Stopping container kube-rbac-proxy
19:57:53	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:57:53	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
19:57:53	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Created	Created container: storage-initializer
19:57:53	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-fc758d9c9	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh
19:57:53	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-fc758d9c9 from 0 to 1
19:57:53	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Started	Started container storage-initializer
19:57:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:57:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Started	Started container kserve-container
19:57:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Created	Created container: kserve-container
19:57:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:57:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:57:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Started	Started container kube-rbac-proxy
19:57:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Created	Created container: kube-rbac-proxy
19:57:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:36 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:58:36 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:58:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
19:58:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
19:58:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:43	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Killing	Stopping container kube-rbac-proxy
19:58:43	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh	Killing	Stopping container kserve-container
19:58:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Unhealthy	Readiness probe failed: Get "https://10.134.0.53:8643/healthz": dial tcp 10.134.0.53:8643: connect: connection refused
19:58:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Killing	Stopping container kube-rbac-proxy
19:58:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Killing	Stopping container kserve-container
19:58:46 (x10)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
19:58:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-7f8b8b68f	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc
19:58:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-7f8b8b68f from 0 to 1
19:58:48 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:58:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Created	Created container: storage-initializer
19:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:58:49	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
19:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Started	Started container storage-initializer
19:58:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
19:58:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-78f7f6bcc9	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-78f7f6bcc9-24q46
19:58:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-78f7f6bcc9 from 0 to 1
19:58:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-predictor-serving-cert" not found
19:58:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Created	Created container: storage-initializer
19:58:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Started	Started container storage-initializer
19:58:51	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
19:58:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Started	Started container kserve-container
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Created	Created container: kserve-container
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Started	Started container kserve-container
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Created	Created container: kserve-container
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:58:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Created	Created container: kube-rbac-proxy
19:58:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Started	Started container kube-rbac-proxy
19:58:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.643s (2.643s including waiting). Image size: 239481897 bytes.
19:58:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Created	Created container: kserve-agent
19:58:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Started	Started container kserve-agent
19:58:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:58:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Created	Created container: kube-rbac-proxy
19:58:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Started	Started container kube-rbac-proxy
19:59:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Unhealthy	Liveness probe failed: Get "http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.132.0.57:8080: connect: connection refused
19:59:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:03 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:00:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
20:00:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Killing	Stopping container kserve-container
20:00:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Killing	Stopping container kube-rbac-proxy
20:00:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Killing	Stopping container kserve-agent
20:00:13	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-69579d8675	SuccessfulCreate	Created pod: isvc-paddle-predictor-69579d8675-rc888
20:00:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
20:00:13	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-69579d8675 from 0 to 1
20:00:14	kserve-ci-e2e-test	multus	isvc-paddle-predictor-69579d8675-rc888	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
20:00:14	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Created	Created container: storage-initializer
20:00:14	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Started	Started container storage-initializer
20:00:14	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:00:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:20	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Pulling	Pulling image "kserve/paddleserver:latest"
20:00:22 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:00:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
20:00:31	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Started	Started container kserve-container
20:00:31	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t
20:00:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8643/healthz": dial tcp 10.134.0.56:8643: connect: connection refused
20:00:31	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:00:31	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.594s (11.594s including waiting). Image size: 1204389121 bytes.
20:00:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Killing	Stopping container kserve-container
20:00:31	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Created	Created container: kserve-container
20:00:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Killing	Stopping container kube-rbac-proxy
20:00:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
20:00:31	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-688c74c647 from 0 to 1
20:00:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Created	Created container: storage-initializer
20:00:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:00:32 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-78f7f6bcc9-24q46	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
20:00:32	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Created	Created container: kube-rbac-proxy
20:00:32	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
20:00:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Started	Started container storage-initializer
20:00:32	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Started	Started container kube-rbac-proxy
20:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Created	Created container: kserve-container
20:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Created	Created container: kube-rbac-proxy
20:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Started	Started container kserve-container
20:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Started	Started container kube-rbac-proxy
20:00:33 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Unhealthy	Readiness probe failed: Get "http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.132.0.57:8080: connect: connection refused
20:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
20:00:38 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc	Unhealthy	Readiness probe failed: Get "https://10.132.0.57:8643/healthz": dial tcp 10.132.0.57:8643: connect: connection refused
20:00:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:01:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:01:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:01:19 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Unhealthy	Readiness probe failed: dial tcp 10.134.0.57:8080: connect: connection refused
20:01:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:01:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:01:29 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:01:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
20:01:35	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Killing	Stopping container kube-rbac-proxy
20:01:35	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-69579d8675-rc888	Killing	Stopping container kserve-container
20:01:40	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-6fd5dfd5ff	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw
20:01:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
20:01:40	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-6fd5dfd5ff from 0 to 1
20:01:40	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-runtime-predictor-serving-cert" not found
20:01:41	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Started	Started container storage-initializer
20:01:41	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Created	Created container: storage-initializer
20:01:41	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:01:41	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
20:01:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:01:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Pulled	Container image "kserve/paddleserver:latest" already present on machine
20:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Created	Created container: kserve-container
20:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Started	Started container kserve-container
20:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Created	Created container: kube-rbac-proxy
20:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Started	Started container kube-rbac-proxy
20:01:50 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
20:02:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:02:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
20:02:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Killing	Stopping container kserve-container
20:02:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Killing	Stopping container kube-rbac-proxy
20:02:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
20:02:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
20:02:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q
20:02:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc from 0 to 1
20:02:02 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
20:02:03	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
20:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t	Unhealthy	Readiness probe failed: Get "https://10.134.0.58:8643/healthz": dial tcp 10.134.0.58:8643: connect: connection refused
20:02:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	Created	Created container: storage-initializer
20:02:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:02:09 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:02:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	Started	Started container storage-initializer
20:02:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q	Killing	Stopping container storage-initializer
20:02:13	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z
20:02:13	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9 from 0 to 1
20:02:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
20:02:14	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
20:02:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Started	Started container storage-initializer
20:02:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Created	Created container: storage-initializer
20:02:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:02:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Started	Started container kube-rbac-proxy
20:02:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Created	Created container: kube-rbac-proxy
20:02:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
20:02:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Created	Created container: kserve-container
20:02:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Started	Started container kserve-container
20:02:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:02:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:02:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:02:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:44 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:02:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
20:02:51	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Killing	Stopping container kube-rbac-proxy
20:02:51	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Killing	Stopping container kserve-container
20:02:54	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
20:02:54 (x8)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.59:8080: connect: connection refused
20:02:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:02:56	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-65d45d6447 from 0 to 1
20:02:57	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
20:02:57	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-65d45d6447	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-65d45d6447-828vt
20:02:58	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Created	Created container: storage-initializer
20:02:58	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
20:02:58	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:02:58	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Started	Started container storage-initializer
20:03:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Started	Started container kube-rbac-proxy
20:03:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Created	Created container: kserve-container
20:03:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Pulled	Container image "kserve/paddleserver:latest" already present on machine
20:03:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Created	Created container: kube-rbac-proxy
20:03:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:03:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Started	Started container kserve-container
20:03:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
20:03:46 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:03:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8643/healthz": dial tcp 10.134.0.60:8643: connect: connection refused
20:03:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Killing	Stopping container kube-rbac-proxy
20:03:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Killing	Stopping container kserve-container
20:03:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7
20:03:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf from 0 to 1
20:03:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
20:03:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
20:03:55 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z	Unhealthy	Readiness probe failed: dial tcp 10.134.0.60:8080: connect: connection refused
20:03:55	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7	AddedInterface	Add eth0 [10.132.0.59/23] from ovn-kubernetes
20:03:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7	Started	Started container storage-initializer
20:03:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:03:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7	Created	Created container: storage-initializer
20:04:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
20:04:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:04:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_kserve-ci-e2e-test(7b1fab5e-c65f-4f66-9aa5-b5a2c6217420)
20:04:04 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:04:05	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2
20:04:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
20:04:05	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d from 0 to 1
20:04:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
20:04:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Created	Created container: storage-initializer
20:04:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Started	Started container storage-initializer
20:04:07	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
20:04:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:04:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Killing	Stopping container kserve-container
20:04:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Created	Created container: kserve-container
20:04:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
20:04:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Started	Started container kserve-container
20:04:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Started	Started container kube-rbac-proxy
20:04:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Created	Created container: kube-rbac-proxy
20:04:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Killing	Stopping container kube-rbac-proxy
20:04:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Unhealthy	Readiness probe failed: Get "https://10.134.0.61:8643/healthz": dial tcp 10.134.0.61:8643: connect: connection refused
20:04:10 (x8)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-65d45d6447-828vt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.61:8080: connect: connection refused
20:04:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
20:04:13	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-57685d4d84 from 0 to 1
20:04:13	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-57685d4d84	SuccessfulCreate	Created pod: isvc-pmml-predictor-57685d4d84-bx4tr
20:04:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Started	Started container storage-initializer
20:04:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Created	Created container: storage-initializer
20:04:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:04:14	kserve-ci-e2e-test	multus	isvc-pmml-predictor-57685d4d84-bx4tr	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
20:04:19	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Pulling	Pulling image "kserve/pmmlserver:latest"
20:04:27	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Started	Started container kube-rbac-proxy
20:04:27	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:27	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Started	Started container kserve-container
20:04:27	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Created	Created container: kube-rbac-proxy
20:04:27	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.617s (7.617s including waiting). Image size: 853725565 bytes.
20:04:27	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Created	Created container: kserve-container
20:04:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:05:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:05:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:05:25 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.62:8080: connect: connection refused
20:05:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
20:05:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:05:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Killing	Stopping container kube-rbac-proxy
20:05:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Killing	Stopping container kserve-container
20:05:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78 from 0 to 1
20:05:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
20:05:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d
20:05:37	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
20:05:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2	Unhealthy	Readiness probe failed: Get "https://10.134.0.62:8643/healthz": dial tcp 10.134.0.62:8643: connect: connection refused
20:05:41 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d	Created	Created container: storage-initializer
20:05:41 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d	Started	Started container storage-initializer
20:05:41 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:05:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:05:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_kserve-ci-e2e-test(c1fb55ce-a411-4157-8c7f-b4ab4358b2bb)
20:05:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:05:48	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Created	Created container: storage-initializer
20:05:48	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:05:48	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
20:05:48	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Started	Started container storage-initializer
20:05:48	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-86649cc556	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-86649cc556-bd42t
20:05:48	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-86649cc556 from 0 to 1
20:05:53	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Pulling	Pulling image "kserve/pmmlserver:latest"
20:05:54 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.63:8080: connect: connection refused
20:05:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:05:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:06:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.311s (7.311s including waiting). Image size: 853725565 bytes.
20:06:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Started	Started container kube-rbac-proxy
20:06:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Created	Created container: kube-rbac-proxy
20:06:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:06:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:06:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
20:06:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Started	Started container kserve-container
20:06:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Created	Created container: kserve-container
20:06:05	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Killing	Stopping container kserve-container
20:06:05	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Killing	Stopping container kube-rbac-proxy
20:06:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-57685d4d84-bx4tr	Unhealthy	Readiness probe failed: Get "https://10.134.0.63:8643/healthz": dial tcp 10.134.0.63:8643: connect: connection refused
20:06:11	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Started	Started container storage-initializer
20:06:11	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Created	Created container: storage-initializer
20:06:11	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-58cc747bcd	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-58cc747bcd-bfdcd
20:06:11	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
20:06:11	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:06:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
20:06:11	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-58cc747bcd from 0 to 1
20:06:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Started	Started container kserve-container
20:06:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Created	Created container: kserve-container
20:06:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
20:06:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:06:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Created	Created container: kube-rbac-proxy
20:06:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Started	Started container kube-rbac-proxy
20:06:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:07:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Unhealthy	Readiness probe failed: dial tcp 10.132.0.61:8080: connect: connection refused
20:07:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:07:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:07:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
20:07:27 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:07:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Killing	Stopping container kube-rbac-proxy
20:07:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Killing	Stopping container kserve-container
20:07:32	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-86649cc556-bd42t	Unhealthy	Readiness probe failed: Get "https://10.132.0.61:8643/healthz": dial tcp 10.132.0.61:8643: connect: connection refused
20:07:41 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:07:41 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:07:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
20:07:47 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:07:53	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Killing	Stopping container kube-rbac-proxy
20:07:53	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Killing	Stopping container kserve-container
20:07:53	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8643/healthz": read tcp 10.134.0.2:46824->10.134.0.64:8643: read: connection reset by peer
20:07:56 (x12)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-58cc747bcd-bfdcd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.64:8080: connect: connection refused
20:07:58	kserve-ci-e2e-test	replicaset-controller	isvc-primary-cf00b2-predictor-7d877cbff	SuccessfulCreate	Created pod: isvc-primary-cf00b2-predictor-7d877cbff-pdv62
20:07:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-cf00b2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-cf00b2": the object has been modified; please apply your changes to the latest version and try again
20:07:58	kserve-ci-e2e-test	multus	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
20:07:58	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:07:58	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Created	Created container: storage-initializer
20:07:58	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Started	Started container storage-initializer
20:07:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-cf00b2	UpdateFailed	Failed to update status for InferenceService "isvc-primary-cf00b2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-cf00b2": the object has been modified; please apply your changes to the latest version and try again
20:07:58	kserve-ci-e2e-test	deployment-controller	isvc-primary-cf00b2-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-cf00b2-predictor-7d877cbff from 0 to 1
20:08:03	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Created	Created container: kserve-container
20:08:03	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Started	Started container kube-rbac-proxy
20:08:03	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1970" already present on machine
20:08:03	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Created	Created container: kube-rbac-proxy
20:08:03	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:08:03	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Started	Started container kserve-container
20:08:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-cf00b2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-cf00b2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-cf00b2	InferenceServiceReady	InferenceService [isvc-primary-cf00b2] is Ready
20:09:25 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-cf00b2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:09:43	kserve-ci-e2e-test	deployment-controller	isvc-secondary-cf00b2-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-cf00b2-predictor-656f4cd6fb from 0 to 1
20:09:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-cf00b2	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-cf00b2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-cf00b2": the object has been modified; please apply your changes to the latest version and try again
20:09:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-cf00b2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-cf00b2": the object has been modified; please apply your changes to the latest version and try again
20:09:43	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-cf00b2-predictor-656f4cd6fb	SuccessfulCreate	Created pod: isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr
20:09:44	kserve-ci-e2e-test	multus	isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
20:09:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d02ee9c5751b53c265ac939508c0a6220ab7a2cbf94f22a8d1199e5d5c8a9e3" already present on machine
20:09:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr	Created	Created container: storage-initializer
20:09:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr	Started	Started container storage-initializer
20:09:55	kserve-ci-e2e-test	kubelet	isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_kserve-ci-e2e-test(6785a832-68c9-4fe0-8fe5-d10cce162ad1)
20:09:56 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-cf00b2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:09:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-cf00b2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-cf00b2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:59 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-cf00b2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:59 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-cf00b2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:10:01	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Killing	Stopping container kserve-container
20:10:01	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Killing	Stopping container kube-rbac-proxy
20:10:04 (x11)	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Unhealthy	Readiness probe failed: dial tcp 10.134.0.65:8080: connect: connection refused
20:10:05	kserve-ci-e2e-test	kubelet	isvc-primary-cf00b2-predictor-7d877cbff-pdv62	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8643/healthz": dial tcp 10.134.0.65:8643: connect: connection refused