KUBEFLOW_SKIP_INSTALL is set. Skipping SDK installation to use the version baked into the notebook image. Input Notebook: /opt/app-root/notebooks/mnist.ipynb Output Notebook: /opt/app-root/src/out.ipynb [IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption. Executing notebook with kernel: python3 Executing Cell 1--------------------------------------- Ending Cell 1------------------------------------------ Executing Cell 2--------------------------------------- [notebook] S3 not configured: missing endpoint or bucket env vars [notebook] Dataset not complete, attempting internet download... [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte [notebook] Internet download completed successfully Ending Cell 2------------------------------------------ Executing Cell 3--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Trainer control-plane version info is not available: unable to read 'kubeflow_trainer_version' from ConfigMap 'kubeflow-trainer-public' in namespace 'kubeflow-system': (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '8c323c2e-3dbc-40c8-8e1f-0e4ee6d8ea28', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '77f5472a-d825-4035-bcb9-80563d58e8d5', 'X-Kubernetes-Pf-Prioritylevel-Uid': '750844ff-09bd-4f0c-a196-970589515e39', 'Date': 'Fri, 23 Oct 2026 04:10:39 GMT', 'Content-Length': '388'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubeflow-trainer-public\" is forbidden: User \"system:serviceaccount:redhat-ods-applications:trainer-e2e\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow-system\"","reason":"Forbidden","details":{"name":"kubeflow-trainer-public","kind":"configmaps"},"code":403} . If your Trainer control-plane is installed in a different namespace, set the KUBEFLOW_SYSTEM_NAMESPACE environment variable accordingly. Ending Cell 3------------------------------------------ Executing Cell 4--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 4------------------------------------------ Executing Cell 5--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [notebook] Using PVC: pvc-ggsvt Kueue enabled with LocalQueue: lq-h9lmc [notebook] Job submitted: v5a437fc2ded Ending Cell 5------------------------------------------ Executing Cell 6--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Training job final status: Complete ✓ Training job 'v5a437fc2ded' completed successfully /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 6------------------------------------------ Executing Cell 7--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Step: node-0, Status: Succeeded, Devices: cpu x 2 Step: node-1, Status: Succeeded, Devices: cpu x 2 /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 7------------------------------------------ Executing Cell 8--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( W1023 04:10:44.149000 1 torch/distributed/run.py:874] W1023 04:10:44.149000 1 torch/distributed/run.py:874] ***************************************** W1023 04:10:44.149000 1 torch/distributed/run.py:874] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1023 04:10:44.149000 1 torch/distributed/run.py:874] ***************************************** Using Device: cpu, Backend: gloo Using Device: cpu, Backend: gloo Distributed Training for WORLD_SIZE: 4, RANK: 0, LOCAL_RANK: 0 Distributed Training for WORLD_SIZE: 4, RANK: 1, LOCAL_RANK: 1 Using dataset from shared PVC at /mnt/shared Using dataset from shared PVC at /mnt/shared Train Epoch: 1 [0/60000 (0%)] Loss: 2.303642 Train Epoch: 1 [1000/60000 (7%)] Loss: 2.041021 Train Epoch: 1 [2000/60000 (13%)] Loss: 2.177464 Train Epoch: 1 [3000/60000 (20%)] Loss: 1.618907 Train Epoch: 1 [4000/60000 (27%)] Loss: 1.123514 Train Epoch: 1 [5000/60000 (33%)] Loss: 0.790678 Train Epoch: 1 [6000/60000 (40%)] Loss: 0.700863 Train Epoch: 1 [7000/60000 (47%)] Loss: 0.728214 Train Epoch: 1 [8000/60000 (53%)] Loss: 0.500510 Train Epoch: 1 [9000/60000 (60%)] Loss: 0.623691 Train Epoch: 1 [10000/60000 (67%)] Loss: 0.374186 Train Epoch: 1 [11000/60000 (73%)] Loss: 0.493790 Train Epoch: 1 [12000/60000 (80%)] Loss: 0.449228 Train Epoch: 1 [13000/60000 (87%)] Loss: 0.472200 Train Epoch: 1 [14000/60000 (93%)] Loss: 0.488852 Train Epoch: 2 [0/60000 (0%)] Loss: 0.520867 Train Epoch: 2 [1000/60000 (7%)] Loss: 0.410515 Train Epoch: 2 [2000/60000 (13%)] Loss: 0.491099 Train Epoch: 2 [3000/60000 (20%)] Loss: 0.311137 Train Epoch: 2 [4000/60000 (27%)] Loss: 0.395999 Train Epoch: 2 [5000/60000 (33%)] Loss: 0.404434 Train Epoch: 2 [6000/60000 (40%)] Loss: 0.507730 Train Epoch: 2 [7000/60000 (47%)] Loss: 0.549229 Train Epoch: 2 [8000/60000 (53%)] Loss: 0.284400 Train Epoch: 2 [9000/60000 (60%)] Loss: 0.388193 Train Epoch: 2 [10000/60000 (67%)] Loss: 0.275778 Train Epoch: 2 [11000/60000 (73%)] Loss: 0.400973 Train Epoch: 2 [12000/60000 (80%)] Loss: 0.288587 Train Epoch: 2 [13000/60000 (87%)] Loss: 0.346585 Train Epoch: 2 [14000/60000 (93%)] Loss: 0.304414 Training is finished Ending Cell 8------------------------------------------ Executing Cell 9--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4025c7ccf93e49d0946af1abba28101-bdc626a586a91815.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 9------------------------------------------ NOTEBOOK_STATUS: SUCCESS