KUBEFLOW_SKIP_INSTALL is set. Skipping SDK installation to use the version baked into the notebook image. Input Notebook: /opt/app-root/notebooks/mnist.ipynb Output Notebook: /opt/app-root/src/out.ipynb [IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption. Executing notebook with kernel: python3 Executing Cell 1--------------------------------------- Ending Cell 1------------------------------------------ Executing Cell 2--------------------------------------- [notebook] S3 not configured: missing endpoint or bucket env vars [notebook] Dataset not complete, attempting internet download... [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte [notebook] Internet download completed successfully Ending Cell 2------------------------------------------ Executing Cell 3--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Trainer control-plane version info is not available: unable to read 'kubeflow_trainer_version' from ConfigMap 'kubeflow-trainer-public' in namespace 'kubeflow-system': (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b7e519af-ebae-4ec8-aafa-5511032a590a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'c4f4e71a-b1ab-4d71-a976-8e26b02d6065', 'X-Kubernetes-Pf-Prioritylevel-Uid': '8c7c762e-1d56-483e-ae8c-b60ad844742d', 'Date': 'Mon, 02 Nov 2026 10:04:07 GMT', 'Content-Length': '376'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubeflow-trainer-public\" is forbidden: User \"system:serviceaccount:opendatahub:trainer-e2e\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow-system\"","reason":"Forbidden","details":{"name":"kubeflow-trainer-public","kind":"configmaps"},"code":403} . If your Trainer control-plane is installed in a different namespace, set the KUBEFLOW_SYSTEM_NAMESPACE environment variable accordingly. Ending Cell 3------------------------------------------ Executing Cell 4--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 4------------------------------------------ Executing Cell 5--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [notebook] Using PVC: pvc-xw6hs KUEUE_QUEUE_NAME not set; Kueue will use the default LocalQueue if the namespace is Kueue-managed [notebook] Job submitted: w2b9dc528159 Ending Cell 5------------------------------------------ Executing Cell 6--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Training job final status: Complete ✓ Training job 'w2b9dc528159' completed successfully /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 6------------------------------------------ Executing Cell 7--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Step: node-0, Status: Succeeded, Devices: cpu x 2 Step: node-1, Status: Succeeded, Devices: cpu x 2 Ending Cell 7------------------------------------------ Executing Cell 8--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( W1102 10:04:12.295000 1 torch/distributed/run.py:874] W1102 10:04:12.295000 1 torch/distributed/run.py:874] ***************************************** W1102 10:04:12.295000 1 torch/distributed/run.py:874] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1102 10:04:12.295000 1 torch/distributed/run.py:874] ***************************************** Using Device: cpu, Backend: gloo Using Device: cpu, Backend: gloo Distributed Training for WORLD_SIZE: 4, RANK: 0, LOCAL_RANK: 0 Distributed Training for WORLD_SIZE: 4, RANK: 1, LOCAL_RANK: 1 Using dataset from shared PVC at /mnt/shared Using dataset from shared PVC at /mnt/shared Train Epoch: 1 [0/60000 (0%)] Loss: 2.301080 Train Epoch: 1 [1000/60000 (7%)] Loss: 2.014548 Train Epoch: 1 [2000/60000 (13%)] Loss: 1.971865 Train Epoch: 1 [3000/60000 (20%)] Loss: 1.354196 Train Epoch: 1 [4000/60000 (27%)] Loss: 1.103706 Train Epoch: 1 [5000/60000 (33%)] Loss: 0.827480 Train Epoch: 1 [6000/60000 (40%)] Loss: 0.729299 Train Epoch: 1 [7000/60000 (47%)] Loss: 0.801915 Train Epoch: 1 [8000/60000 (53%)] Loss: 0.619357 Train Epoch: 1 [9000/60000 (60%)] Loss: 0.637561 Train Epoch: 1 [10000/60000 (67%)] Loss: 0.456242 Train Epoch: 1 [11000/60000 (73%)] Loss: 0.612026 Train Epoch: 1 [12000/60000 (80%)] Loss: 0.514852 Train Epoch: 1 [13000/60000 (87%)] Loss: 0.559816 Train Epoch: 1 [14000/60000 (93%)] Loss: 0.460431 Train Epoch: 2 [0/60000 (0%)] Loss: 0.517589 Train Epoch: 2 [1000/60000 (7%)] Loss: 0.427320 Train Epoch: 2 [2000/60000 (13%)] Loss: 0.442692 Train Epoch: 2 [3000/60000 (20%)] Loss: 0.382887 Train Epoch: 2 [4000/60000 (27%)] Loss: 0.452895 Train Epoch: 2 [5000/60000 (33%)] Loss: 0.322344 Train Epoch: 2 [6000/60000 (40%)] Loss: 0.562502 Train Epoch: 2 [7000/60000 (47%)] Loss: 0.481187 Train Epoch: 2 [8000/60000 (53%)] Loss: 0.325220 Train Epoch: 2 [9000/60000 (60%)] Loss: 0.467202 Train Epoch: 2 [10000/60000 (67%)] Loss: 0.302082 Train Epoch: 2 [11000/60000 (73%)] Loss: 0.351012 Train Epoch: 2 [12000/60000 (80%)] Loss: 0.366964 Train Epoch: 2 [13000/60000 (87%)] Loss: 0.428310 Train Epoch: 2 [14000/60000 (93%)] Loss: 0.338910 Training is finished Ending Cell 8------------------------------------------ Executing Cell 9--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4fd2f2f345e6423da355455404832b7-a7e88af7f5f1bd0c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 9------------------------------------------ NOTEBOOK_STATUS: SUCCESS