KUBEFLOW_SKIP_INSTALL is set. Skipping SDK installation to use the version baked into the notebook image. Input Notebook: /opt/app-root/notebooks/mnist.ipynb Output Notebook: /opt/app-root/src/out.ipynb [IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption. Executing notebook with kernel: python3 Executing Cell 1--------------------------------------- Ending Cell 1------------------------------------------ Executing Cell 2--------------------------------------- [notebook] S3 not configured: missing endpoint or bucket env vars [notebook] Dataset not complete, attempting internet download... [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte [notebook] Internet download completed successfully Ending Cell 2------------------------------------------ Executing Cell 3--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Trainer control-plane version info is not available: unable to read 'kubeflow_trainer_version' from ConfigMap 'kubeflow-trainer-public' in namespace 'kubeflow-system': (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '47b1f2c0-7940-45fe-8767-04873ab5a82c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2d074802-94bf-414c-b6cf-93c703a89c5f', 'X-Kubernetes-Pf-Prioritylevel-Uid': '2be1c82c-dc6a-488b-9825-dd9916f1410b', 'Date': 'Tue, 03 Nov 2026 03:18:03 GMT', 'Content-Length': '376'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubeflow-trainer-public\" is forbidden: User \"system:serviceaccount:opendatahub:trainer-e2e\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow-system\"","reason":"Forbidden","details":{"name":"kubeflow-trainer-public","kind":"configmaps"},"code":403} . If your Trainer control-plane is installed in a different namespace, set the KUBEFLOW_SYSTEM_NAMESPACE environment variable accordingly. Ending Cell 3------------------------------------------ Executing Cell 4--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 4------------------------------------------ Executing Cell 5--------------------------------------- [notebook] Using PVC: pvc-lw95s Kueue enabled with LocalQueue: lq-9825f [notebook] Job submitted: q1af0b51ecf8 /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 5------------------------------------------ Executing Cell 6--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Training job final status: Complete ✓ Training job 'q1af0b51ecf8' completed successfully /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 6------------------------------------------ Executing Cell 7--------------------------------------- Step: node-0, Status: Succeeded, Devices: cpu x 2 Step: node-1, Status: Succeeded, Devices: cpu x 2 /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 7------------------------------------------ Executing Cell 8--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( W1103 03:18:07.310000 1 torch/distributed/run.py:874] W1103 03:18:07.310000 1 torch/distributed/run.py:874] ***************************************** W1103 03:18:07.310000 1 torch/distributed/run.py:874] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1103 03:18:07.310000 1 torch/distributed/run.py:874] ***************************************** Using Device: cpu, Backend: gloo Using Device: cpu, Backend: gloo Distributed Training for WORLD_SIZE: 4, RANK: 0, LOCAL_RANK: 0 Distributed Training for WORLD_SIZE: 4, RANK: 1, LOCAL_RANK: 1 Using dataset from shared PVC at /mnt/sharedUsing dataset from shared PVC at /mnt/shared Train Epoch: 1 [0/60000 (0%)] Loss: 2.298278 Train Epoch: 1 [1000/60000 (7%)] Loss: 1.538986 Train Epoch: 1 [2000/60000 (13%)] Loss: 1.679610 Train Epoch: 1 [3000/60000 (20%)] Loss: 2.066158 Train Epoch: 1 [4000/60000 (27%)] Loss: 1.277263 Train Epoch: 1 [5000/60000 (33%)] Loss: 0.919222 Train Epoch: 1 [6000/60000 (40%)] Loss: 0.902695 Train Epoch: 1 [7000/60000 (47%)] Loss: 0.807271 Train Epoch: 1 [8000/60000 (53%)] Loss: 0.535251 Train Epoch: 1 [9000/60000 (60%)] Loss: 0.679820 Train Epoch: 1 [10000/60000 (67%)] Loss: 0.395503 Train Epoch: 1 [11000/60000 (73%)] Loss: 0.599357 Train Epoch: 1 [12000/60000 (80%)] Loss: 0.504139 Train Epoch: 1 [13000/60000 (87%)] Loss: 0.603782 Train Epoch: 1 [14000/60000 (93%)] Loss: 0.426077 Train Epoch: 2 [0/60000 (0%)] Loss: 0.533425 Train Epoch: 2 [1000/60000 (7%)] Loss: 0.490611 Train Epoch: 2 [2000/60000 (13%)] Loss: 0.438730 Train Epoch: 2 [3000/60000 (20%)] Loss: 0.420892 Train Epoch: 2 [4000/60000 (27%)] Loss: 0.451934 Train Epoch: 2 [5000/60000 (33%)] Loss: 0.400441 Train Epoch: 2 [6000/60000 (40%)] Loss: 0.589565 Train Epoch: 2 [7000/60000 (47%)] Loss: 0.529287 Train Epoch: 2 [8000/60000 (53%)] Loss: 0.357473 Train Epoch: 2 [9000/60000 (60%)] Loss: 0.498512 Train Epoch: 2 [10000/60000 (67%)] Loss: 0.334760 Train Epoch: 2 [11000/60000 (73%)] Loss: 0.361286 Train Epoch: 2 [12000/60000 (80%)] Loss: 0.357403 Train Epoch: 2 [13000/60000 (87%)] Loss: 0.461732 Train Epoch: 2 [14000/60000 (93%)] Loss: 0.305610 Training is finished /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 8------------------------------------------ Executing Cell 9--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1129: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a822b528666d34de89ef89f7d062589f-9e12cded71ab669c.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 9------------------------------------------ NOTEBOOK_STATUS: SUCCESS