KUBEFLOW_SKIP_INSTALL is set. Skipping SDK installation to use the version baked into the notebook image. Input Notebook: /opt/app-root/notebooks/mnist.ipynb Output Notebook: /opt/app-root/src/out.ipynb [IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption. Executing notebook with kernel: python3 Executing Cell 1--------------------------------------- Ending Cell 1------------------------------------------ Executing Cell 2--------------------------------------- [notebook] S3 not configured: missing endpoint or bucket env vars [notebook] Dataset not complete, attempting internet download... [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte [notebook] Internet download completed successfully Ending Cell 2------------------------------------------ Executing Cell 3--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Trainer control-plane version info is not available: unable to read 'kubeflow_trainer_version' from ConfigMap 'kubeflow-trainer-public' in namespace 'kubeflow-system': (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a12224d2-7b4f-4815-a4f9-83c0f30c72de', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '72ecb3cd-829a-4dce-9805-e4771356a1a8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '8057a325-292e-4936-a19d-e29347ef4c5e', 'Date': 'Tue, 03 Nov 2026 04:09:15 GMT', 'Content-Length': '388'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubeflow-trainer-public\" is forbidden: User \"system:serviceaccount:redhat-ods-applications:trainer-e2e\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow-system\"","reason":"Forbidden","details":{"name":"kubeflow-trainer-public","kind":"configmaps"},"code":403} . If your Trainer control-plane is installed in a different namespace, set the KUBEFLOW_SYSTEM_NAMESPACE environment variable accordingly. Ending Cell 3------------------------------------------ Executing Cell 4--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 4------------------------------------------ Executing Cell 5--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [notebook] Using PVC: pvc-szrvj Kueue enabled with LocalQueue: lq-gntg5 [notebook] Job submitted: v21bdac94b06 Ending Cell 5------------------------------------------ Executing Cell 6--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Training job final status: Complete ✓ Training job 'v21bdac94b06' completed successfully /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 6------------------------------------------ Executing Cell 7--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Step: node-0, Status: Succeeded, Devices: cpu x 2 Step: node-1, Status: Succeeded, Devices: cpu x 2 Ending Cell 7------------------------------------------ Executing Cell 8--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( W1103 04:09:21.895000 1 torch/distributed/run.py:874] W1103 04:09:21.895000 1 torch/distributed/run.py:874] ***************************************** W1103 04:09:21.895000 1 torch/distributed/run.py:874] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1103 04:09:21.895000 1 torch/distributed/run.py:874] ***************************************** Using Device: cpu, Backend: gloo Using Device: cpu, Backend: gloo Distributed Training for WORLD_SIZE: 4, RANK: 1, LOCAL_RANK: 1 Distributed Training for WORLD_SIZE: 4, RANK: 0, LOCAL_RANK: 0 Using dataset from shared PVC at /mnt/sharedUsing dataset from shared PVC at /mnt/shared Train Epoch: 1 [0/60000 (0%)] Loss: 2.299648 Train Epoch: 1 [1000/60000 (7%)] Loss: 1.446134 Train Epoch: 1 [2000/60000 (13%)] Loss: 1.888945 Train Epoch: 1 [3000/60000 (20%)] Loss: 1.641615 Train Epoch: 1 [4000/60000 (27%)] Loss: 1.298665 Train Epoch: 1 [5000/60000 (33%)] Loss: 1.056461 Train Epoch: 1 [6000/60000 (40%)] Loss: 0.900710 Train Epoch: 1 [7000/60000 (47%)] Loss: 0.904406 Train Epoch: 1 [8000/60000 (53%)] Loss: 0.773076 Train Epoch: 1 [9000/60000 (60%)] Loss: 0.690305 Train Epoch: 1 [10000/60000 (67%)] Loss: 0.571852 Train Epoch: 1 [11000/60000 (73%)] Loss: 0.609075 Train Epoch: 1 [12000/60000 (80%)] Loss: 0.561538 Train Epoch: 1 [13000/60000 (87%)] Loss: 0.586279 Train Epoch: 1 [14000/60000 (93%)] Loss: 0.517328 Train Epoch: 2 [0/60000 (0%)] Loss: 0.649400 Train Epoch: 2 [1000/60000 (7%)] Loss: 0.496013 Train Epoch: 2 [2000/60000 (13%)] Loss: 0.552539 Train Epoch: 2 [3000/60000 (20%)] Loss: 0.435076 Train Epoch: 2 [4000/60000 (27%)] Loss: 0.538732 Train Epoch: 2 [5000/60000 (33%)] Loss: 0.438765 Train Epoch: 2 [6000/60000 (40%)] Loss: 0.619764 Train Epoch: 2 [7000/60000 (47%)] Loss: 0.658453 Train Epoch: 2 [8000/60000 (53%)] Loss: 0.430869 Train Epoch: 2 [9000/60000 (60%)] Loss: 0.502742 Train Epoch: 2 [10000/60000 (67%)] Loss: 0.332181 Train Epoch: 2 [11000/60000 (73%)] Loss: 0.472969 Train Epoch: 2 [12000/60000 (80%)] Loss: 0.410370 Train Epoch: 2 [13000/60000 (87%)] Loss: 0.464713 Train Epoch: 2 [14000/60000 (93%)] Loss: 0.337480 Training is finished Ending Cell 8------------------------------------------ Executing Cell 9--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a6dc1da2bd83f4dd4812c7b4adaf3540-96f73515b39f48bb.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 9------------------------------------------ NOTEBOOK_STATUS: SUCCESS