KUBEFLOW_SKIP_INSTALL is set. Skipping SDK installation to use the version baked into the notebook image. Input Notebook: /opt/app-root/notebooks/mnist.ipynb Output Notebook: /opt/app-root/src/out.ipynb [IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption. Executing notebook with kernel: python3 Executing Cell 1--------------------------------------- Ending Cell 1------------------------------------------ Executing Cell 2--------------------------------------- [notebook] S3 not configured: missing endpoint or bucket env vars [notebook] Dataset not complete, attempting internet download... [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/train-labels-idx1-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-images-idx3-ubyte [notebook] Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz ... [notebook] Decompressing /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz ... [notebook] Done: /opt/app-root/src/FashionMNIST/raw/t10k-labels-idx1-ubyte [notebook] Internet download completed successfully Ending Cell 2------------------------------------------ Executing Cell 3--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Trainer control-plane version info is not available: unable to read 'kubeflow_trainer_version' from ConfigMap 'kubeflow-trainer-public' in namespace 'kubeflow-system': (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '495b9426-9248-4766-b58d-ef127b18e2bb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '4524ed50-e414-4c7f-86a2-a7d11d022402', 'X-Kubernetes-Pf-Prioritylevel-Uid': '45a9baa8-1884-4230-aa31-b51cf7bf986f', 'Date': 'Tue, 03 Nov 2026 13:25:13 GMT', 'Content-Length': '388'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubeflow-trainer-public\" is forbidden: User \"system:serviceaccount:redhat-ods-applications:trainer-e2e\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow-system\"","reason":"Forbidden","details":{"name":"kubeflow-trainer-public","kind":"configmaps"},"code":403} . If your Trainer control-plane is installed in a different namespace, set the KUBEFLOW_SYSTEM_NAMESPACE environment variable accordingly. Ending Cell 3------------------------------------------ Executing Cell 4--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 4------------------------------------------ Executing Cell 5--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( [notebook] Using PVC: pvc-hzdgr Kueue enabled with LocalQueue: lq-5s27t [notebook] Job submitted: d240fbd6a54b Ending Cell 5------------------------------------------ Executing Cell 6--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Training job final status: Complete ✓ Training job 'd240fbd6a54b' completed successfully /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 6------------------------------------------ Executing Cell 7--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Step: node-0, Status: Succeeded, Devices: cpu x 2 Step: node-1, Status: Succeeded, Devices: cpu x 2 Ending Cell 7------------------------------------------ Executing Cell 8--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( W1103 13:25:19.956000 1 torch/distributed/run.py:874] W1103 13:25:19.956000 1 torch/distributed/run.py:874] ***************************************** W1103 13:25:19.956000 1 torch/distributed/run.py:874] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1103 13:25:19.956000 1 torch/distributed/run.py:874] ***************************************** Using Device: cpu, Backend: gloo Using Device: cpu, Backend: gloo Distributed Training for WORLD_SIZE: 4, RANK: 0, LOCAL_RANK: 0 Distributed Training for WORLD_SIZE: 4, RANK: 1, LOCAL_RANK: 1 Using dataset from shared PVC at /mnt/sharedUsing dataset from shared PVC at /mnt/shared Train Epoch: 1 [0/60000 (0%)] Loss: 2.307306 Train Epoch: 1 [1000/60000 (7%)] Loss: 1.529632 Train Epoch: 1 [2000/60000 (13%)] Loss: 1.767696 Train Epoch: 1 [3000/60000 (20%)] Loss: 1.238528 Train Epoch: 1 [4000/60000 (27%)] Loss: 0.868303 Train Epoch: 1 [5000/60000 (33%)] Loss: 0.647511 Train Epoch: 1 [6000/60000 (40%)] Loss: 0.729530 Train Epoch: 1 [7000/60000 (47%)] Loss: 0.629830 Train Epoch: 1 [8000/60000 (53%)] Loss: 0.433136 Train Epoch: 1 [9000/60000 (60%)] Loss: 0.580601 Train Epoch: 1 [10000/60000 (67%)] Loss: 0.394585 Train Epoch: 1 [11000/60000 (73%)] Loss: 0.522614 Train Epoch: 1 [12000/60000 (80%)] Loss: 0.479691 Train Epoch: 1 [13000/60000 (87%)] Loss: 0.531461 Train Epoch: 1 [14000/60000 (93%)] Loss: 0.434972 Train Epoch: 2 [0/60000 (0%)] Loss: 0.495159 Train Epoch: 2 [1000/60000 (7%)] Loss: 0.454348 Train Epoch: 2 [2000/60000 (13%)] Loss: 0.470757 Train Epoch: 2 [3000/60000 (20%)] Loss: 0.321238 Train Epoch: 2 [4000/60000 (27%)] Loss: 0.471672 Train Epoch: 2 [5000/60000 (33%)] Loss: 0.367303 Train Epoch: 2 [6000/60000 (40%)] Loss: 0.563059 Train Epoch: 2 [7000/60000 (47%)] Loss: 0.506511 Train Epoch: 2 [8000/60000 (53%)] Loss: 0.266898 Train Epoch: 2 [9000/60000 (60%)] Loss: 0.365605 Train Epoch: 2 [10000/60000 (67%)] Loss: 0.290397 Train Epoch: 2 [11000/60000 (73%)] Loss: 0.365026 Train Epoch: 2 [12000/60000 (80%)] Loss: 0.319503 Train Epoch: 2 [13000/60000 (87%)] Loss: 0.350930 Train Epoch: 2 [14000/60000 (93%)] Loss: 0.290739 Training is finished Ending Cell 8------------------------------------------ Executing Cell 9--------------------------------------- /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'a4065eeaf9336427bab0e61e0a91d253-2eee41dc4484d68d.elb.us-east-2.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( Ending Cell 9------------------------------------------ NOTEBOOK_STATUS: SUCCESS