LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 2026-10-23 04:46:09 | 2026-10-23 04:46:09 | pvc-crwkj.18e10f23058dfb5a | - | Normal | ExternalProvisioning | Waiting for a volume to be created either by the external provisioner 'example.com/nfs' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. | 2026-10-23 04:46:09 | 2026-10-23 04:46:09 | pvc-crwkj.18e10f2306e93c99 | - | Normal | Provisioning | External provisioner is provisioning volume for claim "test-ns-bnmwd/pvc-crwkj" | 2026-10-23 04:46:09 | 2026-10-23 04:46:09 | pvc-crwkj.18e10f230a0af528 | - | Normal | ProvisioningSucceeded | Successfully provisioned volume pvc-350998b6-a1ec-4ec9-87db-2167eb901630 | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg.18e10f236220f126 | - | Normal | Scheduled | Successfully assigned test-ns-bnmwd/test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg to ip-10-0-139-48.ec2.internal | 2026-10-23 04:46:11 | 2026-10-23 04:46:11 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg.18e10f2384b961d6 | - | Normal | AddedInterface | Add eth0 [10.132.0.57/23] from ovn-kubernetes | 2026-10-23 04:46:11 | 2026-10-23 04:46:11 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg.18e10f23882453ec | spec.containers{dataset-initializer} | Normal | Pulled | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod | 2026-10-23 04:46:12 | 2026-10-23 04:46:12 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg.18e10f238b4d21ed | spec.containers{dataset-initializer} | Normal | Created | Container created | 2026-10-23 04:46:12 | 2026-10-23 04:46:12 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg.18e10f238c12324e | spec.containers{dataset-initializer} | Normal | Started | Container started | 2026-10-23 04:46:10 | 2026-10-23 04:46:10 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0.18e10f232fdff178 | - | Normal | Suspended | Job suspended | 2026-10-23 04:46:11 | 2026-10-23 04:46:11 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0.18e10f2361c1d285 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0-0-77xfg | 2026-10-23 04:46:11 | 2026-10-23 04:46:11 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0.18e10f2361c22800 | - | Normal | Resumed | Job resumed | 2026-10-23 04:46:36 | 2026-10-23 04:46:36 | test-fashion-mnist-trainjob-4gqmq-dataset-initializer-0.18e10f294ce3940d | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt.18e10f29543f828c | - | Normal | Scheduled | Successfully assigned test-ns-bnmwd/test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt to ip-10-0-139-48.ec2.internal | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt.18e10f2976a86f17 | - | Normal | AddedInterface | Add eth0 [10.132.0.58/23] from ovn-kubernetes | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt.18e10f297a409915 | spec.containers{node} | Normal | Pulled | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt.18e10f297d49029b | spec.containers{node} | Normal | Created | Container created | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt.18e10f297dc42746 | spec.containers{node} | Normal | Started | Container started | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k.18e10f295766fee9 | - | Normal | Scheduled | Successfully assigned test-ns-bnmwd/test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k to ip-10-0-133-176.ec2.internal | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k.18e10f298bbb2028 | - | Normal | AddedInterface | Add eth0 [10.133.0.45/23] from ovn-kubernetes | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k.18e10f298f4fd820 | spec.containers{node} | Normal | Pulled | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k.18e10f2992521998 | spec.containers{node} | Normal | Created | Container created | 2026-10-23 04:46:37 | 2026-10-23 04:46:37 | test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k.18e10f2992c88e91 | spec.containers{node} | Normal | Started | Container started | 2026-10-23 04:46:36 | 2026-10-23 04:46:36 | test-fashion-mnist-trainjob-4gqmq-node-0.18e10f2952a41100 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-4gqmq-node-0-0-gqdwt | 2026-10-23 04:46:36 | 2026-10-23 04:46:36 | test-fashion-mnist-trainjob-4gqmq-node-0.18e10f2957030547 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-4gqmq-node-0-1-bc87k | 2026-10-23 04:47:21 | 2026-10-23 04:47:21 | test-fashion-mnist-trainjob-4gqmq-node-0.18e10f33d627e310 | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4gqmq.18e10f232eb8aadc | - | Normal | CreatedWorkload | Created Workload: test-ns-bnmwd/trainjob-test-fashion-mnist-trainjob-4gqmq-bae23 | 2026-10-23 04:46:10 | 2026-10-23 04:46:10 | test-fashion-mnist-trainjob-4gqmq.18e10f23328f98f8 | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4gqmq.18e10f2348194b80 | - | Normal | Started | Admitted by clusterQueue cq-tbmg8 | 2026-10-23 04:46:10 | 2026-10-23 04:46:10 | test-fashion-mnist-trainjob-4gqmq.18e10f234df99296 | - | Normal | ResumeJobs | jobset is resumed | 2026-10-23 04:47:22 | 2026-10-23 04:47:22 | test-fashion-mnist-trainjob-4gqmq.18e10f33d840fc73 | - | Normal | AllJobsCompleted | jobset completed successfully | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4gqmq.18e10f33e154341d | - | Normal | FinishedWorkload | Workload 'test-ns-bnmwd/trainjob-test-fashion-mnist-trainjob-4gqmq-bae23' is declared finished | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-fashion-mnist-trainjob-4gqmq-bae23.18e10f2331441d81 | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-tbmg8, wait time since queued was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-fashion-mnist-trainjob-4gqmq-bae23.18e10f233144d548 | - | Normal | Admitted | Admitted by ClusterQueue cq-tbmg8, wait time since reservation was 0s |