LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 2026-10-30 07:51:56 | 2026-10-30 07:51:56 | pvc-g5lwt.18e33f564497e668 | - | Normal | ExternalProvisioning | Waiting for a volume to be created either by the external provisioner 'example.com/nfs' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. | 2026-10-30 07:51:56 | 2026-10-30 07:51:56 | pvc-g5lwt.18e33f5646294f19 | - | Normal | Provisioning | External provisioner is provisioning volume for claim "test-ns-mjc7h/pvc-g5lwt" | 2026-10-30 07:51:56 | 2026-10-30 07:51:56 | pvc-g5lwt.18e33f56499ea286 | - | Normal | ProvisioningSucceeded | Successfully provisioned volume pvc-badee086-6c1f-4fe7-869d-9a7ed8f1b1e9 | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz.18e33f569b4c24df | - | Normal | Scheduled | Successfully assigned test-ns-mjc7h/test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz to ip-10-0-137-1.ec2.internal | 2026-10-30 07:51:58 | 2026-10-30 07:51:58 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz.18e33f56bd9505e9 | - | Normal | AddedInterface | Add eth0 [10.132.0.65/23] from ovn-kubernetes | 2026-10-30 07:51:58 | 2026-10-30 07:51:58 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz.18e33f56c110cd23 | spec.containers{dataset-initializer} | Normal | Pulled | Container image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:8e3f9821cae060ad0c77776db829e93c999362caff3e1b0279c2ae386a59ba31" already present on machine and can be accessed by the pod | 2026-10-30 07:51:58 | 2026-10-30 07:51:58 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz.18e33f56c3cadd58 | spec.containers{dataset-initializer} | Normal | Created | Container created | 2026-10-30 07:51:58 | 2026-10-30 07:51:58 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz.18e33f56c43b1c9a | spec.containers{dataset-initializer} | Normal | Started | Container started | 2026-10-30 07:51:57 | 2026-10-30 07:51:57 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0.18e33f5675f230ef | - | Normal | Suspended | Job suspended | 2026-10-30 07:51:57 | 2026-10-30 07:51:57 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0.18e33f569ade1e55 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-kgktm-dataset-initializer-0-0-tkjwz | 2026-10-30 07:51:57 | 2026-10-30 07:51:57 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0.18e33f569ade78b0 | - | Normal | Resumed | Job resumed | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-dataset-initializer-0.18e33f5cf85717fd | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5.18e33f5cfe6896e4 | - | Normal | Scheduled | Successfully assigned test-ns-mjc7h/test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5 to ip-10-0-137-1.ec2.internal | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5.18e33f5d224cc431 | - | Normal | AddedInterface | Add eth0 [10.132.0.66/23] from ovn-kubernetes | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5.18e33f5d25e03f86 | spec.containers{node} | Normal | Pulled | Container image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:8e3f9821cae060ad0c77776db829e93c999362caff3e1b0279c2ae386a59ba31" already present on machine and can be accessed by the pod | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5.18e33f5d28b54019 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5.18e33f5d292daa6b | spec.containers{node} | Normal | Started | Container started | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z.18e33f5d024ad539 | - | Normal | Scheduled | Successfully assigned test-ns-mjc7h/test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z to ip-10-0-128-38.ec2.internal | 2026-10-30 07:52:26 | 2026-10-30 07:52:26 | test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z.18e33f5d48991c4d | - | Normal | AddedInterface | Add eth0 [10.133.0.30/23] from ovn-kubernetes | 2026-10-30 07:52:26 | 2026-10-30 07:52:26 | test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z.18e33f5d4c620757 | spec.containers{node} | Normal | Pulling | Pulling image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:8e3f9821cae060ad0c77776db829e93c999362caff3e1b0279c2ae386a59ba31" | 2026-10-30 07:53:34 | 2026-10-30 07:53:34 | test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z.18e33f6d38a343d3 | spec.containers{node} | Normal | Pulled | Successfully pulled image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:8e3f9821cae060ad0c77776db829e93c999362caff3e1b0279c2ae386a59ba31" in 1m8.388s (1m8.388s including waiting). Image size: 8468893571 bytes. | 2026-10-30 07:53:35 | 2026-10-30 07:53:35 | test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z.18e33f6d3dd283b4 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 07:53:35 | 2026-10-30 07:53:35 | test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z.18e33f6d409f1fe6 | spec.containers{node} | Normal | Started | Container started | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-node-0.18e33f5cfdfca989 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-kgktm-node-0-0-ssmp5 | 2026-10-30 07:52:25 | 2026-10-30 07:52:25 | test-fashion-mnist-trainjob-kgktm-node-0.18e33f5d01be6426 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-kgktm-node-0-1-xg65z | 2026-10-30 07:54:14 | 2026-10-30 07:54:14 | test-fashion-mnist-trainjob-kgktm-node-0.18e33f7661523a00 | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-kgktm.18e33f5672a076a3 | - | Normal | CreatedWorkload | Created Workload: test-ns-mjc7h/trainjob-test-fashion-mnist-trainjob-kgktm-b46ca | 2026-10-30 07:51:57 | 2026-10-30 07:51:57 | test-fashion-mnist-trainjob-kgktm.18e33f5677c5d038 | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-kgktm.18e33f568a6acfce | - | Normal | Started | Admitted by clusterQueue cq-9r8lr | 2026-10-30 07:51:57 | 2026-10-30 07:51:57 | test-fashion-mnist-trainjob-kgktm.18e33f5690cb7954 | - | Normal | ResumeJobs | jobset is resumed | 2026-10-30 07:54:14 | 2026-10-30 07:54:14 | test-fashion-mnist-trainjob-kgktm.18e33f766340dc0b | - | Normal | AllJobsCompleted | jobset completed successfully | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-kgktm.18e33f766df82036 | - | Normal | FinishedWorkload | Workload 'test-ns-mjc7h/trainjob-test-fashion-mnist-trainjob-kgktm-b46ca' is declared finished | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-fashion-mnist-trainjob-kgktm-b46ca.18e33f567581ed1d | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-9r8lr, wait time since queued was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-fashion-mnist-trainjob-kgktm-b46ca.18e33f5675829693 | - | Normal | Admitted | Admitted by ClusterQueue cq-9r8lr, wait time since reservation was 0s |