LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 2026-10-30 14:45:58 | 2026-10-30 14:45:58 | pvc-ftrbw.18e355ee33cf1203 | - | Normal | ExternalProvisioning | Waiting for a volume to be created either by the external provisioner 'example.com/nfs' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. | 2026-10-30 14:45:58 | 2026-10-30 14:45:58 | pvc-ftrbw.18e355ee34e4ed52 | - | Normal | Provisioning | External provisioner is provisioning volume for claim "test-ns-27d9w/pvc-ftrbw" | 2026-10-30 14:45:58 | 2026-10-30 14:45:58 | pvc-ftrbw.18e355ee37c64064 | - | Normal | ProvisioningSucceeded | Successfully provisioned volume pvc-803e0de5-4b4a-43e1-b12a-634287e80a87 | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc.18e355ee7df5eb21 | - | Normal | Scheduled | Successfully assigned test-ns-27d9w/test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc to ip-10-0-132-248.ec2.internal | 2026-10-30 14:46:00 | 2026-10-30 14:46:00 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc.18e355eea080a7af | - | Normal | AddedInterface | Add eth0 [10.132.0.63/23] from ovn-kubernetes | 2026-10-30 14:46:00 | 2026-10-30 14:46:00 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc.18e355eea3ad43b7 | spec.containers{dataset-initializer} | Normal | Pulled | Container image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1327" already present on machine and can be accessed by the pod | 2026-10-30 14:46:00 | 2026-10-30 14:46:00 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc.18e355eea634ab2b | spec.containers{dataset-initializer} | Normal | Created | Container created | 2026-10-30 14:46:00 | 2026-10-30 14:46:00 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc.18e355eea6aa599a | spec.containers{dataset-initializer} | Normal | Started | Container started | 2026-10-30 14:45:58 | 2026-10-30 14:45:58 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0.18e355ee5a89a1af | - | Normal | Suspended | Job suspended | 2026-10-30 14:45:59 | 2026-10-30 14:45:59 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0.18e355ee7d91816d | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0-0-gm5kc | 2026-10-30 14:45:59 | 2026-10-30 14:45:59 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0.18e355ee7d92481a | - | Normal | Resumed | Job resumed | 2026-10-30 14:46:26 | 2026-10-30 14:46:26 | test-fashion-mnist-trainjob-4ptdq-dataset-initializer-0.18e355f4e1ed0c49 | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt.18e355f4e7e8a419 | - | Normal | Scheduled | Successfully assigned test-ns-27d9w/test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt to ip-10-0-132-248.ec2.internal | 2026-10-30 14:46:27 | 2026-10-30 14:46:27 | test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt.18e355f50a68493d | - | Normal | AddedInterface | Add eth0 [10.132.0.64/23] from ovn-kubernetes | 2026-10-30 14:46:27 | 2026-10-30 14:46:27 | test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt.18e355f50d7bd8e0 | spec.containers{node} | Normal | Pulled | Container image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1327" already present on machine and can be accessed by the pod | 2026-10-30 14:46:27 | 2026-10-30 14:46:27 | test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt.18e355f510d2def4 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 14:46:27 | 2026-10-30 14:46:27 | test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt.18e355f5114f4009 | spec.containers{node} | Normal | Started | Container started | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4.18e355f4eb978422 | - | Normal | Scheduled | Successfully assigned test-ns-27d9w/test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4 to ip-10-0-132-161.ec2.internal | 2026-10-30 14:46:28 | 2026-10-30 14:46:28 | test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4.18e355f532418b1d | - | Normal | AddedInterface | Add eth0 [10.134.0.32/23] from ovn-kubernetes | 2026-10-30 14:46:28 | 2026-10-30 14:46:28 | test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4.18e355f53644ed53 | spec.containers{node} | Normal | Pulling | Pulling image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1327" | 2026-10-30 14:47:32 | 2026-10-30 14:47:32 | test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4.18e356043257b90e | spec.containers{node} | Normal | Pulled | Successfully pulled image "quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1327" in 1m4.358s (1m4.358s including waiting). Image size: 7740913465 bytes. | 2026-10-30 14:47:32 | 2026-10-30 14:47:32 | test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4.18e35604393ec157 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 14:47:32 | 2026-10-30 14:47:32 | test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4.18e356043a5ee23d | spec.containers{node} | Normal | Started | Container started | 2026-10-30 14:46:26 | 2026-10-30 14:46:26 | test-fashion-mnist-trainjob-4ptdq-node-0.18e355f4e781fbb8 | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-4ptdq-node-0-0-t8lvt | 2026-10-30 14:46:27 | 2026-10-30 14:46:27 | test-fashion-mnist-trainjob-4ptdq-node-0.18e355f4eb17500f | - | Normal | SuccessfulCreate | Created pod: test-fashion-mnist-trainjob-4ptdq-node-0-1-dfpw4 | 2026-10-30 14:48:14 | 2026-10-30 14:48:14 | test-fashion-mnist-trainjob-4ptdq-node-0.18e3560ddbc4f69c | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4ptdq.18e355ee5a67752a | - | Normal | CreatedWorkload | Created Workload: test-ns-27d9w/trainjob-test-fashion-mnist-trainjob-4ptdq-b20aa | 2026-10-30 14:45:58 | 2026-10-30 14:45:58 | test-fashion-mnist-trainjob-4ptdq.18e355ee5cc79123 | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4ptdq.18e355ee6d1c773e | - | Normal | Started | Admitted by clusterQueue cq-kclrv | 2026-10-30 14:45:59 | 2026-10-30 14:45:59 | test-fashion-mnist-trainjob-4ptdq.18e355ee758276f2 | - | Normal | ResumeJobs | jobset is resumed | 2026-10-30 14:48:14 | 2026-10-30 14:48:14 | test-fashion-mnist-trainjob-4ptdq.18e3560ddd783906 | - | Normal | AllJobsCompleted | jobset completed successfully | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-fashion-mnist-trainjob-4ptdq.18e3560de67bbf5e | - | Normal | FinishedWorkload | Workload 'test-ns-27d9w/trainjob-test-fashion-mnist-trainjob-4ptdq-b20aa' is declared finished | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-fashion-mnist-trainjob-4ptdq-b20aa.18e355ee5d1a56dc | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-kclrv, wait time since queued was 1s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-fashion-mnist-trainjob-4ptdq-b20aa.18e355ee5d1b243b | - | Normal | Admitted | Admitted by ClusterQueue cq-kclrv, wait time since reservation was 1s |