LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-x4fmd-node-0-0-h96kr.18e10a3df5e8a42b | - | Normal | Scheduled | Successfully assigned test-ns-zdcmw/test-preemption-trainjob-x4fmd-node-0-0-h96kr to ip-10-0-129-58.ec2.internal | 2026-10-23 03:16:28 | 2026-10-23 03:16:28 | test-preemption-trainjob-x4fmd-node-0-0-h96kr.18e10a3e1878f905 | - | Normal | AddedInterface | Add eth0 [10.133.0.37/23] from ovn-kubernetes | 2026-10-23 03:16:28 | 2026-10-23 03:16:28 | test-preemption-trainjob-x4fmd-node-0-0-h96kr.18e10a3e1b9c3dad | spec.containers{node} | Normal | Pulling | Pulling image "quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable" | 2026-10-23 03:22:00 | 2026-10-23 03:22:00 | test-preemption-trainjob-x4fmd-node-0-0-h96kr.18e10a8b48dbfc7e | spec.containers{node} | Normal | Pulled | Successfully pulled image "quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable" in 5m31.471s (5m31.471s including waiting). Image size: 27111602746 bytes. | 2026-10-23 03:22:00 | 2026-10-23 03:22:00 | test-preemption-trainjob-x4fmd-node-0-0-h96kr.18e10a8b4dc63958 | spec.containers{node} | Normal | Created | Container created | 2026-10-23 03:22:00 | 2026-10-23 03:22:00 | test-preemption-trainjob-x4fmd-node-0-0-h96kr.18e10a8b4f1c0ba6 | spec.containers{node} | Normal | Started | Container started | 2026-10-23 03:22:01 | 2026-10-23 03:16:27 | test-preemption-trainjob-x4fmd-node-0.18e10a3ddc7ceaa6 | - | Normal | Suspended | Job suspended | 2026-10-23 03:16:27 | 2026-10-23 03:16:27 | test-preemption-trainjob-x4fmd-node-0.18e10a3df55c5b82 | - | Normal | SuccessfulCreate | Created pod: test-preemption-trainjob-x4fmd-node-0-0-h96kr | 2026-10-23 03:16:27 | 2026-10-23 03:16:27 | test-preemption-trainjob-x4fmd-node-0.18e10a3df55cd927 | - | Normal | Resumed | Job resumed | 2026-10-23 03:22:01 | 2026-10-23 03:22:01 | test-preemption-trainjob-x4fmd-node-0.18e10a8bbca0b8c9 | - | Normal | SuccessfulDelete | Deleted pod: test-preemption-trainjob-x4fmd-node-0-0-h96kr | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-x4fmd.18e10a3ddb617ebf | - | Normal | CreatedWorkload | Created Workload: test-ns-zdcmw/trainjob-test-preemption-trainjob-x4fmd-0171d | 2026-10-23 03:22:01 | 2026-10-23 03:16:27 | test-preemption-trainjob-x4fmd.18e10a3ddef6e03c | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-x4fmd.18e10a3dee0eed45 | - | Normal | Started | Admitted by clusterQueue cq-9lzcg | 2026-10-23 03:16:27 | 2026-10-23 03:16:27 | test-preemption-trainjob-x4fmd.18e10a3df37e2e38 | - | Normal | ResumeJobs | jobset is resumed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-x4fmd.18e10a8bb13d8444 | - | Normal | Stopped | The workload is deactivated | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-x4fmd.18e10a8bc9fb13ac | - | Warning | TrainJobResourcesCreationFailed | TrainJob resources reconciliation failed: JobSet.jobset.x-k8s.io "test-preemption-trainjob-x4fmd" is invalid: [spec.replicatedJobs[0].template.spec.template.metadata: Invalid value: "null": spec.replicatedJobs[0].template.spec.template.metadata in body must be of type object: "null", : Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation] | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-x4fmd-0171d.18e10a3ddd8066db | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-9lzcg, wait time since queued was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-x4fmd-0171d.18e10a3ddd8114b3 | - | Normal | Admitted | Admitted by ClusterQueue cq-9lzcg, wait time since reservation was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-x4fmd-0171d.18e10a8ba3714a27 | - | Normal | EvictedDueToDeactivated | The workload is deactivated |