LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-q4vbx-node-0-0-5k9bv.18e47540837286ad | - | Normal | Scheduled | Successfully assigned test-ns-rlhfp/test-preemption-trainjob-q4vbx-node-0-0-5k9bv to ip-10-0-134-19.ec2.internal | 2026-11-03 06:31:12 | 2026-11-03 06:31:12 | test-preemption-trainjob-q4vbx-node-0-0-5k9bv.18e47540a5fddcc0 | - | Normal | AddedInterface | Add eth0 [10.132.0.36/23] from ovn-kubernetes | 2026-11-03 06:31:12 | 2026-11-03 06:31:12 | test-preemption-trainjob-q4vbx-node-0-0-5k9bv.18e47540a8bedc45 | spec.containers{node} | Normal | Pulling | Pulling image "quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable" | 2026-11-03 06:35:28 | 2026-11-03 06:35:28 | test-preemption-trainjob-q4vbx-node-0-0-5k9bv.18e4757c4feed57f | spec.containers{node} | Normal | Pulled | Successfully pulled image "quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable" in 4m16.207s (4m16.207s including waiting). Image size: 24712531635 bytes. | 2026-11-03 06:35:28 | 2026-11-03 06:35:28 | test-preemption-trainjob-q4vbx-node-0-0-5k9bv.18e4757c54d6c92c | spec.containers{node} | Normal | Created | Container created | 2026-11-03 06:35:28 | 2026-11-03 06:35:28 | test-preemption-trainjob-q4vbx-node-0-0-5k9bv.18e4757c55e88e41 | spec.containers{node} | Normal | Started | Container started | 2026-11-03 06:35:29 | 2026-11-03 06:31:11 | test-preemption-trainjob-q4vbx-node-0.18e4754069886d58 | - | Normal | Suspended | Job suspended | 2026-11-03 06:31:11 | 2026-11-03 06:31:11 | test-preemption-trainjob-q4vbx-node-0.18e4754083003e42 | - | Normal | SuccessfulCreate | Created pod: test-preemption-trainjob-q4vbx-node-0-0-5k9bv | 2026-11-03 06:31:11 | 2026-11-03 06:31:11 | test-preemption-trainjob-q4vbx-node-0.18e475408300918f | - | Normal | Resumed | Job resumed | 2026-11-03 06:35:29 | 2026-11-03 06:35:29 | test-preemption-trainjob-q4vbx-node-0.18e4757c9189bf75 | - | Normal | SuccessfulDelete | Deleted pod: test-preemption-trainjob-q4vbx-node-0-0-5k9bv | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-q4vbx.18e47540688c50de | - | Normal | CreatedWorkload | Created Workload: test-ns-rlhfp/trainjob-test-preemption-trainjob-q4vbx-4d43a | 2026-11-03 06:35:29 | 2026-11-03 06:31:11 | test-preemption-trainjob-q4vbx.18e475406b3c3356 | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-q4vbx.18e475407b875f88 | - | Normal | Started | Admitted by clusterQueue cq-tnxzs | 2026-11-03 06:31:11 | 2026-11-03 06:31:11 | test-preemption-trainjob-q4vbx.18e4754080e89cd8 | - | Normal | ResumeJobs | jobset is resumed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-q4vbx.18e4757c85dc7f7d | - | Normal | Stopped | The workload is deactivated | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-q4vbx.18e4757ca0ba18f0 | - | Warning | TrainJobResourcesCreationFailed | TrainJob resources reconciliation failed: JobSet.jobset.x-k8s.io "test-preemption-trainjob-q4vbx" is invalid: [spec.replicatedJobs[0].template.spec.template.metadata: Invalid value: "null": spec.replicatedJobs[0].template.spec.template.metadata in body must be of type object: "null", : Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation] | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-q4vbx-4d43a.18e475406ac739db | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-tnxzs, wait time since queued was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-q4vbx-4d43a.18e475406ac7c458 | - | Normal | Admitted | Admitted by ClusterQueue cq-tnxzs, wait time since reservation was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-q4vbx-4d43a.18e4757c785c70e6 | - | Normal | EvictedDueToDeactivated | The workload is deactivated |