LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-ct8vf-node-0-0-rhq6h.18e3645c75c7b485 | - | Normal | Scheduled | Successfully assigned test-ns-8vdb5/test-preemption-trainjob-ct8vf-node-0-0-rhq6h to ip-10-0-131-175.ec2.internal | 2026-10-30 19:10:25 | 2026-10-30 19:10:25 | test-preemption-trainjob-ct8vf-node-0-0-rhq6h.18e3645c9806438f | - | Normal | AddedInterface | Add eth0 [10.132.0.51/23] from ovn-kubernetes | 2026-10-30 19:10:25 | 2026-10-30 19:10:25 | test-preemption-trainjob-ct8vf-node-0-0-rhq6h.18e3645c9b4632aa | spec.containers{node} | Normal | Pulled | Container image "quay.io/opendatahub/odh-th-torch-cuda-py312@sha256:f96ad5f66d5523c6fe7b2e83c53d081ee367949ef60750705f8231aa86919343" already present on machine and can be accessed by the pod | 2026-10-30 19:10:25 | 2026-10-30 19:10:25 | test-preemption-trainjob-ct8vf-node-0-0-rhq6h.18e3645c9ea6af23 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 19:10:25 | 2026-10-30 19:10:25 | test-preemption-trainjob-ct8vf-node-0-0-rhq6h.18e3645c9f1a594c | spec.containers{node} | Normal | Started | Container started | 2026-10-30 19:10:28 | 2026-10-30 19:10:28 | test-preemption-trainjob-ct8vf-node-0-0-rhq6h.18e3645d49e11b93 | spec.containers{node} | Normal | Killing | Stopping container node | 2026-10-30 19:10:27 | 2026-10-30 19:10:24 | test-preemption-trainjob-ct8vf-node-0.18e3645c5b63edf9 | - | Normal | Suspended | Job suspended | 2026-10-30 19:10:24 | 2026-10-30 19:10:24 | test-preemption-trainjob-ct8vf-node-0.18e3645c74d629df | - | Normal | SuccessfulCreate | Created pod: test-preemption-trainjob-ct8vf-node-0-0-rhq6h | 2026-10-30 19:10:24 | 2026-10-30 19:10:24 | test-preemption-trainjob-ct8vf-node-0.18e3645c74d68560 | - | Normal | Resumed | Job resumed | 2026-10-30 19:10:27 | 2026-10-30 19:10:27 | test-preemption-trainjob-ct8vf-node-0.18e3645d1cba47bb | - | Normal | SuccessfulDelete | Deleted pod: test-preemption-trainjob-ct8vf-node-0-0-rhq6h | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-ct8vf.18e3645c59ee50e6 | - | Normal | CreatedWorkload | Created Workload: test-ns-8vdb5/trainjob-test-preemption-trainjob-ct8vf-b468f | 2026-10-30 19:10:27 | 2026-10-30 19:10:24 | test-preemption-trainjob-ct8vf.18e3645c5df75845 | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-ct8vf.18e3645c6d33c606 | - | Normal | Started | Admitted by clusterQueue cq-rmfk4 | 2026-10-30 19:10:24 | 2026-10-30 19:10:24 | test-preemption-trainjob-ct8vf.18e3645c72a26936 | - | Normal | ResumeJobs | jobset is resumed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-ct8vf.18e3645d16c374ca | - | Normal | Stopped | The workload is deactivated | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-ct8vf.18e3645d2b8ef36f | - | Warning | TrainJobResourcesCreationFailed | TrainJob resources reconciliation failed: JobSet.jobset.x-k8s.io "test-preemption-trainjob-ct8vf" is invalid: [spec.replicatedJobs[0].template.spec.template.metadata: Invalid value: "null": spec.replicatedJobs[0].template.spec.template.metadata in body must be of type object: "null", : Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation] | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-ct8vf-b468f.18e3645c5c85acce | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-rmfk4, wait time since queued was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-ct8vf-b468f.18e3645c5c862ca4 | - | Normal | Admitted | Admitted by ClusterQueue cq-rmfk4, wait time since reservation was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-ct8vf-b468f.18e3645d10c51f36 | - | Normal | EvictedDueToDeactivated | The workload is deactivated |