LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-crfrs-node-0-0-9wlbj.18e46abf18b98726 | - | Normal | Scheduled | Successfully assigned test-ns-2j9c4/test-preemption-trainjob-crfrs-node-0-0-9wlbj to ip-10-0-135-211.us-east-2.compute.internal | 2026-11-03 03:18:41 | 2026-11-03 03:18:41 | test-preemption-trainjob-crfrs-node-0-0-9wlbj.18e46abf3b2bdafe | - | Normal | AddedInterface | Add eth0 [10.132.0.52/23] from ovn-kubernetes | 2026-11-03 03:18:41 | 2026-11-03 03:18:41 | test-preemption-trainjob-crfrs-node-0-0-9wlbj.18e46abf3c7ba174 | spec.containers{node} | Normal | Pulled | Container image "quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable" already present on machine and can be accessed by the pod | 2026-11-03 03:18:41 | 2026-11-03 03:18:41 | test-preemption-trainjob-crfrs-node-0-0-9wlbj.18e46abf4014f827 | spec.containers{node} | Normal | Created | Container created | 2026-11-03 03:18:41 | 2026-11-03 03:18:41 | test-preemption-trainjob-crfrs-node-0-0-9wlbj.18e46abf408a9097 | spec.containers{node} | Normal | Started | Container started | 2026-11-03 03:18:41 | 2026-11-03 03:18:40 | test-preemption-trainjob-crfrs-node-0.18e46abf0dc05d3b | - | Normal | Suspended | Job suspended | 2026-11-03 03:18:40 | 2026-11-03 03:18:40 | test-preemption-trainjob-crfrs-node-0.18e46abf184e7e48 | - | Normal | SuccessfulCreate | Created pod: test-preemption-trainjob-crfrs-node-0-0-9wlbj | 2026-11-03 03:18:40 | 2026-11-03 03:18:40 | test-preemption-trainjob-crfrs-node-0.18e46abf184ee400 | - | Normal | Resumed | Job resumed | 2026-11-03 03:18:41 | 2026-11-03 03:18:41 | test-preemption-trainjob-crfrs-node-0.18e46abf5676798b | - | Normal | SuccessfulDelete | Deleted pod: test-preemption-trainjob-crfrs-node-0-0-9wlbj | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-crfrs.18e46abf0c3d1b69 | - | Normal | CreatedWorkload | Created Workload: test-ns-2j9c4/trainjob-test-preemption-trainjob-crfrs-98238 | 2026-11-03 03:18:41 | 2026-11-03 03:18:40 | test-preemption-trainjob-crfrs.18e46abf0e7545ae | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-crfrs.18e46abf14d60957 | - | Normal | Started | Admitted by clusterQueue cq-mdcv4 | 2026-11-03 03:18:40 | 2026-11-03 03:18:40 | test-preemption-trainjob-crfrs.18e46abf17117cfb | - | Normal | ResumeJobs | jobset is resumed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-crfrs.18e46abf53929d6a | - | Normal | Stopped | The workload is deactivated | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-crfrs.18e46abf5beace62 | - | Warning | TrainJobResourcesCreationFailed | TrainJob resources reconciliation failed: JobSet.jobset.x-k8s.io "test-preemption-trainjob-crfrs" is invalid: [spec.replicatedJobs[0].template.spec.template.metadata: Invalid value: "null": spec.replicatedJobs[0].template.spec.template.metadata in body must be of type object: "null", : Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation] | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-crfrs-98238.18e46abf0cf8643f | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-mdcv4, wait time since queued was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-crfrs-98238.18e46abf0cf8c44a | - | Normal | Admitted | Admitted by ClusterQueue cq-mdcv4, wait time since reservation was 0s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-crfrs-98238.18e46abf51d5bdb5 | - | Normal | EvictedDueToDeactivated | The workload is deactivated |