LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-pnbjw-node-0-0-9xwkx.18e353eaa564c394 | - | Normal | Scheduled | Successfully assigned test-ns-xw62p/test-preemption-trainjob-pnbjw-node-0-0-9xwkx to ip-10-0-132-248.ec2.internal | 2026-10-30 14:09:04 | 2026-10-30 14:09:04 | test-preemption-trainjob-pnbjw-node-0-0-9xwkx.18e353eac840450b | - | Normal | AddedInterface | Add eth0 [10.132.0.51/23] from ovn-kubernetes | 2026-10-30 14:09:04 | 2026-10-30 14:09:04 | test-preemption-trainjob-pnbjw-node-0-0-9xwkx.18e353eacaf2137f | spec.containers{node} | Normal | Pulled | Container image "quay.io/rhoai/odh-th-torch-cuda-py312-rhel9:on-pr-1327" already present on machine and can be accessed by the pod | 2026-10-30 14:09:04 | 2026-10-30 14:09:04 | test-preemption-trainjob-pnbjw-node-0-0-9xwkx.18e353eacde48011 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 14:09:04 | 2026-10-30 14:09:04 | test-preemption-trainjob-pnbjw-node-0-0-9xwkx.18e353eace640d4f | spec.containers{node} | Normal | Started | Container started | 2026-10-30 14:09:05 | 2026-10-30 14:09:03 | test-preemption-trainjob-pnbjw-node-0.18e353ea91c5f2a2 | - | Normal | Suspended | Job suspended | 2026-10-30 14:09:03 | 2026-10-30 14:09:03 | test-preemption-trainjob-pnbjw-node-0.18e353eaa4fef02b | - | Normal | SuccessfulCreate | Created pod: test-preemption-trainjob-pnbjw-node-0-0-9xwkx | 2026-10-30 14:09:03 | 2026-10-30 14:09:03 | test-preemption-trainjob-pnbjw-node-0.18e353eaa4ff6b41 | - | Normal | Resumed | Job resumed | 2026-10-30 14:09:05 | 2026-10-30 14:09:05 | test-preemption-trainjob-pnbjw-node-0.18e353eb1ce6b4fa | - | Normal | SuccessfulDelete | Deleted pod: test-preemption-trainjob-pnbjw-node-0-0-9xwkx | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-pnbjw.18e353ea9235ac4b | - | Normal | CreatedWorkload | Created Workload: test-ns-xw62p/trainjob-test-preemption-trainjob-pnbjw-68f4e | 2026-10-30 14:09:05 | 2026-10-30 14:09:03 | test-preemption-trainjob-pnbjw.18e353ea9379b262 | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-pnbjw.18e353ea9cada77c | - | Normal | Started | Admitted by clusterQueue cq-xmqrz | 2026-10-30 14:09:03 | 2026-10-30 14:09:03 | test-preemption-trainjob-pnbjw.18e353eaa31034a6 | - | Normal | ResumeJobs | jobset is resumed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-pnbjw.18e353eb176c6764 | - | Normal | Stopped | The workload is deactivated | 2026-10-30 14:09:06 | 2026-10-30 14:09:06 | test-preemption-trainjob-pnbjw.18e353eb31c62a1b | - | Warning | TrainJobResourcesCreationFailed | TrainJob resources reconciliation failed: JobSet.jobset.x-k8s.io "test-preemption-trainjob-pnbjw" is invalid: [spec.replicatedJobs[0].template.spec.template.metadata: Invalid value: "null": spec.replicatedJobs[0].template.spec.template.metadata in body must be of type object: "null", : Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation] | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-pnbjw-68f4e.18e353ea9483882f | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-xmqrz, wait time since queued was 1s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-pnbjw-68f4e.18e353ea94840f4c | - | Normal | Admitted | Admitted by ClusterQueue cq-xmqrz, wait time since reservation was 1s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-pnbjw-68f4e.18e353eb1138f212 | - | Normal | EvictedDueToDeactivated | The workload is deactivated |