LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-stn5s-node-0-0-jngkp.18e33d02e5f24c14 | - | Normal | Scheduled | Successfully assigned test-ns-crcw7/test-preemption-trainjob-stn5s-node-0-0-jngkp to ip-10-0-137-1.ec2.internal | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s-node-0-0-jngkp.18e33d0308bd419a | - | Normal | AddedInterface | Add eth0 [10.132.0.51/23] from ovn-kubernetes | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s-node-0-0-jngkp.18e33d030b488bfa | spec.containers{node} | Normal | Pulled | Container image "quay.io/rhoai/odh-th-torch-cuda-py312-rhel9@sha256:3333121b0725af5d23fca0df967cfb12fb6c51b57b3c10147bc90f1b2f034278" already present on machine and can be accessed by the pod | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s-node-0-0-jngkp.18e33d030e43e8d8 | spec.containers{node} | Normal | Created | Container created | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s-node-0-0-jngkp.18e33d030ec9c6f0 | spec.containers{node} | Normal | Started | Container started | 2026-10-30 07:09:22 | 2026-10-30 07:09:22 | test-preemption-trainjob-stn5s-node-0-0-jngkp.18e33d03b9a4d990 | spec.containers{node} | Normal | Killing | Stopping container node | 2026-10-30 07:09:22 | 2026-10-30 07:09:18 | test-preemption-trainjob-stn5s-node-0.18e33d02c8303f35 | - | Normal | Suspended | Job suspended | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s-node-0.18e33d02e5853777 | - | Normal | SuccessfulCreate | Created pod: test-preemption-trainjob-stn5s-node-0-0-jngkp | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s-node-0.18e33d02e5858c7f | - | Normal | Resumed | Job resumed | 2026-10-30 07:09:22 | 2026-10-30 07:09:22 | test-preemption-trainjob-stn5s-node-0.18e33d0389f4865f | - | Normal | SuccessfulDelete | Deleted pod: test-preemption-trainjob-stn5s-node-0-0-jngkp | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-stn5s.18e33d02c6cd65bd | - | Normal | CreatedWorkload | Created Workload: test-ns-crcw7/trainjob-test-preemption-trainjob-stn5s-c6a8c | 2026-10-30 07:09:22 | 2026-10-30 07:09:18 | test-preemption-trainjob-stn5s.18e33d02ca5fe1ca | - | Normal | SuspendedJobs | jobset is suspended | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-stn5s.18e33d02db79cb57 | - | Normal | Started | Admitted by clusterQueue cq-cdjkj | 2026-10-30 07:09:19 | 2026-10-30 07:09:19 | test-preemption-trainjob-stn5s.18e33d02e3605909 | - | Normal | ResumeJobs | jobset is resumed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-stn5s.18e33d03845887cd | - | Normal | Stopped | The workload is deactivated | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-preemption-trainjob-stn5s.18e33d0397f40868 | - | Warning | TrainJobResourcesCreationFailed | TrainJob resources reconciliation failed: JobSet.jobset.x-k8s.io "test-preemption-trainjob-stn5s" is invalid: [spec.replicatedJobs[0].template.spec.template.metadata: Invalid value: "null": spec.replicatedJobs[0].template.spec.template.metadata in body must be of type object: "null", : Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation] | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-stn5s-c6a8c.18e33d02c90c8141 | - | Normal | QuotaReserved | Quota reserved in ClusterQueue cq-cdjkj, wait time since queued was 1s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-stn5s-c6a8c.18e33d02c90d0d21 | - | Normal | Admitted | Admitted by ClusterQueue cq-cdjkj, wait time since reservation was 1s | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | trainjob-test-preemption-trainjob-stn5s-c6a8c.18e33d037de05518 | - | Normal | EvictedDueToDeactivated | The workload is deactivated |