LAST_SEEN            | FIRST_SEEN           | NAME                                                         | SUBOBJECT              | TYPE     | REASON            | MESSAGE                                                                                                                                          | 
0001-01-01 00:00:00  | 0001-01-01 00:00:00  | test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d.18e10a8f829465d0  | -                      | Normal   | Scheduled         | Successfully assigned test-ns-k5wpw/test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d to ip-10-0-140-146.ec2.internal                                   | 
2026-10-23 03:22:18  | 2026-10-23 03:22:18  | test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d.18e10a8fa4fb969d  | -                      | Normal   | AddedInterface    | Add eth0 [10.132.0.38/23] from ovn-kubernetes                                                                                                    | 
2026-10-23 03:22:30  | 2026-10-23 03:22:18  | test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d.18e10a8fa7c1da92  | spec.containers{node}  | Normal   | Pulled            | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod                | 
2026-10-23 03:22:30  | 2026-10-23 03:22:18  | test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d.18e10a8faaed4312  | spec.containers{node}  | Normal   | Created           | Container created                                                                                                                                | 
2026-10-23 03:22:30  | 2026-10-23 03:22:18  | test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d.18e10a8fab64c865  | spec.containers{node}  | Normal   | Started           | Container started                                                                                                                                | 
2026-10-23 03:22:21  | 2026-10-23 03:22:20  | test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d.18e10a8ffa2143a0  | spec.containers{node}  | Warning  | BackOff           | Back-off restarting failed container node in pod test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d_test-ns-k5wpw(ff65c288-5a00-4f82-8634-6647d8767dcd)  | 
2026-10-23 03:22:18  | 2026-10-23 03:22:18  | test-mpi-trainjob-r8fhq-launcher-0.18e10a8f822490e8          | -                      | Normal   | SuccessfulCreate  | Created pod: test-mpi-trainjob-r8fhq-launcher-0-0-k9f6d                                                                                          | 
2026-10-23 03:22:36  | 2026-10-23 03:22:36  | test-mpi-trainjob-r8fhq-launcher-0.18e10a93bcd9ad37          | -                      | Normal   | Completed         | Job completed                                                                                                                                    | 
0001-01-01 00:00:00  | 0001-01-01 00:00:00  | test-mpi-trainjob-r8fhq-node-0-0-l8cjq.18e10a8c5233d866      | -                      | Normal   | Scheduled         | Successfully assigned test-ns-k5wpw/test-mpi-trainjob-r8fhq-node-0-0-l8cjq to ip-10-0-140-14.ec2.internal                                        | 
2026-10-23 03:22:05  | 2026-10-23 03:22:05  | test-mpi-trainjob-r8fhq-node-0-0-l8cjq.18e10a8c748ff709      | -                      | Normal   | AddedInterface    | Add eth0 [10.134.0.42/23] from ovn-kubernetes                                                                                                    | 
2026-10-23 03:22:05  | 2026-10-23 03:22:05  | test-mpi-trainjob-r8fhq-node-0-0-l8cjq.18e10a8c775556e9      | spec.containers{node}  | Normal   | Pulled            | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod                | 
2026-10-23 03:22:05  | 2026-10-23 03:22:05  | test-mpi-trainjob-r8fhq-node-0-0-l8cjq.18e10a8c7a42e9ee      | spec.containers{node}  | Normal   | Created           | Container created                                                                                                                                | 
2026-10-23 03:22:05  | 2026-10-23 03:22:05  | test-mpi-trainjob-r8fhq-node-0-0-l8cjq.18e10a8c7b0fa039      | spec.containers{node}  | Normal   | Started           | Container started                                                                                                                                | 
2026-10-23 03:22:36  | 2026-10-23 03:22:36  | test-mpi-trainjob-r8fhq-node-0-0-l8cjq.18e10a93c45ccab9      | spec.containers{node}  | Normal   | Killing           | Stopping container node                                                                                                                          | 
2026-10-23 03:22:04  | 2026-10-23 03:22:04  | test-mpi-trainjob-r8fhq-node-0.18e10a8c51b01eb2              | -                      | Normal   | SuccessfulCreate  | Created pod: test-mpi-trainjob-r8fhq-node-0-0-l8cjq                                                                                              | 
2026-10-23 03:22:36  | 2026-10-23 03:22:36  | test-mpi-trainjob-r8fhq.18e10a93c06e2c38                     | -                      | Normal   | AllJobsCompleted  | jobset completed successfully                                                                                                                    |