LAST_SEEN            | FIRST_SEEN           | NAME                                                         | SUBOBJECT              | TYPE     | REASON            | MESSAGE                                                                                                                                          | 
0001-01-01 00:00:00  | 0001-01-01 00:00:00  | test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h.18e10d3f46a7b233  | -                      | Normal   | Scheduled         | Successfully assigned test-ns-rsbb2/test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h to ip-10-0-133-176.ec2.internal                                   | 
2026-10-23 04:11:32  | 2026-10-23 04:11:32  | test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h.18e10d3f68f9e013  | -                      | Normal   | AddedInterface    | Add eth0 [10.133.0.39/23] from ovn-kubernetes                                                                                                    | 
2026-10-23 04:11:44  | 2026-10-23 04:11:32  | test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h.18e10d3f6bcb3c61  | spec.containers{node}  | Normal   | Pulled            | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod                | 
2026-10-23 04:11:44  | 2026-10-23 04:11:32  | test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h.18e10d3f6ebf2b54  | spec.containers{node}  | Normal   | Created           | Container created                                                                                                                                | 
2026-10-23 04:11:44  | 2026-10-23 04:11:32  | test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h.18e10d3f6f2aac37  | spec.containers{node}  | Normal   | Started           | Container started                                                                                                                                | 
2026-10-23 04:11:35  | 2026-10-23 04:11:34  | test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h.18e10d3fd5db8b3e  | spec.containers{node}  | Warning  | BackOff           | Back-off restarting failed container node in pod test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h_test-ns-rsbb2(0aa55064-9599-4b25-b61e-4630347371d9)  | 
2026-10-23 04:11:32  | 2026-10-23 04:11:32  | test-mpi-trainjob-ppqcj-launcher-0.18e10d3f46403a8c          | -                      | Normal   | SuccessfulCreate  | Created pod: test-mpi-trainjob-ppqcj-launcher-0-0-2vv9h                                                                                          | 
2026-10-23 04:11:50  | 2026-10-23 04:11:50  | test-mpi-trainjob-ppqcj-launcher-0.18e10d439b95fffc          | -                      | Normal   | Completed         | Job completed                                                                                                                                    | 
0001-01-01 00:00:00  | 0001-01-01 00:00:00  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d3c3e0247b5      | -                      | Normal   | Scheduled         | Successfully assigned test-ns-rsbb2/test-mpi-trainjob-ppqcj-node-0-0-2dg9h to ip-10-0-139-48.ec2.internal                                        | 
2026-10-23 04:11:19  | 2026-10-23 04:11:19  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d3c603f3170      | -                      | Normal   | AddedInterface    | Add eth0 [10.132.0.49/23] from ovn-kubernetes                                                                                                    | 
2026-10-23 04:11:19  | 2026-10-23 04:11:19  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d3c63473514      | spec.containers{node}  | Normal   | Pulled            | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod                | 
2026-10-23 04:11:19  | 2026-10-23 04:11:19  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d3c65f631b6      | spec.containers{node}  | Normal   | Created           | Container created                                                                                                                                | 
2026-10-23 04:11:19  | 2026-10-23 04:11:19  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d3c666816cf      | spec.containers{node}  | Normal   | Started           | Container started                                                                                                                                | 
2026-10-23 04:11:50  | 2026-10-23 04:11:50  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d43a2ab36a8      | spec.containers{node}  | Normal   | Killing           | Stopping container node                                                                                                                          | 
2026-10-23 04:11:50  | 2026-10-23 04:11:50  | test-mpi-trainjob-ppqcj-node-0-0-2dg9h.18e10d43ab871fbb      | spec.containers{node}  | Warning  | Unhealthy         | Readiness probe failed: dial tcp 10.132.0.49:2222: connect: connection refused                                                                   | 
2026-10-23 04:11:19  | 2026-10-23 04:11:19  | test-mpi-trainjob-ppqcj-node-0.18e10d3c3d48bb55              | -                      | Normal   | SuccessfulCreate  | Created pod: test-mpi-trainjob-ppqcj-node-0-0-2dg9h                                                                                              | 
2026-10-23 04:11:50  | 2026-10-23 04:11:50  | test-mpi-trainjob-ppqcj.18e10d439eeaf572                     | -                      | Normal   | AllJobsCompleted  | jobset completed successfully                                                                                                                    |