LAST_SEEN | FIRST_SEEN | NAME | SUBOBJECT | TYPE | REASON | MESSAGE | 2026-11-03 18:23:26 | 2026-11-03 18:23:26 | pvc-zbj5h.18e49c1e96fcca6e | - | Normal | WaitForFirstConsumer | waiting for first consumer to be created before binding | 2026-11-03 18:23:26 | 2026-11-03 18:23:26 | pvc-zbj5h.18e49c1ea263e921 | - | Normal | ExternalProvisioning | Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. | 2026-11-03 18:23:26 | 2026-11-03 18:23:26 | pvc-zbj5h.18e49c1ea26c4c88 | - | Normal | Provisioning | External provisioner is provisioning volume for claim "test-ns-zzs58/pvc-zbj5h" | 2026-11-03 18:23:29 | 2026-11-03 18:23:29 | pvc-zbj5h.18e49c1f236481cd | - | Normal | ProvisioningSucceeded | Successfully provisioned volume pvc-37b70fbc-aa7e-4442-8745-aab7277bebf6 | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c1f553b3a2b | - | Normal | Scheduled | Successfully assigned test-ns-zzs58/test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr to ip-10-0-143-104.us-east-2.compute.internal | 2026-11-03 18:23:33 | 2026-11-03 18:23:33 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c202c297983 | - | Normal | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-37b70fbc-aa7e-4442-8745-aab7277bebf6" | 2026-11-03 18:23:35 | 2026-11-03 18:23:35 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c20958c7657 | - | Normal | AddedInterface | Add eth0 [10.134.0.27/23] from ovn-kubernetes | 2026-11-03 18:23:35 | 2026-11-03 18:23:35 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c2096eb2500 | spec.containers{dataset-initializer} | Normal | Pulling | Pulling image "quay.io/ksuta/alpaca-dataset@sha256:2e90f631180c7b2c916f9569b914b336b612e8ae86efad82546adc5c9fcbbb8d" | 2026-11-03 18:23:40 | 2026-11-03 18:23:40 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c21af18c34a | spec.containers{dataset-initializer} | Normal | Pulled | Successfully pulled image "quay.io/ksuta/alpaca-dataset@sha256:2e90f631180c7b2c916f9569b914b336b612e8ae86efad82546adc5c9fcbbb8d" in 4.7s (4.7s including waiting). Image size: 266627809 bytes. | 2026-11-03 18:23:40 | 2026-11-03 18:23:40 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c21b2342d20 | spec.containers{dataset-initializer} | Normal | Created | Container created | 2026-11-03 18:23:40 | 2026-11-03 18:23:40 | test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr.18e49c21b29460b8 | spec.containers{dataset-initializer} | Normal | Started | Container started | 2026-11-03 18:23:26 | 2026-11-03 18:23:26 | test-trainjob-vt9hl-dataset-initializer-0.18e49c1ea12b1ffd | - | Normal | SuccessfulCreate | Created pod: test-trainjob-vt9hl-dataset-initializer-0-0-7qlsr | 2026-11-03 18:23:47 | 2026-11-03 18:23:47 | test-trainjob-vt9hl-dataset-initializer-0.18e49c23701b0334 | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-trainjob-vt9hl-model-initializer-0-0-cb9fm.18e49c2373056e1d | - | Normal | Scheduled | Successfully assigned test-ns-zzs58/test-trainjob-vt9hl-model-initializer-0-0-cb9fm to ip-10-0-143-104.us-east-2.compute.internal | 2026-11-03 18:23:55 | 2026-11-03 18:23:55 | test-trainjob-vt9hl-model-initializer-0-0-cb9fm.18e49c2554b52896 | - | Normal | AddedInterface | Add eth0 [10.134.0.28/23] from ovn-kubernetes | 2026-11-03 18:23:55 | 2026-11-03 18:23:55 | test-trainjob-vt9hl-model-initializer-0-0-cb9fm.18e49c2556169044 | spec.containers{model-initializer} | Normal | Pulling | Pulling image "quay.io/ksuta/bloom-560m@sha256:f6db02bb7b5d09a8d698c04994d747bfb9e581bbb4c07d00290244d207623733" | 2026-11-03 18:24:47 | 2026-11-03 18:24:47 | test-trainjob-vt9hl-model-initializer-0-0-cb9fm.18e49c316e9001b1 | spec.containers{model-initializer} | Normal | Pulled | Successfully pulled image "quay.io/ksuta/bloom-560m@sha256:f6db02bb7b5d09a8d698c04994d747bfb9e581bbb4c07d00290244d207623733" in 51.95s (51.95s including waiting). Image size: 3589759710 bytes. | 2026-11-03 18:24:47 | 2026-11-03 18:24:47 | test-trainjob-vt9hl-model-initializer-0-0-cb9fm.18e49c3173b25847 | spec.containers{model-initializer} | Normal | Created | Container created | 2026-11-03 18:24:47 | 2026-11-03 18:24:47 | test-trainjob-vt9hl-model-initializer-0-0-cb9fm.18e49c3174e256cf | spec.containers{model-initializer} | Normal | Started | Container started | 2026-11-03 18:23:47 | 2026-11-03 18:23:47 | test-trainjob-vt9hl-model-initializer-0.18e49c23728cff1c | - | Normal | SuccessfulCreate | Created pod: test-trainjob-vt9hl-model-initializer-0-0-cb9fm | 2026-11-03 18:25:00 | 2026-11-03 18:25:00 | test-trainjob-vt9hl-model-initializer-0.18e49c3482e01215 | - | Normal | Completed | Job completed | 0001-01-01 00:00:00 | 0001-01-01 00:00:00 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c3485f69927 | - | Normal | Scheduled | Successfully assigned test-ns-zzs58/test-trainjob-vt9hl-node-0-0-b4g5q to ip-10-0-129-160.us-east-2.compute.internal | 2026-11-03 18:25:01 | 2026-11-03 18:25:01 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c348b97da54 | - | Warning | FailedAttachVolume | Waiting for detach for volume "pvc-37b70fbc-aa7e-4442-8745-aab7277bebf6" Volume is already exclusively attached to one node, waiting on detach before it can be attached to another node | 2026-11-03 18:25:13 | 2026-11-03 18:25:13 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c3785423449 | - | Normal | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-37b70fbc-aa7e-4442-8745-aab7277bebf6" | 2026-11-03 18:25:14 | 2026-11-03 18:25:14 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c37a97f6c42 | - | Normal | AddedInterface | Add eth0 [10.132.0.51/23] from ovn-kubernetes | 2026-11-03 18:25:14 | 2026-11-03 18:25:14 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c37aac556d9 | spec.containers{node} | Normal | Pulled | Container image "quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable" already present on machine and can be accessed by the pod | 2026-11-03 18:25:14 | 2026-11-03 18:25:14 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c37ad947cb4 | spec.containers{node} | Normal | Created | Container created | 2026-11-03 18:25:14 | 2026-11-03 18:25:14 | test-trainjob-vt9hl-node-0-0-b4g5q.18e49c37adf35889 | spec.containers{node} | Normal | Started | Container started | 2026-11-03 18:25:00 | 2026-11-03 18:25:00 | test-trainjob-vt9hl-node-0.18e49c3485572192 | - | Normal | SuccessfulCreate | Created pod: test-trainjob-vt9hl-node-0-0-b4g5q | 2026-11-03 18:25:16 | 2026-11-03 18:25:16 | test-trainjob-vt9hl-node-0.18e49c3838b83861 | - | Normal | Completed | Job completed | 2026-11-03 18:25:16 | 2026-11-03 18:25:16 | test-trainjob-vt9hl.18e49c383983e183 | - | Normal | AllJobsCompleted | jobset completed successfully |