→ 6/27 実際に頓挫気味なので､本格始動中
以下のscriptで一応は変換ができた｡
ppやtp (= mp)は学習用に変更しておく必要がある
hf_to_megatron_llama3-8b_0627.sh
といあえず動いた｡lossもきちんと下がった｡
[2024-06-27 02:15:12] iteration 8/ 12500 | consumed samples: 12288 | elapsed time per iteration (ms): 68381.5 | throughput per GPU (TFLOP/s/GPU): 268.5 | iteration time: 68.382 s samples/sec: 22.5 | learning rate: 8.000000E-07 | global batch size: 1536 | lm loss: 2.549062E+00 | loss scale: 1.0 | grad norm: 6.729 | number of skipped iterations: 0 | number of nan iterations: 0 | [2024-06-27 02:16:18] iteration 9/ 12500 | consumed samples: 13824 | elapsed time per iteration (ms): 65940.6 | throughput per GPU (TFLOP/s/GPU): 278.5 | iteration time: 65.941 s samples/sec: 23.3 | learning rate: 9.000000E-07 | global batch size: 1536 | lm loss: 2.531539E+00 | loss scale: 1.0 | grad norm: 7.008 | number of skipped iterations: 0 | number of nan iterations: 0 | [2024-06-27 02:17:25] iteration 10/ 12500 | consumed samples: 15360 | elapsed time per iteration (ms): 67065.4 | throughput per GPU (TFLOP/s/GPU): 273.8 | iteration time: 67.065 s samples/sec: 22.9 | learning rate: 1.000000E-06 | global batch size: 1536 | lm loss: 2.526659E+00 | loss scale: 1.0 | grad norm: 6.969 | number of skipped iterations: 0 | number of nan iterations: 0 |
1 nodeでtp=2, mp=1にしたら､530 TFLOPSとなった｡30日で230 B tokenを学習できる｡
2nodeで460程度まで上がれば､good
動いた