Star207
313
代码
代码
Issues71
Pull Requests91
流水线
项目讨论
Wiki
分析
Star207
313
已合并
Qwen2.5-Math系列模型适配 #1913
gcw_yyB1VUP3创建于 2024年11月19日
Qwen2.5-Math系列模型适配 #1913
已合并
gcw_yyB1VUP3创建于 2024年11月19日
从refs/pull/1913/head合入到master
共 13 个文件变更+777-1
Mdocs/models/evaluation.md+44-1
@@ -49,6 +49,49 @@ MindSpeed-LLM 已支持的大模型评估数据统计如下：
49| InternLM2.5-1.8b | MMLU | 51.3% | [53.5%](https://huggingface.co/internlm/internlm2_5-1_8b) | InternLM2.5-7B | MMLU | 69.7% | [71.6%](https://huggingface.co/internlm/internlm2_5-7b) | 49| InternLM2.5-1.8b | MMLU | 51.3% | [53.5%](https://huggingface.co/internlm/internlm2_5-1_8b) | InternLM2.5-7B | MMLU | 69.7% | [71.6%](https://huggingface.co/internlm/internlm2_5-7b) |
50| InternLM2.5-20b | MMLU | 73.3% | [74.2%](https://huggingface.co/internlm/internlm2_5-20b) | Yi1.5-6B | MMLU | 63.2% | [63.5%](https://huggingface.co/01-ai/Yi-1.5-6B/tree/main) |50| InternLM2.5-20b | MMLU | 73.3% | [74.2%](https://huggingface.co/internlm/internlm2_5-20b) | Yi1.5-6B | MMLU | 63.2% | [63.5%](https://huggingface.co/01-ai/Yi-1.5-6B/tree/main) |
51| Yi1.5-9B | MMLU | 69.2% | [69.5%](https://huggingface.co/01-ai/Yi-1.5-9B/tree/main) | Yi1.5-34B | MMLU | 76.9% | [77.1%](https://huggingface.co/01-ai/Yi-1.5-34B/tree/main) | 51| Yi1.5-9B | MMLU | 69.2% | [69.5%](https://huggingface.co/01-ai/Yi-1.5-9B/tree/main) | Yi1.5-34B | MMLU | 76.9% | [77.1%](https://huggingface.co/01-ai/Yi-1.5-34B/tree/main) |
52-| CodeQWen2.5-7B | Human. | 66.5% | [61.6%](https://modelscope.cn/models/Qwen/Qwen2.5-Coder-7B) | 52+| CodeQWen2.5-7B | Human. | 66.5% | [61.6%](https://modelscope.cn/models/Qwen/Qwen2.5-Coder-7B) | Qwen2.5-Math-7B |MMLU-STEM| 65.1% | [67.8%](https://github.com/QwenLM/Qwen2.5-Math/tree/main/) |
53+| Qwen2.5-Math-72B |MMLU-STEM| 83.7% | [82.8%](https://github.com/QwenLM/Qwen2.5-Math/tree/main/) |
53 54 
54具体的权重转换功能命令介绍见 [MindSpeed-LLM 大模型使用指南分布式评估](../USER_GUIDE.md/#大模型分布式评估)章节.55具体的权重转换功能命令介绍见 [MindSpeed-LLM 大模型使用指南分布式评估](../USER_GUIDE.md/#大模型分布式评估)章节.
56+ 
57+ 
58+## 社区BUG列表
59+ 
60+1.Qwen2.5-Math:评估集使用MMLU-STEM评估集，MMLU-STEM为MMLU评估集的子集，可以在下载MMLU评估集后自行删除无关学科，并将评估脚本`TASK`配置设置为**mmlu**进行评估任务。
61+按照Qwen2.5-Math官方评估代码中的定义，MMLU-STEM的学科范围如下:
62+ 
63+> abstract_algebra(抽象代数)
64+>
65+> astronomy(天文学)
66+>
67+> college_biology(大学生物学)
68+>
69+> college_chemistry(大学化学)
70+>
71+> college_computer_science(大学计算机科学)
72+>
73+> college_mathematics(大学数学)
74+>
75+> college_physics(大学物理学)
76+>
77+> computer_security(计算机安全)
78+>
79+> conceptual_physics(概念物理学)
80+>
81+> electrical_engineering(电气工程学)
82+>
83+> elementary_mathematics(初等数学)
84+>
85+> high_school_biology(高中生物学)
86+>
87+> high_school_chemistry(高中化学)
88+>
89+> high_school_computer_science(高中计算机科学)
90+>
91+> high_school_mathematics(高中数学)
92+>
93+> high_school_physics(高中物理学)
94+>
95+> high_school_statistics(高中统计学)
96+>
97+> machine_learning(机器学习)
Mdocs/models/pretrain.md+25-0
@@ -526,6 +526,31 @@
526 <td>8x8</td>526 <td>8x8</td>
527 <td>【GTS】</td>527 <td>【GTS】</td>
528 <td>【Test】</td>528 <td>【Test】</td>
529+ </tr>
530+ <tr>
531+ <td rowspan="3"><a href="https://huggingface.co/Qwen">Qwen2.5-Math</a></td>
532+ <td><a href="https://huggingface.co/Qwen/Qwen2.5-Math-1.5B/tree/main">1.5B</a></td>
533+ <td> 4K </td>
534+ <th>Mcore</th>
535+ <td>1x8</td>
536+ <td>【GTS】</td>
537+ <td>【Test】</td>
538+ </tr>
539+ <tr>
540+ <td><a href="https://huggingface.co/Qwen/Qwen2.5-Math-7B/tree/main">7B</a></td>
541+ <td> 4K </td>
542+ <th>Mcore</th>
543+ <td>1x8</td>
544+ <td>【GTS】</td>
545+ <td>【Test】</td>
546+ </tr>
547+ <tr>
548+ <td><a href="https://huggingface.co/Qwen/Qwen2.5-Math-72B/tree/main">72B</a></td>
549+ <td> 4K </td>
550+ <th>Mcore</th>
551+ <td>4x8</td>
552+ <td>【GTS】</td>
553+ <td>【Test】</td>
529 </tr>554 </tr>
530 </tr>555 </tr>
531 <td rowspan="1"><a href="https://huggingface.co/Qwen">CodeQwen2.5</a></td>556 <td rowspan="1"><a href="https://huggingface.co/Qwen">CodeQwen2.5</a></td>
Aexamples/mcore/qwen25_math/ckpt_convert_qwen25_math_hf2mcore.sh+17-0
@@ -0,0 +1,17 @@
1+# 修改 ascend-toolkit 路径
2+source /usr/local/Ascend/ascend-toolkit/set_env.sh
3+ 
4+# 设置需要的权重转换参数
5+python convert_ckpt.py \
6+ --use-mcore-models \
7+ --model-type GPT \
8+ --load-model-type hf \
9+ --save-model-type mg \
10+ --target-tensor-parallel-size 4 \
11+ --target-pipeline-parallel-size 2 \
12+ --add-qkv-bias \
13+ --load-dir ./model_from_hf/qwen2.5_math_hf/ \
14+ --save-dir ./model_weights/qwen2.5_math_mcore/ \
15+ --tokenizer-model ./model_from_hf/qwen2.5_math_hf/tokenizer.json \
16+ --model-type-hf llama2 \
17+ --params-dtype bf16 # --num-layers-per-virtual-pipeline-stage 2 等参数根据模型需要添加
Aexamples/mcore/qwen25_math/ckpt_convert_qwen25_math_mcore2hf.sh+15-0
@@ -0,0 +1,15 @@
1+# 修改 ascend-toolkit 路径
2+source /usr/local/Ascend/ascend-toolkit/set_env.sh
3+ 
4+# 设置并行策略
5+python convert_ckpt.py \
6+ --use-mcore-models \
7+ --model-type GPT \
8+ --model-type-hf llama2 \
9+ --load-model-type mg \
10+ --save-model-type hf \
11+ --target-tensor-parallel-size 1 \
12+ --target-pipeline-parallel-size 1 \
13+ --add-qkv-bias \
14+ --load-dir ./model_weights/qwen2.5_math_mcore/ \
15+ --save-dir ./model_from_hf/qwen2.5_math_hf/ # 需要填入原始HF模型路径，新权重会存于./model_from_hf/qwen2.5_math_hf/mg2hg/
Aexamples/mcore/qwen25_math/data_convert_qwen25_math_pretrain.sh+12-0
@@ -0,0 +1,12 @@
1+ 
2+# 请按照您的真实环境修改 set_env.sh 路径
3+source /usr/local/Ascend/ascend-toolkit/set_env.sh
4+mkdir ./dataset
5+ 
6+python ./preprocess_data.py \
7+ --input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
8+ --tokenizer-name-or-path ./model_from_hf/qwen2.5_math_hf/ \
9+ --output-prefix ./dataset/alpaca \
10+ --tokenizer-type PretrainedFromHF \
11+ --workers 4 \
12+ --log-interval 1000
Aexamples/mcore/qwen25_math/evaluate_qwen25_math_72b_ptd.sh+70-0
@@ -0,0 +1,70 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6003
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=8
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CHECKPOINT="Your ckpt file path"
14+TOKENIZER_PATH="Your vocab file path"
15+DATA_PATH="Your data path (such as ./mmlu/test/)"
16+TASK="mmlu"
17+ 
18+TP=8
19+PP=1
20+MBS=1
21+SEQ_LEN=4096
22+ 
23+DISTRIBUTED_ARGS="
24+ --nproc_per_node $NPUS_PER_NODE \
25+ --nnodes $NNODES \
26+ --node_rank $NODE_RANK \
27+ --master_addr $MASTER_ADDR \
28+ --master_port $MASTER_PORT
29+"
30+ 
31+# Different task needs different max_new_tokens value, please follow the instruction in readme.
32+torchrun $DISTRIBUTED_ARGS evaluation.py \
33+ --use-mcore-models \
34+ --task-data-path $DATA_PATH \
35+ --task ${TASK} \
36+ --tensor-model-parallel-size ${TP} \
37+ --pipeline-model-parallel-size ${PP} \
38+ --micro-batch-size ${MBS} \
39+ --seq-length ${SEQ_LEN} \
40+ --max-position-embeddings ${SEQ_LEN} \
41+ --tokenizer-type PretrainedFromHF \
42+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
43+ --max-new-tokens 1 \
44+ --make-vocab-size-divisible-by 1 \
45+ --padded-vocab-size 152064 \
46+ --rotary-base 10000 \
47+ --untie-embeddings-and-output-weights \
48+ --num-layers 80 \
49+ --hidden-size 8192 \
50+ --ffn-hidden-size 29568 \
51+ --num-attention-heads 64 \
52+ --group-query-attention \
53+ --num-query-groups 8 \
54+ --add-qkv-bias \
55+ --disable-bias-linear \
56+ --swiglu \
57+ --position-embedding-type rope \
58+ --load ${CHECKPOINT} \
59+ --normalization RMSNorm \
60+ --norm-epsilon 1e-06 \
61+ --tokenizer-not-use-fast \
62+ --exit-on-missing-checkpoint \
63+ --no-load-rng \
64+ --no-load-optim \
65+ --no-gradient-accumulation-fusion \
66+ --attention-softmax-in-fp32 \
67+ --seed 42 \
68+ --bf16 \
69+ --no-chat-template \
70+ | tee logs/eval_mcore_qwen25_72b_${TASK}.log
Aexamples/mcore/qwen25_math/evaluate_qwen25_math_7b_ptd.sh+71-0
@@ -0,0 +1,71 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6000
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=4
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CHECKPOINT="Your ckpt file path"
14+TOKENIZER_PATH="Your tokenizer path"
15+DATA_PATH="Your data path (such as ./mmlu/test/)"
16+TASK="mmlu"
17+ 
18+TP=4
19+PP=1
20+SEQ_LENGTH=4096
21+ 
22+DISTRIBUTED_ARGS="
23+ --nproc_per_node $NPUS_PER_NODE \
24+ --nnodes $NNODES \
25+ --node_rank $NODE_RANK \
26+ --master_addr $MASTER_ADDR \
27+ --master_port $MASTER_PORT
28+"
29+ 
30+# Different task needs different max_new_tokens value, please follow the instruction in readme.
31+torchrun $DISTRIBUTED_ARGS evaluation.py \
32+ --use-mcore-models \
33+ --task ${TASK} \
34+ --task-data-path $DATA_PATH \
35+ --no-chat-template \
36+ --tensor-model-parallel-size ${TP} \
37+ --pipeline-model-parallel-size ${PP} \
38+ --num-layers 28 \
39+ --hidden-size 3584 \
40+ --ffn-hidden-size 18944 \
41+ --num-attention-heads 28 \
42+ --max-position-embeddings ${SEQ_LENGTH} \
43+ --seq-length ${SEQ_LENGTH} \
44+ --disable-bias-linear \
45+ --add-qkv-bias \
46+ --group-query-attention \
47+ --num-query-groups 4 \
48+ --swiglu \
49+ --use-fused-swiglu \
50+ --normalization RMSNorm \
51+ --norm-epsilon 1e-6 \
52+ --use-fused-rmsnorm \
53+ --position-embedding-type rope \
54+ --rotary-base 10000 \
55+ --use-fused-rotary-pos-emb \
56+ --make-vocab-size-divisible-by 1 \
57+ --padded-vocab-size 152064 \
58+ --micro-batch-size 1 \
59+ --max-new-tokens 1 \
60+ --tokenizer-type PretrainedFromHF \
61+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
62+ --tokenizer-not-use-fast \
63+ --untie-embeddings-and-output-weights \
64+ --no-gradient-accumulation-fusion \
65+ --attention-softmax-in-fp32 \
66+ --seed 42 \
67+ --load ${CHECKPOINT} \
68+ --exit-on-missing-checkpoint \
69+ --no-load-rng \
70+ --no-load-optim \
71+ | tee logs/evaluation_mcore_qwen25_math_7b_${TASK}.log
Aexamples/mcore/qwen25_math/generate_qwen25_math_1point5b_ptd.sh+63-0
@@ -0,0 +1,63 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6002
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=1
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CHECKPOINT="your model ckpt path"
14+TOKENIZER_PATH="your tokenizer path"
15+ 
16+TP=1
17+PP=1
18+MBS=1
19+SEQ_LEN=4096
20+ 
21+DISTRIBUTED_ARGS="
22+ --nproc_per_node $NPUS_PER_NODE \
23+ --nnodes $NNODES \
24+ --node_rank $NODE_RANK \
25+ --master_addr $MASTER_ADDR \
26+ --master_port $MASTER_PORT
27+"
28+ 
29+torchrun $DISTRIBUTED_ARGS inference.py \
30+ --use-mcore-models \
31+ --tensor-model-parallel-size ${TP} \
32+ --pipeline-model-parallel-size ${PP} \
33+ --num-layers 28 \
34+ --hidden-size 1536 \
35+ --ffn-hidden-size 8960 \
36+ --num-attention-heads 12 \
37+ --group-query-attention \
38+ --num-query-groups 2 \
39+ --tokenizer-type PretrainedFromHF \
40+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
41+ --max-position-embeddings ${SEQ_LEN} \
42+ --seq-length ${SEQ_LEN} \
43+ --make-vocab-size-divisible-by 1 \
44+ --padded-vocab-size 151936 \
45+ --rotary-base 10000 \
46+ --micro-batch-size ${MBS} \
47+ --swiglu \
48+ --add-qkv-bias \
49+ --disable-bias-linear \
50+ --load ${CHECKPOINT} \
51+ --normalization RMSNorm \
52+ --norm-epsilon 1e-6 \
53+ --position-embedding-type rope \
54+ --hidden-dropout 0 \
55+ --attention-dropout 0 \
56+ --tokenizer-not-use-fast \
57+ --max-new-tokens 256 \
58+ --no-gradient-accumulation-fusion \
59+ --exit-on-missing-checkpoint \
60+ --attention-softmax-in-fp32 \
61+ --seed 42 \
62+ --bf16 \
63+ | tee logs/generate_mcore_qwen25_1point5b.log
Aexamples/mcore/qwen25_math/generate_qwen25_math_72b_ptd.sh+64-0
@@ -0,0 +1,64 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6002
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=8
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CHECKPOINT="your model ckpt path"
14+TOKENIZER_PATH="your tokenizer path"
15+ 
16+TP=8
17+PP=1
18+MBS=1
19+SEQ_LEN=4096
20+ 
21+DISTRIBUTED_ARGS="
22+ --nproc_per_node $NPUS_PER_NODE \
23+ --nnodes $NNODES \
24+ --node_rank $NODE_RANK \
25+ --master_addr $MASTER_ADDR \
26+ --master_port $MASTER_PORT
27+"
28+ 
29+torchrun $DISTRIBUTED_ARGS inference.py \
30+ --use-mcore-models \
31+ --tensor-model-parallel-size ${TP} \
32+ --pipeline-model-parallel-size ${PP} \
33+ --num-layers 80 \
34+ --hidden-size 8192 \
35+ --ffn-hidden-size 29568 \
36+ --num-attention-heads 64 \
37+ --group-query-attention \
38+ --num-query-groups 8 \
39+ --tokenizer-type PretrainedFromHF \
40+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
41+ --max-position-embeddings ${SEQ_LEN} \
42+ --seq-length ${SEQ_LEN} \
43+ --make-vocab-size-divisible-by 1 \
44+ --padded-vocab-size 152064 \
45+ --rotary-base 10000 \
46+ --untie-embeddings-and-output-weights \
47+ --micro-batch-size ${MBS} \
48+ --swiglu \
49+ --add-qkv-bias \
50+ --disable-bias-linear \
51+ --load ${CHECKPOINT} \
52+ --normalization RMSNorm \
53+ --norm-epsilon 1e-6 \
54+ --position-embedding-type rope \
55+ --hidden-dropout 0 \
56+ --attention-dropout 0 \
57+ --tokenizer-not-use-fast \
58+ --max-new-tokens 256 \
59+ --no-gradient-accumulation-fusion \
60+ --exit-on-missing-checkpoint \
61+ --attention-softmax-in-fp32 \
62+ --seed 42 \
63+ --bf16 \
64+ | tee logs/generate_mcore_qwen25_72b.log
Aexamples/mcore/qwen25_math/generate_qwen25_math_7b_ptd.sh+64-0
@@ -0,0 +1,64 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6000
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=4
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CHECKPOINT="your model ckpt path"
14+TOKENIZER_PATH="your tokenizer path"
15+ 
16+TP=4
17+PP=1
18+MBS=1
19+SEQ_LENGTH=4096
20+ 
21+DISTRIBUTED_ARGS="
22+ --nproc_per_node $NPUS_PER_NODE \
23+ --nnodes $NNODES \
24+ --node_rank $NODE_RANK \
25+ --master_addr $MASTER_ADDR \
26+ --master_port $MASTER_PORT
27+"
28+ 
29+torchrun $DISTRIBUTED_ARGS inference.py \
30+ --use-mcore-models \
31+ --input-layernorm-in-fp32 \
32+ --tensor-model-parallel-size ${TP} \
33+ --pipeline-model-parallel-size ${PP} \
34+ --num-layers 28 \
35+ --hidden-size 3584 \
36+ --num-attention-heads 28 \
37+ --ffn-hidden-size 18944 \
38+ --max-position-embeddings ${SEQ_LENGTH} \
39+ --seq-length ${SEQ_LENGTH} \
40+ --disable-bias-linear \
41+ --add-qkv-bias \
42+ --group-query-attention \
43+ --num-query-groups 4 \
44+ --swiglu \
45+ --normalization RMSNorm \
46+ --norm-epsilon 1e-6 \
47+ --position-embedding-type rope \
48+ --rotary-base 10000 \
49+ --make-vocab-size-divisible-by 1 \
50+ --padded-vocab-size 152064 \
51+ --micro-batch-size ${MBS} \
52+ --max-new-tokens 256 \
53+ --tokenizer-type PretrainedFromHF \
54+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
55+ --tokenizer-not-use-fast \
56+ --hidden-dropout 0 \
57+ --attention-dropout 0 \
58+ --untie-embeddings-and-output-weights \
59+ --no-gradient-accumulation-fusion \
60+ --attention-softmax-in-fp32 \
61+ --seed 42 \
62+ --load ${CHECKPOINT} \
63+ --exit-on-missing-checkpoint \
64+ | tee logs/generate_mcore_qwen25_math_7b.log
Aexamples/mcore/qwen25_math/pretrain_qwen25_math_1point5b_ptd.sh+107-0
@@ -0,0 +1,107 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6001
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=8
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CKPT_LOAD_DIR="your model ckpt path"
14+CKPT_SAVE_DIR="your model save ckpt path"
15+DATA_PATH="your data path"
16+TOKENIZER_PATH="your tokenizer path"
17+ 
18+TP=1
19+PP=1
20+MBS=2
21+GBS=32
22+SEQ_LEN=4096
23+ 
24+DISTRIBUTED_ARGS="
25+ --nproc_per_node $NPUS_PER_NODE \
26+ --nnodes $NNODES \
27+ --node_rank $NODE_RANK \
28+ --master_addr $MASTER_ADDR \
29+ --master_port $MASTER_PORT
30+"
31+ 
32+GPT_ARGS="
33+ --use-mcore-models \
34+ --tensor-model-parallel-size ${TP} \
35+ --pipeline-model-parallel-size ${PP} \
36+ --sequence-parallel \
37+ --num-layers 28 \
38+ --hidden-size 1536 \
39+ --ffn-hidden-size 8960 \
40+ --num-attention-heads 12 \
41+ --group-query-attention \
42+ --num-query-groups 2 \
43+ --tokenizer-type PretrainedFromHF \
44+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
45+ --seq-length ${SEQ_LEN} \
46+ --max-position-embeddings ${SEQ_LEN} \
47+ --micro-batch-size ${MBS} \
48+ --global-batch-size ${GBS} \
49+ --make-vocab-size-divisible-by 1 \
50+ --padded-vocab-size 151936 \
51+ --rotary-base 10000 \
52+ --train-iters 2000 \
53+ --lr 1.25e-6 \
54+ --min-lr 1.25e-7 \
55+ --weight-decay 1e-1 \
56+ --lr-decay-style cosine \
57+ --lr-warmup-fraction 0.01 \
58+ --clip-grad 1.0 \
59+ --adam-beta1 0.9 \
60+ --adam-beta2 0.95 \
61+ --add-qkv-bias \
62+ --disable-bias-linear \
63+ --attention-dropout 0.0 \
64+ --init-method-std 0.01 \
65+ --hidden-dropout 0.0 \
66+ --position-embedding-type rope \
67+ --normalization RMSNorm \
68+ --norm-epsilon 1e-06 \
69+ --swiglu \
70+ --use-distributed-optimizer \
71+ --use-flash-attn \
72+ --use-fused-rotary-pos-emb \
73+ --use-rotary-position-embeddings \
74+ --use-fused-swiglu \
75+ --use-fused-rmsnorm \
76+ --overlap-grad-reduce \
77+ --no-masked-softmax-fusion \
78+ --attention-softmax-in-fp32 \
79+ --initial-loss-scale 4096 \
80+ --no-gradient-accumulation-fusion \
81+ --no-load-optim \
82+ --no-load-rng \
83+ --seed 42 \
84+ --bf16
85+"
86+ 
87+DATA_ARGS="
88+ --data-path $DATA_PATH \
89+ --split 100,0,0
90+"
91+ 
92+OUTPUT_ARGS="
93+ --log-interval 1 \
94+ --save-interval 1000 \
95+ --eval-interval 1000 \
96+ --eval-iters 0 \
97+ --log-throughput
98+"
99+ 
100+torchrun $DISTRIBUTED_ARGS pretrain_gpt.py \
101+ $GPT_ARGS \
102+ $DATA_ARGS \
103+ $OUTPUT_ARGS \
104+ --distributed-backend nccl \
105+ --load ${CKPT_LOAD_DIR} \
106+ --save ${CKPT_SAVE_DIR} \
107+ | tee logs/train_mcore_qwen25_math_1point5b.log
Aexamples/mcore/qwen25_math/pretrain_qwen25_math_72b_ptd_32p.sh+110-0
@@ -0,0 +1,110 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6001
7+NNODES=4
8+NODE_RANK=0
9+NPUS_PER_NODE=8
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CKPT_LOAD_DIR="your model ckpt path"
14+CKPT_SAVE_DIR="your model save ckpt path"
15+DATA_PATH="your data path"
16+TOKENIZER_PATH="your tokenizer path"
17+ 
18+TP=8
19+PP=4
20+VPP=2
21+MBS=1
22+GBS=64
23+SEQ_LEN=4096
24+ 
25+DISTRIBUTED_ARGS="
26+ --nproc_per_node $NPUS_PER_NODE \
27+ --nnodes $NNODES \
28+ --node_rank $NODE_RANK \
29+ --master_addr $MASTER_ADDR \
30+ --master_port $MASTER_PORT
31+"
32+ 
33+GPT_ARGS="
34+ --use-mcore-models \
35+ --tensor-model-parallel-size ${TP} \
36+ --pipeline-model-parallel-size ${PP} \
37+ --num-layers-per-virtual-pipeline-stage ${VPP} \
38+ --sequence-parallel \
39+ --num-layers 80 \
40+ --hidden-size 8192 \
41+ --ffn-hidden-size 29568 \
42+ --num-attention-heads 64 \
43+ --group-query-attention \
44+ --num-query-groups 8 \
45+ --tokenizer-type PretrainedFromHF \
46+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
47+ --seq-length ${SEQ_LEN} \
48+ --max-position-embeddings ${SEQ_LEN} \
49+ --micro-batch-size ${MBS} \
50+ --global-batch-size ${GBS} \
51+ --make-vocab-size-divisible-by 1 \
52+ --padded-vocab-size 152064 \
53+ --rotary-base 10000 \
54+ --train-iters 2000 \
55+ --lr 1.25e-6 \
56+ --min-lr 1.25e-7 \
57+ --weight-decay 1e-1 \
58+ --lr-decay-style cosine \
59+ --lr-warmup-fraction 0.01 \
60+ --untie-embeddings-and-output-weights \
61+ --clip-grad 1.0 \
62+ --adam-beta1 0.9 \
63+ --adam-beta2 0.95 \
64+ --add-qkv-bias \
65+ --disable-bias-linear \
66+ --attention-dropout 0.0 \
67+ --init-method-std 0.01 \
68+ --hidden-dropout 0.0 \
69+ --position-embedding-type rope \
70+ --normalization RMSNorm \
71+ --norm-epsilon 1e-05 \
72+ --swiglu \
73+ --use-distributed-optimizer \
74+ --use-flash-attn \
75+ --use-fused-rotary-pos-emb \
76+ --use-rotary-position-embeddings \
77+ --use-fused-swiglu \
78+ --use-fused-rmsnorm \
79+ --overlap-grad-reduce \
80+ --no-masked-softmax-fusion \
81+ --attention-softmax-in-fp32 \
82+ --initial-loss-scale 4096 \
83+ --no-gradient-accumulation-fusion \
84+ --no-load-optim \
85+ --no-load-rng \
86+ --seed 42 \
87+ --bf16
88+"
89+ 
90+DATA_ARGS="
91+ --data-path $DATA_PATH \
92+ --split 100,0,0
93+"
94+ 
95+OUTPUT_ARGS="
96+ --log-interval 1 \
97+ --save-interval 1000 \
98+ --eval-interval 1000 \
99+ --eval-iters 0 \
100+ --log-throughput
101+"
102+ 
103+torchrun $DISTRIBUTED_ARGS pretrain_gpt.py \
104+ $GPT_ARGS \
105+ $DATA_ARGS \
106+ $OUTPUT_ARGS \
107+ --distributed-backend nccl \
108+ --load ${CKPT_LOAD_DIR} \
109+ --save ${CKPT_SAVE_DIR} \
110+ | tee logs/train_mcore_qwen25_math_72b.log
Aexamples/mcore/qwen25_math/pretrain_qwen25_math_7b_ptd.sh+115-0
@@ -0,0 +1,115 @@
1+#!/bin/bash
2+export CUDA_DEVICE_MAX_CONNECTIONS=1
3+ 
4+# Change for multinode config
5+MASTER_ADDR=localhost
6+MASTER_PORT=6001
7+NNODES=1
8+NODE_RANK=0
9+NPUS_PER_NODE=8
10+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
11+ 
12+# please fill these path configurations
13+CKPT_LOAD_DIR="your model ckpt path"
14+CKPT_SAVE_DIR="your model save ckpt path"
15+DATA_PATH="your data path"
16+TOKENIZER_PATH="your tokenizer path"
17+ 
18+ 
19+TP=1
20+PP=2
21+SEQ_LEN=4096
22+ 
23+ 
24+DISTRIBUTED_ARGS="
25+ --nproc_per_node $NPUS_PER_NODE \
26+ --nnodes $NNODES \
27+ --node_rank $NODE_RANK \
28+ --master_addr $MASTER_ADDR \
29+ --master_port $MASTER_PORT
30+"
31+ 
32+ 
33+GPT_ARGS="
34+ --use-mcore-models \
35+ --use-distributed-optimizer \
36+ --overlap-grad-reduce \
37+ --overlap-param-gather \
38+ --tensor-model-parallel-size ${TP} \
39+ --pipeline-model-parallel-size ${PP} \
40+ --sequence-parallel \
41+ --num-layers 28 \
42+ --hidden-size 3584 \
43+ --ffn-hidden-size 18944 \
44+ --num-attention-heads 28 \
45+ --max-position-embeddings ${SEQ_LEN} \
46+ --seq-length ${SEQ_LEN} \
47+ --disable-bias-linear \
48+ --add-qkv-bias \
49+ --group-query-attention \
50+ --num-query-groups 4 \
51+ --use-flash-attn \
52+ --swiglu \
53+ --use-fused-swiglu \
54+ --normalization RMSNorm \
55+ --norm-epsilon 1e-6 \
56+ --use-fused-rmsnorm \
57+ --position-embedding-type rope \
58+ --rotary-base 10000 \
59+ --use-fused-rotary-pos-emb \
60+ --untie-embeddings-and-output-weights \
61+ --micro-batch-size 1 \
62+ --global-batch-size 8 \
63+ --make-vocab-size-divisible-by 1 \
64+ --padded-vocab-size 152064 \
65+ --tokenizer-type PretrainedFromHF \
66+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
67+ --attention-dropout 0.0 \
68+ --hidden-dropout 0.0 \
69+ --train-iters 2000 \
70+ --lr 1.25e-6 \
71+ --lr-decay-style cosine \
72+ --min-lr 1.25e-7 \
73+ --lr-warmup-fraction 0.01 \
74+ --init-method-std 0.01 \
75+ --weight-decay 1e-1 \
76+ --clip-grad 1.0 \
77+ --adam-beta1 0.9 \
78+ --adam-beta2 0.95 \
79+ --initial-loss-scale 4096 \
80+ --no-gradient-accumulation-fusion \
81+ --no-masked-softmax-fusion \
82+ --attention-softmax-in-fp32 \
83+ --bf16
84+"
85+ 
86+DATA_ARGS="
87+ --data-path $DATA_PATH \
88+ --split 100,0,0
89+"
90+ 
91+CKPT_ARGS="
92+ --no-load-optim \
93+ --no-load-rng \
94+ --no-save-optim \
95+ --no-save-rng \
96+ --seed 1234 \
97+ --load ${CKPT_LOAD_DIR} \
98+ --save ${CKPT_SAVE_DIR}
99+"
100+ 
101+OUTPUT_ARGS="
102+ --log-interval 1 \
103+ --save-interval 1000 \
104+ --eval-interval 1000 \
105+ --eval-iters 0 \
106+ --log-throughput
107+"
108+ 
109+torchrun $DISTRIBUTED_ARGS pretrain_gpt.py \
110+ $GPT_ARGS \
111+ $DATA_ARGS \
112+ $CKPT_ARGS \
113+ $OUTPUT_ARGS \
114+ --distributed-backend nccl \
115+ | tee logs/train_mcore_qwen25_math_7b.log