已合并
[DTS2025022427363] readme修改 #2310
AtomGit-Bot创建于 2025年2月27日
[DTS2025022427363] readme修改 #2310
已合并
从refs/pull/2310/head合入到master
共 4 个文件变更+68-29
| @@ -2,7 +2,7 @@ | |||
| 2 | 该特性是基于GPRO+ORM基础上扩展打分器进行混合打分而来，可用于复现DeepSeek-R1-Zero的工作 | 2 | 该特性是基于GPRO+ORM基础上扩展打分器进行混合打分而来，可用于复现DeepSeek-R1-Zero的工作 |
| 3 | 3 | ||
| 4 | ## 整体流程示意图 | 4 | ## 整体流程示意图 |
| 5 | - | 5 | + |
| 6 | 6 | ||
| 7 | 7 | ||
| 8 | 8 | ||
| @@ -15,32 +15,37 @@ | |||
| 15 | 该模型指令遵从度高，有一定概率能引导模型输出`<think>...</think><answer>...$\boxed{}</answer>`格式回复 | 15 | 该模型指令遵从度高，有一定概率能引导模型输出`<think>...</think><answer>...$\boxed{}</answer>`格式回复 |
| 16 | 16 | ||
| 17 | ##### 权重转换 | 17 | ##### 权重转换 |
| 18 | -同MindSpeed-LLM仓一样，模型需要从HuggingFace权重转换为MindSpeed权重，可参考[**这里**](../../../docs/features/checkpoint.md) | 18 | +同MindSpeed-LLM仓一样，模型需要从HuggingFace权重转换为megatron权重，可参考[**权重转换特性**](../../../../docs/features/checkpoint.md) |
| 19 | 19 | ||
| 20 | 20 | ||
| 21 | ## 模板构造 | 21 | ## 模板构造 |
| 22 | 22 | ||
| 23 | -* Qwen2.5-Math-7B | 23 | +**Qwen2.5-Math-7B** |
| 24 | -本身指令遵从度低，仅使用官方系统模板，不做格式要求 | ||
| 25 | -``` | ||
| 26 | -<|im_start|>system\nPlease reason step by step, and put your final answer within \\boxed{}.<|im_end|>\n<|im_start|>user\n{你真正的问题}<|im_end|>\n<|im_start|>assistant\n{模型真正的回答} | ||
| 27 | -``` | ||
| 28 | 24 | ||
| 29 | -* Qwen2.5-7B-Instruct | 25 | +* 本身指令遵从度低，仅使用官方系统模板，不做格式要求 |
| 30 | -需要编写prompt模板激发`<think>...</think><answer>...$\boxed{}</answer>` | 26 | + ``` |
| 31 | -``` | 27 | + <|im_start|>system\nPlease reason step by step, and put your final answer within \\boxed{}.<|im_end|>\n<|im_start|>user\n{你真正的问题}<|im_end|>\n<|im_start|>assistant\n{模型真正的回答} |
| 32 | -<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nA conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>Put your final answer within \\boxed{}.\n{你真正的问题}<|im_end|>\n<|im_start|>assistant\n{模型真正的回答} | 28 | + ``` |
| 33 | -``` | ||
| 34 | 29 | ||
| 35 | -用户可以在`configs/finetune/templates.json`添加自己的自定义模板，添加的新数据模板需要在`preprocess_data.py`与`mindspeed_llm/training/arguments.py`中的`prompt-type`参数choices里加上响应的自定义模板名字 | 30 | +**Qwen2.5-7B与Qwen2.5-7B-Instruct** |
| 31 | + | ||
| 32 | +* 需要编写prompt模板激发`<think>...</think><answer>...$\boxed{}</answer>` | ||
| 33 | + ``` | ||
| 34 | + <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nA conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>Put your final answer within \\boxed{}.\n{你真正的问题}<|im_end|>\n<|im_start|>assistant\n{模型真正的回答} | ||
| 35 | + ``` | ||
| 36 | + | ||
| 37 | +* 用户可以在`configs/finetune/templates.json`添加自己的自定义模板，添加的新数据模板需要在`preprocess_data.py`与`mindspeed_llm/training/arguments.py`中的`prompt-type`参数choices里加上响应的自定义模板名字 | ||
| 36 | 38 | ||
| 37 | 39 | ||
| 38 | ## 数据集 | 40 | ## 数据集 |
| 39 | 使用SimpleRL仓默认的8K数据集 | 41 | 使用SimpleRL仓默认的8K数据集 |
| 40 | -* [pe-nlp/math_level3to5_data_processed_with_qwen_prompt](https://huggingface.co/datasets/pe-nlp/math_level3to5_data_processed_with_qwen_prompt?row=0) | 42 | +* [**pe-nlp/math_level3to5_data_processed_with_qwen_prompt**](https://huggingface.co/datasets/pe-nlp/math_level3to5_data_processed_with_qwen_prompt?row=0) |
| 41 | 43 | ||
| 42 | -* Qwen2.5-Math-7B | 44 | +自定义数据集需要设置--map-keys映射，具体参考[**Alpaca数据集处理**](../../../../docs/features/alpaca_dataset.md)与[**ShareGPT数据集处理**](../../../../docs/features/sharegpt_dataset.md)章节 |
| 43 | - 处理的时候要使用qwen_math_r1的模板（或者自行参考上一节进行构造，自定义数据集需要设置--map-keys映射，具体参考[**这里**](../../../docs/features/alpaca_dataset.md)） | 45 | + |
| 46 | +**Qwen2.5-Math-7B** | ||
| 47 | + | ||
| 48 | +* 处理的时候默认使用qwen_math_r1的模板 | ||
| 44 | 49 | ||
| 45 | ``` | 50 | ``` |
| 46 | python ./preprocess_data.py \ | 51 | python ./preprocess_data.py \ |
| @@ -52,13 +57,13 @@ | |||
| 52 | --workers 4 \ | 57 | --workers 4 \ |
| 53 | --log-interval 1000 \ | 58 | --log-interval 1000 \ |
| 54 | --prompt-type qwen_math_r1 \ | 59 | --prompt-type qwen_math_r1 \ |
| 55 | - --dataset-additional-keys answer subject \ | 60 | + --dataset-additional-keys labels \ |
| 56 | --map-keys '{"prompt":"question", "query":"", "response": "ground_truth_answer", "system":""}' \ | 61 | --map-keys '{"prompt":"question", "query":"", "response": "ground_truth_answer", "system":""}' \ |
| 57 | 62 | ||
| 58 | ``` | 63 | ``` |
| 59 | 64 | ||
| 60 | -* Qwen2.5-7B-Instruct | 65 | +**Qwen2.5-7B与Qwen2.5-7B-Instruct** |
| 61 | - 处理的时候要使用qwen_r1的模板（或者自行参考上一节进行构造，自定义数据集需要设置--map-keys映射，具体参考[**这里**](../../../docs/features/alpaca_dataset.md)） | 66 | +* 处理的时候默认使用qwen_r1的模板 |
| 62 | 67 | ||
| 63 | ``` | 68 | ``` |
| 64 | python ./preprocess_data.py \ | 69 | python ./preprocess_data.py \ |
| @@ -70,7 +75,7 @@ | |||
| 70 | --workers 4 \ | 75 | --workers 4 \ |
| 71 | --log-interval 1000 \ | 76 | --log-interval 1000 \ |
| 72 | --prompt-type qwen_r1 \ | 77 | --prompt-type qwen_r1 \ |
| 73 | - --dataset-additional-keys answer subject \ | 78 | + --dataset-additional-keys labels \ |
| 74 | --map-keys '{"prompt":"question", "query":"", "response": "ground_truth_answer", "system":""}' \ | 79 | --map-keys '{"prompt":"question", "query":"", "response": "ground_truth_answer", "system":""}' \ |
| 75 | 80 | ||
| 76 | ``` | 81 | ``` |
| @@ -95,7 +100,7 @@ DeepSeek-R1-Zero训练的过程中仅使用了基于程序的打分器而没有 | |||
| 95 | ``` | 100 | ``` |
| 96 | GRPO通过分组采样n个输出，利用组内的平均奖励作为基线计算每个输出在组内的相对奖励，并基于相对奖励计算优势值，从而避免了引入额外的价值网络（critic model） | 101 | GRPO通过分组采样n个输出，利用组内的平均奖励作为基线计算每个输出在组内的相对奖励，并基于相对奖励计算优势值，从而避免了引入额外的价值网络（critic model） |
| 97 | ``` | 102 | ``` |
| 98 | - | 103 | + |
| 99 | 104 | ||
| 100 | DeepSeek-R1-Zero的训练过程使用GRPO算法，将ORM（结果奖励模型）替换为基于规则的打分器。 | 105 | DeepSeek-R1-Zero的训练过程使用GRPO算法，将ORM（结果奖励模型）替换为基于规则的打分器。 |
| 101 | 106 | ||
| @@ -1,12 +1,27 @@ | |||
| 1 | export HCCL_CONNECT_TIMEOUT=1800 | 1 | export HCCL_CONNECT_TIMEOUT=1800 |
| 2 | export CUDA_DEVICE_MAX_CONNECTIONS=1 | 2 | export CUDA_DEVICE_MAX_CONNECTIONS=1 |
| 3 | +ulimit -n 32768 | ||
| 3 | 4 | ||
| 4 | -# 主节点初始化ray | 5 | +NNODES=2 |
| 5 | -ray start --head --port 6344 --dashboard-host=0.0.0.0 --dashboard-port=8260 --resources='{"NPU": 8}' | 6 | +NPUS_PER_NODE=8 |
| 6 | 7 | ||
| 7 | -# 子节点全部注册上ray后，查看是否状态正常 | 8 | +# 主节点启动 |
| 8 | -ray status | 9 | +ray start --head --port 6344 --dashboard-host=0.0.0.0 --dashboard-port=8260 --resources='{"NPU": '$NPUS_PER_NODE'}' |
| 9 | 10 | ||
| 10 | -# 启动训练 | 11 | +while true; do |
| 11 | -python ray_gpt.py --config-name grpo_trainer_qwen25_7b | tee logs/r1_zero_qwen25_7b_full.log | 12 | + ray_status_output=$(ray status) |
| 13 | + npu_count=$(echo "$ray_status_output" | grep -oP '(?<=/)\d+\.\d+(?=\s*NPU)' | head -n 1) | ||
| 14 | + npu_count_int=$(echo "$npu_count" | awk '{print int($1)}') | ||
| 15 | + device_count=$((npu_count_int / $NPUS_PER_NODE)) | ||
| 12 | 16 | ||
| 17 | + # 判断 device_count 是否与 NNODES 相等 | ||
| 18 | + if [ "$device_count" -eq "$NNODES" ]; then | ||
| 19 | + echo "Ray cluster is ready with $device_count devices (from $npu_count NPU resources), starting Python script." | ||
| 20 | + ray status | ||
| 21 | + python ray_gpt.py --config-name grpo_trainer_qwen25_7b | tee logs/r1_zero_qwen25_7b_full.log | ||
| 22 | + break | ||
| 23 | + else | ||
| 24 | + echo "Waiting for Ray to allocate $NNODES devices. Current device count: $device_count" | ||
| 25 | + sleep 5 | ||
| 26 | + fi | ||
| 27 | +done | ||
| @@ -1,6 +1,25 @@ | |||
| 1 | export HCCL_CONNECT_TIMEOUT=1800 | 1 | export HCCL_CONNECT_TIMEOUT=1800 |
| 2 | export CUDA_DEVICE_MAX_CONNECTIONS=1 | 2 | export CUDA_DEVICE_MAX_CONNECTIONS=1 |
| 3 | +ulimit -n 32768 | ||
| 4 | + | ||
| 5 | +NNODES=2 | ||
| 6 | +NPUS_PER_NODE=8 | ||
| 7 | +MASTER_ADDR="主节点 IP 地址" | ||
| 3 | 8 | ||
| 4 | # 子节点往主节点注册ray | 9 | # 子节点往主节点注册ray |
| 5 | -# IP_ADDRESS 处填写主节点 IP 地址 | 10 | +ray start --address="$MASTER_ADDR:6344" --resources='{"NPU": '$NPUS_PER_NODE'}' |
| 6 | -ray start --address="IP_ADDRESS:6344" --resources='{"NPU": 8}' | 11 | + # 重新尝试连接到主节点直到成功 |
| 12 | + while true; do | ||
| 13 | + # 尝试连接 Ray 集群 | ||
| 14 | + ray start --address="$MASTER_ADDR:6344" --resources='{"NPU": '$NPUS_PER_NODE'}' | ||
| 15 | + | ||
| 16 | + # 检查连接是否成功 | ||
| 17 | + ray status | ||
| 18 | + if [ $? -eq 0 ]; then | ||
| 19 | + echo "Successfully connected to the Ray cluster!" | ||
| 20 | + break | ||
| 21 | + else | ||
| 22 | + echo "Failed to connect to the Ray cluster. Retrying in 5 seconds..." | ||
| 23 | + sleep 5 | ||
| 24 | + fi | ||
| 25 | + done | ||