Pull Request已成功合入, 合并人@ascend-robot
(感谢 yeqm 的贡献)变更摘要
本次变更针对 docs/zh/features/pretrain.md 中的 FSDP2 文本预训练配置指南进行修正,使其与当前 master 实现保持一致。核心问题在于原示例仅配置 attr.pretrain: true,而 FSDP2 后端实际通过 basic_parameters.stage 选择 PretrainDatasetProcessor;缺少 stage: pretrain 时默认回退为 sft,导致无 response 字段的纯文本样本被丢弃并最终触发 StopIteration。文档同步补充了必需的 stage 配置、更新废弃参数、澄清后端差异,并修正 packing 相关描述。
主要改动
- 补充
basic_parameters.stage: pretrain配置: 在 FSDP2 后端示例中新增stage: pretrain,确保后端正确选择PretrainDatasetProcessor,避免默认回退到sft导致纯文本样本被丢弃并触发StopIteration。 - 更新废弃的 collate 参数: 将已废弃的
collate_param.model_name替换为collate_param.collator_id,并明确使用llm_pretrain。 - 澄清 FSDP2 与 Megatron 后端差异: 在文档中说明两种后端在文本预训练配置上的不同要求,避免用户沿用统一配置造成误解。
- 修正 packing 描述: 更正为文本在每次预处理 batch 内进行拼接打包,每个 batch 至少需包含
cutoff_len个有效 token 才能生成一个训练样本。 - 改进文档格式与可读性: 对指南的排版和表述进行优化,使配置说明更清晰易读。


🤖 本次改动均为数据 / 生成 / 二进制文件(如测试用例、锁文件、媒体资源),没有可审查的代码逻辑,已跳过 AI 代码审查。


Thanks for your pull-request.
The full list of commands accepted by me can be found at here。
You can get sig-info at here
PR Approval Progress
✅ Congratulations! All modules have met the lgtm and approve requirements.
Module Approval Details
| module | lgtm status | approve status |
|---|---|---|
| docs | ✅ yaoyaoxu, 陈志国, 王海涛 (3/2) | ✅ 王海涛 (1/1) |
💡 Tip:
- Committer can comment
/approveor/lgtm- Commenting
/approveimplies both code review (lgtm) and intent to merge (approve)
CLA Signature Pass
ye_qm, thanks for your pull request. All authors of the commits have signed the CLA. 👍


Pull Request 已合并或已关闭。
If you want to solve this problem, you can click here to do it in the FAQs.


Pull Request 已合并或已关闭。
If you want to solve this problem, you can click here to do it in the FAQs.


What this PR does / why we need it?
The FSDP2 text pretraining example in
docs/zh/features/pretrain.mddoes not match the current implementation.The existing example only sets
attr.pretrain: true. However, the FSDP2 backend selectsPretrainDatasetProcessorthroughbasic_parameters.stage. Withoutstage: pretrain, the stage defaults tosft, causing text-only samples without a response field to be dropped and eventually resulting inStopIteration.This PR:
basic_parameters.stage: pretrainsetting for the FSDP2 backend.collate_param.model_namewithcollate_param.collator_id.cutoff_lenvalid tokens to generate a training sample.Related issue: N/A. The problem was found while verifying the documented workflow against the current
masterimplementation.Does this PR introduce any user-facing change?
Yes. This PR corrects the user-facing configuration required to run text-only pretraining with the FSDP2 backend.
Users should configure:
data: dataset_param: attr: formatting: alpaca prompt: text basic_parameters: stage: pretrain template: default dataloader_param: collate_param: collator_id: llm_pretrain