已合并
docs: fix text pretraining configuration guide #2974
docs: fix text pretraining configuration guide #2974
已合并
yeqm创建于 5 天前
yeqm成员
5 天前

What this PR does / why we need it?

The FSDP2 text pretraining example in docs/zh/features/pretrain.md does not match the current implementation.

The existing example only sets attr.pretrain: true. However, the FSDP2 backend selects PretrainDatasetProcessor through basic_parameters.stage. Without stage: pretrain, the stage defaults to sft, causing text-only samples without a response field to be dropped and eventually resulting in StopIteration.

This PR:

  • Adds the required basic_parameters.stage: pretrain setting for the FSDP2 backend.
  • Replaces the deprecated collate_param.model_name with collate_param.collator_id.
  • Clarifies the configuration differences between the FSDP2 and Megatron backends.
  • Corrects the packing description: text is packed within each preprocessing batch, so each batch must contain at least cutoff_len valid tokens to generate a training sample.
  • Improves the formatting and readability of the guide.

Related issue: N/A. The problem was found while verifying the documented workflow against the current master implementation.

Does this PR introduce any user-facing change?

Yes. This PR corrects the user-facing configuration required to run text-only pretraining with the FSDP2 backend.

Users should configure:

data:
  dataset_param:
    attr:
      formatting: alpaca
      prompt: text
    basic_parameters:
      stage: pretrain
      template: default
  dataloader_param:
    collate_param:
      collator_id: llm_pretrain
likedislike
Pull Request已成功合入, 合并人@ascend-robot
(感谢 yeqm 的贡献)
Yyeqm成员
5 天前 创建了 pull request,commit 01e222d4
atomgit-bot
atomgit-bot
5 天前 评论:

变更摘要

本次变更针对 docs/zh/features/pretrain.md 中的 FSDP2 文本预训练配置指南进行修正,使其与当前 master 实现保持一致。核心问题在于原示例仅配置 attr.pretrain: true,而 FSDP2 后端实际通过 basic_parameters.stage 选择 PretrainDatasetProcessor;缺少 stage: pretrain 时默认回退为 sft,导致无 response 字段的纯文本样本被丢弃并最终触发 StopIteration。文档同步补充了必需的 stage 配置、更新废弃参数、澄清后端差异,并修正 packing 相关描述。

主要改动

  • 补充 basic_parameters.stage: pretrain 配置: 在 FSDP2 后端示例中新增 stage: pretrain,确保后端正确选择 PretrainDatasetProcessor,避免默认回退到 sft 导致纯文本样本被丢弃并触发 StopIteration
  • 更新废弃的 collate 参数: 将已废弃的 collate_param.model_name 替换为 collate_param.collator_id,并明确使用 llm_pretrain
  • 澄清 FSDP2 与 Megatron 后端差异: 在文档中说明两种后端在文本预训练配置上的不同要求,避免用户沿用统一配置造成误解。
  • 修正 packing 描述: 更正为文本在每次预处理 batch 内进行拼接打包,每个 batch 至少需包含 cutoff_len 个有效 token 才能生成一个训练样本。
  • 改进文档格式与可读性: 对指南的排版和表述进行优化,使配置说明更清晰易读。
likedislike
atomgit-bot
atomgit-bot
5 天前 评论:

🤖 本次改动均为数据 / 生成 / 二进制文件(如测试用例、锁文件、媒体资源),没有可审查的代码逻辑,已跳过 AI 代码审查。

likedislike
ascend-robotascend-robot成员
5 天前 添加了label:ascend-cla/yes
ascend-robot
ascend-robot成员
5 天前 评论:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here
You can get sig-info at here


PR Approval Progress

Congratulations! All modules have met the lgtm and approve requirements.

Module Approval Details

module lgtm status approve status
docs yaoyaoxu, 陈志国, 王海涛 (3/2) 王海涛 (1/1)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

ye_qm, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
此处折叠了103条消息 查看更多
ascend-robotascend-robot成员
5 天前 添加了label:approved
ascend-robotascend-robot成员
5 天前 关闭了关联的issue
ascend-robotascend-robot成员
5 天前 合入了pull request
ascend-robot
ascend-robot成员
5 天前 评论:

Pull Request 已合并或已关闭。

If you want to solve this problem, you can click here to do it in the FAQs.

likedislike
ascend-robot
ascend-robot成员
5 天前 评论:

Pull Request 已合并或已关闭。

If you want to solve this problem, you can click here to do it in the FAQs.

likedislike