| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
docs: modify the structure of documents Co-authored-by: Keilo_W<wangkaiyu11@h-partners.com> # message auto-generated for no-merge-commit merge: !3287 merge docs into master docs: modify the structure of documents Created-by: Keilo_W Commit-by: Keilo_W Merged-by: ascend-robot Description: 1、Modify the structure of documents 2、Pass the ci of Doc Tools See merge request: Ascend/MindSpeed!3287 | 4 个月前 | |
feat(qad): implement QAD (Quantization-Aware Distillation) Co-authored-by: n_nddddd<lanshangwei1@huawei.com> # message auto-generated for no-merge-commit merge: !3638 merge feat_qad_1 into master feat(qad): implement QAD (Quantization-Aware Distillation) Created-by: n_nddddd Commit-by: n_nddddd Merged-by: ascend-robot Description: --- title: "feat(backend): implement QAD (Quantization-Aware Distillation)" labels: ["feat", "backend"] assignees: 'n_nddddd' --- ## What this PR does / why we need it? 大模型量化到 4-bit（MXFP4）后精度显著退化，传统 QAT 用交叉熵训练会改变输出分布，影响 RL 模型。 本 PR 实现 QAD（量化感知蒸馏）：用 BF16 全精度教师模型通过 KL 散度引导 MXFP4 W4A4 学生模型训练，忠实保留原始分布：L = α·KL。 核心组件： QADConfig — 配置数据类与校验 TeacherModelManager — 教师模型懒加载、检查点加载、参数冻结、单例管理 LogitsKLLoss — KL 损失，支持温度缩放和 TP 感知 softmax QADQuantEngineFeature — CLI 参数注册与快速失败校验 QADForwardStepPatch — 包装 pretrain() 注入 QAD 逻辑，KL 损失 支持模式：纯 KL。 ## Does this PR introduce any user-facing change? 是。新增 5个 CLI 参数（qad 参数组）： 参数 默认值 说明 --qad-enable False 启用 QAD --qad-teacher-load "" 教师检查点路径（启用时必填） --kl-temperature 1.0 KL 温度参数 --kl-loss-weight 1.0 KL 损失权重 α --kl-loss-reduction "mean" KL 归约方式（mean/sum） 文档：docs/zh/features/qad.md 约束：需相同架构的 BF16 教师检查点；纯 KL 模式要求 pretrain 入口可导入 get_batch。 ## How was this patch tested? 设计文档： https://wiki.huawei.com/domains/159368/wiki/325164/WIKI2026070911785637 实验数据： https://wiki.huawei.com/domains/159368/wiki/325164/WIKI2026062311569830?title=5--Core-Components 55 个单元测试全部通过： python -m pytest tests_extend/unit_tests/features/qad/ tests_extend/unit_tests/mindspeed/qad/ -v tached PASSED [100%] ============================== slowest durations ============================== (165 durations < 1s hidden.) ============================= 55 passed in 9.54s ============================== ruff check...............................................................Passed ruff format..............................................................Passed 覆盖：QADConfig 校验、LogitsKLLoss 输出/温度/归约/梯度、TeacherModelManager 生命周期、参数校验、QADForwardStepPatch 损失合并、TP 感知 softmax（TP=1 等价性 + TP=2 all-reduce 路径）。 代码质量：ruff check / ruff format / pylint 全部通过，0 warnings，0 # noqa 和 # pylint: disable 注释。 See merge request: Ascend/MindSpeed!3638 | 2 天前 | |
docs: add directory structure doc, contribution guide and license docs Co-authored-by: fanlu5<fanlu5@huawei.com> # message auto-generated for no-merge-commit merge: !3485 merge master into master docs: add directory structure doc, contribution guide and license docs Created-by: fanlu5 Commit-by: fanlu5 Merged-by: ascend-robot Description: What this PR does / why we need it? https://gitcode.com/Ascend/MindSpeed/issues/146 docs: add directory structure doc, contribution guide and license docs 1. add project directory structure documentation 2. add CONTRIBUTING.md contribution guide 3. add CC-BY 4.0 license for docs directory 4. update README with new sections including directory structure, contribution guide and license info Does this PR introduce any user-facing change? NA How was this patch tested? NA See merge request: Ascend/MindSpeed!3485 | 2 个月前 | |
| 1 年前 |
