| feat(agents): add FSDP2 expert parallelism (EP) onboarding skill for MoE models Co-authored-by: vasileone<hanyue42@huawei.com> # message auto-generated for no-merge-commit merge: !3091 merge ep_skill_push into master feat(agents): add FSDP2 expert parallelism (EP) onboarding skill for MoE models Created-by: vasileone Commit-by: vasileone Merged-by: ascend-robot Description: ## What this PR does / why we need it? 本 PR 为 MindSpeed-MM 新增面向 Agent 的 FSDP2 专家并行（EP）接入技能，将 MoE 模型接入 EP 的完整方法论沉淀为可复用流程，降低新模型开启专家并行的门槛与出错概率。 主要改动如下： - 新增 mindspeed-mm-fsdp2-ep-parallel 技能（SKILL.md），定义五阶段工作流：模型探测（模块树 / 权重布局 / forward 契约）→ 按分支 A（零模型改动）/ B（自定义 ep_forward ）/ C（权重融合改造）决策 → ep_plan 配置 → 六项预检 → 验证（生效确认先于数值对照，EP-off/EP-on loss 平均相对差 ≤ 1% 为完成判据）。 - 新增参考文档 ep-module-contract.md ：默认 EP forward 的契约假设与 A/B 分支判定、权重布局与 forward 的配对规则、per-expert 子模块到融合专家权重的改造步骤。 - 新增参考文档 ep-config-coupling.md ：EP 相关七个配置字段间的六条隐式耦合及排障对照表（如 EP 静默跳过、EFSDP 条目剔除导致的重复 fully_shard 等）。 - 更新 .agents/README.md 与 knowledge/task-flow-guide.md ，登记该技能入口并纳入任务路由表。 ## Does this PR introduce any user-facing change? 否。本 PR 仅新增/更新 .agents/ 下的文档，不改动任何训练代码、模型实现、配置格式或命令行接口。 ## How was this patch tested? 以 MiniMax-M3-VL（已完成 FSDP2 迁移的 MoE 模型）为端到端案例，按该技能全流程执行验证： - 模型侧实现自定义 MiniMaxM3VLExperts.ep_forward ：仅表达相对 dispatcher 的差异——DTensor local tensor 提取、 [E, out, in] 显式转置、复用框架既有 alltoall/allgather/mc2 dispatcher、以 _apply_gate 作为自定义 activation 传入。 - 生效确认：pattern 实际命中唯一专家容器（未命中 Router 与 shared expert）、确认使用自定义 ep_forward、4 卡专家 mesh 为 EP2/EFSDP2、专家参数为按专家维切分的 DTensor、日志进入 Building EP+FSDP2 optimizer 路径。 - 数值验证：共同逻辑初始权重、相同 seed / 数据顺序 / world_size 的 10 步 BF16 对照，EP-off 与 EP-on 平均逐步 loss 相对差 0.105394% （最大 0.560276%，首步 0.000065%），满足 ≤ 1% 判据。 See merge request: Ascend/MindSpeed-MM!3091 | 1 个月前 |
| feat: Bump PyTorch to 2.10.0 and Python to 3.12 Co-authored-by: zhangxubin<zhangxubin3@h-partners.com> # message auto-generated for no-merge-commit merge: !3030 merge master into master feat: Bump PyTorch to 2.10.0 and Python to 3.12 Created-by: MoCuishle-M Commit-by: zhangxubin Merged-by: ascend-robot Description: ## What this PR does / why we need it? 完成"升级pytorch到2.10.0，python到3.12"收尾工作 1. mm_ci_trigger.sh 适配当前CI镜像 2. qwen3.5 模型ST开启FLA算子ascend c实现。（配置文件从examples\qwen3_5\qwen3_5_35B_config.yaml复制，简单修改后得到） 3. pyproject.toml 里将pytorch版本限制到2.10.0，python版本限制到3.12。 4. 修改grpo UT里需要的卡数（从8卡 --> 4卡，因为现在UT资源只有4卡） ## Does this PR introduce any user-facing change? pytorch、python版本的修改已经在https://gitcode.com/Ascend/MindSpeed-MM/pull/3020/discuss 中说明 ## How was this patch tested? CI环境镜像已经更新。ST、UT通过即无问题。 See merge request: Ascend/MindSpeed-MM!3030 | 1 个月前 |
| feat: add gitcode workflow Co-authored-by: w30057661<wangshusheng7@h-partners.com> # message auto-generated for no-merge-commit merge: !3048 merge master into master feat: add gitcode workflow Created-by: WSs_321 Commit-by: w30057661 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. issue已关联[https://gitcode.com/Ascend/MindSpeed-MM/issues/677](url) See merge request: Ascend/MindSpeed-MM!3048 | 1 个月前 |
| docs: 整改UserGuide/config中相关文档的说明 Co-authored-by: weixin_44031810<gaojie75@huawei.com> # message auto-generated for no-merge-commit merge: !3000 merge master into master docs: 整改UserGuide/config中相关文档的说明 Created-by: gaojie_ Commit-by: weixin_44031810 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 本 PR 属于纯文档变更，目标修正 UserGuide/config 相关文档的说明内容。变更统计显示共修改 7 个文件、新增 1035 行、删除 0 行； 主要改动 修正 UserGuide/config 文档说明: 本次改动聚焦于更正 UserGuide/config 相关文档中的表述，属于文档说明层面的修正。 纯新增型文档变更: 变更以新增内容为主（新增 1035 行、删除 0 行），未涉及代码逻辑、接口、数据结构或配置项的实际修改。 https://gitcode.com/Ascend/MindSpeed-MM/issues/565 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!3000 | 1 个月前 |
| 修复 qwen25vl_7B mbridge 特性报错的问题 Co-authored-by: bonjour_rzq1998<ruanzeqi1@huawei.com> # message auto-generated for no-merge-commit merge: !2906 merge feature/fix-mbridge-qwen25vl into master 修复 qwen25vl_7B mbridge 特性报错的问题 Created-by: bonjour_rzq1998 Commit-by: bonjour_rzq1998 Merged-by: ascend-robot Description: What this PR does / why we need it? 1、修改bridge/models/qwen_vl/__init__.py中对于异常情况的容忍范围。 Does this PR introduce any user-facing change? 不涉及 How was this patch tested? 不涉及 See merge request: Ascend/MindSpeed-MM!2906 | 2 个月前 |
| fix: save generation_config.json during dcp_to_hf conversion Co-authored-by: LiuYuan001<liuyuan203@huawei.com> # message auto-generated for no-merge-commit merge: !3002 merge 0818 into master fix: save generation_config.json during dcp_to_hf conversion Created-by: LiuYuan001 Commit-by: LiuYuan001 Merged-by: ascend-robot Description: ## What this PR does / why we need it? NPU训练保存的模型通过dcp2hf时未将generation_config.json保存至目标hf权重路径，导致评测时因为缺少该文件导致vLLM实际推理参数不同，进而导致评测结果指标不同. 在agentic 评测（jingyan_agent_benchmark）请求里不传任何采样参数，使用服务端默认值，temperature=1 → 实际走的是"采样"路径，top_k/top_p 的宽松会让每步都有更大概率抽到低概率的噪声 token；而该任务要求严格的多轮结构化输出（<tool_call> XML 标签），一步错就解析失败，且误差会跨轮累积 → 显著变差 ## Does this PR introduce any user-facing change? 转换后的 HF 模型目录将新增 generation_config.json 文件，用户无需额外操作即可获得正确的生成参数. ## How was this patch tested? 使用dcp2hf能正常转换 mm-convert Qwen35Converter dcp_to_hf --save_hf_dir ./save_test --dcp_dir ./iter_0000720 --origin_hf_dir ./Qwen/Qwen3.5-122B-A10B --to_bf16 false See merge request: Ascend/MindSpeed-MM!3002 | 2 个月前 |
| feat: Bump PyTorch to 2.10.0 and Python to 3.12 Co-authored-by: zhangxubin<zhangxubin3@h-partners.com> # message auto-generated for no-merge-commit merge: !3030 merge master into master feat: Bump PyTorch to 2.10.0 and Python to 3.12 Created-by: MoCuishle-M Commit-by: zhangxubin Merged-by: ascend-robot Description: ## What this PR does / why we need it? 完成"升级pytorch到2.10.0，python到3.12"收尾工作 1. mm_ci_trigger.sh 适配当前CI镜像 2. qwen3.5 模型ST开启FLA算子ascend c实现。（配置文件从examples\qwen3_5\qwen3_5_35B_config.yaml复制，简单修改后得到） 3. pyproject.toml 里将pytorch版本限制到2.10.0，python版本限制到3.12。 4. 修改grpo UT里需要的卡数（从8卡 --> 4卡，因为现在UT资源只有4卡） ## Does this PR introduce any user-facing change? pytorch、python版本的修改已经在https://gitcode.com/Ascend/MindSpeed-MM/pull/3020/discuss 中说明 ## How was this patch tested? CI环境镜像已经更新。ST、UT通过即无问题。 See merge request: Ascend/MindSpeed-MM!3030 | 1 个月前 |
| docs: Modify docker/supported_tags.md based on the doc template Co-authored-by: xubin-wsl-codex<1656631289@qq.com> # message auto-generated for no-merge-commit merge: !3046 merge master into master docs: Modify docker/supported_tags.md based on the doc template Created-by: MoCuishle-M Commit-by: xubin-wsl-codex Merged-by: ascend-robot Description: ## What this PR does / why we need it? 根据文档模板修改docker/supported_tags.md ## Does this PR introduce any user-facing change? 不涉及 ## How was this patch tested? 不涉及 See merge request: Ascend/MindSpeed-MM!3046 | 1 个月前 |
| docs:modify Qwen3.5-35B Readme and related document Co-authored-by: songke029<songke29@huawei.com> # message auto-generated for no-merge-commit merge: !3037 merge master into master docs:modify Qwen3.5-35B Readme and related document Created-by: songke029 Commit-by: songke029 Merged-by: ascend-robot Description: ## What this PR does / why we need it? There are some usability issues with the current installation and fine-tuning guide for Qwen3.5, as well as its associated documentation and shell scripts. 当前Qwen3.5的安装和微调指南以及其相关文档和.sh文件存在一些易用性问题 ## Does this PR introduce any user-facing change? examples/qwen3_5/README.md docs/zh/features/building_data_for_VLModel.md ## How was this patch tested? Run bash scripts/install.sh --msbranch master to test if the change in install.sh that skips the .postX suffix in version checks works as expected. bash scripts/install.sh --msbranch master用于测试install.sh修改版本检查时忽略 .postX 后缀是否生效 See merge request: Ascend/MindSpeed-MM!3037 | 1 个月前 |
| fix: Fix dataloader state mismatch with enable_preload for checkpoint resume Co-authored-by: WendongPang<pangwendong@huawei.com> # message auto-generated for no-merge-commit merge: !3065 merge preload into master fix: Fix dataloader state mismatch with enable_preload for checkpoint resume Created-by: WendongPang Commit-by: WendongPang Merged-by: ascend-robot Description: ## What this PR does / why we need it? 开启enable_preload时，检查点保存时会出现 1 个 batch 的偏移。保存在检查点中的 DataLoader 状态是预取超前的位置，并非真实已消费的训练样本位置。因此恢复训练的第一步会使用第 k+1 个 batch，而非第 k 个 batch，造成损失、梯度范数与连续训练的结果不一致。 ## Does this PR introduce any user-facing change? 将以下几项默认置为false： training: no_load_optim: false # Do not load optimizer state; load by default. no_load_rng: false # Do not load RNG state; load by default. no_save_optim: false # Do not save optimizer state; save by default. no_save_rng: false # Do not save RNG state; save by default. ## How was this patch tested? 修正后开启preload断点续训精度对齐  See merge request: Ascend/MindSpeed-MM!3065 | 1 个月前 |
| fix: Fix dataloader state mismatch with enable_preload for checkpoint resume Co-authored-by: WendongPang<pangwendong@huawei.com> # message auto-generated for no-merge-commit merge: !3065 merge preload into master fix: Fix dataloader state mismatch with enable_preload for checkpoint resume Created-by: WendongPang Commit-by: WendongPang Merged-by: ascend-robot Description: ## What this PR does / why we need it? 开启enable_preload时，检查点保存时会出现 1 个 batch 的偏移。保存在检查点中的 DataLoader 状态是预取超前的位置，并非真实已消费的训练样本位置。因此恢复训练的第一步会使用第 k+1 个 batch，而非第 k 个 batch，造成损失、梯度范数与连续训练的结果不一致。 ## Does this PR introduce any user-facing change? 将以下几项默认置为false： training: no_load_optim: false # Do not load optimizer state; load by default. no_load_rng: false # Do not load RNG state; load by default. no_save_optim: false # Do not save optimizer state; save by default. no_save_rng: false # Do not save RNG state; save by default. ## How was this patch tested? 修正后开启preload断点续训精度对齐  See merge request: Ascend/MindSpeed-MM!3065 | 1 个月前 |
| feat: add gitleaks secret detection Co-authored-by: zhangxubin<zhangxubin3@h-partners.com> # message auto-generated for no-merge-commit merge: !2817 merge master into master feat: add gitleaks secret detection Created-by: MoCuishle-M Commit-by: zhangxubin Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. 引入gitleaks二进制离线扫描工具 2. 新增pre-commit/.gitleaks.toml配置，继承官方全部检测规则 3. 配置pre-commit钩子，提交前自动扫描密钥硬编码风险。 ## Does this PR introduce any user-facing change? 无 ## How was this patch tested? PR流水线pre-commit检测新增敏感信息检测. See merge request: Ascend/MindSpeed-MM!2817 | 3 个月前 |
| docs:modify Qwen3.5-35B Readme and related document Co-authored-by: songke029<songke29@huawei.com> # message auto-generated for no-merge-commit merge: !3037 merge master into master docs:modify Qwen3.5-35B Readme and related document Created-by: songke029 Commit-by: songke029 Merged-by: ascend-robot Description: ## What this PR does / why we need it? There are some usability issues with the current installation and fine-tuning guide for Qwen3.5, as well as its associated documentation and shell scripts. 当前Qwen3.5的安装和微调指南以及其相关文档和.sh文件存在一些易用性问题 ## Does this PR introduce any user-facing change? examples/qwen3_5/README.md docs/zh/features/building_data_for_VLModel.md ## How was this patch tested? Run bash scripts/install.sh --msbranch master to test if the change in install.sh that skips the .postX suffix in version checks works as expected. bash scripts/install.sh --msbranch master用于测试install.sh修改版本检查时忽略 .postX 后缀是否生效 See merge request: Ascend/MindSpeed-MM!3037 | 1 个月前 |
| docs: 整改UserGuide/config中相关文档的说明 Co-authored-by: weixin_44031810<gaojie75@huawei.com> # message auto-generated for no-merge-commit merge: !3000 merge master into master docs: 整改UserGuide/config中相关文档的说明 Created-by: gaojie_ Commit-by: weixin_44031810 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 本 PR 属于纯文档变更，目标修正 UserGuide/config 相关文档的说明内容。变更统计显示共修改 7 个文件、新增 1035 行、删除 0 行； 主要改动 修正 UserGuide/config 文档说明: 本次改动聚焦于更正 UserGuide/config 相关文档中的表述，属于文档说明层面的修正。 纯新增型文档变更: 变更以新增内容为主（新增 1035 行、删除 0 行），未涉及代码逻辑、接口、数据结构或配置项的实际修改。 https://gitcode.com/Ascend/MindSpeed-MM/issues/565 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!3000 | 1 个月前 |
| [feat] add logging modules Co-authored-by: htwang<wanghaitao60@huawei.com> # message auto-generated for no-merge-commit merge: !3092 merge refactor into master [feat] add logging modules Created-by: htwang Commit-by: htwang Merged-by: ascend-robot Description: ## What this PR does / why we need it? 支持日志模块 ## Does this PR introduce any user-facing change? 当前不影响，只是增加了个目录，还未被调用 ## How was this patch tested? NA See merge request: Ascend/MindSpeed-MM!3092 | 1 个月前 |
| docs: fix master doc ci Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2651 merge doc-scan-fix into master docs: fix master doc ci Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? fix doc ci ## Does this PR introduce any user-facing change? fix doc ci ## How was this patch tested? fix doc ci See merge request: Ascend/MindSpeed-MM!2651 | 4 个月前 |
| 修改完善pre-commit工具配置 Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !2470 merge master into master 修改完善pre-commit工具配置 Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? pre-commit 工具规则配置修改：试运行阶段发现python bandit工具检测过严，超出昇腾编程规范。 此次修改重点修改bandit规则。 ## Does this PR introduce any user-facing change? 门禁codecheck检测规则精简，提升代码合入效率 ## How was this patch tested? 代码扫描工具配置，不涉及代码仓功能 See merge request: Ascend/MindSpeed-MM!2470 | 5 个月前 |
| feat: Update Docker image overview & refactor image build process Co-authored-by: zhangxubin<zhangxubin3@h-partners.com> # message auto-generated for no-merge-commit merge: !2975 merge master into master feat: Update Docker image overview & refactor image build process Created-by: MoCuishle-M Commit-by: zhangxubin Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. 重构镜像构建过程 1.1 改变dockerfile写法，使得镜像大小从25.9GB 缩减至 14.9GB。 通过指令顺序与缓存优化，多阶段构建（最终镜像只保留运行时必需的产物）实现镜像瘦身。 1.2 删除一些不使用的接口 1.3 剥离CI镜像构建逻辑 1.4 删除qwen3vl GRPO环境构建。该模型已经通过verl仓库承载。 2. 调整CI镜像构建参数，为后续的版本升级做准备 3. 根据用户反馈与相关要求更新overview文档，提供镜像易用性。 ## Does this PR introduce any user-facing change? 更改了镜像构建使用方法，已经在overview里更新说明。 ## How was this patch tested? 本地验证镜像构建没问题。 验证了三种chip、两种OS。 See merge request: Ascend/MindSpeed-MM!2975 | 2 个月前 |
| chore: add VS Code devcontainer configuration. Co-authored-by: zhangxubin<zhangxubin3@h-partners.com> # message auto-generated for no-merge-commit merge: !2911 merge master into master chore: add VS Code devcontainer configuration. Created-by: MoCuishle-M Commit-by: zhangxubin Merged-by: ascend-robot Description: ## What this PR does / why we need it? 当前仓库缺乏 Dev Container 配置（.devcontainer/devcontainer.json 及配套 Dockerfile），导致开发者在不同环境中需要手动配置依赖，降低了协作效率。 为 MindSpeed-MM 新增 VS Code Dev Container 配置（.devcontainer/），一键启动标准化的容器开发环境。 同时对dockerfile做一定适配简化。 ## Does this PR introduce any user-facing change? 可选使用，不影响已有代码。VS Code 打开仓库 → F1 → Reopen in Container 即可。 ## How was this patch tested? 在 arm64 NPU 服务器上 Reopen in Container 验证通过：镜像构建、依赖安装、pre-commit hooks、torch_npu.npu.is_available() 均正常。 See merge request: Ascend/MindSpeed-MM!2911 | 2 个月前 |
| docs: 移除文档中 MindSpore 相关展示 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2670 merge docs-fix into master docs: 移除文档中 MindSpore 相关展示 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? 移除文档中 MindSpore 相关展示(目录树/Profiler项),仅保留 PyTorch 移除出海描述 ## Does this PR introduce any user-facing change? 移除文档中 MindSpore 相关展示(目录树/Profiler项),仅保留 PyTorch 移除出海描述 ## How was this patch tested? read See merge request: Ascend/MindSpeed-MM!2670 | 4 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| !325 【测试】添加InternVL2-8B ST & build打包内容完善 Merge pull request !325 from 陆劲夫/master | 1 年前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| docs: 增加新增模型描述 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !3115 merge doc_models into master docs: 增加新增模型描述 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? 增加新增模型描述 ## Does this PR introduce any user-facing change? 增加新增模型描述 ## How was this patch tested? 不需要 See merge request: Ascend/MindSpeed-MM!3115 | 1 个月前 |
| docs:rename Ascend Extension for PyTorch Co-authored-by: z60112595<zhaotao68@h-partners.com> # message auto-generated for no-merge-commit merge: !2836 merge master into master docs:rename Ascend Extension for PyTorch Created-by: kali00 Commit-by: z60112595 Merged-by: ascend-robot Description: ## What this PR does / why we need it? https://gitcode.com/Ascend/MindSpeed-MM/issues/506 1、更新Ascend Extension for PyTorch名称为TorchNPU 2、补充快速入门环境最低配置 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2836 | 3 个月前 |
| feature: Add MiniMax-H3 FSDP2 finetuning for T2VA, FL2VA, and Ref2VA Co-authored-by: vasileone<hanyue42@huawei.com> # message auto-generated for no-merge-commit merge: !3026 merge minimax_h3_adpter into master feature: Add MiniMax-H3 FSDP2 finetuning for T2VA, FL2VA, and Ref2VA Created-by: vasileone Commit-by: vasileone Merged-by: ascend-robot Description: ## What this PR does / why we need it? 本 PR 将 MiniMax-H3 的全参数 DiT 微调接入 MindSpeed-MM FSDP2 后端，支持 T2VA、FL2VA 和 Ref2VA 三类任务。 MiniMax-H3 的文本编码、Video VAE、Audio VAE 和数据预处理仍由 DiffSynth-Studio 离线完成；MindSpeed-MM 读取 DiffSynth-Studio 生成的预 计算 .pth 缓存并负责 DiT 训练，从而避免在训练路径中引入不必要的在线预处理依赖。 主要改动如下： - 新增 MiniMax-H3 T2VA、FL2VA、Ref2VA 的 FSDP2 全参数微调支持； - 新增 DiffSynth-Studio 预计算缓存的数据集插件； - 新增 Ref2VA 的 visual/audio reference anchor 处理； - 集成 NPU Flash Attention、RMSNorm、RoPE、SwiGLU 融合算子； - 支持 MindSpeed-MM 重计算和 Ulysses Context Parallel； - 提供 T2VA、FL2VA、Ref2VA 三套可直接启动的配置文件和脚本； - MiniMax-H3 README，补充环境安装、权重下载、缓存生成、微调和正式训练说明。 MiniMax-H3 DiT 结构及训练语义参考 DiffSynth-Studio： https://github.com/modelscope/DiffSynth-Studio/tree/0ad850953156c028c6334ea8c17c50032a1ef169/diffsynth ## Does this PR introduce any user-facing change? 是。 用户可通过以下脚本启动对应任务的 FSDP2 微调： - examples/minimax_h3/finetune_minimax_h3_t2va.sh - examples/minimax_h3/finetune_minimax_h3_fl2va.sh - examples/minimax_h3/finetune_minimax_h3_ref2va.sh 对应配置文件： - examples/minimax_h3/minimax_h3_t2va.yaml - examples/minimax_h3/minimax_h3_fl2va.yaml - examples/minimax_h3/minimax_h3_ref2va.yaml 用户文档路径： - examples/minimax_h3/README.md ## How was this patch tested? 以 DiffSynth-Studio 为参考实现，在相同预计算缓存、初始 Transformer 权重及固定随机种子条件下，进行 loss 精度对齐验证。 精度对齐使用 DiffSynth-Studio 基于 HQ-OpenHumanVid 生成的预计算 .pth 缓存。MindSpeed-MM 与 DiffSynth-Studio 训练时读取完全相同的缓存，并按固定路径顺序加载。 - FL2VA：使用 130 条真实视频样本，分辨率为 704 x 1280、243 帧；每条样本包含 prompt、目标视频及同步音频，并预计算首尾帧条件。 - Ref2VA：使用同一批 130 条真实视频样本。每条样本的 reference 由目标首帧图像、目标开头 1 秒的同步视频和音频构成。 ### 测试结果 | 任务 | 测试配置 | Loss 平均相对误差 | Loss 最大相对误差 | 结果 | | ------ | --------------------------------------------------------------------- | ----------------: | ----------------: | ------------------------ | | FL2VA | 5 个 DiT Block，500 step，8 NPU | 0.015047% | 0.090183% | 误差低于 1% | | Ref2VA | 5 个 DiT Block，500 step，8 NPU，Ulysses CP=2 | 0.039115% | 0.192966% | 误差低于 1% | ### GPU、NPU精度对齐实验  ### 性能表现 相同配置及数据下 - Diffsynth 源仓微调的单步耗时是33~34s - MindSpeed-MM 仓微调的单步耗时是29S (性能≈1.11 x GPU A100) 限制说明： - Ulysses CP 需要使用 flash_attention_2 或 flash_attention_3； - T2VA、FL2VA 必须使用 FL2VA transformer 权重；Ref2VA 必须使用 Ref2VA transformer 权重； - 不同任务类型、分辨率或帧数需要使用独立生成的 DiffSynth-Studio 缓存。 See merge request: Ascend/MindSpeed-MM!3026 | 1 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| bugfix: 合法合规风险数据修复 Co-authored-by: yaoyaoxu<xuyaoyao.824404@huawei.com> # message auto-generated for no-merge-commit merge: !2763 merge codecheck_260702 into master bugfix: 合法合规风险数据修复 Created-by: yaoyaoxu Commit-by: yaoyaoxu Merged-by: ascend-robot Description: ## What this PR does / why we need it? 合法合规风险数据修复：补充缺少的Copyright ## Does this PR introduce any user-facing change? 不涉及用户界面修改 ## How was this patch tested? 通过已有ut和st See merge request: Ascend/MindSpeed-MM!2763 | 3 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| [Bugfix] fix protobuf resource conflict bug Co-authored-by: LKONE<wanglikai4@huawei.com> # message auto-generated for no-merge-commit merge: !2343 merge master into master [Bugfix] fix protobuf resource conflict bug Created-by: wanglikai1019 Commit-by: LKONE Merged-by: ascend-robot Description: ## What this PR does / why we need it? 当环境中存在tensorflow时，直接拉起训练脚本，在import transformers时会因框架不兼容导致段错误，程序异常退出 ## Does this PR introduce any user-facing change? 无 ## How was this patch tested? 在装有tensorflow库的环境中拉起训练脚本，看是否能够正常运行 See merge request: Ascend/MindSpeed-MM!2343 | 6 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| [modify] modify the threshold for gc in qwen3_vl 30B Co-authored-by: ye_qm<yeqiangmao@huawei.com> # message auto-generated for no-merge-commit merge: !2593 merge fix-qwen3vl-gc into master [modify] modify the threshold for gc in qwen3_vl 30B Created-by: ye_qm Commit-by: ye_qm Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional GC threshold setting for Qwen3-VL 30B LoRA fine-tuning. Background: During Qwen3-VL 30B training, Python GC behavior may introduce performance fluctuation in some runs. This PR allows the Qwen3-VL 30B LoRA training script to explicitly enable a tuned GC threshold before training starts. Changes: - Adds ENABLE_GC_THRESHOLD=1 to examples/qwen3vl/finetune_lora_qwen3vl_30B.sh. - Updates pretrain_transformers.py to read ENABLE_GC_THRESHOLD. - When ENABLE_GC_THRESHOLD is set to "1", pretrain_transformers.py applies gc.set_threshold(700, 10, 1000). - If the environment variable is not set, the default behavior is unchanged. ## Does this PR introduce any user-facing change? Yes, but it is limited to an optional environment variable. Users can enable the GC threshold behavior by setting: export ENABLE_GC_THRESHOLD=1 The Qwen3-VL 30B LoRA example script enables this variable by default. Other training scripts are not affected unless they also set ENABLE_GC_THRESHOLD=1. No public API is changed. Related script: examples/qwen3vl/finetune_lora_qwen3vl_30B.sh ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2593 | 4 个月前 |
| style: pre-commit autofix cleancode (base check) Co-authored-by: liyingxuan<liyingxuan3@huawei.com> # message auto-generated for no-merge-commit merge: !2616 merge master into master style: pre-commit autofix cleancode (base check) Created-by: liyx616 Commit-by: liyingxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2616 | 4 个月前 |
| [Refactor]: replace optional dependencies with standard library implementations Co-authored-by: AZe_404<wangze62@h-partners.com> # message auto-generated for no-merge-commit merge: !3075 merge site_package into master [Refactor]: replace optional dependencies with standard library implementations Created-by: AZe_404 Commit-by: AZe_404 Merged-by: ascend-robot Description: ## What this PR does / why we need it? MindSpeed-MM 三方非必要依赖消减，以及少量非必要三方接口调用重构. ## Does this PR introduce any user-facing change? 1. pyproject.toml 依赖清理：从运行时依赖中移除 20 个无使用或可替代的依赖 （six、decorator、attrs、gpytorch、psutil、SQLAlchemy、scikit-learn、 pybind11、wheel、ninja、protobuf、regex、bs4、orjson、beartype、 pandarallel、imageio、imageio-ffmpeg、scipy、pytest-mock）；pytest-mock 移 入 [project.optional-dependencies] test。ftfy 因 mojibake 修复算法复杂、功 能不可替代，予以保留。 2. 代码适配改造（先改后删，功能等价）： - data/data_utils/utils.py、fsdp/data/data_utils/generative_data_utils/ utils.py：orjson 替换为标准库 json（含 shared memory 写入的 UTF-8 编码 适配），BeautifulSoup 替换为标准库 html/re 解析； - utils/data_balance/balance_sorting_algo.py：protobuf INT64_MAX 替换为 sys.maxsize； - tasks/inference/pipeline/wan_pipeline.py：regex 替换为标准库 re； - models/common/embeddings/pos_embeddings.py：移除 beartype，类型注解改为 typing，删除 @beartype 装饰器； - data/dataloader/sampler.py：pandarallel 替换为 pandas 原生 apply； - fsdp/distributed/context_parallel/ring_context_parallel/utils.py：删除 scipy 未使用的 import； - 新增 utils/video_io.py：基于已保留依赖 av 统一实现视频 MP4/图片 PNG 保 存，替换 imageio/imageio-ffmpeg；tasks/inference/pipeline/utils/ sora_utils.py、tasks/evaluation/gen_impl/vbench_gen.py 同步改造。 ## How was this patch tested? 使用测试环境验证部分模型（Wan2.2 TI2V 5B、Qwen2.5VL 7B、Qwen2.5Omni 7B、Flux kontext）训练功能性，全量测试依赖合入后全量流水验证. See merge request: Ascend/MindSpeed-MM!3075 | 1 个月前 |
