Star206
303
代码
代码
Issues71
Pull Requests95
流水线
项目讨论
Wiki
分析
Star206
303
ascend-robotascend-robotdocs(pytorch): Add document for en
052cd35a创建于 7月9日历史提交

GitHub Documentation

Introduction

MindSpeed LLM is a distributed large language model training toolkit built on the Ascend ecosystem. It aims to provide ecosystem partners in the Huawei Ascend chips ecosystem with an end-to-end large language model training solution, including distributed pretraining, distributed instruction fine-tuning, and the corresponding development toolchain, such as data preprocessing, weight conversion, online inference, and baseline evaluation.

Note: The original repository name, ModelLink, has been changed to MindSpeed LLM, and the original package name, modellink, has been changed to mindspeed_llm.

Roadmap

The latest roadmap updates are published in MindSpeed LLM RoadMap. Visit there for the latest LLM plans and updates.

Community Meetings

Please see the Ascend Meeting Center for the schedule of MindSpeed LLM TC and SIG meetings.

Join Us

To share development experience, exchange usage tips, and stay up to date on project releases, we created the MindSpeed LLM community group. Whether you already use this project or have new ideas, you are welcome to join.

Ways to join:

  1. Scan the QR code to join the WeChat group directly. The QR code is valid for 7 days and is updated regularly.
  2. Add the Ascend open-source assistant to get the group link and join the MindSpeed LLM community group.
MindSpeed LLM community group
MindSpeed LLM WeChat group
Ascend open-source assistant
Ascend assistant WeChat

Latest News

Note: 【Prototype】 indicates that the feature has not been fully validated. If you encounter any issues while using it, please report them in the issue tracker.

Directory Structure

The MindSpeed LLM project code is organized according to modular design principles. For details, see the Project Guide.

MindSpeed-LLM/
 ├── ci                        # CI watchdog
 ├── configs                   # Configuration files
 ├── docs                      # Project documentation
 ├── examples                  # Model example scripts
 ├── mindspeed_llm             # Core code
 ├── tests                     # Test cases
 ├── convert_ckpt.py           # Weight conversion tool
 ├── convert_ckpt_v2.py        # Weight conversion tool v2
 ├── preprocess_data.py        # Data preprocessing tool
 ├── pretrain_gpt.py           # Pretraining workflow
 ├── pretrain_mamba.py         # Pretraining workflow for Mamba models
 ├── posttrain_gpt.py          # Post-training workflow
 ├── preprocess_prompt.py      # Prompt preprocessing tool
 ├── rlhf_gpt.py               # RLHF training workflow
 ├── train_fsdp2.py            # FSDP2 training workflow
 ├── inference.py              # Model inference tool
 ├── evaluation.py             # Model evaluation tool
 ├── setup.py                  # Installation configuration file
 ├── README.md                 # Project overview document

Documentation Navigation

The Documentation Guide provides the complete usage guide for MindSpeed LLM and covers the following core topics:

  • Environment setup guide: installation and configuration instructions for MindSpeed LLM.
  • Quick start: beginner guidance from environment setup to launching training.
  • Model list: models supported by the PyTorch and MindSpore frameworks.
  • Feature list: performance optimization and memory optimization features.
  • Training solutions: complete solutions for pretraining, fine-tuning, inference, and evaluation.
  • Toolchain: usage instructions for weight conversion, dataset processing, performance collection and analysis, deterministic computation, and other tools.

Release Notes

See the release notes.

Installation

Quick Start

This section helps you quickly launch large language model pretraining and fine-tuning tasks. See the following guides for details:

Supported Models

MindSpeed LLM now includes built-in support for pretraining and fine-tuning more than 100 widely used large language models. See the supported model lists:

Training Solutions and Features

MindSpeed LLM includes training solutions such as distributed pretraining and distributed fine-tuning.

Distributed Pretraining

The measured pretraining performance based on MindSpeed LLM is as follows.

Model Series Experimental Model Hardware Cluster Size MFU
Llama-2 Llama-2-7B Atlas 900 A2 PODc 1x8 69.0%
Llama-2-13B Atlas 900 A2 PODc 1x8 64.7%
Llama-2-70B Atlas 900 A2 PODc 4x8 44.1%
Mixtral Mixtral-8x7B Atlas 900 A2 PODc 8x8 31.7%

Pretraining Solutions

Solution Category MCore Released Contributor
Multi-dataset pretraining ✅ ✅ [Ascend]
Multi-sample pack-mode pretraining ✅ ❌

Acceleration Features

Scenario Feature MCore Released Contributor
SPTD parallelism Tensor parallelism ✅ ✅ [Ascend]
Pipeline parallelism ✅ ✅
Virtual pipeline parallelism ✅ ✅
Sequence parallelism ✅ ✅
noop layers ✅ ✅
Long-sequence parallelism Ascend Ring Attention Long-sequence parallelism ✅ ✅
Ulysses Long-sequence parallelism ✅ ✅
Hybrid long-sequence parallelism ✅ ✅
MoE MoE parallelism ✅ ✅
MoE rerouting communication optimization ✅ ✅
Memory optimization Parameter copy reuse ✅ ✅
Distributed optimizer ✅ ✅
Swap Attention ✅ ✅
Recomputation ✅ ✅
Norm recomputation ✅ ✅
O2 BF16 Optimizer ✅ ❌
Fusion operators Flash attention ✅ ✅
Flash attention variable length ✅ ✅
Fused rmsnorm ✅ ✅
Fused swiglu ✅ ✅
Fused rotary position embedding ✅ ✅
GMM ✅ ✅
Matmul Add ✅ ✅
Communication optimization Gradient reduce communication-compute overlap ✅ ✅
Recompute in advance ✅ ✅
Weight all-gather communication-compute overlap ✅ ✅
MC2 ✅ ❌
CoC ✅ ❌
Ascend Gloo archive flushing optimization ✅ ✅

Distributed Fine-Tuning

The measured instruction fine-tuning performance based on MindSpeed LLM is as follows.

Model Hardware Cluster Solution Sequence Performance MFU
Llama-2-7B Atlas 900 A2 PODc 1x8 Full-parameter dynamic 15.87 samples/s -
Full-parameter 16K 1.14 samples/s 37.4%
Full-parameter 32K 0.51 samples/s 48.4%
Llama-2-13B Atlas 900 A2 PODc 1x8 Full-parameter dynamic 50.4 samples/s -
Llama-2-70B Atlas 900 A2 PODc 1x8 LoRA dynamic 15.2 samples/s -

Fine-Tuning Solution

Solution MCore LoRA QLoRA Released Contributor
Single-sample fine-tuning ✅ ✅ ✅ ✅ [Ascend]
Multi-sample pack fine-tuning ✅ ✅ ❌ ❌ [NAIE]
Multi-turn conversation fine-tuning ✅ ✅ ❌ ❌ [Ascend]

Acceleration Features

Scenario Feature MCore Released Contributor
LoRA fine-tuning CCLoRA ✅ ✅ [Ascend]
QLoRA fine-tuning CCLoRA ❌ ❌ [NAIE]
Long-sequence fine-tuning Long-sequence CP ✅ ❌ [Ascend]

Online Inference

Feature MCore Released Contributor
Streaming inference ✅ ✅ [NAIE]
Chat conversation ✅ ✅ [NAIE]
YARN context extension ✅ ❌ [Ascend]

Open Dataset Evaluation

See the open dataset evaluation baselines for model baselines in the repository.

Scenario Dataset MCore Released Contributor
Evaluation MMLU ✅ ❌ [NAIE]
CEval ✅ ❌ [NAIE]
BoolQ ✅ ❌ [NAIE]
BBH ✅ ❌ [NAIE]
AGIEval ✅ ❌ [NAIE]
HumanEval ✅ ❌ [NAIE]

Developer Toolchain

Weight Conversion

MindSpeed LLM supports two-way weight conversion between Hugging Face and Megatron-core formats, and it supports merging LoRA weights. For parameters and usage instructions for the weight conversion feature, see Weight Conversion.

Source Format Target Format Sharding Features LoRA Contributor Released
Hugging Face Megatron-core tp, pp, dpp, vpp, cp, ep, loop layer ❌ [Ascend] ❌
Megatron-core Hugging Face ✅
Megatron-core tp, pp, dpp, vpp, cp, ep, loop layer ✅

Data Preprocessing

MindSpeed LLM supports data preprocessing for pretraining, instruction fine-tuning, and other tasks.

Task Scenario Dataset MCore Released Contributor
Pretraining Pretraining data processing ✅ ✅ [Ascend]
Fine-tuning Alpaca style ✅ ✅
ShareGPT style ✅ ✅

Performance Collection

Scenario Feature MCore Released Contributor
Performance collection Collect profiling data on Ascend chips ✅ ❌ [Ascend]

High Availability

Scenario Feature MCore Released Contributor
High availability Enable deterministic computation on Ascend chips ✅ ❌ [Ascend]

Version Maintenance Policy

MindSpeed LLM versions go through the following five maintenance stages:

Status Time Description
Planning 1 to 3 months Planned features.
Development 3 months Features under development.
Maintenance 6 to 12 months Merge all resolved issues and release versions. Different MindSpeed LLM versions follow different maintenance strategies. The maintenance period is six months for regular releases and twelve months for long-term support releases.
No maintenance 0 to 3 months Merge all resolved issues. No dedicated maintenance staff. No version release.
End of life (EOL) N/A The branch no longer accepts any changes.

MindSpeed LLM released version maintenance policy:

MindSpeed LLM Version Corresponding Tag Maintenance Policy Current Status Release Date Next Status EOL Date
26.0.0 v26.0.0 Regular release Maintenance Mar 30, 2026 Expected to enter no-maintenance status starting Sep 30, 2026
2.3.0 v2.3.0 Regular release Maintenance Dec 30, 2025 Expected to enter no-maintenance status starting Jun 30, 2026
2.2.0 v2.2.0 Regular release Maintenance Sep 30, 2025 Expected to enter no-maintenance status starting Mar 30, 2026
2.1.0 v2.1.0 Regular release EOL Jun 30, 2025 End of life Dec 30, 2025
2.0.0 v2.0.0 Regular release EOL Mar 30, 2025 End of life Sep 30, 2025
1.0.0 v1.0.0 Regular release EOL Dec 30, 2024 End of life Jun 30, 2025
1.0.RC3 v1.0.RC3.0 Regular release EOL Sep 30, 2024 End of life Mar 30, 2025
1.0.RC2 v1.0.RC2.0 Regular release EOL Jun 30, 2024 End of life Dec 30, 2024
1.0.RC1 v1.0.RC1.0 Regular release EOL Mar 30, 2024 End of life Sep 30, 2024
bk_origin_23 \ Demo EOL 2023 End of life Jun 30, 2024

Security Statement

MindSpeed LLM Security Statement

Disclaimer

To MindSpeed LLM Users

  1. The models provided by MindSpeed LLM are for non-commercial use only.
  2. Third-party open source software that MindSpeed LLM depends on, such as Megatron, is provided and maintained by third-party communities. Fixes for problems caused by third-party open source software depend on contributions from and feedback to the relevant communities. You should understand that the MindSpeed LLM repository does not guarantee fixes for problems in the third-party open source software itself, and it also does not guarantee that it tests or corrects all vulnerabilities and errors in third-party open source software.
  3. For each model, the MindSpeed LLM platform only suggests datasets that you can use for training. Huawei does not provide any datasets. If you use these datasets for training, you must especially ensure that you comply with the license of the corresponding dataset. If you become involved in an infringement dispute because of your use of a dataset, Huawei bears no responsibility.
  4. If you find any issues while using MindSpeed LLM models, including but not limited to functional issues and compliance issues, please submit an issue on GitCode. We will review and address it in a timely manner.

To Dataset Owners

If you do not want your dataset to be mentioned in MindSpeed LLM model descriptions, or if you want to update the description of your dataset in MindSpeed LLM, please submit an issue on GitCode. We will delete or update the dataset description according to your request. We sincerely thank you for your understanding and contribution to MindSpeed LLM.

License Statement

  • The license for the MindSpeed LLM product is described in LICENSE.
  • The documents in the docs directory of the MindSpeed LLM tool are subject to the CC-BY 4.0 license. See LICENSE for details.

Contribution Statement

1. Reporting Issues

  • If you find any issues, first check the repository issues list and try to find similar issues or solutions.

  • If the existing issues list does not contain your issue, you can submit a new issue and provide a clear problem description, reproduction steps, and environment information where possible.

2. Performance Optimization and New Features

  • For performance optimization proposals, use the Performance label when submitting the issue and describe the performance optimization feature and usage scenario.

  • For new feature suggestions or discussions, use the Feature label when submitting the issue and describe the background, expectations, and proposal.

3. Code Contribution Process

To submit code changes, follow these brief steps:

  • Develop and commit on your personal branch, then open a Pull Request (PR) to this repository.

  • Register for a PR review slot at our SIG regular meetings by filling in the specified format, and attend the corresponding review meeting on time.

  • Modify the PR according to review comments and update it.

  • After the PR passes review, enter compile in the comment area to trigger the gated CI pipeline.

  • After the PR CI passes and the PR obtains enough labels, the repository Committer will perform the final review and merge it into the development branch.

Thank you for your participation and contribution. We look forward to advancing the project together with you.

Acknowledgments

MindSpeed LLM is jointly contributed by the following Huawei departments and Ascend ecosystem partners:

Huawei:

  • Computing Product Line: Ascend
  • Public Development Department: NAIE
  • Global Technical Service Department: GTS
  • Huawei Cloud Computing: Cloud

Ecosystem partners:

  • Mobile Cloud (China Mobile Cloud): Dayun Zhenze Intelligent Computing Platform
  • Big Data and Artificial Intelligence Lab of the Software Development Center of Industrial and Commercial Bank of China

Thank you for every PR from the community. Contributions to MindSpeed LLM are welcome.