HyperParallel Examples
This directory contains usage examples for the HyperParallel distributed training library, demonstrating how to apply various parallel strategies in PyTorch and MindSpore frameworks.
Directory Structure
examples/
├── torch/ # PyTorch examples
│ ├── fully_shard/ # FSDP (Fully Sharded Data Parallel) examples
│ ├── llama3/ # Llama3-style tensor parallel + sequence parallel
│ └── moe/ # MoE expert parallel + pipeline parallel examples
└── mindspore/ # MindSpore examples
├── fully_shard/ # FSDP (Fully Sharded Data Parallel) examples
└── llama3/ # Llama3-style tensor parallel + sequence parallel
Environment Requirements
MindSpore
| Component | Version |
|---|---|
| Python | >=3.9 |
| MindSpore | nightly (202603) |
| CANN | 8.3.0 |
| HyperParallel | nightly (202603) |
PyTorch
| Component | Version |
|---|---|
| Python | >=3.9 |
| torch & torch_npu | 2.6.0 |
| CANN | 8.3.0 |
| HyperParallel | nightly (202603) |
AutoModel Trainer Example
The delivered Qwen3-MoE AutoModel training recipe lives at
hyper_parallel/models/qwen3_moe/recipes/train.yaml inside the
package (shipped in the wheel). Model, dataset, optimizer, and parallel
components are selected through typed targets in train.yaml.
PyTorch MoE Examples
The torch/moe/ directory provides Mixture-of-Experts (MoE) distributed training examples, demonstrating expert parallelism and pipeline parallelism with correctness verification.
Expert Parallelism (EP)
Shards MoE experts across devices using ExpertParallel, each device holds a subset of experts.
torchrun --nproc_per_node=2 expert_parallel_example.py
Pipeline Parallelism + Expert Parallelism (PP + EP)
Combines pipeline parallelism (1F1B schedule) with expert parallelism, verifying that the distributed forward loss matches a standalone reference.
torchrun --nproc_per_node=4 pp_ep_example.py
Environment variables:
| Variable | Default | Description |
|---|---|---|
MOE_PP_SIZE |
2 | Pipeline parallel width |
MOE_EP_SIZE |
2 | Expert parallel width |
MOE_NUM_STEPS |
5 | Number of verification steps |
MOE_DEVICE_TYPE |
npu | Device type (npu or cuda) |
Requirements: world_size == MOE_PP_SIZE * MOE_EP_SIZE, num_experts % MOE_EP_SIZE == 0, n_layers >= MOE_PP_SIZE.