HyperParallel Examples

This directory contains usage examples for the HyperParallel distributed training library, demonstrating how to apply various parallel strategies in PyTorch and MindSpore frameworks.

Directory Structure

examples/
├── torch/              # PyTorch examples
│   ├── fully_shard/    # FSDP (Fully Sharded Data Parallel) examples
│   ├── llama3/         # Llama3-style tensor parallel + sequence parallel
│   └── moe/            # MoE expert parallel + pipeline parallel examples
└── mindspore/          # MindSpore examples
    ├── fully_shard/    # FSDP (Fully Sharded Data Parallel) examples
    └── llama3/         # Llama3-style tensor parallel + sequence parallel

Environment Requirements

MindSpore

Component	Version
Python	>=3.9
MindSpore	nightly (202603)
CANN	8.3.0
HyperParallel	nightly (202603)

PyTorch

Component	Version
Python	>=3.9
torch & torch_npu	2.6.0
CANN	8.3.0
HyperParallel	nightly (202603)

AutoModel Trainer Example

The delivered Qwen3-MoE AutoModel training recipe lives at hyper_parallel/models/qwen3_moe/recipes/train.yaml inside the package (shipped in the wheel). Model, dataset, optimizer, and parallel components are selected through typed targets in train.yaml.

PyTorch MoE Examples

The torch/moe/ directory provides Mixture-of-Experts (MoE) distributed training examples, demonstrating expert parallelism and pipeline parallelism with correctness verification.

Expert Parallelism (EP)

Shards MoE experts across devices using ExpertParallel, each device holds a subset of experts.

torchrun --nproc_per_node=2 expert_parallel_example.py

Pipeline Parallelism + Expert Parallelism (PP + EP)

Combines pipeline parallelism (1F1B schedule) with expert parallelism, verifying that the distributed forward loss matches a standalone reference.

torchrun --nproc_per_node=4 pp_ep_example.py

Environment variables:

Variable	Default	Description
MOE_PP_SIZE	2	Pipeline parallel width
MOE_EP_SIZE	2	Expert parallel width
MOE_NUM_STEPS	5	Number of verification steps
MOE_DEVICE_TYPE	npu	Device type (npu or cuda)

Requirements: world_size == MOE_PP_SIZE * MOE_EP_SIZE, num_experts % MOE_EP_SIZE == 0, n_layers >= MOE_PP_SIZE.