| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(examples): add MindSpore llama3 TP+FSDP demo and local embedding Replace nn.Embedding with Llama3LocalEmbedding to avoid Gather layout-infer gaps on MindSpore, add fsdp_tp_example on a (dp,tp) mesh, and wire autograd compat plus build_dp_tp_mesh for multi-rank smoke runs. | 5 个月前 | |
feat(trainer+models): training stack + qwen3.5 dense/moe/vl-moe aligned Add an in-tree training stack as part of HyperParallel: ModelSpec contract (build_model_fn / parallelize_fn / state_dict_adapter / clip_grad_fn) + 3-layer YAML (model/data/train) + DeviceMesh aliases (fsdp/dp/loss). Per-model parallelize.py owns the full AC / FSDP / Prefetch composition. Three Qwen3.5 models registered (folder names follow HF transformers convention): - hyper_parallel/models/qwen3_5/ (dense) - hyper_parallel/models/qwen3_5_moe/ (MoE w/ SharedExpertMoE) - hyper_parallel/models/qwen3_vl_moe/ (VL-MoE w/ vision tower + DeepStack) For uneven dim-0 params (e.g. shared_expert_gate.weight [1, hidden]), moe / vlmoe parallelize prefer Shard(d>0) along the first divisible non-zero dim via shard_placement_fn, falling back to replicate_params only when no non-zero dim is divisible. This avoids to_sharded()'s copy-back path (added by upstream fix for in-place forward writes on replicate state) when forward is read-only, which otherwise costs +/-1 bf16 ULP/step on NPU. Self-consistency (HP 1c vs HP FSDP4, avg_loss): - dense: 20/20 byte-equal - moe: 20/20 byte-equal (after Shard(d>0) preference) - vlmoe: 11/20 byte-equal + 9/20 sub-ULP <=2e-3 (NPU vision-tower fp noise) Cross-framework (HP-self vs LF + HF-native + HP-backend, FSDP4): - dense: 20/20 byte-equal - moe: 17/20 byte-equal + 3/20 1-bf16-ULP (LF integration kept conservative to preserve correctness for HF models that may in-place write replicate state in forward; root-cause is upstream's unconditional copy in fully_shard, follow-up to make it conditional via autograd version counter) - vlmoe: out-of-scope (NPU bf16 noise floor + LF compute_loss_func) LF integration provides Accelerate FSDP2 compat layer with auto replicate_params for shape[0]<shard_size weights -- kept conservative on purpose to preserve correctness for arbitrary HF models. Training-correctness fixes: - train() epoch loop now uses while global_step < max_steps driven by the live counter; multi-epoch + resume runs no longer exceed max_steps. - train_step increments global_step AFTER next(data_iterator) so StopIteration does not bump checkpoint / log indices. - rank_average loss reporting divides by the DP-group size instead of global world_size; previously TP / EP / PP / CP > 1 systematically under-reported the logged loss. Code quality: trainer + model files use module-top imports for stdlib / torch / hyper internals; optional deps (datasets / transformers / safetensors / torchdata / tqdm) and torch_npu remain lazy with # pylint: disable=C0415 annotations. Forward/backward and interface-conformance methods carry # pylint: disable=W0613. _materialize_and_init_shards / _load_weights split into single-responsibility helpers (each <= 10 CCN, <= 40 NLOC). Latent bug in _maybe_(up|down)cast_*_params fixed: the isinstance check used torch's DTensor instead of hyper's, so the rebuild branch never ran. Known limitations (P2-P4 stub): EP / PP / CP fail-fast; streaming / packing / dynamic batching not implemented; Profiler / W&B / TB / MemoryMonitor / Eval / MoEMonitor callbacks reduced to stubs; Float8 / ChunkedCELoss not wired. | 5 个月前 | |
feat(trainer+models): training stack + qwen3.5 dense/moe/vl-moe aligned Add an in-tree training stack as part of HyperParallel: ModelSpec contract (build_model_fn / parallelize_fn / state_dict_adapter / clip_grad_fn) + 3-layer YAML (model/data/train) + DeviceMesh aliases (fsdp/dp/loss). Per-model parallelize.py owns the full AC / FSDP / Prefetch composition. Three Qwen3.5 models registered (folder names follow HF transformers convention): - hyper_parallel/models/qwen3_5/ (dense) - hyper_parallel/models/qwen3_5_moe/ (MoE w/ SharedExpertMoE) - hyper_parallel/models/qwen3_vl_moe/ (VL-MoE w/ vision tower + DeepStack) For uneven dim-0 params (e.g. shared_expert_gate.weight [1, hidden]), moe / vlmoe parallelize prefer Shard(d>0) along the first divisible non-zero dim via shard_placement_fn, falling back to replicate_params only when no non-zero dim is divisible. This avoids to_sharded()'s copy-back path (added by upstream fix for in-place forward writes on replicate state) when forward is read-only, which otherwise costs +/-1 bf16 ULP/step on NPU. Self-consistency (HP 1c vs HP FSDP4, avg_loss): - dense: 20/20 byte-equal - moe: 20/20 byte-equal (after Shard(d>0) preference) - vlmoe: 11/20 byte-equal + 9/20 sub-ULP <=2e-3 (NPU vision-tower fp noise) Cross-framework (HP-self vs LF + HF-native + HP-backend, FSDP4): - dense: 20/20 byte-equal - moe: 17/20 byte-equal + 3/20 1-bf16-ULP (LF integration kept conservative to preserve correctness for HF models that may in-place write replicate state in forward; root-cause is upstream's unconditional copy in fully_shard, follow-up to make it conditional via autograd version counter) - vlmoe: out-of-scope (NPU bf16 noise floor + LF compute_loss_func) LF integration provides Accelerate FSDP2 compat layer with auto replicate_params for shape[0]<shard_size weights -- kept conservative on purpose to preserve correctness for arbitrary HF models. Training-correctness fixes: - train() epoch loop now uses while global_step < max_steps driven by the live counter; multi-epoch + resume runs no longer exceed max_steps. - train_step increments global_step AFTER next(data_iterator) so StopIteration does not bump checkpoint / log indices. - rank_average loss reporting divides by the DP-group size instead of global world_size; previously TP / EP / PP / CP > 1 systematically under-reported the logged loss. Code quality: trainer + model files use module-top imports for stdlib / torch / hyper internals; optional deps (datasets / transformers / safetensors / torchdata / tqdm) and torch_npu remain lazy with # pylint: disable=C0415 annotations. Forward/backward and interface-conformance methods carry # pylint: disable=W0613. _materialize_and_init_shards / _load_weights split into single-responsibility helpers (each <= 10 CCN, <= 40 NLOC). Latent bug in _maybe_(up|down)cast_*_params fixed: the isinstance check used torch's DTensor instead of hyper's, so the rebuild branch never ran. Known limitations (P2-P4 stub): EP / PP / CP fail-fast; streaming / packing / dynamic batching not implemented; Profiler / W&B / TB / MemoryMonitor / Eval / MoEMonitor callbacks reduced to stubs; Float8 / ChunkedCELoss not wired. | 5 个月前 | |
feat(trainer+models): training stack + qwen3.5 dense/moe/vl-moe aligned Add an in-tree training stack as part of HyperParallel: ModelSpec contract (build_model_fn / parallelize_fn / state_dict_adapter / clip_grad_fn) + 3-layer YAML (model/data/train) + DeviceMesh aliases (fsdp/dp/loss). Per-model parallelize.py owns the full AC / FSDP / Prefetch composition. Three Qwen3.5 models registered (folder names follow HF transformers convention): - hyper_parallel/models/qwen3_5/ (dense) - hyper_parallel/models/qwen3_5_moe/ (MoE w/ SharedExpertMoE) - hyper_parallel/models/qwen3_vl_moe/ (VL-MoE w/ vision tower + DeepStack) For uneven dim-0 params (e.g. shared_expert_gate.weight [1, hidden]), moe / vlmoe parallelize prefer Shard(d>0) along the first divisible non-zero dim via shard_placement_fn, falling back to replicate_params only when no non-zero dim is divisible. This avoids to_sharded()'s copy-back path (added by upstream fix for in-place forward writes on replicate state) when forward is read-only, which otherwise costs +/-1 bf16 ULP/step on NPU. Self-consistency (HP 1c vs HP FSDP4, avg_loss): - dense: 20/20 byte-equal - moe: 20/20 byte-equal (after Shard(d>0) preference) - vlmoe: 11/20 byte-equal + 9/20 sub-ULP <=2e-3 (NPU vision-tower fp noise) Cross-framework (HP-self vs LF + HF-native + HP-backend, FSDP4): - dense: 20/20 byte-equal - moe: 17/20 byte-equal + 3/20 1-bf16-ULP (LF integration kept conservative to preserve correctness for HF models that may in-place write replicate state in forward; root-cause is upstream's unconditional copy in fully_shard, follow-up to make it conditional via autograd version counter) - vlmoe: out-of-scope (NPU bf16 noise floor + LF compute_loss_func) LF integration provides Accelerate FSDP2 compat layer with auto replicate_params for shape[0]<shard_size weights -- kept conservative on purpose to preserve correctness for arbitrary HF models. Training-correctness fixes: - train() epoch loop now uses while global_step < max_steps driven by the live counter; multi-epoch + resume runs no longer exceed max_steps. - train_step increments global_step AFTER next(data_iterator) so StopIteration does not bump checkpoint / log indices. - rank_average loss reporting divides by the DP-group size instead of global world_size; previously TP / EP / PP / CP > 1 systematically under-reported the logged loss. Code quality: trainer + model files use module-top imports for stdlib / torch / hyper internals; optional deps (datasets / transformers / safetensors / torchdata / tqdm) and torch_npu remain lazy with # pylint: disable=C0415 annotations. Forward/backward and interface-conformance methods carry # pylint: disable=W0613. _materialize_and_init_shards / _load_weights split into single-responsibility helpers (each <= 10 CCN, <= 40 NLOC). Latent bug in _maybe_(up|down)cast_*_params fixed: the isinstance check used torch's DTensor instead of hyper's, so the rebuild branch never ran. Known limitations (P2-P4 stub): EP / PP / CP fail-fast; streaming / packing / dynamic batching not implemented; Profiler / W&B / TB / MemoryMonitor / Eval / MoEMonitor callbacks reduced to stubs; Float8 / ChunkedCELoss not wired. | 5 个月前 | |
feat(examples): add MoE Expert Parallel correctness demo Add a minimal MoE decoder model and EP verification script under examples/torch/moe/. The demo compares standalone vs EP-distributed forward/backward outputs to validate numerical equivalence. Includes model (GQA + RoPE + MoE FFN), EP/TP parallelization helpers, demo utilities, and a torchrun-based correctness check script. | 5 个月前 | |
feat(mindspore): Llama3 TP+SP example and Cell parity in TP distribute_module | 5 个月前 |
HyperParallel Examples
This directory contains usage examples for the HyperParallel distributed training library, demonstrating how to apply various parallel strategies in PyTorch and MindSpore frameworks.
Directory Structure
examples/
├── torch/ # PyTorch examples
│ ├── fully_shard/ # FSDP (Fully Sharded Data Parallel) examples
│ └── llama3/ # Llama3-style tensor parallel + sequence parallel
└── mindspore/ # MindSpore examples
├── fully_shard/ # FSDP (Fully Sharded Data Parallel) examples
└── llama3/ # Llama3-style tensor parallel + sequence parallel
Environment Requirements
MindSpore
| Component | Version |
|---|---|
| Python | >=3.9 |
| MindSpore | nightly (202603) |
| CANN | 8.3.0 |
| HyperParallel | nightly (202603) |
PyTorch
| Component | Version |
|---|---|
| Python | >=3.9 |
| torch & torch_npu | 2.6.0 |
| CANN | 8.3.0 |
| HyperParallel | nightly (202603) |