| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: add sapp nd search modules | 4 个月前 | |
fix device mesh bugs | 7 个月前 | |
feat(dmodule): add M1 declarative Configurable and Module layer Introduce Configurable in hyper_parallel/config/ and declarative Module, BaseModel, ShardingConfig, and ModelSpec under hyper_parallel.dmodule. ModelSpec supports Config-based registration alongside legacy build_model_fn. Module.parallelize applies DTensor sharding on a single DeviceMesh; local_map is deferred to M9. Expand Google-style docstrings with usage examples; add resolve_placements unit tests (case sensitivity, empty mesh, Partial). Add tests/ut/dmodule and dist_module parallelize coverage without wiring the new path into LLMTrainer or package __init__ exports. | 5 个月前 | |
[fix][fully_shard] adjust prefetch, unshard position | 4 个月前 | |
test: improve UT coverage for core/shard and custom_ops/experimental Extract experimental_ops.py from custom_ops/experimental/__init__.py so __init__.py becomes a pure re-export. Add 100% UT coverage for the new module. Raise core/shard coverage: _op_dispatch.py 31%->70%, api.py 8%->82%, custom_shard.py 18%->100%. Add new UT files for parallel_flash_attention_score, parallel_ops, parallel_tuple_elementwise, parallel_scaled_dot_product_attention, and extend embedding/conv3d/ one_hot_ext/ms_flash_attention/npu_flash_attention/sdpa tests. | 5 个月前 | |
feat(dmodule): add M1 declarative Configurable and Module layer Introduce Configurable in hyper_parallel/config/ and declarative Module, BaseModel, ShardingConfig, and ModelSpec under hyper_parallel.dmodule. ModelSpec supports Config-based registration alongside legacy build_model_fn. Module.parallelize applies DTensor sharding on a single DeviceMesh; local_map is deferred to M9. Expand Google-style docstrings with usage examples; add resolve_placements unit tests (case sensitivity, empty mesh, Partial). Add tests/ut/dmodule and dist_module parallelize coverage without wiring the new path into LLMTrainer or package __init__ exports. | 5 个月前 | |
feat(trainer+models): training stack + qwen3.5 dense/moe/vl-moe aligned Add an in-tree training stack as part of HyperParallel: ModelSpec contract (build_model_fn / parallelize_fn / state_dict_adapter / clip_grad_fn) + 3-layer YAML (model/data/train) + DeviceMesh aliases (fsdp/dp/loss). Per-model parallelize.py owns the full AC / FSDP / Prefetch composition. Three Qwen3.5 models registered (folder names follow HF transformers convention): - hyper_parallel/models/qwen3_5/ (dense) - hyper_parallel/models/qwen3_5_moe/ (MoE w/ SharedExpertMoE) - hyper_parallel/models/qwen3_vl_moe/ (VL-MoE w/ vision tower + DeepStack) For uneven dim-0 params (e.g. shared_expert_gate.weight [1, hidden]), moe / vlmoe parallelize prefer Shard(d>0) along the first divisible non-zero dim via shard_placement_fn, falling back to replicate_params only when no non-zero dim is divisible. This avoids to_sharded()'s copy-back path (added by upstream fix for in-place forward writes on replicate state) when forward is read-only, which otherwise costs +/-1 bf16 ULP/step on NPU. Self-consistency (HP 1c vs HP FSDP4, avg_loss): - dense: 20/20 byte-equal - moe: 20/20 byte-equal (after Shard(d>0) preference) - vlmoe: 11/20 byte-equal + 9/20 sub-ULP <=2e-3 (NPU vision-tower fp noise) Cross-framework (HP-self vs LF + HF-native + HP-backend, FSDP4): - dense: 20/20 byte-equal - moe: 17/20 byte-equal + 3/20 1-bf16-ULP (LF integration kept conservative to preserve correctness for HF models that may in-place write replicate state in forward; root-cause is upstream's unconditional copy in fully_shard, follow-up to make it conditional via autograd version counter) - vlmoe: out-of-scope (NPU bf16 noise floor + LF compute_loss_func) LF integration provides Accelerate FSDP2 compat layer with auto replicate_params for shape[0]<shard_size weights -- kept conservative on purpose to preserve correctness for arbitrary HF models. Training-correctness fixes: - train() epoch loop now uses while global_step < max_steps driven by the live counter; multi-epoch + resume runs no longer exceed max_steps. - train_step increments global_step AFTER next(data_iterator) so StopIteration does not bump checkpoint / log indices. - rank_average loss reporting divides by the DP-group size instead of global world_size; previously TP / EP / PP / CP > 1 systematically under-reported the logged loss. Code quality: trainer + model files use module-top imports for stdlib / torch / hyper internals; optional deps (datasets / transformers / safetensors / torchdata / tqdm) and torch_npu remain lazy with # pylint: disable=C0415 annotations. Forward/backward and interface-conformance methods carry # pylint: disable=W0613. _materialize_and_init_shards / _load_weights split into single-responsibility helpers (each <= 10 CCN, <= 40 NLOC). Latent bug in _maybe_(up|down)cast_*_params fixed: the isinstance check used torch's DTensor instead of hyper's, so the rebuild branch never ran. Known limitations (P2-P4 stub): EP / PP / CP fail-fast; streaming / packing / dynamic batching not implemented; Profiler / W&B / TB / MemoryMonitor / Eval / MoEMonitor callbacks reduced to stubs; Float8 / ChunkedCELoss not wired. | 5 个月前 | |
feat(trainer+models): training stack + qwen3.5 dense/moe/vl-moe aligned Add an in-tree training stack as part of HyperParallel: ModelSpec contract (build_model_fn / parallelize_fn / state_dict_adapter / clip_grad_fn) + 3-layer YAML (model/data/train) + DeviceMesh aliases (fsdp/dp/loss). Per-model parallelize.py owns the full AC / FSDP / Prefetch composition. Three Qwen3.5 models registered (folder names follow HF transformers convention): - hyper_parallel/models/qwen3_5/ (dense) - hyper_parallel/models/qwen3_5_moe/ (MoE w/ SharedExpertMoE) - hyper_parallel/models/qwen3_vl_moe/ (VL-MoE w/ vision tower + DeepStack) For uneven dim-0 params (e.g. shared_expert_gate.weight [1, hidden]), moe / vlmoe parallelize prefer Shard(d>0) along the first divisible non-zero dim via shard_placement_fn, falling back to replicate_params only when no non-zero dim is divisible. This avoids to_sharded()'s copy-back path (added by upstream fix for in-place forward writes on replicate state) when forward is read-only, which otherwise costs +/-1 bf16 ULP/step on NPU. Self-consistency (HP 1c vs HP FSDP4, avg_loss): - dense: 20/20 byte-equal - moe: 20/20 byte-equal (after Shard(d>0) preference) - vlmoe: 11/20 byte-equal + 9/20 sub-ULP <=2e-3 (NPU vision-tower fp noise) Cross-framework (HP-self vs LF + HF-native + HP-backend, FSDP4): - dense: 20/20 byte-equal - moe: 17/20 byte-equal + 3/20 1-bf16-ULP (LF integration kept conservative to preserve correctness for HF models that may in-place write replicate state in forward; root-cause is upstream's unconditional copy in fully_shard, follow-up to make it conditional via autograd version counter) - vlmoe: out-of-scope (NPU bf16 noise floor + LF compute_loss_func) LF integration provides Accelerate FSDP2 compat layer with auto replicate_params for shape[0]<shard_size weights -- kept conservative on purpose to preserve correctness for arbitrary HF models. Training-correctness fixes: - train() epoch loop now uses while global_step < max_steps driven by the live counter; multi-epoch + resume runs no longer exceed max_steps. - train_step increments global_step AFTER next(data_iterator) so StopIteration does not bump checkpoint / log indices. - rank_average loss reporting divides by the DP-group size instead of global world_size; previously TP / EP / PP / CP > 1 systematically under-reported the logged loss. Code quality: trainer + model files use module-top imports for stdlib / torch / hyper internals; optional deps (datasets / transformers / safetensors / torchdata / tqdm) and torch_npu remain lazy with # pylint: disable=C0415 annotations. Forward/backward and interface-conformance methods carry # pylint: disable=W0613. _materialize_and_init_shards / _load_weights split into single-responsibility helpers (each <= 10 CCN, <= 40 NLOC). Latent bug in _maybe_(up|down)cast_*_params fixed: the isinstance check used torch's DTensor instead of hyper's, so the rebuild branch never ran. Known limitations (P2-P4 stub): EP / PP / CP fail-fast; streaming / packing / dynamic batching not implemented; Profiler / W&B / TB / MemoryMonitor / Eval / MoEMonitor callbacks reduced to stubs; Float8 / ChunkedCELoss not wired. | 5 个月前 | |
[fix][fully_shard] adjust prefetch, unshard position | 4 个月前 | |
| 4 个月前 | ||
feat(context_parallel): add async DSA context parallel support | 5 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 4 个月前 | ||
| 7 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 5 个月前 |