| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Resolve #2099 Add GLM5 trainer integration | 3 个月前 | |
refactor: behavior-neutral code cleanup Behavior-neutral cleanups across the trainer and models — no functional change: relocate module __all__ above imports and move the entry-script deferred imports to the top; mark the instance-free StateDictAdapter load/save methods and _make_amp_context as @staticmethod; declare BaseTrainer attributes at class level (switching the sampler / _last_global_tokens reads to is-None / getattr fallbacks); use dict.get for guarded lookups; return a namedtuple from _build_grad_groups; extract _compute_shard_overrides; minor docstring/formatting. Callback lifecycle hooks stay instance methods (overridable interface — keeping them consistent with subclass overrides). | 4 个月前 | |
feat: support Qwen3.5 and Qwen3-VL-MoE training Add dense, MoE, and vision-language model integration with checkpoint conversion, data loading, and unified trainer execution. Correct TP, CP, EP, PP, FSDP, and HSDP synchronization semantics so supported BF16 single-card and hybrid-parallel configurations follow the same training behavior. | 3 个月前 | |
feat: support Qwen3.5 and Qwen3-VL-MoE training Add dense, MoE, and vision-language model integration with checkpoint conversion, data loading, and unified trainer execution. Correct TP, CP, EP, PP, FSDP, and HSDP synchronization semantics so supported BF16 single-card and hybrid-parallel configurations follow the same training behavior. | 3 个月前 | |
feat: support Qwen3.5 and Qwen3-VL-MoE training Add dense, MoE, and vision-language model integration with checkpoint conversion, data loading, and unified trainer execution. Correct TP, CP, EP, PP, FSDP, and HSDP synchronization semantics so supported BF16 single-card and hybrid-parallel configurations follow the same training behavior. | 3 个月前 | |
feat: support Qwen3.5 and Qwen3-VL-MoE training Add dense, MoE, and vision-language model integration with checkpoint conversion, data loading, and unified trainer execution. Correct TP, CP, EP, PP, FSDP, and HSDP synchronization semantics so supported BF16 single-card and hybrid-parallel configurations follow the same training behavior. | 3 个月前 | |
feat: support Qwen3.5 and Qwen3-VL-MoE training Add dense, MoE, and vision-language model integration with checkpoint conversion, data loading, and unified trainer execution. Correct TP, CP, EP, PP, FSDP, and HSDP synchronization semantics so supported BF16 single-card and hybrid-parallel configurations follow the same training behavior. | 3 个月前 | |
feat(trainer+models): training stack + qwen3.5 dense/moe/vl-moe aligned Add an in-tree training stack as part of HyperParallel: ModelSpec contract (build_model_fn / parallelize_fn / state_dict_adapter / clip_grad_fn) + 3-layer YAML (model/data/train) + DeviceMesh aliases (fsdp/dp/loss). Per-model parallelize.py owns the full AC / FSDP / Prefetch composition. Three Qwen3.5 models registered (folder names follow HF transformers convention): - hyper_parallel/models/qwen3_5/ (dense) - hyper_parallel/models/qwen3_5_moe/ (MoE w/ SharedExpertMoE) - hyper_parallel/models/qwen3_vl_moe/ (VL-MoE w/ vision tower + DeepStack) For uneven dim-0 params (e.g. shared_expert_gate.weight [1, hidden]), moe / vlmoe parallelize prefer Shard(d>0) along the first divisible non-zero dim via shard_placement_fn, falling back to replicate_params only when no non-zero dim is divisible. This avoids to_sharded()'s copy-back path (added by upstream fix for in-place forward writes on replicate state) when forward is read-only, which otherwise costs +/-1 bf16 ULP/step on NPU. Self-consistency (HP 1c vs HP FSDP4, avg_loss): - dense: 20/20 byte-equal - moe: 20/20 byte-equal (after Shard(d>0) preference) - vlmoe: 11/20 byte-equal + 9/20 sub-ULP <=2e-3 (NPU vision-tower fp noise) Cross-framework (HP-self vs LF + HF-native + HP-backend, FSDP4): - dense: 20/20 byte-equal - moe: 17/20 byte-equal + 3/20 1-bf16-ULP (LF integration kept conservative to preserve correctness for HF models that may in-place write replicate state in forward; root-cause is upstream's unconditional copy in fully_shard, follow-up to make it conditional via autograd version counter) - vlmoe: out-of-scope (NPU bf16 noise floor + LF compute_loss_func) LF integration provides Accelerate FSDP2 compat layer with auto replicate_params for shape[0]<shard_size weights -- kept conservative on purpose to preserve correctness for arbitrary HF models. Training-correctness fixes: - train() epoch loop now uses while global_step < max_steps driven by the live counter; multi-epoch + resume runs no longer exceed max_steps. - train_step increments global_step AFTER next(data_iterator) so StopIteration does not bump checkpoint / log indices. - rank_average loss reporting divides by the DP-group size instead of global world_size; previously TP / EP / PP / CP > 1 systematically under-reported the logged loss. Code quality: trainer + model files use module-top imports for stdlib / torch / hyper internals; optional deps (datasets / transformers / safetensors / torchdata / tqdm) and torch_npu remain lazy with # pylint: disable=C0415 annotations. Forward/backward and interface-conformance methods carry # pylint: disable=W0613. _materialize_and_init_shards / _load_weights split into single-responsibility helpers (each <= 10 CCN, <= 40 NLOC). Latent bug in _maybe_(up|down)cast_*_params fixed: the isinstance check used torch's DTensor instead of hyper's, so the rebuild branch never ran. Known limitations (P2-P4 stub): EP / PP / CP fail-fast; streaming / packing / dynamic batching not implemented; Profiler / W&B / TB / MemoryMonitor / Eval / MoEMonitor callbacks reduced to stubs; Float8 / ChunkedCELoss not wired. | 5 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 个月前 | ||
| 4 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 5 个月前 |