hyper-parallel/docs · david-he91/hyper-parallel

Ddavid-he91refactor: remove legacy custom ops feature

Files	Last commit	Last update
api	refactor: remove legacy custom ops feature	1 month ago
contributing	refactor: remove legacy custom ops feature	1 month ago
guide	refactor: remove legacy custom ops feature	1 month ago
images	docs: refresh project and AutoModels guides	2 months ago
agent_workflow.md	docs: correct API and workflow details in faq and agent_workflow	4 months ago
faq.md	refactor: remove legacy custom ops feature	1 month ago
feature_status.md	refactor: remove legacy custom ops feature	1 month ago
index.md	refactor: remove legacy custom ops feature	1 month ago
installation.md	refactor: remove legacy custom ops feature	1 month ago
mpipe_transpose.md	feat(mpipe): MPipe support for Qwen3-VL MPipe support for Qwen3-VL: the MPipe Transpose pipeline schedule and its integration into the existing Qwen3-VL-MoE model — everything needed to run the Qwen3-VL-30B-A3B demo, and nothing more. Schedule: under PP the preprocess block (for Qwen3-VL the frozen visual tower) is transposed across pipeline ranks — each rank encodes its own micro-batch during the warmup bubble and ships the features to stage 0, shrinking the bubble. Includes round-robin distribution of M > NT overflow micros (full/min knob), opt-in owner-does-backward for a trainable transposed tower, in-schedule dataload (data.load: single, with a per-PP-rank sampler), interleaved (VPP) support including the shallow-warmup pp_less_memory variant, and the trainer/config wiring this needs (pp_schedule: mpipe and its knobs, the PP dataload flavors, activation checkpointing honored under pp>1). Both platform backends track the executor interface; only the torch backend ships tests — MindSpore mpipe is untested and claims no accuracy. Qwen3-VL integration (built on the existing model): explicit pp_layer_split stage builder with materialize/lazy-init fixes so PP+FSDP works at full model depth, MPipe visual-payload injection with a fail-fast guard for ragged pixel/grid splits, activation checkpointing over the trainable visual tower with recompute-depth knobs (num_layers / visual_num_layers), the AC wrap guard for the shared parameter-free rotary embedding, Qwen3-VL-specific FSDP all-gather/ compute overlap (fsdp_forward_prefetch / fsdp_backward_prefetch), and get_device in the DTensor dispatch whitelist. Demo config: examples/qwen3_vl_30b_a3b_instruct/train_pp_mpipe.yaml — the cluster-validated 128-die shape: full 48-layer model, pp=8 x dp_shard=16, 16 micro-batches with round-robin overflow, data.load: single, bf16, FSDP forward prefetch depth 1, per-stage recompute depths and the less-memory warmup. Scope: the Qwen3-VL-only wirings (AC under pp>1, pp_layer_split, FSDP prefetch, pp_less_memory) are deliberate demo-focused implementations, temporary until proper general ones land on master; other models under pp>1 silently ignore activation_checkpoint (master rejected it). data.load: single works with any PP schedule (non-mpipe schedules load on stage 0 and ship raw inputs stage-to-stage) and requires gbs == pp_micro_batch_num x micro_batch_size x dp (no gradient accumulation) and single-epoch runs; the visual transpose expects all-image, sample-uniform micro-batches. Code-check fixes folded in: the visual-AC unit test no longer leaks transformers' logger rebinding and lazy-module state into the rest of tests/ut, which had turned six previously-green tests red; the pp>1 activation_checkpoint guard is narrowed via a ModelSpec capability flag rather than dropped, so qwen3_5 and qwen3_5_moe fail loudly instead of silently skipping recompute; the dataload-only (T=0) suppression of the inline MPIPE_TRANSPOSE_FWD is restored with its regression test; pylint is back to the master baseline over the changed files; mid-function comments are held to two lines; and the six functions above the lizard CCN/NLOC thresholds are declared in whitelizard.txt. Co-authored-by: Nelson Lossing <nelson.lossing@huawei.com> Co-authored-by: Thibaut Tachon <thibaut.tachon@huawei.com> Co-authored-by: Etienne Fihol <etienne.fihol@huawei.com> Co-authored-by: Pierre Leca <pierre.leca@huawei.com>	2 months ago