| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Merge upstream/master into trainer_dev Bring master's content into trainer_dev, resolving conflicts between master's platform-abstraction removal and resolver API naming (normalize_value / annotation / ConfigResolutionError) and trainer_dev's omni data lifecycle, runtime input adapter, DSA attention support, and model integration diagnostics. | 1 个月前 | |
feat(trainer): wire optimizer-state swap into the optimizer build The swap runtime (core/optimizer/swap_optimizer.py + swap_optimizer_base.py) already exists, but nothing in the trainer could turn it on: the optimizer section's swap key was passed to the optimizer target constructor, and a built optimizer was never wrapped. * config/optimization.py: OptimizerSwapConfig (enabled, swap_times, min_numel, state_keys, include_master_params, packed_swap) as OptimizerConfig.swap, and its to_dict serialization. * config/resolver.py: pop swap out of the optimizer target node and resolve it as OptimizerSwapConfig, exactly like the existing fp32_main_params policy. * trainer/base.py: map the trainer section onto the core SwapOptimizerConfig and wrap every Adam/AdamW leaf of the built (optionally mixed-precision) optimizer after the fp32 main-parameter wrap. * tests/ut/trainer/test_optimizer_swap.py: 14 cases covering the config surface, the build path, and the core interaction. Measured need: at edp=2 (2 supernodes) AdamW state is what pushes the step over the memory ceiling -- 56.59 GB without swap versus 41.49 GB with it. | 1 个月前 | |
Merge upstream/master into trainer_dev Bring master's content into trainer_dev, resolving conflicts between master's platform-abstraction removal and resolver API naming (normalize_value / annotation / ConfigResolutionError) and trainer_dev's omni data lifecycle, runtime input adapter, DSA attention support, and model integration diagnostics. | 1 个月前 | |
refactor!: reorganize auto_models into top-level models/components/distributed/trainer/data Squashed 58 commits of the AutoModels directory restructure (M0-M5, S4a-S8d, top-level split): - characterization tests first (M0): UT collection baseline, public API contract snapshots, build-order golden cases, shared fixtures, training YAML snapshots - distributed split (S4a-S4g): sharding_config/planner/applier/injection split into distributed/{plan,recipe_spec,apply,compile, activation_checkpoint} + private _builder/; CP/EP into context_parallel/ and expert_parallel/; fsdp2 into fsdp_adapter + source_shard - models adapter layer (M1-M5): adapter_spec/registry skeleton, Qwen3-MoE replacements and CP/EP contracts migrated into models/qwen3_moe/adapter, single recipe converged onto models/qwen3_moe/recipes/train.yaml, legacy fusion files deleted; per-family sharding_rules declared via ModelAdapterSpec with directory-convention auto-discovery registration - build pipeline (S5a-S5g): checkpoint finalization into checkpoint_loader, plan DTO boundary normalized, god module split into model_builder, registry split into config_resolver + models/registry - data stack (S6a-S6f): indexed/text/online/batching/parallel/vlm/tools migrated to top-level hyper_parallel/data - trainer (S7a-S7e): config tree/callbacks/runtime/base+text+vlm migrated to top-level hyper_parallel/trainer, utils split eliminated - closeout (S8a-S8d): legacy core/shard API, legacy hyper_parallel/models and components compatibility package deleted; import boundary gates - top-level split: auto_models/ dissolved into models/ (adapter layer + api/build_options/replacement/_transformers, external interface), components/ (modules/functional/quantization/losses/optim/checkpoint) and distributed/; references rewritten across py/yaml/sh and setup.py; boundary gates updated (distributed -> models limited to the build_options leaf DTO plus the planner's function-level registry lookup) Verified: full UT 4562 passed / 39 failed (identical to the pre-refactor environment-failure baseline), characterization suites green. | 1 个月前 | |
feat(trainer): wire optimizer-state swap into the optimizer build The swap runtime (core/optimizer/swap_optimizer.py + swap_optimizer_base.py) already exists, but nothing in the trainer could turn it on: the optimizer section's swap key was passed to the optimizer target constructor, and a built optimizer was never wrapped. * config/optimization.py: OptimizerSwapConfig (enabled, swap_times, min_numel, state_keys, include_master_params, packed_swap) as OptimizerConfig.swap, and its to_dict serialization. * config/resolver.py: pop swap out of the optimizer target node and resolve it as OptimizerSwapConfig, exactly like the existing fp32_main_params policy. * trainer/base.py: map the trainer section onto the core SwapOptimizerConfig and wrap every Adam/AdamW leaf of the built (optionally mixed-precision) optimizer after the fp32 main-parameter wrap. * tests/ut/trainer/test_optimizer_swap.py: 14 cases covering the config surface, the build path, and the core interaction. Measured need: at edp=2 (2 supernodes) AdamW state is what pushes the step over the memory ceiling -- 56.59 GB without swap versus 41.49 GB with it. | 1 个月前 | |
refactor!: reorganize auto_models into top-level models/components/distributed/trainer/data Squashed 58 commits of the AutoModels directory restructure (M0-M5, S4a-S8d, top-level split): - characterization tests first (M0): UT collection baseline, public API contract snapshots, build-order golden cases, shared fixtures, training YAML snapshots - distributed split (S4a-S4g): sharding_config/planner/applier/injection split into distributed/{plan,recipe_spec,apply,compile, activation_checkpoint} + private _builder/; CP/EP into context_parallel/ and expert_parallel/; fsdp2 into fsdp_adapter + source_shard - models adapter layer (M1-M5): adapter_spec/registry skeleton, Qwen3-MoE replacements and CP/EP contracts migrated into models/qwen3_moe/adapter, single recipe converged onto models/qwen3_moe/recipes/train.yaml, legacy fusion files deleted; per-family sharding_rules declared via ModelAdapterSpec with directory-convention auto-discovery registration - build pipeline (S5a-S5g): checkpoint finalization into checkpoint_loader, plan DTO boundary normalized, god module split into model_builder, registry split into config_resolver + models/registry - data stack (S6a-S6f): indexed/text/online/batching/parallel/vlm/tools migrated to top-level hyper_parallel/data - trainer (S7a-S7e): config tree/callbacks/runtime/base+text+vlm migrated to top-level hyper_parallel/trainer, utils split eliminated - closeout (S8a-S8d): legacy core/shard API, legacy hyper_parallel/models and components compatibility package deleted; import boundary gates - top-level split: auto_models/ dissolved into models/ (adapter layer + api/build_options/replacement/_transformers, external interface), components/ (modules/functional/quantization/losses/optim/checkpoint) and distributed/; references rewritten across py/yaml/sh and setup.py; boundary gates updated (distributed -> models limited to the build_options leaf DTO plus the planner's function-level registry lookup) Verified: full UT 4562 passed / 39 failed (identical to the pre-refactor environment-failure baseline), characterization suites green. | 1 个月前 | |
Merge upstream/master into trainer_dev Bring master's content into trainer_dev, resolving conflicts between master's platform-abstraction removal and resolver API naming (normalize_value / annotation / ConfigResolutionError) and trainer_dev's omni data lifecycle, runtime input adapter, DSA attention support, and model integration diagnostics. | 1 个月前 | |
feat(models): add the Kimi-K2.6 multimodal family (kimi_k26) Adapt the Kimi-K2.6 multimodal family onto trainer_dev under the registered identity kimi_k26: * models/kimi_k26/: family adapter (registration with MLA/vision sharding rules, chunk-loss forward adapter, context-parallel input sharding, model builder), family README and the 4-supernode recipe with its multi-node launch script and demo-data generator. * data/omni/kimi_transform.py: Omni transform for the native processor's image_grid_thw protocol (eager padding and token-budget packing). * registry: map the checkpoint's native kimi_k25 spelling onto kimi_k26 via _FAMILY_DIR_ALIASES, so model_type, architecture and directory spellings all resolve through get_model_adapter(). * trainer/vlm_trainer.py: forward model.trust_remote_code to the Omni model_assets processor build so the native implementation is used. * UT: family adapter contracts (chunk loss, CP input) plus the Kimi transform (dynamic padding, image budget, truncation, CP batches). The recipe's chunk loss requires tp_size=1: the vocabulary-sharded chunk loss is not part of this change, and the core loss rejects tp_size != 1 at bind time. EP archetype parity: kimi_k26_ep_compute_fn now forwards overlap_shared_expert and capacity_factor to build_ep_compute, and the recipe turns the shared-expert overlap on. Without it the family entry point built a non-overlapped plan: the 2-SN ladder on a merge of these five PRs peaked at 43.77 GB / 25.03 s at GBS 256 instead of the recorded 41.49 GB / 20.50 s. | 30 天前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 30 天前 |