| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
refactor: 去除 platform 抽象，改用 torch 原生接口 删除 hyper_parallel/platform/ 整个双后端抽象层，相关实现直接使用 torch 原生接口；仅保留的 Torch-only 组件迁至 components/。 主要变更： - 删除 hyper_parallel/platform/**（mindspore 后端、platform.py、 swap_optimizer、activation_checkpoint、loss_parallel_ops、 function_override、init_weights 等） - hyper_parallel/platform/torch/common/moe.py 重命名至 hyper_parallel/components/modules/moe.py（内容不变），并同步更新 examples/torch、tests/torch、核心模块中的 20 余处 import - hyper_parallel/__init__.py 移除 get_platform 导出 - destroy_process_group 中 _P2P_MULTI_STREAM_GROUPS 改从 hyper_parallel.core.pipeline_parallel._p2p 导入 - 删除 examples/mindspore/**、tests/ut/platform/** - scripts/pylint_hyperparallel.py：PLATFORM_ALLOWED_PARTS 移除 hyper_parallel/platform/；删除 instance-platform-assignment (C9003) 规则 - 更新 .agent 规则、docs、docker、CODEOWNERS 等配套内容 327 files changed, 648 insertions(+), 20321 deletions(-) | 1 个月前 | |
chore: purge MindSpore residue after backend removal The platform abstraction layer (and with it the MindSpore backend) was deleted earlier, but references to it survived throughout the tree. This removes them; it changes no behavior on the surviving Torch path. Removed: - docker/Dockerfile.mindspore, the MindSpore build branch and examples in docker/build_hyper-parallel_npu.sh / docker/README.md - .jenkins/test/config/dependent_packages.yaml (a single MindSpore wheel URL with no remaining consumer) - MindSporeCase in tests/common/parallel_case.py and msrun_case in tests/common/distributed_launcher.py, plus msrun / MS_NODE_ID rank-discovery fallbacks - dead _MS_PRIMITIVE_OP_NAMES branches in the shard ops (none of those op names are registered in any yaml) and their tests - the stale aclop packed-args notes in parallel_elementwise.py / parallel_gather.py (_normalize_aclop_args no longer exists) - three pylint errors the Atomgit gate reports on files this commit already touches: an unused os import in _pytorch_reference.py, a missing docstring on NoAcceleratorGuard._check_init_process_group and trailing blank lines in test_parallel_topk.py - mindspore from the multicore boundary test, pylint's generated-members, from the autogit lint backend list and from the docs, examples and .agent rules Kept on purpose: - _MS_POSITIONAL_DTYPE_OP_NAMES in parallel_reduce.py — SumExt / MeanExt are live torch registrations in torch_reduce.yaml, so the trailing positional dtype slot is still reachable - test_runtime_sources_import_no_platform_layer — the boundary test still asserts production sources import no Platform layer - URLs under the project's own mindspore.org / gitcode / atomgit namespaces, including the links to the MindSpore repo's contribution rules - hyper_parallel_v1.0.0_release_notes.md (historical) | 1 个月前 | |
feat(distributed): add encoder parallelism support for deepseek_v41 VLM Introduce a dedicated encoder_parallel module (mesh, config, contracts, planning, exchange, attention, runtime) that shards ViT encoder computation across ranks and exchanges image token/metadata with the LLM side, and wire it through the deepseek_v41 adapter, planner, forward rewriter, batching, and VLM trainer. - Add encoder_parallel core: Ulysses-style attention context, work-item contracts, planning/merge, and runtime exchange of image fields - Bind deepseek_v41 vision attention into the encoder-parallel path via a new adapter distributed module and registration entries - Extend planner/forward_rewriter and parallelism config to schedule encoder work items alongside existing parallelisms - Adjust engram module and MoE-engram expert parallel for the new path - Add unit and torch-distributed tests plus a VLM encoder-parallel training example, and update skill references style: satisfy gate pylint, lizard, and markdownlint checks - Type EncoderAttentionContext.cp_mesh as DeviceMesh and drop trailing blank lines in encoder_parallel/attention.py - Add full type hints to rewrap_declared_outputs and to the nested reduce_sum helper in test_precision.py - Split compare_matrix, run_vlm_matrix main, and test_e0_runtime to bring CCN/NLOC under the gate lizard limits - Add blank lines around a fenced block in code-review SKILL.md test(distributed): package encoder_parallel UT dir to fix pytest module collision tests/ut/distributed/encoder_parallel/test_runtime.py collided with tests/ut/core/multicore/shmem/test_runtime.py during gate collection (import file mismatch: both import as top-level test_runtime). Add an __init__.py so the directory is collected as the encoder_parallel package, matching tests/torch/encoder_parallel. | 15 天前 | |
chore: purge MindSpore residue after backend removal The platform abstraction layer (and with it the MindSpore backend) was deleted earlier, but references to it survived throughout the tree. This removes them; it changes no behavior on the surviving Torch path. Removed: - docker/Dockerfile.mindspore, the MindSpore build branch and examples in docker/build_hyper-parallel_npu.sh / docker/README.md - .jenkins/test/config/dependent_packages.yaml (a single MindSpore wheel URL with no remaining consumer) - MindSporeCase in tests/common/parallel_case.py and msrun_case in tests/common/distributed_launcher.py, plus msrun / MS_NODE_ID rank-discovery fallbacks - dead _MS_PRIMITIVE_OP_NAMES branches in the shard ops (none of those op names are registered in any yaml) and their tests - the stale aclop packed-args notes in parallel_elementwise.py / parallel_gather.py (_normalize_aclop_args no longer exists) - three pylint errors the Atomgit gate reports on files this commit already touches: an unused os import in _pytorch_reference.py, a missing docstring on NoAcceleratorGuard._check_init_process_group and trailing blank lines in test_parallel_topk.py - mindspore from the multicore boundary test, pylint's generated-members, from the autogit lint backend list and from the docs, examples and .agent rules Kept on purpose: - _MS_POSITIONAL_DTYPE_OP_NAMES in parallel_reduce.py — SumExt / MeanExt are live torch registrations in torch_reduce.yaml, so the trailing positional dtype slot is still reachable - test_runtime_sources_import_no_platform_layer — the boundary test still asserts production sources import no Platform layer - URLs under the project's own mindspore.org / gitcode / atomgit namespaces, including the links to the MindSpore repo's contribution rules - hyper_parallel_v1.0.0_release_notes.md (historical) | 1 个月前 | |
chore: purge MindSpore residue after backend removal The platform abstraction layer (and with it the MindSpore backend) was deleted earlier, but references to it survived throughout the tree. This removes them; it changes no behavior on the surviving Torch path. Removed: - docker/Dockerfile.mindspore, the MindSpore build branch and examples in docker/build_hyper-parallel_npu.sh / docker/README.md - .jenkins/test/config/dependent_packages.yaml (a single MindSpore wheel URL with no remaining consumer) - MindSporeCase in tests/common/parallel_case.py and msrun_case in tests/common/distributed_launcher.py, plus msrun / MS_NODE_ID rank-discovery fallbacks - dead _MS_PRIMITIVE_OP_NAMES branches in the shard ops (none of those op names are registered in any yaml) and their tests - the stale aclop packed-args notes in parallel_elementwise.py / parallel_gather.py (_normalize_aclop_args no longer exists) - three pylint errors the Atomgit gate reports on files this commit already touches: an unused os import in _pytorch_reference.py, a missing docstring on NoAcceleratorGuard._check_init_process_group and trailing blank lines in test_parallel_topk.py - mindspore from the multicore boundary test, pylint's generated-members, from the autogit lint backend list and from the docs, examples and .agent rules Kept on purpose: - _MS_POSITIONAL_DTYPE_OP_NAMES in parallel_reduce.py — SumExt / MeanExt are live torch registrations in torch_reduce.yaml, so the trailing positional dtype slot is still reachable - test_runtime_sources_import_no_platform_layer — the boundary test still asserts production sources import no Platform layer - URLs under the project's own mindspore.org / gitcode / atomgit namespaces, including the links to the MindSpore repo's contribution rules - hyper_parallel_v1.0.0_release_notes.md (historical) | 1 个月前 | |
test: 清理 MindSpore 用例及悬空引用 后续只支持 torch 平台，移除 MindSpore 测试资产： - 删除 tests/mindspore/ 整个 ST 目录（112 文件） - 删除 tests/ut/platform/mindspore/ 下的 MindSpore UT（保留 _ensure_mindspore_platform.py，UT 仍依赖其 restore_torch_platform_for_ut） - 清理悬空引用：shard_ops 框架去掉 mindspore 的 _FW_CFG 与 --framework 选项、 CODEOWNERS、.jenkins pylint 过滤规则、AGENTS.md 与 .agent 规则/技能文档 - autogit 的 ST gate 目标从不存在的 tests/torch/st、tests/mindspore/st 改为 tests/torch - 顺带修掉 pre-commit 拦截的问题：orch ST 用例补 docstring 与 wrong-import-position 抑制，markdown 补空行/尾换行/表格列数 | 1 个月前 | |
feat(distributed): add encoder parallelism support for deepseek_v41 VLM Introduce a dedicated encoder_parallel module (mesh, config, contracts, planning, exchange, attention, runtime) that shards ViT encoder computation across ranks and exchanges image token/metadata with the LLM side, and wire it through the deepseek_v41 adapter, planner, forward rewriter, batching, and VLM trainer. - Add encoder_parallel core: Ulysses-style attention context, work-item contracts, planning/merge, and runtime exchange of image fields - Bind deepseek_v41 vision attention into the encoder-parallel path via a new adapter distributed module and registration entries - Extend planner/forward_rewriter and parallelism config to schedule encoder work items alongside existing parallelisms - Adjust engram module and MoE-engram expert parallel for the new path - Add unit and torch-distributed tests plus a VLM encoder-parallel training example, and update skill references style: satisfy gate pylint, lizard, and markdownlint checks - Type EncoderAttentionContext.cp_mesh as DeviceMesh and drop trailing blank lines in encoder_parallel/attention.py - Add full type hints to rewrap_declared_outputs and to the nested reduce_sum helper in test_precision.py - Split compare_matrix, run_vlm_matrix main, and test_e0_runtime to bring CCN/NLOC under the gate lizard limits - Add blank lines around a fenced block in code-review SKILL.md test(distributed): package encoder_parallel UT dir to fix pytest module collision tests/ut/distributed/encoder_parallel/test_runtime.py collided with tests/ut/core/multicore/shmem/test_runtime.py during gate collection (import file mismatch: both import as top-level test_runtime). Add an __init__.py so the directory is collected as the encoder_parallel package, matching tests/torch/encoder_parallel. | 15 天前 | |
feat(deepseek_v41): add CPU-resident sparse Engram table with host-side offload - Add engram host table/lookup/replacement/state modules under the deepseek_v41 adapter - Add host sparse grad and host sparse optimizer support - Add host sidecar checkpoint save/restore and external state plumbing - Wire host offload through trainer, checkpoint callback, optim builders, parameter groups, clip_grad, fsdp adapter, and sharding policies - Add training demo configs and prepare script for host online mode - Add unit and torch tests for host table, engram sparse, and optimizer restore | 13 天前 | |
chore: purge MindSpore residue after backend removal The platform abstraction layer (and with it the MindSpore backend) was deleted earlier, but references to it survived throughout the tree. This removes them; it changes no behavior on the surviving Torch path. Removed: - docker/Dockerfile.mindspore, the MindSpore build branch and examples in docker/build_hyper-parallel_npu.sh / docker/README.md - .jenkins/test/config/dependent_packages.yaml (a single MindSpore wheel URL with no remaining consumer) - MindSporeCase in tests/common/parallel_case.py and msrun_case in tests/common/distributed_launcher.py, plus msrun / MS_NODE_ID rank-discovery fallbacks - dead _MS_PRIMITIVE_OP_NAMES branches in the shard ops (none of those op names are registered in any yaml) and their tests - the stale aclop packed-args notes in parallel_elementwise.py / parallel_gather.py (_normalize_aclop_args no longer exists) - three pylint errors the Atomgit gate reports on files this commit already touches: an unused os import in _pytorch_reference.py, a missing docstring on NoAcceleratorGuard._check_init_process_group and trailing blank lines in test_parallel_topk.py - mindspore from the multicore boundary test, pylint's generated-members, from the autogit lint backend list and from the docs, examples and .agent rules Kept on purpose: - _MS_POSITIONAL_DTYPE_OP_NAMES in parallel_reduce.py — SumExt / MeanExt are live torch registrations in torch_reduce.yaml, so the trailing positional dtype slot is still reachable - test_runtime_sources_import_no_platform_layer — the boundary test still asserts production sources import no Platform layer - URLs under the project's own mindspore.org / gitcode / atomgit namespaces, including the links to the MindSpore repo's contribution rules - hyper_parallel_v1.0.0_release_notes.md (historical) | 1 个月前 | |
feat(auto_parallel): add the sapp-parallel-strategy and the timing round it plans ND prices every valid layout of a job off the device. This adds the packaging that lets somebody ask it a question, the tools that measure the few layouts worth measuring, and a report of what more recompute would cost. The skill, under .agent/skills/sapp-parallel-strategy, turns a request into ND's inputs, runs ND once, and reads the ranking back: the layout to launch, the next two, why, and what the record says may be claimed about them. Templates for the Qwen3.5-MoE family and for the deepseek_v41 validation crop ship with it, with the model files each needs. One command installs it: python .agent/skills/sapp-parallel-strategy/install.py It links the skill into the folders assistants read, says what the checkout can price, and prints a first command. --check reports without changing anything, --copy copies where links cannot be made, and a copy keeps the path of its checkout beside it, which the scripts read. For the deepseek_v41 crop the round that times layouts is now in the tree: a preflight that reads a host before a window and launches nothing, a ladder tool that prints the points no round has measured and the command that runs them, the repeat script that launches each point several times with no profiler and writes one row a step, and a describe tool that states what the launcher's factory builds as facts ND can read and verifies them against the tensors it builds. The round reads its own launcher, nd_train_deepseek_v41_timing.yaml. It trains what the released launcher trains, with checkpointing off, the grouped expert path off and no drafter overrides, which is what ND prices: with the released launcher the skill refuses to write a round's command, because a run that saves its state and takes the grouped path is not the job that was priced. The repeat script takes the yaml as LAUNCHER, so another job can be timed deliberately, and a unit test holds the shipped launcher and the template together. New in the search itself, behind -rp: for every configuration that fits, the plans that fully recompute one more layer at the same degrees and the same memory budget, written as a CSV and two comparison plots, with the recompute mode and the layers that depart from it now shown in the results table. Every figure there is the cost model's estimate of a step and not a measurement, and a measured round can disagree with it. The flag is off by default and the search does none of that work without it, since pricing those plans costs roughly thirteen times the search. A plan still states the mode that recomputes and names the layers that depart from it, because the other way round is a plan that a sweep forcing the mode silently turns into recomputing everything. Three fixes in ND itself: a run's debug CSVs start afresh instead of growing across runs, an offloading stage is trimmed to the whole megabytes its budget holds, and the census spec builds from the configuration cropped to the number of layers a request states. Ninety-three comment and docstring lines in ND named an item in a tracker outside this repository, as "(I12)". The sentence around each one says what it has to say without the tag, so the tags are gone; the test-case ids of the form BD-H07 are a name for the case itself and stay. Checked: the auto-parallel and skill unit tests pass, with the one failure master also has, and so do they from a copy of the tests tree beside an installed package; the golden harness reports all 23157 numbers identical to the tree before this change, so nothing it prices moved; pylint 3.3.7 under the repository and the gate rcfiles reports nothing new on any touched file; no function added here is above complexity 17, Lizard's threshold being 20; codespell, markdownlint and shellcheck are clean; and the launcher the timer reads differs in nothing from the yaml ND prices, which is what the round depends on. The crop's Engram assets ship compressed. The code check reads every text file of a change for sensitive words, and the assets' token map holds vocabulary ids it reads as a product number, so the check failed on data that names no product. The archive holds the same bytes, 311 KB in place of 875, and the asking script unpacks it into the cost model's cache before it names the file, since the model's builder reads that asset as text. ND_ASSET_CACHE names another folder. Co-authored-by: Thibaut Tachon <tachon.thibaut@gmail.com> | 10 天前 | |
chore: make agent tooling assistant-agnostic for any AI coding assistant Make the in-repo agent tooling work with any AI coding assistant instead of being tied to a specific vendor, so contributors can use whichever assistant they prefer. | 4 个月前 |
Skills 管理指南
创建 Skill
- 按 Skill 规范编写：每个 Skill 必须包含
SKILL.md入口文件。 - 推荐使用 Skill Creator skill 生成初始模板，确保结构和格式符合规范。
修改 Skill
- 优先联系 Skill 的 owner（上传者）提需求或讨论修改方案，避免独自大改引入问题。
- Skill 难以编写自动化测试用例，修改后务必人工验证核心流程。
迭代策略
- 小改动：在原有 Skill 上修复即可。
- 大迭代：不建议直接改原有 Skill，建议基于原有 Skill 新建一个 Skill（如
autogit-v2/），保持原版本可回退。
安装 Skill
- 将 Skill 仓库 clone 到本地。
- 告知本地 Agent，Agent 会自动将 Skill 路径注册到其可识别的配置中。
# 示例：clone 后告诉 Agent
git clone <skill-repo-url> ~/.local/skills/my-skill
# Agent 会自动识别并注册