Yyangzhenzhang02chore: purge MindSpore residue after backend removal
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
refactor: 将 EXISTING_COMM_GROUPS 从 platform 迁移至 core.utils 为后续去除 MindSpore 双平台支持做准备。platform 层的职责是屏蔽 torch/mindspore 差异，而 EXISTING_COMM_GROUPS 只是进程组缓存，属于 纯数据，不应挂在 platform 上。 新增 hyper_parallel/core/utils/communication.py 承载该常量，并刷新全部 引用（源文件 8 处、测试文件 73 处）。core/utils/__init__.py 未改动。 platform.py 在包初始化早期即被加载，而 core.utils.__init__ 会急切加载 shape_utils -> core.dtensor，core/dtensor/dtensor.py 又反向引用 core.utils， 顶部 import 构成循环，故这三处改为方法内局部导入。 | 1 个月前 | |
refactor(dtensor): remove dead code and fix static check warnings Delete unreachable async wrappers, the sync-hook autograd function (now in pipeline_parallel/_sync_hook.py) and their orphaned helpers. Extract a stage reporting helper and add docstrings so the touched functions stay within the local-variable and protected-access limits. Refs: #242 | 1 个月前 | |
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
fix codecheck and fix optim state name in ActivationWrapper | 1 个月前 | |
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
refactor(dcp): drop the platform abstraction from distributed_checkpoint distributed_checkpoint no longer supports MindSpore, so every get_platform() call is replaced by the torch API it resolved to, and the per-backend branches go with it. - Platform calls become direct torch calls: dist.get_rank / get_world_size / barrier / broadcast / new_group / all_gather_object, tensor.detach(), isinstance(x, torch.Tensor), torch.empty. What torch has no single call for is wrapped inside this package instead of reaching through the platform layer. - Split utils.py by concern: checkpoint_io.py (one checkpoint file in, one out) and broadcast.py (replica-set broadcast: group build and teardown, the sends, the batching). The rest stays in utils.py. - Drop what the checkpoint core does not use: loader.py / saver.py (single-file wrappers), layout.py (layout serialisation) and reshard.py's ReshardHandler. infer_intersection was the only piece still needed and moves into utils.py. - Remove the MindSpore-only branches: _fetch_ms_tensor_file, _MS_FILES_KEPT, _broadcast_batch_bytes and src_platform="mindspore". - acquire_tensor now detaches on every branch. The ragged branch never did and filesystem_storage was covering for it, so the requirement moves into the LoadPlanner.acquire_tensor contract and the redundant guard goes. - Rename util.py to utils.py and ragged_utils.py to ragged.py, so the package has one utils module and no second name ending in the same word. - Drop ensure_broadcast_groups. It wrapped _build_broadcast_groups to hand the groups over without destroying them, for a caller that owns their lifetime; no such caller was ever written, and the load path takes the broadcast_groups_for_load scope instead. Its cases now cover the builder. - Register distributed_checkpoint as a torch-only core area in the pylint plugin, alongside core/dtensor. - MindSpore ST cases move to level3. | 1 个月前 | |
fix codecheck and fix optim state name in ActivationWrapper | 1 个月前 | |
refactor(fsdp): lift torch FSDP into core and remove mindspore backend Lift the Torch-side FSDP implementation verbatim into core/fully_shard and drop the MindSpore FSDP backend entirely. - Move platform/torch/fully_shard sources and UTs into core/fully_shard (import paths only); delete platform/mindspore/fully_shard and its UTs. - De-platform core/fully_shard: it no longer imports hyper_parallel.platform. - Merge the HSDP state/param base classes into single HSDPStateV2 and HSDPParamV2, dropping the NotImplementedError abstraction layer and the dead group-info builders (_init_group_infos, _build_layout_driven_group_info, _normalize_unsharded_grad_to_local, _build_group_info_from_rank_list). - Rename param.py/state.py to hsdp_param.py/hsdp_state.py. - Drop the torch-only ModuleClass/TensorClass/ParameterClass aliases. - Remove the now-unused process-group helper cluster from fully_shard/utils.py. - Add protected-access/unused-variable to the core/fully_shard pylint filter. Co-Authored-By: Claude <noreply@anthropic.com> | 1 个月前 | |
perf(dcp): overlap the replicated-shard broadcast with the read that feeds it A shard several ranks load identically was already read by one of them and sent to the rest. It went, though, only once the whole read had finished: the planner marked the tensors it wanted broadcast, the reader worked through every file, and one round at the end of execute_read moved them all. Nothing was sent while there was still reading to do, and nothing was read while the sending went on. The send now happens as each shard lands, so it overlaps the reads behind it. What lets it is where the decision is carried: on the read item rather than on the state dict entry, so the plan settles who sends and who receives before the read starts, per shard rather than per tensor. Who reads is then decided by the bytes a rank has already been given rather than by its position in the group, which keeps the lowest ranks of every group from taking all of it. What travels is the destination shard's whole local buffer, so a shard that a reshard puts together out of several reads is marked on every one of them, weighed by all of them, and goes only once its last piece is in place. The read keeps checkpoint files open across shards instead of reopening one per shard and sends shards under 6 MiB together, so that a load spends its time moving data rather than starting broadcasts. What no other rank shares is read a file at a time rather than a shard at a time: nothing waits on those, so nothing holds them to the order the ranks agreed on. That matters most to the pickled entries, which the save side spreads over the ranks like everything else while the load side never broadcasts one, so every rank reads every entry it wants out of whatever file it landed in - eighty entries over three files cost eighty opens where three do. Two checks in front of the read are gone, each of which only said what the line after it says. Listing a file's names to report a missing tensor brought every name across from the reader, 6.5 ms on a file of nine thousand, where safetensors reports the same thing itself. Stat-ing a file to report it missing cost a round trip per file on shared storage, where both safetensors and the builtin open raise FileNotFoundError naming the path. - replace should_load_shard and the BROADCAST_INFO attribute with BroadcastSource on ReadItem, and broadcast_loaded_tensors with a send per shard - rename load()'s broadcast_from_minimum_rank to broadcast_replicated_tensors, the reader no longer being the lowest rank of its group - build_global_plan hands back the plan of the rank that runs it, every rank deriving the same division from the same gathered plans, and only the identity of each read crosses the all-gather that feeds it - add Platform.broadcast_async, copy_each and get_world_group to the base class and the torch backend. The ms backend is left as it was, new_group included, so a ms load that broadcasts raises rather than a backend on its way out being adapted for it - take the exchange file delete out of the block whose FileNotFoundError means "not written yet". A file another reader had already removed was reported as not ready, sending the caller back for what it was holding until its timeout - gather the shards no other rank shares by the file holding them, and drop the key-set listing and the existence check that stood in front of every read - have load() exchange plans and broadcast by default, as save() already gathers by default. The trainer calls it without either, so the path this commit is about was never taken by a real run. What it asks in return is what save() already asks: every rank of the default group calls load together, a rank with nothing to read passing an empty state dict - leave a shard whose destination stayed in host memory to every rank that wants it, marked on the read item so the decision crosses the gather with it. What a load keeps on the host is little and few - an optimizer's step counter, one scalar per parameter tensor, while the moments beside it follow the parameter onto the accelerator - so reading it costs less than moving it, and the group it would move through has no backend for host memory to move it with | 1 个月前 | |
chore: purge MindSpore residue after backend removal The platform abstraction layer (and with it the MindSpore backend) was deleted earlier, but references to it survived throughout the tree. This removes them; it changes no behavior on the surviving Torch path. Removed: - docker/Dockerfile.mindspore, the MindSpore build branch and examples in docker/build_hyper-parallel_npu.sh / docker/README.md - .jenkins/test/config/dependent_packages.yaml (a single MindSpore wheel URL with no remaining consumer) - MindSporeCase in tests/common/parallel_case.py and msrun_case in tests/common/distributed_launcher.py, plus msrun / MS_NODE_ID rank-discovery fallbacks - dead _MS_PRIMITIVE_OP_NAMES branches in the shard ops (none of those op names are registered in any yaml) and their tests - the stale aclop packed-args notes in parallel_elementwise.py / parallel_gather.py (_normalize_aclop_args no longer exists) - three pylint errors the Atomgit gate reports on files this commit already touches: an unused os import in _pytorch_reference.py, a missing docstring on NoAcceleratorGuard._check_init_process_group and trailing blank lines in test_parallel_topk.py - mindspore from the multicore boundary test, pylint's generated-members, from the autogit lint backend list and from the docs, examples and .agent rules Kept on purpose: - _MS_POSITIONAL_DTYPE_OP_NAMES in parallel_reduce.py — SumExt / MeanExt are live torch registrations in torch_reduce.yaml, so the trailing positional dtype slot is still reachable - test_runtime_sources_import_no_platform_layer — the boundary test still asserts production sources import no Platform layer - URLs under the project's own mindspore.org / gitcode / atomgit namespaces, including the links to the MindSpore repo's contribution rules - hyper_parallel_v1.0.0_release_notes.md (historical) | 1 个月前 |