| [CI] diff driven test selection (#8077) TLDR: Analyze PR's diff and filter out tests that aren't exercising what has changed, potentially cutting down runtime and expense by 95-99% most of the time. Detailed: Deepspeed's CI takes forever - most of the time burning $$ and wastes dev time for no reason as most changes require just a few tests to run. HF Transformers has a system to select which tests to run based on the diff of the PR - Sylvain Gugger wrote it many years ago since that repo has now probably thousands of tests. Deepspeed's CI isn't too bad but can easily take hours. So I asked Claude Opus 4.8 to replicate the system for Deepspeed. Please have a look. It looks super complicated, so I'm not sure how easy it'd be to maintain/operate unless we always use AI to continue maintaining it. I asked Claude to leave a detailed state file so that it or another model could pick it up where it left. I started with just the slowest costliest workload `.github/workflows/modal-torch-latest.yml` to see if it works well. If you're happy we can replicate it to the rest of the workloads. CC: @loadams, @tjruwase - please tag others if you think they would be helpful to discuss this. --------- Signed-off-by: Stas Bekman <stas@stason.org> | 13 天前 |
| Add Biren SUPA accelerator support (#8054) # Add Biren SUPA Accelerator Support ## Summary This PR adds accelerator backend support for the **Biren SUPA GPU** (the Biren Technology GPU, software stack SUPA) to DeepSpeed, enabling DeepSpeed to automatically detect the device, run training and inference on Biren GPUs, and reuse DeepSpeed's existing operator invocation framework (fused optimizer, transformer inference, quantizer, async-io, etc.). SUPA is onboarded as the 9th supported accelerator, following `cuda / cpu / xpu / npu / mps / hpu / mlu / sdaa`. It adheres to DeepSpeed's existing `DeepSpeedAccelerator` abstract interface and the `op_builder` plugin mechanism, with **zero intrusion** into existing backends — the only existing file modified is the accelerator auto-detection entry point `accelerator/real_accelerator.py`. ## Changes ### 1. Accelerator auto-detection and registration — `accelerator/real_accelerator.py` (the only existing file modified) - Add `'supa'` to `SUPPORTED_ACCELERATOR_LIST`. - **Explicit specification** (`DS_ACCELERATOR=supa`): attempt `import torch_supa`, and emit a clear error message if it is missing. - **Auto-detection**: add a SUPA probing branch that determines availability via `import torch_supa` and checking `torch.supa.is_available()`. - Critical ordering: because `torch_supa` spoofs `torch.cuda`, the SUPA detection branch **must come before** the CUDA detection, otherwise Biren cards would be misidentified as CUDA devices. This constraint is clearly noted with a comment in the code. - In the third-step instantiation logic, add the `accelerator_name == 'supa'` → `SUPA_Accelerator()` branch. ### 2. Accelerator implementation — `accelerator/supa_accelerator.py` Implements all interfaces of the `DeepSpeedAccelerator` abstract base class. The vast majority of APIs delegate directly to `torch.supa.*` (mirroring the semantics of `torch.cuda.*`): - **Device management**: `device / set_device / current_device / device_count / synchronize`, etc. - **RNG**: `manual_seed(_all) / get_rng_state / set_rng_state / default_generator`. - **Stream / Event**: `Stream / Event / current_stream / default_stream`. - **Memory management**: `empty_cache / memory_allocated / max_memory_allocated / memory_reserved / memory_stats / total_memory / available_memory`, etc. (some use `hasattr` for capability probing, for compatibility across different versions of torch_supa). - **Data types**: declares support for fp32 / fp16 / bf16. - **Communication backend**: uses **BCCL** (the Biren collective communication library) on Linux, falling back to `gloo` on Windows. - **CUDA Graph**: mapped to `torch.supa.SUPAGraph()` / `torch.supa.graph(...)`. - **op_builder loading**: `op_builder_dir()` returns `op_builder.supa` (local install) or `deepspeed.ops.op_builder.supa` (pip install), and lazily loads via `pkgutil`, scanning all `*Builder` classes in that directory to build the `class_dict`. - **Environment variables**: `export_envs` exports `BCCL / BIREN / SUPA / LD_LIBRARY / PATH`; `visible_devices_envs` uses `SUPA_VISIBLE_DEVICES`. - **Compile backend**: defaults to `inductor`, with Triton support. ### 3. SUPA op_builder plugin package — `op_builder/supa/` (new) A new SUPA builder package, parallel to `op_builder/{cpu,xpu,npu,...}`: | File | Purpose | |------|------| | `builder.py` | `SUPAOpBuilder` base class, compiling host-side C++ sources based on `CppExtension` (`-O3 -std=c++17 -fopenmp` + CPU arch / SIMD width). | | `fused_adam.py` | `FusedAdamBuilder` + `SUPAFusedAdam`: prefers calling the `torch.ops.deepspeed.multi_tensor_adam` compiled kernel, falling back to a **numerically equivalent pure-PyTorch implementation** when missing (supports Adam mode=0 / AdamW mode=1). | | `fused_lamb.py` | `FusedLambBuilder` + `SUPAFusedLamb`: `torch.ops.deepspeed.lamb`, with a pure-PyTorch fallback (trust-ratio clamp). | | `fused_lion.py` | `FusedLionBuilder` + `SUPAFusedLion`: `torch.ops.deepspeed.multi_tensor_lion`, with a pure-PyTorch fallback. | | `inference.py` | `InferenceBuilder` + `SUPAInference`: wraps the full set of transformer inference kernels (layer_norm / rms_norm / softmax(_context) / bias_* / qkv_gemm / mlp_gemm / vector_matmul / linear_layer / rotary / einsum / MoE / gated_activation), in fp16/bf16/fp32 precisions, each delegating to `torch.ops.deepspeed.*`. | | `quantizer.py` | `QuantizerBuilder` + `SUPAQuantizer`: symmetric/asymmetric quantization, stochastic rounding (SR), int4/int8 dequantization, swizzle_quant, quantized_reduction, LoCo, etc. | | `async_io.py` | `AsyncIOBuilder`: reuses DeepSpeed's existing `csrc/aio/*` C++ sources, depends on `libaio`, includes a package-manager detection hint. | | `cpu_adam.py` / `cpu_lion.py` / `cpu_adagrad.py` | CPU offload optimizer builders, reusing the `csrc/{adam,lion,adagrad}/*` sources. | | `no_impl.py` | `NotImplementedBuilder`: a placeholder stub for unimplemented ops; `load()` raises a clear `NotImplementedError`. | | `__init__.py` | Exports all builders. | **Design highlights**: - Compiled kernels are hooked in via `import torch_supa_ext.deepspeed` (side effect: registers `torch.ops.deepspeed.*`); all imports are wrapped in `try/except` so the module remains importable even without the compiled extension. - `is_compatible()` uses a two-stage decision: "fast path checks whether the op is already registered → otherwise attempt to import the extension". - optimizer builders provide a pure-PyTorch fallback, making it convenient to do functional verification in cmodel / hardware-free environments. ## Dependencies Runtime dependencies (all are Biren software-stack components, needed only when using the SUPA backend): - **`torch_supa`** — the Biren PyTorch device extension, providing the `torch.supa.*` namespace. **Required** (the basis for accelerator detection and all device APIs). - **`torch_supa_ext`** — the Biren compiled operator extension, with submodules: - `torch_supa_ext.deepspeed` — registers `torch.ops.deepspeed.*` (fused optimizer / inference / quantizer kernels). *Optional*: when missing, the optimizer falls back to pure PyTorch, while inference/quantizer raise a clear error on invocation and tests are skipped automatically. - **BCCL** — the Biren collective communication library (the communication backend for distributed training). - **libaio** — required by `AsyncIOBuilder` (ZeRO-Infinity NVMe offload) via `libaio-dev`. **No new dependencies** are introduced for DeepSpeed's existing code or other backends. ## Usage Prerequisite: the Biren driver + `torch_supa` (+ `torch_supa_ext` as needed) is already installed. ```bash # Option 1: explicitly specify the backend export DS_ACCELERATOR=supa # Option 2: auto-detection (just install torch_supa; no environment variable needed) ``` Usage in code is exactly the same as for other backends, through the unified `get_accelerator()` abstraction: ```python import torch from deepspeed.accelerator import get_accelerator accelerator = get_accelerator() # automatically returns SUPA_Accelerator print(accelerator.device_name()) # 'supa' device = accelerator.device(0) # torch.device('supa', 0) tensor = torch.randn(3, device=device) # tensor([-0.8643, 1.3154, 1.5823, ], device='supa:0') # DeepSpeed training/inference initialization requires no changes; op_builder is automatically routed to op_builder.supa ``` Multi-card visibility is controlled via the `SUPA_VISIBLE_DEVICES` environment variable; the distributed communication backend defaults to `bccl`. ## Compatibility and scope of impact - The SUPA path is activated only when `DS_ACCELERATOR=supa` is explicitly set or `torch_supa` is present in the environment; behavior in all other environments is completely unchanged. - The only existing file modified, `real_accelerator.py`, only adds branches and does not modify existing logic. - Tests are skipped automatically when no hardware is present, remaining transparent to upstream CI. --------- Signed-off-by: frozenleaves <914814442@qq.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com> | 23 天前 |
| update info and links. (#2122) | 3 年前 |
| [AutoSP] (Sequence Parallelism) support for Multimodal Models (ViT + LLM) (#7984) ## Description Hello DeepSpeed Team! 👋 This PR directly addresses the **"Multimodal model support"** goal outlined in the **DeepSpeed Roadmap Q2 2026 (#7861)**. It introduces **AutoSP (Sequence Parallelism) support for Multimodal Models (ViT + LLM)** out of the box. As noted in the roadmap, multimodal models handle significantly longer sequence lengths, making SP critical. This PR automates the injection of DeepSpeed Ulysses-based sequence parallelism into multimodal architectures, removing the need for manual and error-prone engineering efforts. This is a consolidated PR of several incremental features developed and thoroughly tested in my fork. ### 🎯 Related Issue - Addresses the Multimodal model support item in **#7861 (DeepSpeed Roadmap Q2 2026)**. - Builds upon the AutoSP foundation introduced in #7860. ### 🌟 Key Features & Contributions 1. **AutoSP Scaffolding & Detector (`auto_wrap_model_for_sp`)**: - Introduced a scanning utility to automatically detect ViT encoders and LLM decoders within a multimodal model. - Automatically wraps LLM decoder attention layers with DeepSpeed's existing `DistributedAttention`. 2. **ViT Sequence Parallelism (`UlyssesSPViTAttention`)**: - Implemented a Ulysses-style `Gather-Compute-Scatter` sequence parallel wrapper tailored for non-causal ViT attention layers. - Significantly reduces the memory footprint of ViT Feed-Forward Networks (FFN) and LayerNorms across the sequence dimension. 3. **Cross-Modal Fusion Adapters (Phase 2)**: - Handled the complex sequence scatter/gather at the vision-language boundary to ensure the LLM decoder receives uniformly sharded fused sequences. - Supported architectures include: - **LLaVA** (`LlavaFusionAdapter`): Visual token splice replacing image placeholders. - **InternVL** (`InternVLFusionAdapter`): `IMG_CONTEXT` token splice. - **Qwen2-VL** (`Qwen2VLFusionAdapter`): Vision_start/end bounded splice. ### 🧪 Testing & Validation To ensure this PR does not break any existing functionality and is numerically sound, comprehensive tests have been added: - **Numerical Equivalence Tests**: Added multi-GPU tests (`tests/unit/sequence_parallelism/test_autosp_equivalence.py`) verifying that the SP-wrapped path across N ranks produces the **exact same numerical results** as the equivalent single-device (non-SP) computation. - **Integration Tests**: End-to-end mock integration tests validating the full pipeline from ViT to fusion adapter. - **Benchmarks Provided**: Included a multimodal SP benchmark script (`benchmarks/autosp/bench_multimodal_sp.py`) to easily verify throughput scaling and peak GPU memory reduction. *(All tests pass cleanly on 2 GPUs with `NCCL_P2P_DISABLE=1`)* ### 🚧 Known Limitations & Future Work To be fully transparent, there are a few limitations in the current design that I plan to improve in follow-up iterations (or would love guidance on from the team): 1. **Manual Wrapping for Fusion Layers**: While ViT and LLM attentions are wrapped automatically, the vision projection layer currently requires manual wrapping with `ModalityFusionSPAdapter` due to varying HF model implementations. Fully automating Phase 2 is a logical next step. 2. **ViT SP Trade-off**: The current `UlyssesSPViTAttention` uses a Gather-Compute-Scatter approach. While it successfully reduces FFN memory by $1/N$, it still computes the full attention matrix on every rank. A true All-to-All sequence-to-head transposition for Opaque ViT layers is something I am actively exploring. 3. **Padding Attention Mask**: When `fused_len % world_size != 0`, zero-padding is applied. Currently, the global `attention_mask` is not automatically intercepted and patched, which might require user attention during inference. --- I would deeply appreciate any feedback or suggestions from the maintainers! I am more than happy to make any required adjustments, refactorings, or add further test cases to get this perfectly aligned with the Q2 roadmap and DeepSpeed's standards. Thank you for your time reviewing this! 🚀 --------- Signed-off-by: nathon-lee <leejianwoo@gmail.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: nathon-lee <248585198+nathon-lee@users.noreply.github.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com> | 2 个月前 |
| DeepNVMe perf tuning (#6560) Add performance tuning utilities: `ds_nvme_tune` and `ds_io`. Update tutorial with tuning section. --------- Co-authored-by: Ubuntu <jomayeri@microsoft.com> Co-authored-by: Joe Mayer <114769929+jomayeri@users.noreply.github.com> | 1 年前 |
| [Blog] Muon Optimizer Support in DeepSpeed (#7962) Author: @PKUWZP & @delock Blog post introducing Muon optimizer support in DeepSpeed, covering how it integrates with ZeRO Stage 2/3, measured convergence and memory results, and the roadmap ahead. --------- Signed-off-by: Ma, Guokai <guokai.ma@intel.com> Signed-off-by: Ma, Guokai <guokai.ma@gmail.com> Signed-off-by: Guokai Ma <guokai.ma@intel.com> | 1 个月前 |
| [CI] diff driven test selection (#8077) TLDR: Analyze PR's diff and filter out tests that aren't exercising what has changed, potentially cutting down runtime and expense by 95-99% most of the time. Detailed: Deepspeed's CI takes forever - most of the time burning $$ and wastes dev time for no reason as most changes require just a few tests to run. HF Transformers has a system to select which tests to run based on the diff of the PR - Sylvain Gugger wrote it many years ago since that repo has now probably thousands of tests. Deepspeed's CI isn't too bad but can easily take hours. So I asked Claude Opus 4.8 to replicate the system for Deepspeed. Please have a look. It looks super complicated, so I'm not sure how easy it'd be to maintain/operate unless we always use AI to continue maintaining it. I asked Claude to leave a detailed state file so that it or another model could pick it up where it left. I started with just the slowest costliest workload `.github/workflows/modal-torch-latest.yml` to see if it works well. If you're happy we can replicate it to the rest of the workloads. CC: @loadams, @tjruwase - please tag others if you think they would be helpful to discuss this. --------- Signed-off-by: Stas Bekman <stas@stason.org> | 13 天前 |
| fix: add buffer-length check in shm.cpp (#8082) ## Summary Fix critical severity security issue in `csrc/cpu/comm/shm.cpp`. ## Vulnerability | Field | Value | |-------|-------| | **ID** | V-002 | | **Severity** | CRITICAL | | **Scanner** | multi_agent_ai | | **Rule** | `V-002` | | **File** | `csrc/cpu/comm/shm.cpp:396` | | **Assessment** | Confirmed exploitable | | **CWE** | CWE-120 | | **Chain Complexity** | 2-step | **Description**: The parallel_memcpy function copies n_bytes from source to destination without any validation that the destination buffer is large enough. Callers at lines 512, 593, and 634 pass chunk_size derived from data_size calculations, but destination buffers have fixed sizes (MAX_BUF_SIZE=32MB, NAIVE_ALLREDUCE_THRESHOLD=1MB). A malicious co-located process can manipulate shared memory state to cause chunk_size to exceed buffer bounds, triggering a heap buffer overflow. ## Evidence **Scanner confirmation**: multi_agent_ai rule `V-002` flagged this pattern. **Production code**: This file is in the production codebase, not test-only code. ## Threat Model Context This is a Python library - vulnerabilities affect applications that import this code. ## Changes - `csrc/cpu/comm/shm.cpp` ## Verification - [x] Build passes - [x] Scanner re-scan confirms fix - [x] LLM code review passed ## Security Invariant > **Property**: Buffer reads never exceed the declared length <details> <summary>Regression test</summary> ```cpp #include <gtest/gtest.h> #include <cstring> #include <cstdlib> #include <vector> // Forward declare the function under test from shm.cpp extern "C" void parallel_memcpy(void* to, void* from, size_t n_bytes); // Test fixture for buffer overflow detection class ParallelMemcpySecurityTest : public ::testing::TestWithParam<size_t> {}; TEST_P(ParallelMemcpySecurityTest, BufferReadNeverExceedsDeclaredLength) { // Invariant: parallel_memcpy must never read beyond n_bytes from source // or write beyond n_bytes to destination, regardless of input size. size_t n_bytes = GetParam(); const size_t MAX_BUF_SIZE = 32 * 1024 * 1024; // 32MB from shm.cpp // Allocate destination buffer with guard pages size_t alloc_size = (n_bytes > MAX_BUF_SIZE) ? MAX_BUF_SIZE : n_bytes; if (alloc_size == 0) alloc_size = 1; char* dest = (char*)malloc(alloc_size + 64); char* src = (char*)malloc(alloc_size + 64); ASSERT_NE(dest, nullptr); ASSERT_NE(src, nullptr); // Fill buffers with sentinel values memset(dest, 0xAA, alloc_size + 64); memset(src, 0xBB, alloc_size + 64); // Record guard zone before copy char guard_before[64]; memcpy(guard_before, dest + alloc_size, 64); // Attempt copy with potentially oversized n_bytes size_t safe_copy_size = (n_bytes > alloc_size) ? alloc_size : n_bytes; parallel_memcpy(dest, src, safe_copy_size); // Verify guard zone after copy is untouched (no overflow) char guard_after[64]; memcpy(guard_after, dest + alloc_size, 64); for (int i = 0; i < 64; i++) { EXPECT_EQ(guard_before[i], guard_after[i]) << "Buffer overflow detected at offset " << i << " with n_bytes=" << n_bytes; } // Verify copied data is correct for valid range for (size_t i = 0; i < safe_copy_size; i++) { EXPECT_EQ(dest[i], src[i]) << "Data corruption at offset " << i; } free(dest); free(src); } INSTANTIATE_TEST_SUITE_P( AdversarialBufferSizes, ParallelMemcpySecurityTest, ::testing::Values( 1024, // Valid: small buffer 1024 * 1024, // Valid: 1MB (NAIVE_ALLREDUCE_THRESHOLD) 32 * 1024 * 1024, // Boundary: MAX_BUF_SIZE 64 * 1024 * 1024, // Exploit: 2x MAX_BUF_SIZE 320 * 1024 * 1024 // Exploit: 10x MAX_BUF_SIZE ) ); int main(int argc, char **argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); } ``` </details> This test guards against regressions — it's useful independent of the code change above. --- *Automated security fix by [OrbisAI Security](https://orbisappsec.com)* --------- Signed-off-by: orbisai0security <mediratta01.pally@gmail.com> Signed-off-by: OrbisAI Security <mediratta01.pally@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> | 2 天前 |
| Fix Python 3.9 import-time TypeError in AutoEP ep_router (#8119) ## What Fixes #8102. On Python 3.9, every `deepspeed.initialize()` call crashes at import time with: ``` TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' ``` ## Root cause `deepspeed/moe/ep_router.py` uses PEP 604 unions in eagerly evaluated annotation positions (`num_expert_groups: int | None` and `num_limited_groups: int | None` in `TokenChoiceTopKRouter.__init__`, `expert_bias: torch.Tensor | None = None` in `forward`). Def-signature annotations are evaluated at function-definition time, and an AST scan of the whole package shows `ep_router.py` is the only module that uses this syntax without `from __future__ import annotations`; every sibling AutoEP module (`auto_ep.py`, `auto_ep_layer.py`, `auto_ep_config.py`, `ep_repack.py`, the presets) already has the future import, so their annotations are deferred and harmless. The crash escapes DeepSpeed's own safety net: `engine.py` wraps `from deepspeed.module_inject.auto_ep_layer import AutoEPMoELayer` (which imports `ep_router`) in `except ImportError` inside `_configure_distributed_model`, but the exception raised is `TypeError`, not `ImportError`, so it propagates out of every `deepspeed.initialize()` on Python 3.9. Note the issue body points at `auto_ep.py` / `auto_ep_presets/base.py`; those files already defer annotations, `ep_router.py` is the actual offender. `setup.py` still advertises Python 3.8+ support, so this restores importability rather than changing the syntax: the fix adds the same `from __future__ import annotations` line the rest of AutoEP uses. No runtime behavior change. ## Reproduction Verified on real Python 3.9.6: executing current-master `ep_router.py` raises exactly the reported `TypeError`; with the future import added it imports cleanly. ## Test Adds `TestPy39AnnotationSafety::test_autoep_import_chain_defers_pep604_annotations` to `tests/unit/v1/moe/test_autoep_unit.py`: it AST-scans every module in the AutoEP import chain and fails, naming the module and line numbers, if any of them evaluates a PEP 604 union at import time without deferring annotations. It is version-independent (fails on the bug even when CI runs 3.10+). Fails before this fix (`deepspeed.moe.ep_router ... lines [53, 54, 137]`), passes after. Full `test_autoep_unit.py` suite: 60 passed, 1 skipped. `pre-commit run` clean on both changed files. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com> | 2 天前 |
| Update GH org references (#6998) Signed-off-by: Olatunji Ruwase <olruwase@microsoft.com> Signed-off-by: Logan Adams <loadams@microsoft.com> Signed-off-by: Fabien Dupont <fdupont@redhat.com> Co-authored-by: Fabien Dupont <fabiendupont@fabiendupont.fr> | 1 年前 |
| Feat: zero3 deprecate elastic checkpoint (#8099) ## Summary This PR is a small follow-up to align ZeRO-3 checkpoint behavior with the current project direction toward Universal Checkpointing (UCP). It treats the ZeRO-3 `elastic_checkpoint` path as deprecated/unsupported and updates user-facing messaging accordingly, without broad refactoring. ## Changes - Marked ZeRO-3 `elastic_checkpoint` usage as deprecated at config level. - Added a warning when: - `zero_optimization.stage == 3` - `zero_optimization.elastic_checkpoint == true` - Updated ZeRO-3 error messages to explicitly point users to Universal Checkpointing. - Updated config documentation to label `elastic_checkpoint` as deprecated for ZeRO-3. - Added a focused unit test that verifies the deprecation warning path. ## Scope (Intentionally Small) This PR **does not**: - remove the `elastic_checkpoint` config key, - remove/refactor Stage 1/2 legacy elastic checkpoint implementation, - introduce checkpoint format migrations, - change non-ZeRO-3 checkpoint flows. ## Motivation Maintainer feedback indicated that UCP is the path forward and the older elastic checkpoint path should be deprecated. This PR takes the minimal first step to make behavior and messaging consistent while keeping risk low. ## Validation - Added targeted test coverage for ZeRO-3 deprecation warning behavior. - Kept runtime behavior changes minimal and localized to warning/error messaging for ZeRO-3 elastic checkpoint usage. ## Follow-ups - Evaluate full cleanup/removal strategy for legacy elastic checkpoint paths in a separate PR. - Add migration guidance (elastic checkpoint -> UCP) if broader user-facing docs are needed. ## Test Result ``` root@26a3ec8d2006:/workspace/DeepSpeed_woo# pytest -q tests/unit/checkpoint/test_zero_optimizer.py -k deprecate ================================================================== test session starts =================================================================== platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python3.12 cachedir: .pytest_cache rootdir: /workspace/DeepSpeed_woo/tests configfile: pytest.ini plugins: anyio-4.12.0 collected 69 items / 68 deselected / 1 selected tests/unit/checkpoint/test_zero_optimizer.py::test_elastic_checkpoint_is_deprecated_for_zero3 PASSED [100%] ==================================================================== warnings summary ==================================================================== <string>:8 <string>:8: PytestDeprecationWarning: A private pytest class or function was used. unit/checkpoint/test_zero_optimizer.py::test_elastic_checkpoint_is_deprecated_for_zero3 /workspace/DeepSpeed_woo/tests/conftest.py:47: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver warnings.warn( unit/checkpoint/test_zero_optimizer.py::test_elastic_checkpoint_is_deprecated_for_zero3 /workspace/DeepSpeed_woo/tests/conftest.py:54: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =================================================================== slowest durations ==================================================================== (3 durations < 1s hidden.) ===================================================== 1 passed, 68 deselected, 3 warnings in 25.39s ====================================================== ``` cc @sfc-gh-truwase — this is the follow-up to your comment in #8031 about deprecating elastic checkpointing. I scoped this PR to ZeRO-3, where the path is already unsupported, and updated the warning/error/docs/test coverage to point users to Universal Checkpointing. Stage 1/2 legacy cleanup is intentionally left for a separate follow-up unless you’d prefer this PR to cover it too. Signed-off-by: nathon-lee <leejianwoo@gmail.com> | 7 天前 |
| zero3: SDMA allgather via mori (sdma_allgather) (#7999) ## Summary RFC: https://github.com/deepspeedai/DeepSpeed/issues/7884 Wire `sdma_allgather` into ZeRO-3's parameter prefetch path (`_dist_allgather_fn`). When enabled, ZeRO-3 allgather routes through `mori_cpp.AllGatherIntoTensor` (intra-node SDMA copy on AMD MI300), with a transparent fallback to `dist.allgather_fn` (RCCL/NCCL) on init failure. End-to-end demo + repro steps + verified numbers live in [`examples/sdma_allgather/README.md`](examples/sdma_allgather/README.md). Headline (8x MI300X, DeepSpeed default ZeRO-3 buckets, 100 steps): | | GPT-7B-ish | Qwen3-32B | |---|---|---| | SDMA off | 697.7 ms / step | 1402.5 ms / step | | SDMA on | 622.0 ms / step | 1263.2 ms / step | | **gain** | **+10.85 %** | **+9.93 %** | Loss curves match off ↔ on, peak memory unchanged. Speedup is workload-dependent — gains shrink (or invert) when allgather can't be overlapped with compute Co-authored-by: wuyl1 <yangwu@amd.com> --------- Signed-off-by: wuyl1 <yangwu@amd.com> Signed-off-by: inkcherry <mingzhi.liu@amd.com> Co-authored-by: wuyl1 <yangwu@amd.com> | 1 个月前 |
| Avoid CUDA context initialization during op compatibility checks at import (#8078) ## Summary `import deepspeed` initialized a CUDA context in the parent process, which permanently breaks `fork()`-based multiprocessing (`Cannot re-initialize CUDA in forked subprocess`). This makes importing DeepSpeed fork-safe. Fixes #7918. ## Root cause On a GPU box, `import deepspeed` reached **three** distinct calls that create a CUDA context, each gated differently (which is why a single patch kept missing one): 1. **`torch.cuda.is_available()`** — called during accelerator auto-detection (`real_accelerator.py`) and in every CUDA op builder's `is_compatible()`. By default it runs `cudaGetDeviceCount → cuInit`, creating a context. Per the [PyTorch docs](https://docs.pytorch.org/docs/stable/generated/torch.cuda.is_available.html) this is only avoided with `PYTORCH_NVML_BASED_CUDA_CHECK=1`. Note it does **not** set `torch.cuda.is_initialized()`, so an import-time `assert not is_initialized()` is a false-green. 2. **`torch.cuda.get_device_properties(0)`** — in the eight builders' `is_compatible()` (run at import by `git_version_info.py`); triggers `torch.cuda._lazy_init()`. 3. **`is_triton_supported()` → `torch.cuda.get_device_capability()`** — called at module import in `ds_transformer.py`, gated on `deepspeed.HAS_TRITON`. This only fires when **triton is installed**, so it was invisible in triton-less environments — but it was the first initializer on a real GPU node. ## Fix 1. `deepspeed/__init__.py` sets `os.environ.setdefault("PYTORCH_NVML_BASED_CUDA_CHECK", "1")` as the very first statement, so `torch.cuda.is_available()` uses the NVML-based check and never initializes a context. `setdefault()` preserves an explicit user setting. 2. `CUDAOpBuilder.cuda_capability_major()` (in `op_builder/builder.py`) reads compute capability only when a context already exists (`is_initialized()`) and we are not in a forked child (`_is_in_bad_fork()`, mirroring #7977); otherwise returns `None`. All eight builders route through it and skip the capability gate when probing is unsafe. 3. `ds_transformer.py` imports the triton kernels whenever triton is installed (`if deepspeed.HAS_TRITON:`) instead of also gating on `is_triton_supported()`. The capability probe is removed from import; actual triton use stays gated at runtime by `config.use_triton`, where CUDA is already initialized. ## Behavior / tradeoff - NVML-based availability is a slightly weaker assessment than the default runtime check and falls back to `cudaGetDeviceCount` if NVML is unavailable (documented PyTorch behavior); a non-issue on standard NVIDIA boxes. - Dropping the import-time capability gate means triton kernel modules are imported whenever triton is installed (even on pre-Ampere). Importing them has no CUDA side effects; their use is still gated by `config.use_triton`. ## Tests - Three unit tests for `cuda_capability_major()`'s decision tree (not-initialized → skip, initialized → probe, bad-fork → skip), mocked `torch.cuda`, no GPU required. - `test_forked_child_can_use_cuda_after_importing_deepspeed` — forks after `import deepspeed`, the child runs a real CUDA op, parent asserts success. ## Validation Verified on a CUDA GPU node (NVIDIA, torch 2.4.1+cu121). After `import deepspeed`: - `torch.cuda.is_initialized()` → `False` - a forked child runs `torch.ones(1, device="cuda")` successfully (exit 0) - instrumenting `torch.cuda._lazy_init` shows **0** distinct import-time CUDA-touch sites (down from the `ds_transformer.py:17` initializer + its downstream builder probe). ## Docs Updated `CONTRIBUTING.md` and `docs/contributing.md`: `--forked` is safe now that `import deepspeed` no longer initializes CUDA. cc @tjruwase @loadams @tohtana --------- Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com> Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Masahiro Tanaka <mtanaka@anyscale.com> | 10 天前 |
| Replace calls to `python setup.py sdist` with `python -m build --sdist` (#7069) With future changes coming to pip/python/etc, we need to modify to no longer call `python setup.py ...` and replace it instead: https://packaging.python.org/en/latest/guides/modernize-setup-py-project/#should-setup-py-be-deleted  This means we need to install the build package which is added here as well. Additionally, we pass the `--sdist` flag to only build the sdist rather than the wheel as well here. --------- Signed-off-by: Logan Adams <loadams@microsoft.com> | 1 年前 |
| Add optional torchembed RoPE backend to apply_rotary_pos_emb (#8052) Adds `torchembed` as an optional fused RoPE backend for `deepspeed.sequence.layer.apply_rotary_pos_emb()`, following the same pattern used in transformers and vLLM. ## Changes - **`deepspeed/sequence/layer.py`**: Add `try/except ImportError` guard for `torchembed._triton.fused_rope_forward`. When `torchembed` is installed, the tensor is on CUDA, and `rotary_dim` is even, the function dispatches to the fused triton kernel instead of the PyTorch reference path. - **`setup.py`**: Add `torchembed` extras key (`pip install deepspeed[torchembed]`). - **`tests/unit/sequence/test_apply_rotary_pos_emb.py`**: Numerical correctness vs PyTorch reference across seq_len (1/17/128), dim (32/64/128), and various rotary_dim. Gradient flow test. ## Implementation details The torchembed kernel processes `(*leading, seq_len, dim)` tensors with `RotaryEmbedding(use_fused=True)`, applying Neox-style RoPE via triton. The helper reshapes arbitrary leading dims, calls the kernel, and restores the original shape — transparent to callers. ## Testing ```bash pytest tests/unit/sequence/test_apply_rotary_pos_emb.py -v ``` --------- Signed-off-by: py-ai-dev <py.oss.ml@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> | 7 天前 |
| Add AutoEP (#7938) This PR adds AutoEP (Automatic Expert Parallelism) to DeepSpeed training for HuggingFace MoE models. AutoEP detects MoE blocks during `deepspeed.initialize()`, builds the required EP/EDP process groups, and replaces supported MoE blocks with an EP-enabled execution path, so expert parallelism can be enabled with DeepSpeed config only and without model code changes. Current scope in this PR is the base AutoEP feature: - ZeRO stages 0, 1, and 2 support - checkpoint save/load support - universal checkpoint conversion support ZeRO-3 extensions are intentionally left as follow-up work (#7928 should be merged for this work) Supported presets in this PR: - Mixtral - Qwen3-MoE - DeepSeek-V2 - DeepSeek-V3 For end-to-end benchmarking and testing, an AutoEP example is available in DeepSpeedExamples: - <https://github.com/tohtana/DeepSpeedExamples/tree/tohtana/add_auto_ep/training/expert_parallel> ## Attribution This implementation substantially builds on TorchTitan's MoE / expert-parallel implementation, and we want to explicitly acknowledge that prior work. The TorchTitan-derived pieces in this PR are primarily: - `deepspeed/moe/ep_router.py`: adapted from TorchTitan's `TokenChoiceTopKRouter` - `deepspeed/moe/ep_experts.py`: adapted from TorchTitan's `GroupedExperts` and grouped-GEMM expert execution path - `deepspeed/moe/ep_kernels.py`: adapted from TorchTitan's `TokenReorderer`, `generate_permute_indices`, Triton fill-indices kernel, and token-group alignment / padding helpers - `deepspeed/module_inject/auto_ep_layer.py`: adapts the same router -> reorder -> dispatch -> local expert compute -> combine structure used in TorchTitan's MoE / EP flow Relevant TorchTitan sources: - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/common/moe/moe.py> - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/common/moe/kernels.py> - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/common/moe/utils.py> - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/expert_parallel.py> The DeepSpeed-specific work in this PR is the AutoEP integration layer around those building blocks: - HuggingFace MoE detection and structural validation - model-family presets and custom-config path - weight repacking from HF expert layouts into grouped expert tensors - DeepSpeed runtime group setup and module replacement - DeepSpeed checkpoint save/load and universal checkpoint support - DeepSpeed docs and tests ## Design The implementation is split into a few layers: - `deepspeed/module_inject/auto_ep_config.py` - user config parsing - built-in model presets - validation for EP topology and per-model constraints - `deepspeed/module_inject/auto_ep.py` - scans the model for MoE blocks - validates the detected structure - builds a `MoELayerSpec` for each supported MoE layer - replaces the original HF block with `AutoEPMoELayer` - `deepspeed/module_inject/auto_ep_layer.py` - the drop-in execution wrapper for a detected MoE block - implements router execution, token reorder, EP dispatch/combine, local expert compute, and shared-expert merge - `deepspeed/moe/ep_router.py`, `deepspeed/moe/ep_experts.py`, `deepspeed/moe/ep_kernels.py` - reusable MoE runtime pieces for routing, grouped expert compute, token permutation, and aligned grouped-GEMM execution - `deepspeed/moe/ep_repack.py` - converts HF expert weights into the grouped expert layout expected by the runtime - `deepspeed/runtime/engine.py` and checkpoint conversion code - wires AutoEP into `deepspeed.initialize()` - handles checkpoint save/load metadata and universal checkpoint integration At runtime, the execution path is: 1. detect and replace supported HF MoE blocks during initialization 2. route tokens with the EP router 3. reorder tokens by expert assignment 4. perform all-to-all dispatch across the EP group when `autoep_size > 1` 5. run local grouped expert compute 6. all-to-all combine and restore the original token order 7. merge shared experts if the model has them ## Adding new model support There are two supported ways to extend AutoEP to a new MoE model family. 1. Add a preset in `PRESET_MODELS`. This is the preferred path for a model family we want to support out of the box. A preset defines: - MoE layer pattern - router child name - experts child name - expert weight names / layout - `num_experts` and `top_k` config attributes - routing defaults - optional shared-expert structure 2. Use the custom config path. For models that are not yet built into DeepSpeed, AutoEP can be driven from config with: - `moe_layer_pattern` - `router_pattern` - `expert_pattern` - `expert_w1`, `expert_w2`, `expert_w3` - `num_experts_attr` - `top_k_attr` - optional shared-expert fields Once detection can produce a valid `MoELayerSpec`, the replacement, execution, and checkpoint paths are shared. --------- Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Signed-off-by: Ma, Guokai <guokai.ma@gmail.com> Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com> Co-authored-by: Guokai Ma <guokai.ma@intel.com> | 27 天前 |
| Fix Python 3.9 import-time TypeError in AutoEP ep_router (#8119) ## What Fixes #8102. On Python 3.9, every `deepspeed.initialize()` call crashes at import time with: ``` TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' ``` ## Root cause `deepspeed/moe/ep_router.py` uses PEP 604 unions in eagerly evaluated annotation positions (`num_expert_groups: int | None` and `num_limited_groups: int | None` in `TokenChoiceTopKRouter.__init__`, `expert_bias: torch.Tensor | None = None` in `forward`). Def-signature annotations are evaluated at function-definition time, and an AST scan of the whole package shows `ep_router.py` is the only module that uses this syntax without `from __future__ import annotations`; every sibling AutoEP module (`auto_ep.py`, `auto_ep_layer.py`, `auto_ep_config.py`, `ep_repack.py`, the presets) already has the future import, so their annotations are deferred and harmless. The crash escapes DeepSpeed's own safety net: `engine.py` wraps `from deepspeed.module_inject.auto_ep_layer import AutoEPMoELayer` (which imports `ep_router`) in `except ImportError` inside `_configure_distributed_model`, but the exception raised is `TypeError`, not `ImportError`, so it propagates out of every `deepspeed.initialize()` on Python 3.9. Note the issue body points at `auto_ep.py` / `auto_ep_presets/base.py`; those files already defer annotations, `ep_router.py` is the actual offender. `setup.py` still advertises Python 3.8+ support, so this restores importability rather than changing the syntax: the fix adds the same `from __future__ import annotations` line the rest of AutoEP uses. No runtime behavior change. ## Reproduction Verified on real Python 3.9.6: executing current-master `ep_router.py` raises exactly the reported `TypeError`; with the future import added it imports cleanly. ## Test Adds `TestPy39AnnotationSafety::test_autoep_import_chain_defers_pep604_annotations` to `tests/unit/v1/moe/test_autoep_unit.py`: it AST-scans every module in the AutoEP import chain and fails, naming the module and line numbers, if any of them evaluates a PEP 604 union at import time without deferring annotations. It is version-independent (fails on the bug even when CI runs 3.10+). Fails before this fix (`deepspeed.moe.ep_router ... lines [53, 54, 137]`), passes after. Full `test_autoep_unit.py` suite: 60 passed, 1 skipped. `pre-commit run` clean on both changed files. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com> | 2 天前 |
| force set lf instead of crlf (https://github.com/pre-commit/pre-commit-hooks#mixed-line-ending) (#1598) | 4 年前 |
| Re-enable GPT-J unit tests and refactor inference tests (#3618) | 2 年前 |
| [CI] diff driven test selection (#8077) TLDR: Analyze PR's diff and filter out tests that aren't exercising what has changed, potentially cutting down runtime and expense by 95-99% most of the time. Detailed: Deepspeed's CI takes forever - most of the time burning $$ and wastes dev time for no reason as most changes require just a few tests to run. HF Transformers has a system to select which tests to run based on the diff of the PR - Sylvain Gugger wrote it many years ago since that repo has now probably thousands of tests. Deepspeed's CI isn't too bad but can easily take hours. So I asked Claude Opus 4.8 to replicate the system for Deepspeed. Please have a look. It looks super complicated, so I'm not sure how easy it'd be to maintain/operate unless we always use AI to continue maintaining it. I asked Claude to leave a detailed state file so that it or another model could pick it up where it left. I started with just the slowest costliest workload `.github/workflows/modal-torch-latest.yml` to see if it works well. If you're happy we can replicate it to the rest of the workloads. CC: @loadams, @tjruwase - please tag others if you think they would be helpful to discuss this. --------- Signed-off-by: Stas Bekman <stas@stason.org> | 13 天前 |
| DeepSpeed-FastGen (#4604) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com> Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: Masahiro Tanaka <mtanaka@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> | 2 年前 |
| Update pre-commit version (#6821) | 1 年前 |
| Add codespell to pre-commit checks (#1717) | 4 年前 |
| Fix RTD builds (#4558) * Update .readthedocs.yml * Update requirements-readthedocs.txt | 2 年前 |
| update formatter version and style settings (#3098) | 3 年前 |
| Remove Microsoft Corporation copyright from AGENTS.md and CLAUDE.md (#7932) ## Summary - Remove the `# Copyright (c) Microsoft Corporation.` line from the license header template in both AGENTS.md and CLAUDE.md - The project license header should only contain the SPDX identifier and team attribution ## Test plan - [x] Verify AGENTS.md and CLAUDE.md no longer reference Microsoft Corporation copyright Signed-off-by: Zhipeng Wang <zwanga@wustl.edu> Co-authored-by: Zhipeng Wang <zwanga@wustl.edu> Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> | 3 个月前 |
| Remove Microsoft Corporation copyright from AGENTS.md and CLAUDE.md (#7932) ## Summary - Remove the `# Copyright (c) Microsoft Corporation.` line from the license header template in both AGENTS.md and CLAUDE.md - The project license header should only contain the SPDX identifier and team attribution ## Test plan - [x] Verify AGENTS.md and CLAUDE.md no longer reference Microsoft Corporation copyright Signed-off-by: Zhipeng Wang <zwanga@wustl.edu> Co-authored-by: Zhipeng Wang <zwanga@wustl.edu> Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com> | 3 个月前 |
| Update code owners (#6890) Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> | 1 年前 |
| force set lf instead of crlf (https://github.com/pre-commit/pre-commit-hooks#mixed-line-ending) (#1598) | 4 年前 |
| Update TSC Committers (#7517) Update the affiliations and the TSC Committers. Co-authored-by: Zhipeng Wang <zwanga@wustl.edu> | 10 个月前 |
| Avoid CUDA context initialization during op compatibility checks at import (#8078) ## Summary `import deepspeed` initialized a CUDA context in the parent process, which permanently breaks `fork()`-based multiprocessing (`Cannot re-initialize CUDA in forked subprocess`). This makes importing DeepSpeed fork-safe. Fixes #7918. ## Root cause On a GPU box, `import deepspeed` reached **three** distinct calls that create a CUDA context, each gated differently (which is why a single patch kept missing one): 1. **`torch.cuda.is_available()`** — called during accelerator auto-detection (`real_accelerator.py`) and in every CUDA op builder's `is_compatible()`. By default it runs `cudaGetDeviceCount → cuInit`, creating a context. Per the [PyTorch docs](https://docs.pytorch.org/docs/stable/generated/torch.cuda.is_available.html) this is only avoided with `PYTORCH_NVML_BASED_CUDA_CHECK=1`. Note it does **not** set `torch.cuda.is_initialized()`, so an import-time `assert not is_initialized()` is a false-green. 2. **`torch.cuda.get_device_properties(0)`** — in the eight builders' `is_compatible()` (run at import by `git_version_info.py`); triggers `torch.cuda._lazy_init()`. 3. **`is_triton_supported()` → `torch.cuda.get_device_capability()`** — called at module import in `ds_transformer.py`, gated on `deepspeed.HAS_TRITON`. This only fires when **triton is installed**, so it was invisible in triton-less environments — but it was the first initializer on a real GPU node. ## Fix 1. `deepspeed/__init__.py` sets `os.environ.setdefault("PYTORCH_NVML_BASED_CUDA_CHECK", "1")` as the very first statement, so `torch.cuda.is_available()` uses the NVML-based check and never initializes a context. `setdefault()` preserves an explicit user setting. 2. `CUDAOpBuilder.cuda_capability_major()` (in `op_builder/builder.py`) reads compute capability only when a context already exists (`is_initialized()`) and we are not in a forked child (`_is_in_bad_fork()`, mirroring #7977); otherwise returns `None`. All eight builders route through it and skip the capability gate when probing is unsafe. 3. `ds_transformer.py` imports the triton kernels whenever triton is installed (`if deepspeed.HAS_TRITON:`) instead of also gating on `is_triton_supported()`. The capability probe is removed from import; actual triton use stays gated at runtime by `config.use_triton`, where CUDA is already initialized. ## Behavior / tradeoff - NVML-based availability is a slightly weaker assessment than the default runtime check and falls back to `cudaGetDeviceCount` if NVML is unavailable (documented PyTorch behavior); a non-issue on standard NVIDIA boxes. - Dropping the import-time capability gate means triton kernel modules are imported whenever triton is installed (even on pre-Ampere). Importing them has no CUDA side effects; their use is still gated by `config.use_triton`. ## Tests - Three unit tests for `cuda_capability_major()`'s decision tree (not-initialized → skip, initialized → probe, bad-fork → skip), mocked `torch.cuda`, no GPU required. - `test_forked_child_can_use_cuda_after_importing_deepspeed` — forks after `import deepspeed`, the child runs a real CUDA op, parent asserts success. ## Validation Verified on a CUDA GPU node (NVIDIA, torch 2.4.1+cu121). After `import deepspeed`: - `torch.cuda.is_initialized()` → `False` - a forked child runs `torch.ones(1, device="cuda")` successfully (exit 0) - instrumenting `torch.cuda._lazy_init` shows **0** distinct import-time CUDA-touch sites (down from the `ds_transformer.py:17` initializer + its downstream builder probe). ## Docs Updated `CONTRIBUTING.md` and `docs/contributing.md`: `--forked` is safe now that `import deepspeed` no longer initializes CUDA. cc @tjruwase @loadams @tohtana --------- Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com> Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Masahiro Tanaka <mtanaka@anyscale.com> | 10 天前 |
| Adding the governance doc (#6748) Drafted governance doc for the LFAI. Co-authored-by: Minjia Zhang <minjiaz@illinois.edu> | 1 年前 |
| added full apache license (#3119) | 3 年前 |
| [manifest] update mainfest to add hpp file in deepspeed. (#5533) Hi @loadams, Could you please help to review this pr? After add hpp files in csrc, we found sometimes the hpp headers will still be excluded from the op src packaging, so we add the hpp file in deepspeed to make sure the hpp header in deepspeed package, to ensure jit load to compile the xpu/fused_adam ops in 0.14.2. | 2 年前 |
| Abstract accelerator (step 2) (#2560) * Abstract accelerator (step 2) * more flex op_builder path for both installation and runtime * add SpatialInferenceBuilder into cuda_accelerator.py * use reflection to make cuda_accelerator adapt to CUDA op builder change automatically * clean up deepspeed/__init__.py * add comments in cuda_accelerator for no torch path * Update deepspeed/env_report.py Change env_report.py according to suggestion Co-authored-by: Michael Wyatt <mrwyattii@gmail.com> * reduce the range of try...except for better code clarity * Add porting for deepspeed/ops/random_ltd/dropping_utils.py * move accelerator to top directory and create symlink under deepspeed Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Michael Wyatt <mrwyattii@gmail.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> | 3 年前 |
| add `Makefile` to ease maintenance (#7267) adding `Makefile` with `make format` and `make test` to make things easier to maintain. --------- Signed-off-by: Stas Bekman <stas@stason.org> | 1 年前 |
| Update publication list in README.md (#8095) Add EMNLP 2025 paper which leverages DeepSpeed as the training engine for on-policy distillation. Signed-off-by: GitHub <noreply@github.com> | 12 天前 |
| Update SECURITY.md to point to GitHub reporting rather than Microsoft (#7692) Heavily borrowed from https://github.com/vllm-project/vllm/blob/main/SECURITY.md | 7 个月前 |
| Add AutoEP (#7938) This PR adds AutoEP (Automatic Expert Parallelism) to DeepSpeed training for HuggingFace MoE models. AutoEP detects MoE blocks during `deepspeed.initialize()`, builds the required EP/EDP process groups, and replaces supported MoE blocks with an EP-enabled execution path, so expert parallelism can be enabled with DeepSpeed config only and without model code changes. Current scope in this PR is the base AutoEP feature: - ZeRO stages 0, 1, and 2 support - checkpoint save/load support - universal checkpoint conversion support ZeRO-3 extensions are intentionally left as follow-up work (#7928 should be merged for this work) Supported presets in this PR: - Mixtral - Qwen3-MoE - DeepSeek-V2 - DeepSeek-V3 For end-to-end benchmarking and testing, an AutoEP example is available in DeepSpeedExamples: - <https://github.com/tohtana/DeepSpeedExamples/tree/tohtana/add_auto_ep/training/expert_parallel> ## Attribution This implementation substantially builds on TorchTitan's MoE / expert-parallel implementation, and we want to explicitly acknowledge that prior work. The TorchTitan-derived pieces in this PR are primarily: - `deepspeed/moe/ep_router.py`: adapted from TorchTitan's `TokenChoiceTopKRouter` - `deepspeed/moe/ep_experts.py`: adapted from TorchTitan's `GroupedExperts` and grouped-GEMM expert execution path - `deepspeed/moe/ep_kernels.py`: adapted from TorchTitan's `TokenReorderer`, `generate_permute_indices`, Triton fill-indices kernel, and token-group alignment / padding helpers - `deepspeed/module_inject/auto_ep_layer.py`: adapts the same router -> reorder -> dispatch -> local expert compute -> combine structure used in TorchTitan's MoE / EP flow Relevant TorchTitan sources: - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/common/moe/moe.py> - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/common/moe/kernels.py> - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/common/moe/utils.py> - <https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/expert_parallel.py> The DeepSpeed-specific work in this PR is the AutoEP integration layer around those building blocks: - HuggingFace MoE detection and structural validation - model-family presets and custom-config path - weight repacking from HF expert layouts into grouped expert tensors - DeepSpeed runtime group setup and module replacement - DeepSpeed checkpoint save/load and universal checkpoint support - DeepSpeed docs and tests ## Design The implementation is split into a few layers: - `deepspeed/module_inject/auto_ep_config.py` - user config parsing - built-in model presets - validation for EP topology and per-model constraints - `deepspeed/module_inject/auto_ep.py` - scans the model for MoE blocks - validates the detected structure - builds a `MoELayerSpec` for each supported MoE layer - replaces the original HF block with `AutoEPMoELayer` - `deepspeed/module_inject/auto_ep_layer.py` - the drop-in execution wrapper for a detected MoE block - implements router execution, token reorder, EP dispatch/combine, local expert compute, and shared-expert merge - `deepspeed/moe/ep_router.py`, `deepspeed/moe/ep_experts.py`, `deepspeed/moe/ep_kernels.py` - reusable MoE runtime pieces for routing, grouped expert compute, token permutation, and aligned grouped-GEMM execution - `deepspeed/moe/ep_repack.py` - converts HF expert weights into the grouped expert layout expected by the runtime - `deepspeed/runtime/engine.py` and checkpoint conversion code - wires AutoEP into `deepspeed.initialize()` - handles checkpoint save/load metadata and universal checkpoint integration At runtime, the execution path is: 1. detect and replace supported HF MoE blocks during initialization 2. route tokens with the EP router 3. reorder tokens by expert assignment 4. perform all-to-all dispatch across the EP group when `autoep_size > 1` 5. run local grouped expert compute 6. all-to-all combine and restore the original token order 7. merge shared experts if the model has them ## Adding new model support There are two supported ways to extend AutoEP to a new MoE model family. 1. Add a preset in `PRESET_MODELS`. This is the preferred path for a model family we want to support out of the box. A preset defines: - MoE layer pattern - router child name - experts child name - expert weight names / layout - `num_experts` and `top_k` config attributes - routing defaults - optional shared-expert structure 2. Use the custom config path. For models that are not yet built into DeepSpeed, AutoEP can be driven from config with: - `moe_layer_pattern` - `router_pattern` - `expert_pattern` - `expert_w1`, `expert_w2`, `expert_w3` - `num_experts_attr` - `top_k_attr` - optional shared-expert fields Once detection can produce a valid `MoELayerSpec`, the replacement, execution, and checkpoint paths are shared. --------- Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Signed-off-by: Ma, Guokai <guokai.ma@gmail.com> Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com> Co-authored-by: Guokai Ma <guokai.ma@intel.com> | 27 天前 |
| DeepCompile for enhanced compiler integration (#7154) This PR introduces *DeepCompile*, a new feature that efficiently integrates compiler optimizations with other DeepSpeed features. DeepCompile utilizes torch's dynamo to capture the computation graph and modifies it to incorporate DeepSpeed’s optimizations seamlessly. Currently, DeepCompile supports ZeRO-1 and ZeRO-3, with enhancements such as proactive prefetching and selective unsharding to improve performance. (More details will be added later.) --------- Signed-off-by: Masahiro Tanaka <mtanaka@microsoft.com> Signed-off-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: zafarsadiq <zafarsadiq120@gmail.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> | 1 年前 |
| Introduce pydantic_v1 compatibility module for pydantic>=2.0.0 support (#4407) * Introduce pydantic_v1 compatibility module for pydantic>=2.0.0 support | 2 年前 |
| Replace calls to `python setup.py sdist` with `python -m build --sdist` (#7069) With future changes coming to pip/python/etc, we need to modify to no longer call `python setup.py ...` and replace it instead: https://packaging.python.org/en/latest/guides/modernize-setup-py-project/#should-setup-py-be-deleted  This means we need to install the build package which is added here as well. Additionally, we pass the `--sdist` flag to only build the sdist rather than the wheel as well here. --------- Signed-off-by: Logan Adams <loadams@microsoft.com> | 1 年前 |
| Seeded unit tests (#1072) * is not -> != * Use pytest-randomly to seed unit tests. | 5 年前 |
| Add optional torchembed RoPE backend to apply_rotary_pos_emb (#8052) Adds `torchembed` as an optional fused RoPE backend for `deepspeed.sequence.layer.apply_rotary_pos_emb()`, following the same pattern used in transformers and vLLM. ## Changes - **`deepspeed/sequence/layer.py`**: Add `try/except ImportError` guard for `torchembed._triton.fused_rope_forward`. When `torchembed` is installed, the tensor is on CUDA, and `rotary_dim` is even, the function dispatches to the fused triton kernel instead of the PyTorch reference path. - **`setup.py`**: Add `torchembed` extras key (`pip install deepspeed[torchembed]`). - **`tests/unit/sequence/test_apply_rotary_pos_emb.py`**: Numerical correctness vs PyTorch reference across seq_len (1/17/128), dim (32/64/128), and various rotary_dim. Gradient flow test. ## Implementation details The torchembed kernel processes `(*leading, seq_len, dim)` tensors with `RotaryEmbedding(use_fused=True)`, applying Neox-style RoPE via triton. The helper reshapes arbitrary leading dims, calls the kernel, and restores the original shape — transparent to callers. ## Testing ```bash pytest tests/unit/sequence/test_apply_rotary_pos_emb.py -v ``` --------- Signed-off-by: py-ai-dev <py.oss.ml@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> | 7 天前 |
| Update version post release (#8070) Signed-off-by: Logan Adams <loadams@microsoft.com> | 22 天前 |
