已合并
修改中英文一致性问题 #2608
zangyan创建于 15 天前
修改中英文一致性问题 #2608
已合并
共 19 个文件变更+237-234
| @@ -6,7 +6,7 @@ | |||
| 6 | <!-- If this PR does not involve an Issue, enter "NA". --> | 6 | <!-- If this PR does not involve an Issue, enter "NA". --> |
| 7 | 7 | ||
| 8 | ## Testing | 8 | ## Testing |
| 9 | -<!-- Describe what tests you performed to verify your changes. This includes but is not limited to constructing corresponding test cases, secondary smoke tests, operator generalization, and so on. --> | 9 | +<!-- Describe what tests you performed to verify your changes. This includes but is not limited to constructing corresponding xx test cases, secondary smoke tests, operator generalization, and so on. --> |
| 10 | 10 | ||
| 11 | ## Documentation Update | 11 | ## Documentation Update |
| 12 | <!-- If this PR includes documentation updates, indicate them here. For example: Updated the README.md file. --> | 12 | <!-- If this PR includes documentation updates, indicate them here. For example: Updated the README.md file. --> |
| @@ -30,7 +30,7 @@ Community contributions such as new operators, algorithms, or extended features | |||
| 30 | - In principle, **avoid modifying** any files under `src/` as much as possible to prevent polluting stable code. | 30 | - In principle, **avoid modifying** any files under `src/` as much as possible to prevent polluting stable code. |
| 31 | - If modifications to `src/` are necessary, explicitly state the reason and impact scope in the PR description and obtain Committer review. | 31 | - If modifications to `src/` are necessary, explicitly state the reason and impact scope in the PR description and obtain Committer review. |
| 32 | - Code changes in a PR should focus on the `experimental/` directory. | 32 | - Code changes in a PR should focus on the `experimental/` directory. |
| 33 | -- Provide a runtime switch for quick rollback. For details, see [experimental/README.md](./experimental/README_en.md). | 33 | +- Provide a runtime switch for quick rollback. For details, see [experimental/README_en.md](./experimental/README_en.md). |
| 34 | 34 | ||
| 35 | ## Contribution Process | 35 | ## Contribution Process |
| 36 | 36 | ||
| @@ -24,7 +24,7 @@ HCCL is a core component of CANN. It supports multiple AI frameworks upward and | |||
| 24 | HCCL consists of the HCCL collective communication library and the HCOMM (Huawei Communication) basic communication library: | 24 | HCCL consists of the HCCL collective communication library and the HCOMM (Huawei Communication) basic communication library: |
| 25 | 25 | ||
| 26 | - HCCL: includes built-in and extended communication operators, and provides external communication operator APIs. | 26 | - HCCL: includes built-in and extended communication operators, and provides external communication operator APIs. |
| 27 | -- [HCOMM](https://gitcode.com/cann/hcomm): adopts a layered and decoupled design, dividing communication capabilities into a control plane and a data plane. | 27 | +- [HCOMM](https://gitcode.com/cann/hcomm): adopts a layered and decoupled design, dividing communication capabilities into a control plane and a data plane. |
| 28 | 28 | ||
| 29 | ## Directory Structure | 29 | ## Directory Structure |
| 30 | 30 | ||
| @@ -24,8 +24,8 @@ | |||
| 24 | 24 | ||
| 25 | | 类型 | 开源代码地址 | 文件名 | 公网 IP 地址/公网 URL 地址/域名/邮箱地址/压缩文件地址 | 用途说明 | | 25 | | 类型 | 开源代码地址 | 文件名 | 公网 IP 地址/公网 URL 地址/域名/邮箱地址/压缩文件地址 | 用途说明 | |
| 26 | | :--: | :----------: | :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------- | | 26 | | :--: | :----------: | :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------- | |
| 27 | -| 依赖 | 不涉及 | cmake/third_party/makeself-fetch.cmake | https://gitcode.com/cann-src-third-party/makeself/releases/download/release-2.5.0-patch1.0/makeself-release-2.5.0-patch1.tar.gz | 从 gitcode 下载 makeself 源码,作为编译依赖 | | 27 | +| 依赖 | 不涉及 | cmake/third_party/makeself-fetch.cmake | https://gitcode.com/cann-src-third-party/makeself/releases/download/release-2.5.0-patch1.0/makeself-release-2.5.0-patch1.tar.gz | 从 GitCode 下载 makeself 源码,作为编译依赖 | |
| 28 | -| 依赖 | 不涉及 | cmake/third_party/gtest.cmake | https://gitcode.com/cann-src-third-party/googletest/releases/download/v1.14.0/googletest-1.14.0.tar.gz | 从 gitcode 下载 googletest 源码,作为编译依赖 | | 28 | +| 依赖 | 不涉及 | cmake/third_party/gtest.cmake | https://gitcode.com/cann-src-third-party/googletest/releases/download/v1.14.0/googletest-1.14.0.tar.gz | 从 GitCode 下载 Google Test 源码,作为编译依赖 | |
| 29 | 29 | ||
| 30 | --- | 30 | --- |
| 31 | 31 | ||
| @@ -7,7 +7,7 @@ To ensure security and minimize permissions, you are not advised to use administ | |||
| 7 | ## File Permission Control | 7 | ## File Permission Control |
| 8 | 8 | ||
| 9 | - You are advised to set the system `umask` value to `0027` or higher on hosts (including host machines) and containers. This ensures that new folders have a default maximum permission of `750` and new files have a default maximum permission of `640`. | 9 | - You are advised to set the system `umask` value to `0027` or higher on hosts (including host machines) and containers. This ensures that new folders have a default maximum permission of `750` and new files have a default maximum permission of `640`. |
| 10 | -- You are advised to take security measures such as permission control on sensitive files, including personal privacy data, business assets, and source files. For example, permissions for the project installation directory and public input data files must follow the recommendations in [A–Recommended Maximum Permissions for Files and Folders in Different Scenarios](#a-recommended-maximum-permissions-for-files-and-folders-in-different-scenarios). | 10 | +- You are advised to take security measures such as permission control on sensitive files, including personal privacy data, commercial assets, and source files. For example, permissions for the project installation directory and public input data files must follow the recommendations in [A–Recommended Maximum Permissions for Files and Folders in Different Scenarios](#a-recommended-maximum-permissions-for-files-and-folders-in-different-scenarios). |
| 11 | - During installation and usage, you must enforce proper permission control, referring to the same [A–Recommended Maximum Permissions for Files and Folders in Different Scenarios](#a-recommended-maximum-permissions-for-files-and-folders-in-different-scenarios). | 11 | - During installation and usage, you must enforce proper permission control, referring to the same [A–Recommended Maximum Permissions for Files and Folders in Different Scenarios](#a-recommended-maximum-permissions-for-files-and-folders-in-different-scenarios). |
| 12 | 12 | ||
| 13 | ## Build Security Statement | 13 | ## Build Security Statement |
| @@ -1,7 +1,7 @@ | |||
| 1 | # HCCL Document Overview | 1 | # HCCL Document Overview |
| 2 | 2 | ||
| 3 | - [Collective Communication Library User Guide](https://www.hiascend.com/document/detail/en/canncommercial/850/commlib/hcclug/hcclug_000001.html): Introduces basic HCCL concepts, how to use HCCL APIs for collective communication development, how to develop communication operators, and fault handling with common configurations. | 3 | - [Collective Communication Library User Guide](https://www.hiascend.com/document/detail/en/canncommercial/850/commlib/hcclug/hcclug_000001.html): Introduces basic HCCL concepts, how to use HCCL APIs for collective communication development, how to develop communication operators, and fault handling with common configurations. |
| 4 | -- [Communication Operator Development Guide (Chinese)](https://gitcode.com/cann/hcomm/blob/master/docs/zh/comm_op_dev_guide/README.md):Introduces how to develop communication operators based on the APIs provided by the HCOMM communication foundation library. | 4 | +- [Communication Operator Development Guide (Chinese)](https://gitcode.com/cann/hcomm/blob/master/docs/zh/comm_op_dev_guide/README.md): Introduces how to develop communication operators based on the APIs provided by the HCOMM communication foundation library. |
| 5 | 5 | ||
| 6 | ## Technical Articles | 6 | ## Technical Articles |
| 7 | 7 | ||
| @@ -14,7 +14,7 @@ Introduce BIRS (Batchsize Invariant ReduceScatter) — a novel batch-invariant R | |||
| 14 | 14 | ||
| 15 | ### Industry Demand for Deterministic Collective Communication | 15 | ### Industry Demand for Deterministic Collective Communication |
| 16 | 16 | ||
| 17 | -In distributed training and inference, **deterministic collective communication** requires that reduction operations (AllReduce, ReduceScatter, etc.) produce **bit-identical** results for the same input, regardless of batch size, process count, or memory sharding strategy. This requirement has become a hard constraint in multiple industry scenarios: | 17 | +In distributed training and inference, **deterministic collective communication** requires that reduction operations (AllReduce, ReduceScatter, etc.) produce **bit-identical** results for the same input, regardless of batch size, process count, or memory sharding strategy. This requirement has become a hard constraint in multiple industry scenarios. |
| 18 | 18 | ||
| 19 | #### 1. Training Reproducibility and CI/CD | 19 | #### 1. Training Reproducibility and CI/CD |
| 20 | 20 | ||
| @@ -62,9 +62,9 @@ On the A3 server topology (SIO + HCCS hybrid interconnect), RHD cannot simultane | |||
| 62 | 62 | ||
| 63 | The BIRS algorithm is designed for the 2D topology characteristics of A3 servers, maintaining batch invariance while: | 63 | The BIRS algorithm is designed for the 2D topology characteristics of A3 servers, maintaining batch invariance while: |
| 64 | 64 | ||
| 65 | -- **First round**: Performing SendReduce over SIO links (cross X-axis reduction) | 65 | +- **First round**: Performing SendReduce over SIO links (cross X-axis reduction). |
| 66 | -- **Subsequent rounds**: Simultaneously utilizing SIO (reduction) and HCCS (intermediate result transfer) links | 66 | +- **Subsequent rounds**: Simultaneously utilizing SIO (reduction) and HCCS (intermediate result transfer) links. |
| 67 | -- Achieving near-optimal bandwidth utilization, with only the first round not fully utilizing bandwidth | 67 | +- Achieving near-optimal bandwidth utilization, with only the first round not fully utilizing bandwidth. |
| 68 | 68 | ||
| 69 | ## Detailed Design | 69 | ## Detailed Design |
| 70 | 70 | ||
| @@ -163,7 +163,7 @@ HcclReduceScatter(sendBuf, recvBuf, recvCount, dataType, op, comm, stream); | |||
| 163 | 163 | ||
| 164 | BIRS constructs a logical 2D layout over the A3/16P topology: | 164 | BIRS constructs a logical 2D layout over the A3/16P topology: |
| 165 | 165 | ||
| 166 | -``` | 166 | +```text |
| 167 | rankSizeX = 2 // X-axis direction (SIO links) | 167 | rankSizeX = 2 // X-axis direction (SIO links) |
| 168 | rankSizeY = rankSize / rankSizeX // Y-axis direction (HCCS links) | 168 | rankSizeY = rankSize / rankSizeX // Y-axis direction (HCCS links) |
| 169 | ``` | 169 | ``` |
| @@ -183,7 +183,7 @@ Each rank maintains the following topology information: | |||
| 183 | 183 | ||
| 184 | BIRS uses scratch memory to store intermediate reduction results (IM), with a strided layout to satisfy the 910B minimum slice alignment requirement: | 184 | BIRS uses scratch memory to store intermediate reduction results (IM), with a strided layout to satisfy the 910B minimum slice alignment requirement: |
| 185 | 185 | ||
| 186 | -``` | 186 | +```text |
| 187 | localStrideSize = RoundUp(sliceSize, HCCL_MIN_SLICE_ALIGN_910B) | 187 | localStrideSize = RoundUp(sliceSize, HCCL_MIN_SLICE_ALIGN_910B) |
| 188 | 188 | ||
| 189 | Scratch buffer is divided into 2 regions with N slots each: Region A is used to accumulate intermediate results for HCCS, | 189 | Scratch buffer is divided into 2 regions with N slots each: Region A is used to accumulate intermediate results for HCCS, |
| @@ -228,14 +228,15 @@ Inter-thread synchronization is performed via `PreSyncInterThreads` / `PostSyncI | |||
| 228 | The core property of the BIRS algorithm is **batch invariance**: the order of reduction additions on each rank is strictly identical, regardless of batch size or memory slicing. | 228 | The core property of the BIRS algorithm is **batch invariance**: the order of reduction additions on each rank is strictly identical, regardless of batch size or memory slicing. |
| 229 | 229 | ||
| 230 | **Notation**: | 230 | **Notation**: |
| 231 | -- `S(d, i)`: The i-th slice of the input message on device d | 231 | + |
| 232 | -- `rankSizeX = 2`, `rankSizeY = rankSize / 2` | 232 | +- `S(d, i)`: The i-th slice of the input message on device d. |
| 233 | -- `sio_rank = rank XOR 1` (SIO peer) | 233 | +- `rankSizeX = 2`, `rankSizeY = rankSize / 2`. |
| 234 | -- `hccs_ranks[i] = (rank + rankSizeX * i) % rankSize` (HCCS peer sequence) | 234 | +- `sio_rank = rank XOR 1` (SIO peer). |
| 235 | +- `hccs_ranks[i] = (rank + rankSizeX * i) % rankSize` (HCCS peer sequence). | ||
| 235 | 236 | ||
| 236 | #### 4.2 Main Communication Loop | 237 | #### 4.2 Main Communication Loop |
| 237 | 238 | ||
| 238 | -``` | 239 | +```c |
| 239 | // Initial: copy the input slice corresponding to the first HCCS peer into scratch memory | 240 | // Initial: copy the input slice corresponding to the first HCCS peer into scratch memory |
| 240 | LocalCopy(input[S(hccs_ranks[0])], scratch[IM_0]) | 241 | LocalCopy(input[S(hccs_ranks[0])], scratch[IM_0]) |
| 241 | 242 | ||
| @@ -268,7 +269,7 @@ for round in 0 ... hccs_ranks.size(): | |||
| 268 | 269 | ||
| 269 | After all rounds complete, each rank holds `rankSizeY` intermediate results in scratch memory. These are merged via a **tree-based local reduction**: | 270 | After all rounds complete, each rank holds `rankSizeY` intermediate results in scratch memory. These are merged via a **tree-based local reduction**: |
| 270 | 271 | ||
| 271 | -``` | 272 | +```text |
| 272 | // Collect all intermediate result offsets | 273 | // Collect all intermediate result offsets |
| 273 | vec = [IM_0, IM_1, ..., IM_{rankSizeY-1}] // this rank's result is at the correct position | 274 | vec = [IM_0, IM_1, ..., IM_{rankSizeY-1}] // this rank's result is at the correct position |
| 274 | 275 | ||
| @@ -297,14 +298,14 @@ The BIRS algorithm currently has the following constraints: | |||
| 297 | 298 | ||
| 298 | | Constraint | Description | | 299 | | Constraint | Description | |
| 299 | |------------|-------------| | 300 | |------------|-------------| |
| 300 | -| Platform | A3 servers only (SIO + HCCS hybrid topology) | | 301 | +| Platform | A3 servers only (SIO + HCCS hybrid topology). | |
| 301 | -| rankSize | Must be even (`rankSize % 2 == 0`), typical values: 4, 8, 16 | | 302 | +| rankSize | Must be even (`rankSize % 2 == 0`), typical values: 4, 8, 16. | |
| 302 | -| Communication domain | Both Intra-server and Inter-server are supported | | 303 | +| Communication domain | Both Intra-server and Inter-server are supported. | |
| 303 | -| Data alignment | Slice sizes must satisfy `HCCL_MIN_SLICE_ALIGN_910B` alignment requirements | | 304 | +| Data alignment | Slice sizes must satisfy `HCCL_MIN_SLICE_ALIGN_910B` alignment requirements. | |
| 304 | 305 | ||
| 305 | -ReduceScatterBIRS() is the recommended choice for single-server A3 scenario (rankSize <= 16), ReduceScatterBIRSInter() is chosen automatically for multi-server A3 scenario. | 306 | +ReduceScatterBIRS() is the recommended choice for the single-server A3 scenario (rankSize <= 16), ReduceScatterBIRSInter() is chosen automatically for multi-server A3 scenario. |
| 306 | 307 | ||
| 307 | -When conditions are not met, the workflow exit and log error messages using hccl. The user must follow the recommendations in the error logs or manually adjust the parameters to comply with the restrictions. | 308 | +When conditions are not met, the workflow exits and logs error messages via HCCL. The user must follow the recommendations in the error logs or manually adjust the parameters to comply with the restrictions. |
| 308 | 309 | ||
| 309 | #### 5.3 Rollout Strategy | 310 | #### 5.3 Rollout Strategy |
| 310 | 311 | ||
| @@ -325,12 +326,12 @@ When conditions are not met, the workflow exit and log error messages using hccl | |||
| 325 | 326 | ||
| 326 | #### 6.2 Batch Invariance Verification | 327 | #### 6.2 Batch Invariance Verification |
| 327 | 328 | ||
| 328 | -- Execute ReduceScatter with the same input data but different batch sizes | 329 | +- Execute ReduceScatter with the same input data but different batch sizes. |
| 329 | -- Verify that output results are bit-identical | 330 | +- Verify that output results are bit-identical. |
| 330 | 331 | ||
| 331 | #### 6.3 Performance Testing | 332 | #### 6.3 Performance Testing |
| 332 | 333 | ||
| 333 | -- Compare against the RHD algorithm, measuring Task Duration across different message sizes | 334 | +- Compare against the RHD algorithm, measuring Task Duration across different message sizes. |
| 334 | - Expected: For message sizes >= 16MB, BIRS achieves up to 25% improvement over RHD. | 335 | - Expected: For message sizes >= 16MB, BIRS achieves up to 25% improvement over RHD. |
| 335 | - Note: At the moment of this RFC creation kernel submission mechanism in HCCL is slower than the one of HCOMM, so 25% performance improvement applies only to operator execution time (without submission overhead). | 336 | - Note: At the moment of this RFC creation kernel submission mechanism in HCCL is slower than the one of HCOMM, so 25% performance improvement applies only to operator execution time (without submission overhead). |
| 336 | 337 | ||
| @@ -342,10 +343,10 @@ When conditions are not met, the workflow exit and log error messages using hccl | |||
| 342 | 343 | ||
| 343 | | Risk | Impact | Mitigation | | 344 | | Risk | Impact | Mitigation | |
| 344 | |------|--------|------------| | 345 | |------|--------|------------| |
| 345 | -| BIRS only available for specific rankSize (even numbers) | Odd rank scenarios cannot use BIRS | `MatchBIRS()` check auto-falls back to existing algorithms; document constraints clearly | | 346 | +| BIRS only available for specific rankSize (even numbers) | Odd rank scenarios cannot use BIRS | `MatchBIRS()` check auto-falls back to existing algorithms; document constraints clearly. | |
| 346 | -| Experimental code may introduce stability issues | Affects overall HCCL reliability | Dual gating (compile + runtime) isolation; independent `experimental/` directory; disabled by default | | 347 | +| Experimental code may introduce stability issues | Affects overall HCCL reliability | Dual gating (compile + runtime) isolation; independent `experimental/` directory; disabled by default. | |
| 347 | -| Additional scratch memory overhead | Increased memory usage for large messages | Requires `2 * rankSizeY × localStrideSize` scratch space; pre-allocated via `CalcResRequest` | | 348 | +| Additional scratch memory overhead | Increased memory usage for large messages | Requires `2 * rankSizeY × localStrideSize` scratch space; pre-allocated via `CalcResRequest`. | |
| 348 | -| A3 topology assumption (SIO + HCCS) may not apply to other platforms | Cross-platform compatibility | Algorithm explicitly bound to A3 topology characteristics; other platforms require independent adaptation | | 349 | +| A3 topology assumption (SIO + HCCS) may not apply to other platforms | Cross-platform compatibility | Algorithm explicitly bound to A3 topology characteristics; other platforms require independent adaptation. | |
| 349 | 350 | ||
| 350 | ## Alternative Approaches | 351 | ## Alternative Approaches |
| 351 | 352 | ||
| @@ -353,7 +354,7 @@ N/A | |||
| 353 | 354 | ||
| 354 | ## Open Questions | 355 | ## Open Questions |
| 355 | 356 | ||
| 356 | -1. **AllReduce extension**: Batch-invariant AllReduce which follows the same ideas will be submitted in separate PR | 357 | +1. **AllReduce extension**: Batch-invariant AllReduce which follows the same ideas will be submitted in separate PR. |
| 357 | 2. **Efficient support for arbitrary rank enumeration**: Current solution assumes default rank enumeration where rankID of SIO neighbour of RankX can be calculated as (RankX XOR 1). In case of other rank enumerations BIRS is functional but doesn't deliver performance advantage over RHD. Efficient support for custom enumerations have already been implemented and will be submitted in the next PR. | 358 | 2. **Efficient support for arbitrary rank enumeration**: Current solution assumes default rank enumeration where rankID of SIO neighbour of RankX can be calculated as (RankX XOR 1). In case of other rank enumerations BIRS is functional but doesn't deliver performance advantage over RHD. Efficient support for custom enumerations have already been implemented and will be submitted in the next PR. |
| 358 | 359 | ||
| 359 | --- | 360 | --- |
| @@ -1,114 +1,115 @@ | |||
| 1 | -# RFC: Bandwidth-efficient Invariant ReduceScatter (BIRS) Algorithm | 1 | +# RFC:Bandwidth-efficient Invariant ReduceScatter (BIRS)算法 |
| 2 | 2 | ||
| 3 | -- Start Date: 2026-04-24 | 3 | +- 起始日期:2026-04-24 |
| 4 | -- RFC PR: cann/hccl#657 | 4 | +- RFC PR编号:cann/hccl#657 |
| 5 | -- Related Issues: cann/hcomm#139, cann/hccl#96 | 5 | +- 相关Issue:cann/hcomm#139, cann/hccl#96 |
| 6 | 6 | ||
| 7 | --- | 7 | --- |
| 8 | 8 | ||
| 9 | -## Summary | 9 | +## 概要 |
| 10 | 10 | ||
| 11 | -Introduce BIRS (Batchsize Invariant ReduceScatter) — a novel batch-invariant ReduceScatter algorithm for the Ascend A3 server topology in HCCL. While guaranteeing deterministic reduction ordering (bit-level reproducibility), the algorithm achieves up to 25% performance improvement (operator execution time without submission overhead) over the existing RHD (Recursive Halving-Doubling) algorithm for large message sizes by more fully utilizing the SIO + HCCS hybrid interconnect bandwidth. | 11 | +BIRS(Batchsize Invariant ReduceScatter,批大小不变的 ReduceScatter 算法)是HCCL面向Ascend A3 服务器拓扑提出的一种新型batch大小不变的ReduceScatter算法。该算法在保证确定性归约顺序(即比特级可重现性)的前提下,通过更充分地利用SIO + HCCS混合互连带宽,在大消息场景下相比现有的RHD(Recursive Halving-Doubling)算法,可获得最高25%的性能提升(算子执行时间,不含下发开销)。 |
| 12 | 12 | ||
| 13 | -## Background and Motivation | 13 | +## 背景与动机 |
| 14 | 14 | ||
| 15 | -### Industry Demand for Deterministic Collective Communication | 15 | +### 业界对确定性集合通信的需求 |
| 16 | 16 | ||
| 17 | -In distributed training and inference, **deterministic collective communication** requires that reduction operations (AllReduce, ReduceScatter, etc.) produce **bit-identical** results for the same input, regardless of batch size, process count, or memory sharding strategy. This requirement has become a hard constraint in multiple industry scenarios: | 17 | +在分布式训练和推理中,**确定性集合通信**要求归约操作(AllReduce、ReduceScatter等)在输入相同的情况下,无论批大小、进程数或内存分片策略如何变化,都能产生**比特级**完全相同的结果。这一需求已在多个行业场景中成为硬性约束。 |
| 18 | 18 | ||
| 19 | -#### 1. Training Reproducibility & CI/CD | 19 | +#### 1. 训练可复现性与CI/CD |
| 20 | 20 | ||
| 21 | -Reproducible training is essential for trustworthy research and production pipelines. Non‑deterministic reductions introduce floating‑point noise that masks bugs and makes results impossible to compare across runs. | 21 | +可复现的训练是可信研究和生产流水线的基础。非确定性归约会引入浮点噪声,掩盖缺陷并使不同运行之间的结果无法比较。 |
| 22 | 22 | ||
| 23 | -- **Picard (2021)** (“Torch.manual_seed(3407) is all you need”) demonstrates that random seed variations alone can produce statistically significant outliers in final model performance – when reduction ordering is also non‑deterministic, the variance grows even larger. ([arXiv:2109.08203](https://arxiv.org/abs/2109.08203)) | 23 | +- **Picard (2021)**("Torch.manual_seed(3407) is all you need")证明仅随机种子变化就能在最终模型性能上产生统计显著的离群值 —— 当归约顺序也是非确定性时,方差会进一步增大。([arXiv:2109.08203](https://arxiv.org/abs/2109.08203)) |
| 24 | -- **CI/CD & Debugging**: In continuous integration testing and distributed debugging, any non‑determinism turns a reproducible bug into a ghost. Deterministic collectives guarantee that a failing test will fail identically on every rerun, drastically reducing root‑cause analysis time. | 24 | +- **CI/CD与调试**:在持续集成测试和分布式调试中,任何非确定性都会将可复现的缺陷变为“幽灵”问题。确定性集合通信保证失败的测试在每次重运行时以完全相同的方式失败,从而大幅缩短根因分析时间。 |
| 25 | 25 | ||
| 26 | -#### 2. Reinforcement Learning (RL / RLHF / PPO) | 26 | +#### 2. 强化学习(RL、RLHF、PPO) |
| 27 | 27 | ||
| 28 | -Reinforcement learning training is highly sensitive to consistency in policy evaluation. In PPO and RLHF pipelines, when the same policy is evaluated with different batch sizes, a change in ReduceScatter reduction ordering due to sharding can inject floating‑point noise into gradient/reward signals, destabilizing policy updates. | 28 | +强化学习训练对策略评估的一致性高度敏感。在PPO和RLHF流水线中,当同一策略以不同batch size进行评估时,由于分片导致的ReduceScatter归约顺序变化会向梯度或奖励信号中注入浮点噪声,使策略更新不稳定。 |
| 29 | 29 | ||
| 30 | -- **verl** ([github.com/verl-project/verl](https://github.com/verl-project/verl)): A mainstream open‑source RLHF/PPO framework that provides a `full_determinism` configuration option and explicitly sets `HCCL_DETERMINISTIC=1` to guarantee reproducible collective operations. | 30 | +- **verl**([github.com/verl-project/verl](https://github.com/verl-project/verl)):主流开源RLHF或PPO框架,提供了 `full_determinism` 配置选项,并显式设置`HCCL_DETERMINISTIC=1`以保证可复现的集合操作。 |
| 31 | -- **DeepSpeed‑Chat** and derivative frameworks: Require deterministic reductions in RLHF training to keep reward model training consistent across identical inputs. | 31 | +- **DeepSpeed-Chat** 及衍生框架:在RLHF训练中要求确定性归约,以保持奖励模型训练在相同输入上的一致性。 |
| 32 | 32 | ||
| 33 | -#### 3. Inference Consistency & Batch Invariance | 33 | +#### 3. 推理一致性与 Batch 不变性 |
| 34 | 34 | ||
| 35 | -In large‑model serving, users expect the same prompt to always return the same output. However, dynamic batching means a prompt can be grouped with different neighbours on each request. Without deterministic collective communication, floating‑point reduction order can vary with batch composition, breaking this invariance. | 35 | +在大模型服务中,用户期望同一prompt始终返回相同输出。然而,动态batching意味着一个prompt在每次请求时可能与不同的邻居组合。如果没有确定性集合通信,浮点归约顺序会随batch组成而变化,破坏这一不变性。 |
| 36 | 36 | ||
| 37 | -- **vLLM Batch Invariance**: The vLLM project explicitly calls out that non‑deterministic all‑reduce backends (e.g., NCCL) can cause different logits for the same prompt depending on batch mates. Their batch invariance guarantee relies on deterministic communication to ensure “the output for a given prompt is the same regardless of what other prompts are in the batch.” ([Motivation](https://docs.vllm.ai/en/latest/features/batch_invariance/#motivation), [Ascend Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/batch_invariance.html)) | 37 | +- **vLLM Batch不变性**:vLLM项目明确指出非确定性all-reduce后端(如 NCCL)会导致同一prompt因batch中的其他prompt不同而产生不同的logits。其batch不变性保证依赖于确定性通信,以确保"给定prompt的输出不受batch中其他prompt的影响"。([Motivation](https://docs.vllm.ai/en/latest/features/batch_invariance/#motivation), [Ascend Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/batch_invariance.html)) |
| 38 | -- **SGLang**: Provides an `--enable-deterministic-inference` flag that forces deterministic computation and communication ordering, making inference outputs fully reproducible across different batch sizes and request arrival patterns. ([SGLang deterministic inference](https://sgl-project.github.io/advanced_features/deterministic_inference.html)) | 38 | +- **SGLang**:提供`--enable-deterministic-inference`标志,强制确定性计算和通信顺序,使推理输出在不同batch size和请求到达模式下完全可复现。([SGLang deterministic inference](https://sgl-project.github.io/advanced_features/deterministic_inference.html)) |
| 39 | -- **OpenAI Community**: Practitioners have long struggled with non‑deterministic GPU operations in production LLM inference, where bit‑for‑bit reproducibility is expected by end‑users and essential for debugging. ([Defeating Nondeterminism in LLM Inference](https://community.openai.com/t/defeating-nondeterminism-in-llm-inference/1358623)) | 39 | +- **OpenAI 社区**:在生产环境的LLM推理中,从业者长期以来一直受困于非确定性的GPU操作——而终端用户期望的是比特级完全可复现的结果,且这对于调试也至关重要。([Defeating Nondeterminism in LLM Inference](https://community.openai.com/t/defeating-nondeterminism-in-llm-inference/1358623)) |
| 40 | 40 | ||
| 41 | -#### 4. Ecosystem API & Framework Support | 41 | +#### 4. 生态API与框架支持 |
| 42 | 42 | ||
| 43 | -The demand for determinism is reflected in the official APIs and configuration flags of major ML frameworks: | 43 | +对确定性的需求已体现在主流 ML 框架的官方 API 和配置标志中: |
| 44 | 44 | ||
| 45 | -- **PyTorch**: `torch.use_deterministic_algorithms(True)` requires all operations – including collectives – to produce the same output given the same input on the same hardware/software. ([PyTorch docs](https://docs.pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html)) | 45 | +- **PyTorch**:`torch.use_deterministic_algorithms(True)` 要求所有操作——包括集合通信——在相同软硬件环境下对相同输入产生相同输出。([PyTorch docs](https://docs.pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html)) |
| 46 | -- **HuggingFace Transformers / Diffusers**: Provide a standardised `enable_full_determinism()` function that sets `NCCL_DETERMINISTIC=1`, `CUBLAS_WORKSPACE_CONFIG`, and other variables. | 46 | +- **HuggingFace Transformers或Diffusers**:提供标准化的`enable_full_determinism()`函数,用于设置`NCCL_DETERMINISTIC=1`、`CUBLAS_WORKSPACE_CONFIG`等变量。 |
| 47 | -- **LlamaFactory**: Large‑model fine‑tuning framework offering an `enable_full_determinism(seed)` interface for reproducible distributed training. | 47 | +- **LlamaFactory**:大模型微调框架,提供`enable_full_determinism(seed)`接口用于可复现的分布式训练。 |
| 48 | -- **ByteDance VeOmni**: Enforces `--train.enable_full_determinism true` in CI tests, making deterministic collectives a gate for code acceptance. | 48 | +- **ByteDance VeOmni**:在CI测试中强制`--train.enable_full_determinism true`,使确定性集合通信成为代码合入的门槛。 |
| 49 | 49 | ||
| 50 | -### Limitations of Existing Batch-Invariant Algorithms in HCCL | 50 | +### HCCL现有Batch不变算法的局限性 |
| 51 | 51 | ||
| 52 | -HCCL currently provides two batch-invariant algorithms: | 52 | +HCCL目前提供两种batch不变算法: |
| 53 | 53 | ||
| 54 | -| Algorithm | Use Case | Limitation | | 54 | +| 算法 | 适用场景 | 局限性 | |
| 55 | -|-----------|----------|------------| | 55 | +|------|----------|--------| |
| 56 | -| **Mesh + Local Reduce** | Small messages (< several MB) | Low bandwidth utilization for large messages | | 56 | +| **Mesh + Local Reduce** | 小消息(< 数MB) | 大消息场景带宽利用率低 | |
| 57 | -| **RHD (Recursive Halving-Doubling)** | Large messages | Utilizes only ~50% of available bandwidth (only half the nodes communicate per round) | | 57 | +| **RHD(递归半倍-倍增)** | 大消息 | 仅利用约50%的可用带宽(每轮仅一半节点参与通信) | |
| 58 | 58 | ||
| 59 | -On the A3 server topology (SIO + HCCS hybrid interconnect), RHD cannot simultaneously utilize SIO and HCCS links, resulting in insufficient bandwidth utilization for large message scenarios. | 59 | +在A3 服务器拓扑(SIO + HCCS 混合互连)上,RHD无法同时利用SIO和HCCS链路,导致大消息场景下带宽利用不足。 |
| 60 | 60 | ||
| 61 | -### Value of BIRS | 61 | +### BIRS的价值 |
| 62 | 62 | ||
| 63 | -The BIRS algorithm is designed for the 2D topology characteristics of A3 servers, maintaining batch invariance while: | 63 | +BIRS算法针对A3服务器的2D拓扑特性设计,在保持batch不变性的同时: |
| 64 | -- **First round**: Performing SendReduce over SIO links (cross X-axis reduction) | ||
| 65 | -- **Subsequent rounds**: Simultaneously utilizing SIO (reduction) and HCCS (intermediate result transfer) links | ||
| 66 | -- Achieving near-optimal bandwidth utilization, with only the first round not fully utilizing bandwidth | ||
| 67 | 64 | ||
| 68 | -## Detailed Design | 65 | +- **第一轮**:通过SIO链路执行SendReduce(跨X轴归约)。 |
| 66 | +- **后续轮次**:同时利用SIO(归约)和HCCS(中间结果传输)链路。 | ||
| 67 | +- 实现近最优带宽利用,仅第一轮未充分利用带宽。 | ||
| 69 | 68 | ||
| 70 | -### 1. Overall Architecture | 69 | +## 详细设计 |
| 71 | 70 | ||
| 72 | -The BIRS algorithm is integrated into HCCL as an experimental feature, isolated from existing algorithms through an independent code path and build option. | 71 | +### 1. 总体架构 |
| 72 | + | ||
| 73 | +BIRS算法作为实验特性集成到HCCL中,通过独立代码路径和编译选项与现有算法隔离。 | ||
| 73 | 74 | ||
| 74 | ```text | 75 | ```text |
| 75 | HCCL | 76 | HCCL |
| 76 | -├── src/ops/reduce_scatter/ # Existing ReduceScatter implementation | 77 | +├── src/ops/reduce_scatter/ # 现有 ReduceScatter 实现 |
| 77 | -│ └── reduce_scatter_op.cc/.h # Entry function (with BIRS dispatch logic added) | 78 | +│ └── reduce_scatter_op.cc/.h # 入口函数(新增BIRS分发逻辑) |
| 78 | │ | 79 | │ |
| 79 | -├── experimental/ops/ # Experimental features directory (new) | 80 | +├── experimental/ops/ # 实验特性目录(新增) |
| 80 | -│ ├── op_common/ # Common infrastructure | 81 | +│ ├── op_common/ # 公共基础设施 |
| 81 | -│ │ ├── op_common_experimental.cc/.h # Experimental op common logic (ProcessA3, etc.) | 82 | +│ │ ├── op_common_experimental.cc/.h # 实验op公共逻辑(ProcessA3等) |
| 82 | -│ │ ├── template/ # Experimental algorithm template base class | 83 | +│ │ ├── template/ # 实验算法模板基类 |
| 83 | │ │ │ └── alg_template_base_experimental.cc/.h | 84 | │ │ │ └── alg_template_base_experimental.cc/.h |
| 84 | -│ │ └── topo/ # Experimental topology utilities | 85 | +│ │ └── topo/ # 实验拓扑工具 |
| 85 | │ │ └── topo_experimental.cc/.h | 86 | │ │ └── topo_experimental.cc/.h |
| 86 | │ │ | 87 | │ │ |
| 87 | -│ └── reduce_scatter/ # ReduceScatter experimental algorithms | 88 | +│ └── reduce_scatter/ # ReduceScatter实验算法 |
| 88 | -│ ├── reduce_scatter_op_experimental.cc/.h # Experimental entry (MatchBIRS dispatch) | 89 | +│ ├── reduce_scatter_op_experimental.cc/.h # 实验入口(MatchBIRS分发) |
| 89 | -│ └── birs/ # BIRS algorithm implementation | 90 | +│ └── birs/ # BIRS算法实现 |
| 90 | -│ ├── reduce_scatter_birs_executor.cc/.h # Executor layer (resource calc, scheduling) | 91 | +│ ├── reduce_scatter_birs_executor.cc/.h # Executor层(资源计算、调度) |
| 91 | -│ ├── reduce_scatter_executor_base.cc/.h # Executor base class | 92 | +│ ├── reduce_scatter_executor_base.cc/.h # Executor基类 |
| 92 | │ └── template/ | 93 | │ └── template/ |
| 93 | -│ ├── reduce_scatter_birs.cc/.h # Core algorithm template (communication loop) | 94 | +│ ├── reduce_scatter_birs.cc/.h # 核心算法模板(通信循环) |
| 94 | -│ └── reduce_scatter_birs_inter.cc/.h # Intermediate result handling | 95 | +│ └── reduce_scatter_birs_inter.cc/.h # 中间结果处理 |
| 95 | │ | 96 | │ |
| 96 | └── test/st/algorithm/testcase/ | 97 | └── test/st/algorithm/testcase/ |
| 97 | - └── reduce_scatter_testcase_a3.cc # A3 platform test cases (new) | 98 | + └── reduce_scatter_testcase_a3.cc # A3平台测试用例(新增) |
| 98 | ``` | 99 | ``` |
| 99 | 100 | ||
| 100 | -**Data Flow**: | 101 | +**数据流**: |
| 101 | 102 | ||
| 102 | ```text | 103 | ```text |
| 103 | -User calls HcclReduceScatter() | 104 | +用户调用HcclReduceScatter() |
| 104 | │ | 105 | │ |
| 105 | - ├── HCCL_BIRS_ENABLE != TRUE → Take existing HcclReduceScatterInner() path | 106 | + ├── HCCL_BIRS_ENABLE != TRUE → 走现有 HcclReduceScatterInner() 路径 |
| 106 | │ | 107 | │ |
| 107 | └── HCCL_BIRS_ENABLE == TRUE | 108 | └── HCCL_BIRS_ENABLE == TRUE |
| 108 | │ | 109 | │ |
| 109 | └── ReduceScatterExperimental() | 110 | └── ReduceScatterExperimental() |
| 110 | │ | 111 | │ |
| 111 | - ├── Parameter validation (reuse existing CheckReduceScatterInputPara, etc.) | 112 | + ├── 参数校验(复用现有 CheckReduceScatterInputPara 等) |
| 112 | │ | 113 | │ |
| 113 | └── ReduceScatterOutPlaceCustom() | 114 | └── ReduceScatterOutPlaceCustom() |
| 114 | │ | 115 | │ |
| @@ -118,24 +119,24 @@ User calls HcclReduceScatter() | |||
| 118 | │ | 119 | │ |
| 119 | └── ReduceScatterBIRS::RunAsync() | 120 | └── ReduceScatterBIRS::RunAsync() |
| 120 | │ | 121 | │ |
| 121 | - ├── Preprocess() — Preprocessing (slice calc, channel validation) | 122 | + ├── Preprocess() — 预处理(切片计算、通道校验) |
| 122 | - ├── Main comm loop — SIO SendReduce + HCCS transfers | 123 | + ├── Main comm loop — SIO SendReduce + HCCS 传输 |
| 123 | - └── FinalStep() — Local tree reduction + output copy | 124 | + └── FinalStep() — 本地树形归约 + 输出拷贝 |
| 124 | ``` | 125 | ``` |
| 125 | 126 | ||
| 126 | -### 2. Interface Design | 127 | +### 2. 接口设计 |
| 127 | 128 | ||
| 128 | -#### 2.1 Environment Variables | 129 | +#### 2.1 环境变量 |
| 129 | 130 | ||
| 130 | -| Environment Variable | Values | Description | | 131 | +| 环境变量 | 取值 | 说明 | |
| 131 | -|---------------------|--------|-------------| | 132 | +|---------|------|------| |
| 132 | -| `HCCL_BIRS_ENABLE` | `TRUE` / `FALSE` (default) | Enable the BIRS algorithm. When set to `TRUE`, ReduceScatter calls are routed to the BIRS implementation | | 133 | +| `HCCL_BIRS_ENABLE` | `TRUE`/`FALSE`(默认) | 启用BIRS算法。设为`TRUE`时,ReduceScatter调用将路由到BIRS实现。| |
| 133 | 134 | ||
| 134 | -#### 2.2 Build Options | 135 | +#### 2.2 编译选项 |
| 135 | 136 | ||
| 136 | -Run the following command from the root directory of the hccl repository: | 137 | +在hccl仓库根目录下执行以下命令: |
| 137 | 138 | ||
| 138 | -``` | 139 | +```bash |
| 139 | # host + device + experimental | 140 | # host + device + experimental |
| 140 | bash build.sh --pkg --full --experimental | 141 | bash build.sh --pkg --full --experimental |
| 141 | ``` | 142 | ``` |
| @@ -144,103 +145,104 @@ bash build.sh --pkg --full --experimental | |||
| 144 | option(ENABLE_EXPERIMENTAL "Enable experimental features" OFF) | 145 | option(ENABLE_EXPERIMENTAL "Enable experimental features" OFF) |
| 145 | ``` | 146 | ``` |
| 146 | 147 | ||
| 147 | -To enable experimental functions, use the `--experimental option`. This sets the compilation flag `-DENABLE_EXPERIMENTAL=ON`, which in turn causes the `experimental/ops/` subdirectory to be compiled. Disabled by default, with no impact on existing builds. | 148 | +使用`--experimental option`选项启用实验功能。为该选项设置编译标志`-DENABLE_EXPERIMENTAL=ON`,从而编译`experimental/ops/`子目录中的代码。该功能默认关闭,不影响现有构建。 |
| 148 | 149 | ||
| 149 | -#### 2.3 API Compatibility | 150 | +#### 2.3 API 兼容性 |
| 150 | 151 | ||
| 151 | -BIRS introduces no new user-facing APIs. Users call the standard `HcclReduceScatter()` interface, with algorithm selection entirely controlled by environment variables: | 152 | +BIRS未引入新的用户态API。用户调用标准 `HcclReduceScatter()`接口即可,算法选择完全由环境变量控制: |
| 152 | 153 | ||
| 153 | ```c | 154 | ```c |
| 154 | -// No user code changes required — just set the environment variable to enable | 155 | +// 无需修改用户代码 —— 只需设置环境变量即可启用 |
| 155 | // export HCCL_BIRS_ENABLE=TRUE | 156 | // export HCCL_BIRS_ENABLE=TRUE |
| 156 | HcclReduceScatter(sendBuf, recvBuf, recvCount, dataType, op, comm, stream); | 157 | HcclReduceScatter(sendBuf, recvBuf, recvCount, dataType, op, comm, stream); |
| 157 | ``` | 158 | ``` |
| 158 | 159 | ||
| 159 | -### 3. Data Structures | 160 | +### 3. 数据结构 |
| 160 | 161 | ||
| 161 | -#### 3.1 Logical 2D Topology Layout | 162 | +#### 3.1 逻辑2D拓扑布局 |
| 162 | 163 | ||
| 163 | -BIRS constructs a logical 2D layout over the A3/16P topology: | 164 | +BIRS在A3/16P拓扑上构建逻辑2D布局: |
| 164 | 165 | ||
| 165 | -``` | 166 | +```text |
| 166 | -rankSizeX = 2 // X-axis direction (SIO links) | 167 | +rankSizeX = 2 // X 轴方向(SIO 链路) |
| 167 | -rankSizeY = rankSize / rankSizeX // Y-axis direction (HCCS links) | 168 | +rankSizeY = rankSize / rankSizeX // Y 轴方向(HCCS 链路) |
| 168 | ``` | 169 | ``` |
| 169 | 170 | ||
| 170 | -Each rank maintains the following topology information: | 171 | +每个rank维护以下拓扑信息: |
| 171 | 172 | ||
| 172 | -| Member | Type | Description | | 173 | +| 成员 | 类型 | 说明 | |
| 173 | -|--------|------|-------------| | 174 | +|------|------|------| |
| 174 | -| `sio_rank` | `u32` | SIO peer rank (`rank XOR 1`) | | 175 | +| `sio_rank` | `u32` | SIO对端rank(`rank XOR 1`) | |
| 175 | -| `hccs_ranks` | `vector<u32>` | Peer rank list along HCCS direction | | 176 | +| `hccs_ranks` | `vector<u32>` | HCCS方向的对端rank列表 | |
| 176 | -| `hccs_neighbour_rank` | `vector<u32>` | SIO neighbor ranks of HCCS peers | | 177 | +| `hccs_neighbour_rank` | `vector<u32>` | HCCS对端的SIO邻居rank | |
| 177 | -| `sio_link` | `ChannelInfo` | SIO communication channel | | 178 | +| `sio_link` | `ChannelInfo` | SIO通信通道 | |
| 178 | -| `hccs_links` | `vector<ChannelInfo>` | HCCS communication channel list | | 179 | +| `hccs_links` | `vector<ChannelInfo>` | HCCS通信通道列表 | |
| 179 | -| `hccs_links_reversed` | `vector<ChannelInfo>` | Reversed HCCS channels (for receiving) | | 180 | +| `hccs_links_reversed` | `vector<ChannelInfo>` | 反向HCCS通道(用于接收) | |
| 180 | 181 | ||
| 181 | -#### 3.2 Scratch Memory Layout | 182 | +#### 3.2 Scratch 内存布局 |
| 182 | 183 | ||
| 183 | -BIRS uses scratch memory to store intermediate reduction results (IM), with a strided layout to satisfy the 910B minimum slice alignment requirement: | 184 | +BIRS使用scratch内存存储中间归约结果(IM),采用步长布局以满足910B最小切片对齐要求: |
| 184 | 185 | ||
| 185 | -``` | 186 | +```text |
| 186 | localStrideSize = RoundUp(sliceSize, HCCL_MIN_SLICE_ALIGN_910B) | 187 | localStrideSize = RoundUp(sliceSize, HCCL_MIN_SLICE_ALIGN_910B) |
| 187 | 188 | ||
| 188 | -Scratch buffer is divided into 2 regions with N slots each: Region A is used to accumulate intermediate results for HCCS, | 189 | +Scratch 缓冲区分为 2 个区域,各含 N 个槽位:区域 A 用于累积 HCCS 中间结果, |
| 189 | -Region B is used for sendReduce over SIO. | 190 | +区域 B 用于 SIO 上的 sendReduce。 |
| 190 | 191 | ||
| 191 | Scratch Memory: | 192 | Scratch Memory: |
| 192 | ┌─────────────────────────────────────────────┐ | 193 | ┌─────────────────────────────────────────────┐ |
| 193 | -│ IM[0]: offset = 0 * localStrideSize │ ← Region A intermediate result | 194 | +│ IM[0]: offset = 0 * localStrideSize │ ← 区域 A 中间结果 |
| 194 | ├─────────────────────────────────────────────┤ | 195 | ├─────────────────────────────────────────────┤ |
| 195 | -│ IM[1]: offset = 1 * localStrideSize │ ← Region A intermediate result | 196 | +│ IM[1]: offset = 1 * localStrideSize │ ← 区域 A 中间结果 |
| 196 | ├─────────────────────────────────────────────┤ | 197 | ├─────────────────────────────────────────────┤ |
| 197 | │ ... │ | 198 | │ ... │ |
| 198 | ├─────────────────────────────────────────────┤ | 199 | ├─────────────────────────────────────────────┤ |
| 199 | -│ IM[N]: offset = N * localStrideSize │ ← Region A intermediate result | 200 | +│ IM[N]: offset = N * localStrideSize │ ← 区域 A 中间结果 |
| 200 | ├─────────────────────────────────────────────┤ | 201 | ├─────────────────────────────────────────────┤ |
| 201 | -│ SIO[0]: offset = (N+1) * localStrideSize │ ← Region B SIO-intermediate result | 202 | +│ SIO[0]: offset = (N+1) * localStrideSize │ ← 区域 B SIO 中间结果 |
| 202 | ├─────────────────────────────────────────────┤ | 203 | ├─────────────────────────────────────────────┤ |
| 203 | -│ SIO[1]: offset = (N+2) * localStrideSize │ ← Region B SIO-intermediate result | 204 | +│ SIO[1]: offset = (N+2) * localStrideSize │ ← 区域 B SIO 中间结果 |
| 204 | ├─────────────────────────────────────────────┤ | 205 | ├─────────────────────────────────────────────┤ |
| 205 | │ ... │ | 206 | │ ... │ |
| 206 | ├─────────────────────────────────────────────┤ | 207 | ├─────────────────────────────────────────────┤ |
| 207 | -│ SIO[N]: offset = 2 * N * localStrideSize │ ← Region B SIO-intermediate result | 208 | +│ SIO[N]: offset = 2 * N * localStrideSize │ ← 区域 B SIO 中间结果 |
| 208 | └─────────────────────────────────────────────┘ | 209 | └─────────────────────────────────────────────┘ |
| 209 | ``` | 210 | ``` |
| 210 | 211 | ||
| 211 | -#### 3.3 Thread Model | 212 | +#### 3.3 线程模型 |
| 212 | 213 | ||
| 213 | -BIRS uses a 3-thread parallel model: | 214 | +BIRS使用三线程并行模型: |
| 214 | 215 | ||
| 215 | -| Thread | Role | Responsibility | | 216 | +| 线程 | 角色 | 职责 | |
| 216 | -|--------|------|----------------| | 217 | +|------|------|------| |
| 217 | -| `mainThread` | Main thread | SIO SendReduce, final local reduction | | 218 | +| `mainThread` | 主线程 | SIO SendReduce、最终本地归约 | |
| 218 | -| `subThreads[0]` | HCCS sub-thread | HCCS link Send/Notify operations | | 219 | +| `subThreads[0]` | HCCS子线程 | HCCS链路Send或Notify操作 | |
| 219 | -| `subThreads[1]` | Copy sub-thread | Pre-copy of next round's input data | | 220 | +| `subThreads[1]` | 拷贝子线程 | 预拷贝下一轮输入数据 | |
| 220 | 221 | ||
| 221 | -Inter-thread synchronization is performed via `PreSyncInterThreads` / `PostSyncInterThreads`. | 222 | +线程间同步通过 `PreSyncInterThreads`/`PostSyncInterThreads`完成。 |
| 222 | 223 | ||
| 223 | -### 4. Key Logic | 224 | +### 4. 关键逻辑 |
| 224 | 225 | ||
| 225 | -#### 4.1 Algorithm Overview | 226 | +#### 4.1 算法概述 |
| 226 | 227 | ||
| 227 | -The core property of the BIRS algorithm is **batch invariance**: the order of reduction additions on each rank is strictly identical, regardless of batch size or memory slicing. | 228 | +BIRS算法的核心性质是**batch 不变性**:每个rank上的归约加法顺序严格一致,不受batch size或内存分片影响。 |
| 228 | 229 | ||
| 229 | -**Notation**: | 230 | +**符号约定**: |
| 230 | -- `S(d, i)`: The i-th slice of the input message on device d | ||
| 231 | -- `rankSizeX = 2`, `rankSizeY = rankSize / 2` | ||
| 232 | -- `sio_rank = rank XOR 1` (SIO peer) | ||
| 233 | -- `hccs_ranks[i] = (rank + rankSizeX * i) % rankSize` (HCCS peer sequence) | ||
| 234 | 231 | ||
| 235 | -#### 4.2 Main Communication Loop | 232 | +- `S(d, i)`:设备d上输入消息的第i个切片。 |
| 233 | +- `rankSizeX = 2`,`rankSizeY = rankSize / 2`。 | ||
| 234 | +- `sio_rank = rank XOR 1`(SIO 对端)。 | ||
| 235 | +- `hccs_ranks[i] = (rank + rankSizeX * i) % rankSize`(HCCS 对端序列)。 | ||
| 236 | 236 | ||
| 237 | -``` | 237 | +#### 4.2 主通信循环 |
| 238 | -// Initial: copy the input slice corresponding to the first HCCS peer into scratch memory | 238 | + |
| 239 | +```c | ||
| 240 | +// 初始:将第一个 HCCS 对端对应的输入切片拷贝到 scratch 内存 | ||
| 239 | LocalCopy(input[S(hccs_ranks[0])], scratch[IM_0]) | 241 | LocalCopy(input[S(hccs_ranks[0])], scratch[IM_0]) |
| 240 | 242 | ||
| 241 | for round in 0 ... hccs_ranks.size(): | 243 | for round in 0 ... hccs_ranks.size(): |
| 242 | 244 | ||
| 243 | - // ── Sub-thread 0: HCCS transfer (when round > 0) ── | 245 | + // ── 子线程 0:HCCS 传输(round > 0 时) ── |
| 244 | if round > 0: | 246 | if round > 0: |
| 245 | Notify(sio → hccs_ack) | 247 | Notify(sio → hccs_ack) |
| 246 | Wait(hccs_ack) | 248 | Wait(hccs_ack) |
| @@ -248,118 +250,118 @@ for round in 0 ... hccs_ranks.size(): | |||
| 248 | Notify(data_signal) | 250 | Notify(data_signal) |
| 249 | Wait(data_signal) | 251 | Wait(data_signal) |
| 250 | 252 | ||
| 251 | - // ── Main thread: SIO SendReduce ── | 253 | + // ── 主线程:SIO SendReduce ── |
| 252 | Notify(sio_ack) | 254 | Notify(sio_ack) |
| 253 | Wait(sio_ack) | 255 | Wait(sio_ack) |
| 254 | SendReduce( | 256 | SendReduce( |
| 255 | - local: input[S(hccs_neighbour_rank[round])], // or S(sio_rank) in the last round | 257 | + local: input[S(hccs_neighbour_rank[round])], // 最后一轮为 S(sio_rank) |
| 256 | remote: scratch[IM_round on sio_peer] | 258 | remote: scratch[IM_round on sio_peer] |
| 257 | ) → scratch[IM_round on sio_peer] | 259 | ) → scratch[IM_round on sio_peer] |
| 258 | Notify(data_signal) | 260 | Notify(data_signal) |
| 259 | Wait(data_signal) | 261 | Wait(data_signal) |
| 260 | 262 | ||
| 261 | - // ── Sub-thread 1: Pre-copy next round's data ── | 263 | + // ── 子线程 1:预拷贝下一轮数据 ── |
| 262 | if round < hccs_ranks.size() - 1: | 264 | if round < hccs_ranks.size() - 1: |
| 263 | LocalCopy(input[S(hccs_ranks[round+1])], scratch[next_slot]) | 265 | LocalCopy(input[S(hccs_ranks[round+1])], scratch[next_slot]) |
| 264 | ``` | 266 | ``` |
| 265 | 267 | ||
| 266 | -#### 4.3 Final Reduction (FinalStep) | 268 | +#### 4.3 最终归约(FinalStep) |
| 267 | 269 | ||
| 268 | -After all rounds complete, each rank holds `rankSizeY` intermediate results in scratch memory. These are merged via a **tree-based local reduction**: | 270 | +所有轮次完成后,每个rank在scratch内存中持有`rankSizeY`个中间结果。这些结果通过**树形本地归约**合并: |
| 269 | 271 | ||
| 270 | -``` | 272 | +```text |
| 271 | -// Collect all intermediate result offsets | 273 | +// 收集所有中间结果偏移 |
| 272 | -vec = [IM_0, IM_1, ..., IM_{rankSizeY-1}] // this rank's result is at the correct position | 274 | +vec = [IM_0, IM_1, ..., IM_{rankSizeY-1}] // 本 rank 的结果位于正确位置 |
| 273 | 275 | ||
| 274 | -// Tree reduction (guarantees deterministic addition order) | 276 | +// 树形归约(保证确定性加法顺序) |
| 275 | for stride in 1, 2, 4, ...: | 277 | for stride in 1, 2, 4, ...: |
| 276 | for i in stride, stride+stride, ...: | 278 | for i in stride, stride+stride, ...: |
| 277 | LocalReduce(vec[i] → vec[i - stride]) | 279 | LocalReduce(vec[i] → vec[i - stride]) |
| 278 | 280 | ||
| 279 | -// Copy final result to output | 281 | +// 将最终结果拷贝到输出 |
| 280 | LocalCopy(vec[0] → outputMem) | 282 | LocalCopy(vec[0] → outputMem) |
| 281 | ``` | 283 | ``` |
| 282 | 284 | ||
| 283 | -The tree reduction guarantees deterministic addition ordering: for `rankSizeY = 4`, the reduction order is `(IM_0 + IM_1) + (IM_2 + IM_3)`, independent of rank id. ReduceScatterBIRS() supports reduction for rankSize <= 16, for larger rankSize it is recommended to use ReduceScatterBIRSInter(). | 285 | +树形归约保证确定性加法顺序:以`rankSizeY = 4`为例,归约顺序为 `(IM_0 + IM_1) + (IM_2 + IM_3)`,与rank id无关。ReduceScatterBIRS()支持rankSize <= 16的归约,对于更大的rankSize,建议使用ReduceScatterBIRSInter()。 |
| 284 | 286 | ||
| 285 | -### 5. Compatibility Considerations | 287 | +### 5. 兼容性考虑 |
| 286 | 288 | ||
| 287 | -#### 5.1 Backward Compatibility | 289 | +#### 5.1 向后兼容 |
| 288 | 290 | ||
| 289 | -- **Fully backward compatible**: BIRS is disabled by default (`HCCL_BIRS_ENABLE` defaults to `FALSE`), with no impact on existing ReduceScatter behavior. | 291 | +- **完全向后兼容**:BIRS默认关闭(`HCCL_BIRS_ENABLE` 默认为`FALSE`),对现有的ReduceScatter行为无任何影响。 |
| 290 | -- **Build isolation**: Experimental code resides in an independent `experimental/` directory, controlled by the `ENABLE_EXPERIMENTAL` build flag, and is excluded from compilation by default. | 292 | +- **构建隔离**:实验代码位于独立的`experimental/`目录,由`ENABLE_EXPERIMENTAL`编译标志控制,默认不参与编译。 |
| 291 | -- **No API changes**: The user-facing API (`HcclReduceScatter`) remains unchanged; algorithm selection is transparent to users. | 293 | +- **无API变更**:用户态API(`HcclReduceScatter`)保持不变,算法选择对用户透明。 |
| 292 | 294 | ||
| 293 | -#### 5.2 Applicability Conditions | 295 | +#### 5.2 适用条件 |
| 294 | 296 | ||
| 295 | -The BIRS algorithm currently has the following constraints: | 297 | +BIRS 算法当前有以下约束: |
| 296 | 298 | ||
| 297 | -| Constraint | Description | | 299 | +| 约束 | 说明 | |
| 298 | -|------------|-------------| | 300 | +|------|------| |
| 299 | -| Platform | A3 servers only (SIO + HCCS hybrid topology) | | 301 | +| 平台 | 仅A3服务器(SIO + HCCS 混合拓扑)。 | |
| 300 | -| rankSize | Must be even (`rankSize % 2 == 0`), typical values: 4, 8, 16 | | 302 | +| rankSize | 必须为偶数(`rankSize % 2 == 0`),典型值:4、8、16。 | |
| 301 | -| Communication domain | Both Intra-server and Inter-server are supported | | 303 | +| 通信域 | 同时支持单服务器内和跨服务器。| |
| 302 | -| Data alignment | Slice sizes must satisfy `HCCL_MIN_SLICE_ALIGN_910B` alignment requirements | | 304 | +| 数据对齐 | 切片大小须满足`HCCL_MIN_SLICE_ALIGN_910B` 对齐要求。 | |
| 303 | 305 | ||
| 304 | -ReduceScatterBIRS() is the recommended choice for single-server A3 scenario (rankSize <= 16), ReduceScatterBIRSInter() is chosen automatically for multi-server A3 scenario. | 306 | +ReduceScatterBIRS() 是A3单服务器场景(rankSize <= 16)的推荐选择,A3多服务器场景自动选择 ReduceScatterBIRSInter()。 |
| 305 | 307 | ||
| 306 | -When conditions are not met, the workflow exit and log error messages using hccl. The user must follow the recommendations in the error logs or manually adjust the parameters to comply with the restrictions. | 308 | +当条件不满足时,流程退出并通过HCCL记录错误日志。用户需根据错误日志中的建议操作或手动调整参数以满足约束条件。 |
| 307 | 309 | ||
| 308 | -#### 5.3 Rollout Strategy | 310 | +#### 5.3 上线策略 |
| 309 | 311 | ||
| 310 | -1. **Phase 1** (current): As an experimental feature, with dual gating via `ENABLE_EXPERIMENTAL=ON` compile flag + `HCCL_BIRS_ENABLE=TRUE` runtime flag. | 312 | +1. **阶段 1**(当前):作为实验特性,通过`ENABLE_EXPERIMENTAL=ON`编译标志 + `HCCL_BIRS_ENABLE=TRUE`运行时标志双重门控。 |
| 311 | -2. **Phase 2** (post-validation): Remove compile-time gating, retain only environment variable control. | 313 | +2. **阶段 2**(验证后):移除编译期门控,仅保留环境变量控制。 |
| 312 | -3. **Phase 3** (post-stabilization): Automatically select BIRS as the default algorithm when conditions are met; users can disable via `HCCL_BIRS_ENABLE=FALSE`. | 314 | +3. **阶段 3**(稳定后):满足条件时自动选择BIRS作为默认算法;用户可通过`HCCL_BIRS_ENABLE=FALSE`关闭。 |
| 313 | 315 | ||
| 314 | -### 6. Test Plan | 316 | +### 6. 测试方案 |
| 315 | 317 | ||
| 316 | -#### 6.1 Functional Correctness Testing | 318 | +#### 6.1 功能正确性测试 |
| 317 | 319 | ||
| 318 | -- **New test file**: `test/st/algorithm/testcase/reduce_scatter_testcase_a3.cc` | 320 | +- **新增测试文件**:`test/st/algorithm/testcase/reduce_scatter_testcase_a3.cc` |
| 319 | -- **Test dimensions**: | 321 | +- **测试维度**: |
| 320 | - - Different rankSize values (4, 8, 16) | 322 | + - 不同rankSize值(4、8、16) |
| 321 | - - Different data types (FP16, FP32, BF16) | 323 | + - 不同数据类型(FP16、FP32、BF16) |
| 322 | - - Different reduction operations (SUM, MAX, MIN, PROD) | 324 | + - 不同归约操作(SUM、MAX、MIN、PROD) |
| 323 | - - Different message sizes (from KB-level to tens of MB) | 325 | + - 不同消息大小(从KB级到数十MB) |
| 324 | 326 | ||
| 325 | -#### 6.2 Batch Invariance Verification | 327 | +#### 6.2 Batch不变性验证 |
| 326 | 328 | ||
| 327 | -- Execute ReduceScatter with the same input data but different batch sizes | 329 | +- 使用相同输入数据但不同batch size执行ReduceScatter。 |
| 328 | -- Verify that output results are bit-identical | 330 | +- 验证输出结果比特一致。 |
| 329 | 331 | ||
| 330 | -#### 6.3 Performance Testing | 332 | +#### 6.3 性能测试 |
| 331 | 333 | ||
| 332 | -- Compare against the RHD algorithm, measuring Task Duration across different message sizes | 334 | +- 与RHD算法进行对比,测量不同消息大小下的任务执行时长(Task Duration)。 |
| 333 | -- Expected: For message sizes ≥ 16MB, BIRS achieves up to 25% improvement over RHD. | 335 | +- 预期:消息大小 >= 16MB时,BIRS相较RHD实现最高25%的提升。 |
| 334 | -- Note: At the moment of this RFC creation kernel submission mechanism in HCCL is slower than the one of HCOMM, so 25% performance improvement applies only to operator execution time (without submission overhead). | 336 | +- 说明:在本RFC创建时,HCCL的算子下发机制慢于HCOMM,因此25%的性能提升仅适用于算子执行时间(不含下发开销)。 |
| 335 | 337 | ||
| 336 | -#### 6.4 Regression Testing | 338 | +#### 6.4 回归测试 |
| 337 | 339 | ||
| 338 | -- Ensure all existing ReduceScatter test cases are unaffected when `HCCL_BIRS_ENABLE=FALSE` (default) | 340 | +- 确保`HCCL_BIRS_ENABLE=FALSE`(默认)时所有现有 ReduceScatter测试用例不受影响。 |
| 339 | 341 | ||
| 340 | -## Risk Assessment | 342 | +## 风险评估 |
| 341 | 343 | ||
| 342 | -| Risk | Impact | Mitigation | | 344 | +| 风险 | 影响 | 缓解措施 | |
| 343 | -|------|--------|------------| | 345 | +|------|------|----------| |
| 344 | -| BIRS only available for specific rankSize (even numbers) | Odd rank scenarios cannot use BIRS | `MatchBIRS()` check auto-falls back to existing algorithms; document constraints clearly | | 346 | +| BIRS仅适用于特定rankSize(偶数) | 奇数rank场景无法使用BIRS | `MatchBIRS()`检查自动回退到现有算法;明确文档化约束条件。 | |
| 345 | -| Experimental code may introduce stability issues | Affects overall HCCL reliability | Dual gating (compile + runtime) isolation; independent `experimental/` directory; disabled by default | | 347 | +| 实验代码可能引入稳定性问题 | 影响HCCL整体可靠性 | 双重门控(编译 + 运行时)隔离;独立`experimental/`目录;默认关闭。 | |
| 346 | -| Additional scratch memory overhead | Increased memory usage for large messages | Requires `2 * rankSizeY × localStrideSize` scratch space; pre-allocated via `CalcResRequest` | | 348 | +| 额外scratch内存开销 | 大消息场景内存使用增加 | 需要`2 * rankSizeY × localStrideSize`的scratch空间;通过`CalcResRequest`预分配。 | |
| 347 | -| A3 topology assumption (SIO + HCCS) may not apply to other platforms | Cross-platform compatibility | Algorithm explicitly bound to A3 topology characteristics; other platforms require independent adaptation | | 349 | +| A3拓扑假设(SIO + HCCS)可能不适用于其他平台 | 跨平台兼容性 | 算法明确绑定A3拓扑特性;其他平台需独立适配。 | |
| 348 | 350 | ||
| 351 | +## 替代方案 | ||
| 349 | 352 | ||
| 350 | -## Alternative Approaches | 353 | +无 |
| 351 | 354 | ||
| 352 | -N/A | 355 | +## 开放问题 |
| 353 | 356 | ||
| 354 | -## Open Questions | 357 | +1. **AllReduce扩展**:遵循相同思路的batch不变AllReduce将在单独的PR中提交。 |
| 355 | - | 358 | +2. **高效支持任意rank编号**:当前方案假设默认rank编号,即RankX的SIO邻居rankID可通过(RankX XOR 1)计算。在其他rank编号方式下,BIRS功能正常但无法提供相较RHD的性能优势。对自定义编号的高效支持已实现,将在下一个PR中提交。 |
| 356 | -1. **AllReduce extension**: Batch-invariant AllReduce which follows the same ideas will be submitted in separate PR | ||
| 357 | -2. **Efficient support for arbitrary rank enumeration**: Current solution assumes default rank enumeration where rankID of SIO neighbour of RankX can be calculated as (RankX XOR 1). In case of other rank enumerations BIRS is functional but doesn't deliver performance advantage over RHD. Efficient support for custom enumerations have already been implemented and will be submitted in the next PR. | ||
| 358 | 359 | ||
| 359 | --- | 360 | --- |
| 360 | 361 | ||
| 361 | -## Review Records | 362 | +## 评审记录 |
| 363 | + | ||
| 364 | +检视过程在PR评论区进行。详细检视评论请参考对应PR: | ||
| 362 | 365 | ||
| 363 | -The review process takes place in the PR comment section. For detailed review comments, please refer to the corresponding PR: | ||
| 364 | - PR: [cann/hccl#657](https://gitcode.com/cann/hccl/pull/657) | 366 | - PR: [cann/hccl#657](https://gitcode.com/cann/hccl/pull/657) |
| 365 | - Issues: [cann/hcomm#139](https://gitcode.com/cann/hcomm/issues/139), [cann/hccl#96](https://gitcode.com/cann/hccl/issues/96) | 367 | - Issues: [cann/hcomm#139](https://gitcode.com/cann/hcomm/issues/139), [cann/hccl#96](https://gitcode.com/cann/hccl/issues/96) |
| @@ -2,9 +2,9 @@ | |||
| 2 | 2 | ||
| 3 | ## Sample Description | 3 | ## Sample Description |
| 4 | 4 | ||
| 5 | -This sample demonstrates how to call the `HcclAllReduce()` API to perform an `AllReduce` operation. It covers the following functions: | 5 | +This sample demonstrates how to call the `HcclAllReduce()` API to perform an `AllReduce` operation. It covers the following functions: |
| 6 | 6 | ||
| 7 | -- Call `aclrtGetDeviceCount()` to detect devices and query the number of available devices. | 7 | +- Call `aclrtGetDeviceCount()` to detect devices and query the number of available devices. |
| 8 | - Call `HcclGetRootInfo()` and use `rank 0` as the root rank to generate the rootinfo identifier. | 8 | - Call `HcclGetRootInfo()` and use `rank 0` as the root rank to generate the rootinfo identifier. |
| 9 | 9 | ||
| 10 | > The rootinfo identifier contains the device IP address and device ID. This information must be broadcast to all ranks in the cluster to initialize the communicator. | 10 | > The rootinfo identifier contains the device IP address and device ID. This information must be broadcast to all ranks in the cluster to initialize the communicator. |
| @@ -9,8 +9,8 @@ | |||
| 9 | 9 | ||
| 10 | > rootinfo 标识信息主要包含:Device IP、Device ID 等信息,此信息需广播至集群内所有 rank 用来初始化通信域。 | 10 | > rootinfo 标识信息主要包含:Device IP、Device ID 等信息,此信息需广播至集群内所有 rank 用来初始化通信域。 |
| 11 | 11 | ||
| 12 | -- 在每个线程中,基于 rootinfo 标识信息通过 `HcclCommInitRootInfo()` 接口初始化通信域。 | 12 | + - 在每个线程中,基于 rootinfo 标识信息通过 `HcclCommInitRootInfo()` 接口初始化通信域。 |
| 13 | -- 调用 `HcclBroadcast()` 接口,将通信域内 root 节点的数据广播至其他 rank,并打印结果。 | 13 | + - 调用 `HcclBroadcast()` 接口,将通信域内 root 节点的数据广播至其他 rank,并打印结果。 |
| 14 | 14 | ||
| 15 | ## 目录结构 | 15 | ## 目录结构 |
| 16 | 16 | ||
| @@ -4,7 +4,7 @@ | |||
| 4 | 4 | ||
| 5 | This sample demonstrates how to call the `HcclAlltoAll()` API to perform an `AlltoAll` operation. It covers the following functions: | 5 | This sample demonstrates how to call the `HcclAlltoAll()` API to perform an `AlltoAll` operation. It covers the following functions: |
| 6 | 6 | ||
| 7 | -- Call `aclrtGetDeviceCount()` to detect devices and query the number of available devices. | 7 | +- Call `aclrtGetDeviceCount()` to detect devices and query the number of available devices. |
| 8 | - Call `HcclGetRootInfo()` and use `rank 0` as the root rank to generate the rootinfo identifier. | 8 | - Call `HcclGetRootInfo()` and use `rank 0` as the root rank to generate the rootinfo identifier. |
| 9 | 9 | ||
| 10 | > The rootinfo identifier contains the device IP address and device ID. This information must be broadcast to all ranks in the cluster to initialize the communicator. | 10 | > The rootinfo identifier contains the device IP address and device ID. This information must be broadcast to all ranks in the cluster to initialize the communicator. |
| @@ -4,10 +4,10 @@ | |||
| 4 | 4 | ||
| 5 | This sample demonstrates how to use the `HcclScatter()` API to perform collective communication. It covers the following functions: | 5 | This sample demonstrates how to use the `HcclScatter()` API to perform collective communication. It covers the following functions: |
| 6 | 6 | ||
| 7 | -- Call `aclrtGetDeviceCount()` to detect devices and query the number of available devices. | 7 | +- Call `aclrtGetDeviceCount()` to detect devices and query the number of available devices. |
| 8 | - Call `HcclGetRootInfo()` and use `rank 0` as the root rank to generate the rootinfo identifier. | 8 | - Call `HcclGetRootInfo()` and use `rank 0` as the root rank to generate the rootinfo identifier. |
| 9 | 9 | ||
| 10 | -> The rootinfo identifier contains the device IP address and device ID. This information must be broadcast to all ranks in the cluster to initialize the communicator. | 10 | + > The rootinfo identifier contains the device IP address and device ID. This information must be broadcast to all ranks in the cluster to initialize the communicator. |
| 11 | 11 | ||
| 12 | - In each thread, call `HcclCommInitRootInfo()` to initialize the communicator based on the rootinfo identifier. | 12 | - In each thread, call `HcclCommInitRootInfo()` to initialize the communicator based on the rootinfo identifier. |
| 13 | - Call `HcclScatter()` to evenly distribute the data of the root node in the communicator to all other ranks and display the results. | 13 | - Call `HcclScatter()` to evenly distribute the data of the root node in the communicator to all other ranks and display the results. |
| @@ -2,7 +2,7 @@ | |||
| 2 | 2 | ||
| 3 | ## Sample Description | 3 | ## Sample Description |
| 4 | 4 | ||
| 5 | -This sample demonstrates how to perform an AllReduce operation using the PyTorch interface. It covers the following features: | 5 | +This sample demonstrates how to perform an AllReduce operation using the PyTorch interface. It covers the following functions: |
| 6 | 6 | ||
| 7 | - Device detection: Query the number of available devices using the `torch_npu.npu.device_count()` interface. | 7 | - Device detection: Query the number of available devices using the `torch_npu.npu.device_count()` interface. |
| 8 | - Start multiple processes using the `torch.multiprocessing.spawn()` interface. | 8 | - Start multiple processes using the `torch.multiprocessing.spawn()` interface. |
| @@ -2,13 +2,13 @@ | |||
| 2 | 2 | ||
| 3 | ## Sample Description | 3 | ## Sample Description |
| 4 | 4 | ||
| 5 | -This sample demonstrates how to perform an AllReduce operation using the TensorFlow interface. It covers the following features: | 5 | +This sample demonstrates how to perform an AllReduce operation using the TensorFlow interface. It covers the following functions: |
| 6 | 6 | ||
| 7 | - Initialize the communication domain based on the `ranktable.json` configuration file. | 7 | - Initialize the communication domain based on the `ranktable.json` configuration file. |
| 8 | 8 | ||
| 9 | ## Environment Preparation | 9 | ## Environment Preparation |
| 10 | 10 | ||
| 11 | -### Prerequisites | 11 | +### Environment Requirements |
| 12 | 12 | ||
| 13 | This sample supports the following products in a single-server 8-card configuration: | 13 | This sample supports the following products in a single-server 8-card configuration: |
| 14 | 14 | ||
| @@ -54,7 +54,7 @@ Using the root user default installation path as an example: | |||
| 54 | source /usr/local/Ascend/cann/set_env.sh | 54 | source /usr/local/Ascend/cann/set_env.sh |
| 55 | ``` | 55 | ``` |
| 56 | 56 | ||
| 57 | -Additionally, running test cases requires an MPI environment. Ensure that MPI is installed and configured. | 57 | +Additionally, running the test cases requires MPI environment support. Please ensure that MPI has been installed and configured correctly. For MPI configuration, please refer to the "MPI Installation and Configuration" section in the corresponding version of the [Ascend Documentation Center - HCCL Performance Test Tool User Guide](https://hiascend.com/en/document/redirect/CannCommunityToolHcclTest). |
| 58 | 58 | ||
| 59 | ## 2. Compilation and Execution | 59 | ## 2. Compilation and Execution |
| 60 | 60 | ||
| @@ -201,7 +201,7 @@ export LD_LIBRARY_PATH=${ASCEND_HOME_PATH}/opp/vendors/cust/lib64:${LD_LIBRARY_P | |||
| 201 | 201 | ||
| 202 | ```text | 202 | ```text |
| 203 | Found 2 NPU device(s) available | 203 | Found 2 NPU device(s) available |
| 204 | -rankId:1, input: [1 1] | 204 | +rankId: 1, input: [1 1] |
| 205 | rankId: 0, input: [0 0] | 205 | rankId: 0, input: [0 0] |
| 206 | rankId: 0, output: [ 0 0 1 1 ] | 206 | rankId: 0, output: [ 0 0 1 1 ] |
| 207 | rankId: 1, output: [ 0 0 1 1 ] | 207 | rankId: 1, output: [ 0 0 1 1 ] |
| @@ -1 +1 @@ | |||
| 1 | -# readme | 1 | +# README |
| @@ -203,7 +203,7 @@ struct SrcBufDes { | |||
| 203 | - **错误场景:** | 203 | - **错误场景:** |
| 204 |  | 204 |  |
| 205 | 205 | ||
| 206 | -#### 5.1.3 定位思路 | 206 | +#### 5.1.4 定位思路 |
| 207 | 207 | ||
| 208 | 语义校验阶段可以发现两种类型的错误: | 208 | 语义校验阶段可以发现两种类型的错误: |
| 209 | 209 | ||
| @@ -182,11 +182,11 @@ The following example illustrates what semantics calculation is. | |||
| 182 | 182 | ||
| 183 |  | 183 |  |
| 184 | 184 | ||
| 185 | -2. Action state 1: Move a data block from rank0 Input with offset 20 and size 30 to rank0 Output with offset 35. Result: A semantic block is generated on rank0 Output, recording the movement information. | 185 | +2. State 1 action: Move a data block from rank0 Input with offset 20 and size 30 to rank0 Output with offset 35. Result: A semantic block is generated on rank0 Output, recording the movement information. |
| 186 | 186 | ||
| 187 |  | 187 |  |
| 188 | 188 | ||
| 189 | -3. Action state 2: Move a data block from rank1 Input with offset 70 and size 15 to rank0 Output with offset 50. Result: The destination memory overlaps with the existing semantic block. The existing semantic block must be split, generating two semantic blocks. | 189 | +3. State 2 action: Move a data block from rank1 Input with offset 70 and size 15 to rank0 Output with offset 50. Result: The destination memory overlaps with the existing semantic block. The existing semantic block must be split, generating two semantic blocks. |
| 190 | 190 | ||
| 191 |  | 191 |  |
| 192 | 192 | ||