已合并
补充项目文档和样例说明 #21
Hyunbin创建于 7月3日
补充项目文档和样例说明 #21
已合并
共 40 个文件变更+1380-2117
| @@ -1,3 +1,8 @@ | |||
| 1 | build/ | 1 | build/ |
| 2 | oat_reports/ | 2 | oat_reports/ |
| 3 | log/ | 3 | log/ |
| 4 | +AGENTS.md | ||
| 5 | +.claude/ | ||
| 6 | +.opencode/ | ||
| 7 | +.cursor/ | ||
| 8 | +.agent/ | ||
| @@ -0,0 +1,16 @@ | |||
| 1 | +# 更新日志 | ||
| 2 | + | ||
| 3 | +本文档记录asc-comm仓库的主要变更。 | ||
| 4 | + | ||
| 5 | +## Unreleased | ||
| 6 | + | ||
| 7 | +### 新增 | ||
| 8 | + | ||
| 9 | +- 新增项目文档入口、快速开始、构建与测试、三方依赖与兼容性说明。 | ||
| 10 | +- 新增AICore Hcomm点对点通信接口API参考文档，覆盖`Hcomm`、`Init`、`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`、`AtomicCAS`、`Commit`、`Drain`。 | ||
| 11 | +- 新增AICore Hcomm `hcomm_write_read_nbi`样例入口，覆盖`WriteNbi`和`ReadNbi`两卡点对点通信流程。 | ||
| 12 | +- 新增Issue模板、贡献指南和资料贡献说明。 | ||
| 13 | + | ||
| 14 | +### 说明 | ||
| 15 | + | ||
| 16 | +- 当前公开能力主要覆盖AICore Hcomm点对点通信接口。 | ||
| @@ -1,17 +1,17 @@ | |||
| 1 | # 贡献指南 | 1 | # 贡献指南 |
| 2 | 2 | ||
| 3 | -本项目欢迎广大开发者体验并参与贡献，在参与社区贡献之前。请参见 [cann-community](https://gitcode.com/cann/community) 了解行为准则，进行 CLA 协议签署，了解源码仓的贡献流程。 | 3 | +本项目欢迎广大开发者体验并参与贡献，在参与社区贡献之前。请参见[cann-community](https://gitcode.com/cann/community)了解行为准则，进行CLA协议签署，了解源码仓的贡献流程。 |
| 4 | 4 | ||
| 5 | -开发者准备本地代码与提交 PR 时需要重点关注如下几点： | 5 | +开发者准备本地代码与提交PR时需要重点关注如下几点： |
| 6 | 6 | ||
| 7 | -1. 提交 PR 时，请按照 PR 模板仔细填写本次 PR 的业务背景、目的、方案等信息。 | 7 | +1. 提交PR时，请按照PR模板仔细填写本次PR的业务背景、目的、方案等信息。 |
| 8 | -2. 若您的修改不是简单的 bug 修复，而是涉及到新增特性、新增接口、新增配置参数或者修改代码流程等，请务必先通过 Issue 进行方案讨论，以避免您的代码被拒绝合入。若您不确定本次修改是否可被归为“简单的 bug 修复”，亦可通过提交 Issue 进行方案讨论。 | 8 | +2. 若您的修改不是简单的bug修复，而是涉及到新增特性、新增接口、新增配置参数或者修改代码流程等，请务必先通过Issue进行方案讨论，以避免您的代码被拒绝合入。若您不确定本次修改是否可被归为“简单的bug修复”，亦可通过提交Issue进行方案讨论。 |
| 9 | 9 | ||
| 10 | -## API 开发指南 | 10 | +## API开发指南 |
| 11 | 11 | ||
| 12 | -本项目支持 API 相关贡献。开发者在新增或修改 API 前，应先通过 Issue 说明使用场景、接口设计、兼容性影响和测试方案；涉及新增特性、新增接口或行为变化的修改，应在方案讨论清楚后再提交 PR。 | 12 | +本项目支持API相关贡献。开发者在新增或修改API前，应先通过Issue说明使用场景、接口设计、兼容性影响和测试方案；涉及新增特性、新增接口或行为变化的修改，应在方案讨论清楚后再提交PR。 |
| 13 | 13 | ||
| 14 | -API 开发类贡献主要包括： | 14 | +API开发类贡献主要包括： |
| 15 | 15 | ||
| 16 | - 新增接口能力。 | 16 | - 新增接口能力。 |
| 17 | - 修复接口缺陷。 | 17 | - 修复接口缺陷。 |
| @@ -30,30 +30,30 @@ API 开发类贡献主要包括： | |||
| 30 | - 补充构建、测试、依赖或样例说明。 | 30 | - 补充构建、测试、依赖或样例说明。 |
| 31 | - 修复无效链接、格式问题或表述歧义。 | 31 | - 修复无效链接、格式问题或表述歧义。 |
| 32 | 32 | ||
| 33 | -新增或修改 API 文档时，请参考 [API 文档贡献指南](./docs/api_contributing.md)。新增或修改 README、docs、examples 等资料文档时，请参考 [资料贡献指南](./docs/doc_contributing.md)。 | 33 | +新增或修改API文档时，请参考[API文档贡献指南](./docs/api_contributing.md)。新增或修改README、docs、examples等资料文档时，请参考[资料贡献指南](./docs/doc_contributing.md)。 |
| 34 | 34 | ||
| 35 | ## 开发者贡献场景主要包括 | 35 | ## 开发者贡献场景主要包括 |
| 36 | 36 | ||
| 37 | -- Bug 修复 | 37 | +- Bug修复 |
| 38 | 38 | ||
| 39 | - 如果您在本项目中发现了某些 Bug，希望对其进行修复，欢迎您新建 Issue 进行反馈和跟踪处理。 | 39 | + 如果您在本项目中发现了某些Bug，希望对其进行修复，欢迎您新建Issue进行反馈和跟踪处理。 |
| 40 | 40 | ||
| 41 | - 您可以按照 [提交 Issue/处理 Issue 任务](https://gitcode.com/cann/community#提交Issue处理Issue任务) 指引新建 `Bug-Report|缺陷反馈` 类 Issue 对 Bug 进行描述，然后在评论框中输入 `/assign` 或 `/assign @yourself`，将该 Issue 分配给您进行处理。 | 41 | + 您可以按照[提交Issue/处理Issue任务](https://gitcode.com/cann/community#提交Issue处理Issue任务)指引新建`Bug-Report|缺陷反馈`类Issue对Bug进行描述，然后在评论框中输入`/assign`或`/assign @yourself`，将该Issue分配给您进行处理。 |
| 42 | 42 | ||
| 43 | - 代码优化 | 43 | - 代码优化 |
| 44 | 44 | ||
| 45 | 如果您对本项目中某些实现有泛化性增强或优化思路，希望着手实现这些优化点，欢迎您进行优化贡献。 | 45 | 如果您对本项目中某些实现有泛化性增强或优化思路，希望着手实现这些优化点，欢迎您进行优化贡献。 |
| 46 | 46 | ||
| 47 | - 您可以按照 [提交 Issue/处理 Issue 任务](https://gitcode.com/cann/community#提交Issue处理Issue任务) 指引新建 `Requirement|需求建议` 类 Issue 对优化点进行说明，并提供您的设计方案，然后在评论框中输入 `/assign` 或 `/assign @yourself`，将该 Issue 分配给您进行跟踪优化。 | 47 | + 您可以按照[提交Issue/处理Issue任务](https://gitcode.com/cann/community#提交Issue处理Issue任务)指引新建`Requirement|需求建议`类Issue对优化点进行说明，并提供您的设计方案，然后在评论框中输入`/assign`或`/assign @yourself`，将该Issue分配给您进行跟踪优化。 |
| 48 | 48 | ||
| 49 | - 文档纠错 | 49 | - 文档纠错 |
| 50 | 50 | ||
| 51 | - 如果您在本项目中发现某些文档描述错误，欢迎您新建 Issue 进行反馈和修复。 | 51 | + 如果您在本项目中发现某些文档描述错误，欢迎您新建Issue进行反馈和修复。 |
| 52 | 52 | ||
| 53 | - 您可以按照 [提交 Issue/处理 Issue 任务](https://gitcode.com/cann/community#提交Issue处理Issue任务) 指引新建 `Documentation|文档反馈` 类 Issue 指出对应文档的问题，然后在评论框中输入 `/assign` 或 `/assign @yourself`，将该 Issue 分配给您纠正对应文档描述。 | 53 | + 您可以按照[提交Issue/处理Issue任务](https://gitcode.com/cann/community#提交Issue处理Issue任务)指引新建`Documentation|文档反馈`类Issue指出对应文档的问题，然后在评论框中输入`/assign`或`/assign @yourself`，将该Issue分配给您纠正对应文档描述。 |
| 54 | 54 | ||
| 55 | -- 帮助解决他人 Issue | 55 | +- 帮助解决他人Issue |
| 56 | 56 | ||
| 57 | - 如果社区中他人遇到的问题您有合适的解决方法，欢迎您在 Issue 中发表评论交流，帮助他人解决问题和痛点，共同优化易用性。 | 57 | + 如果社区中他人遇到的问题您有合适的解决方法，欢迎您在Issue中发表评论交流，帮助他人解决问题和痛点，共同优化易用性。 |
| 58 | 58 | ||
| 59 | - 如果对应 Issue 需要进行代码修改，您可以在 Issue 评论框中输入 `/assign` 或 `/assign @yourself`，将该 Issue 分配给您，跟踪协助解决问题。 | 59 | + 如果对应Issue需要进行代码修改，您可以在Issue评论框中输入`/assign`或`/assign @yourself`，将该Issue分配给您，跟踪协助解决问题。 |
| @@ -1,27 +1,174 @@ | |||
| 1 | +<div align="center"> | ||
| 2 | + | ||
| 1 | # asc-comm | 3 | # asc-comm |
| 2 | 4 | ||
| 5 | +<h4>面向昇腾AI处理器通信场景，提供Hcomm通信API、AIV直驱实现、样例和验证用例</h4> | ||
| 3 | 6 | ||
| 4 | -## 目录结构说明 | 7 | +[](./docs) |
| 8 | +[](./examples) | ||
| 9 | +[](./LICENSE) | ||
| 10 | +[](./CONTRIBUTING.md) | ||
| 5 | 11 | ||
| 12 | +</div> | ||
| 13 | + | ||
| 14 | +## 🔥Latest News | ||
| 15 | + | ||
| 16 | +- [2026/07] asc-comm项目首次上线 | ||
| 17 | + | ||
| 18 | +### 🚀 当前能力 | ||
| 19 | + | ||
| 20 | +- 提供AICore侧Hcomm点对点通信接口，覆盖`Init`、`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`、`AtomicCAS`、`Commit`、`Drain`。 | ||
| 21 | +- 提供AIV直驱Hcomm RoCE和UBC_CTP/URMA相关实现，主实现位于`src/aicore/hcomm/detail/`。 | ||
| 22 | +- 提供Hcomm UT工程，覆盖`ascend950pr_9599_AIV`的RoCE/URMA路径，以及`ascend910B1_AIC`基础接口用例。 | ||
| 23 | +- 提供`hcomm_write_read_nbi`样例，演示AICore Kernel侧`WriteNbi`和`ReadNbi`点对点通信流程，并包含运行样例所需的Host侧资源准备流程。 | ||
| 24 | + | ||
| 25 | +### 📖 资料文档 | ||
| 26 | + | ||
| 27 | +- 新增[快速开始](./docs/quick_start.md)、[构建与测试](./docs/guide/build_and_test.md)、[三方依赖与兼容性](./docs/guide/dependencies.md)说明。 | ||
| 28 | +- 新增[Hcomm使用说明](./docs/guide/hcomm_usage.md)和[API参考](./docs/api/README.md)，覆盖当前公开的Hcomm接口。 | ||
| 29 | +- 新增[样例目录](./examples/README.md)，提供Hcomm Kernel侧调用和端到端通信样例入口。 | ||
| 30 | + | ||
| 31 | +有关所有历史版本及更新的详细信息，请参阅[CHANGELOG.md](./CHANGELOG.md)。 | ||
| 32 | + | ||
| 33 | +## 🚀概述 | ||
| 34 | + | ||
| 35 | +asc-comm是面向昇腾AI处理器通信场景的开源仓，当前用于承载AICore侧公开API、AIV直驱设备侧实现、API文档、样例和验证能力。 | ||
| 36 | + | ||
| 37 | +当前公开能力以`AscendC::Hcomm`为主，面向算子Kernel侧点对点通信数据路径。使用方通过`AscendC::Hcomm`模板选择通信协议，通过`ChannelHandle`指定通信通道，并调用非阻塞读写接口提交通信任务。任务可按需显式`Commit`提交，并通过`Drain`等待完成。 | ||
| 38 | + | ||
| 39 | +### 数据面能力 | ||
| 40 | + | ||
| 41 | +| 能力 | 当前状态 | | ||
| 42 | +| --- | --- | | ||
| 43 | +| AICore Hcomm公开接口 | 已提供Kernel侧`Init`、`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`、`AtomicCAS`、`Commit`、`Drain`。 | | ||
| 44 | +| AIV直驱实现 | 已提供Hcomm RoCE和UBC_CTP/URMA相关实现，主实现位于`src/aicore/hcomm/detail/`。 | | ||
| 45 | +| 样例配套流程 | `hcomm_write_read_nbi`包含运行样例所需的通信域创建、通信内存注册、P2P通道创建和远端内存获取流程。 | | ||
| 46 | +| 协议能力 | `COMM_PROTOCOL_ROCE`支持读写、提交和等待；`COMM_PROTOCOL_UBC_CTP`支持读写、写通知、原子操作、提交和等待。 | | ||
| 47 | +| UT验证 | UT覆盖`ascend950pr_9599_AIV`的RoCE/URMA路径，以及`ascend910B1_AIC`基础接口用例。 | | ||
| 48 | +| 样例 | 提供`hcomm_write_read_nbi`样例，覆盖两卡`WriteNbi`/`ReadNbi`对称通信和结果校验流程。 | | ||
| 49 | + | ||
| 50 | +### 如何使用Hcomm接口 | ||
| 51 | + | ||
| 52 | +Hcomm Kernel侧使用时包含如下头文件： | ||
| 53 | + | ||
| 54 | +```cpp | ||
| 55 | +#include "hcomm/hcomm.h" | ||
| 6 | ``` | 56 | ``` |
| 57 | + | ||
| 58 | +基本调用流程如下： | ||
| 59 | + | ||
| 60 | +1. 创建`AscendC::Hcomm`对象，并选择通信协议。 | ||
| 61 | +2. 调用`Init`初始化临时工作区。 | ||
| 62 | +3. 通过`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`或`AtomicCAS`提交通信任务。 | ||
| 63 | +4. 如果提交任务时设置`commit = false`，调用`Commit`显式提交通信任务。 | ||
| 64 | +5. 调用`Drain`等待通道上的通信任务完成。 | ||
| 65 | + | ||
| 66 | +协议能力说明： | ||
| 67 | + | ||
| 68 | +| 协议 | 能力说明 | | ||
| 69 | +| --- | --- | | ||
| 70 | +| `COMM_PROTOCOL_ROCE` | RoCE点对点通信路径，支持`ReadNbi`、`WriteNbi`、`Commit`、`Drain`，不支持`WriteWithNotifyNbi`。 | | ||
| 71 | +| `COMM_PROTOCOL_UBC_CTP` | UBC CTP/URMA点对点通信路径，支持`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`、`AtomicCAS`、`Commit`、`Drain`。 | | ||
| 72 | + | ||
| 73 | +详细参数约束和返回值说明请参考[Hcomm使用说明](./docs/guide/hcomm_usage.md)和[API参考](./docs/api/README.md)。 | ||
| 74 | + | ||
| 75 | +## 🔍目录结构说明 | ||
| 76 | + | ||
| 77 | +本仓主要包含asc-comm AICore侧通信数据面API、设备侧实现、样例、文档和UT用例，目录结构如下： | ||
| 78 | + | ||
| 79 | +```text | ||
| 80 | +├── cmake # asc-comm CMake辅助模块 | ||
| 7 | ├── docs # 项目文档介绍 | 81 | ├── docs # 项目文档介绍 |
| 8 | -├── examples # 样例 | 82 | +├── examples # asc-comm API样例目录 |
| 9 | -├── include # asc-comm API 声明源代码 | 83 | +│ └── hcomm_write_read_nbi # Hcomm WriteNbi/ReadNbi两卡P2P通信样例 |
| 10 | -| ├── aicore | 84 | +├── include # asc-comm API声明源代码 |
| 11 | -│ ├── ain | 85 | +│ ├── aicore/hcomm # AICore侧Hcomm公开接口 |
| 12 | -│ ├── collective_comm | 86 | +│ ├── ain # AIN相关API预留目录 |
| 13 | -│ └── direct_drive | ||
| 14 | ├── scripts # 脚本 | 87 | ├── scripts # 脚本 |
| 15 | -├── src # asc-comm API 实现源代码 | 88 | +├── src # asc-comm API实现源代码 |
| 16 | -└── tests # asc-comm API UT 用例 | 89 | +│ ├── aicore/hcomm/detail # AICore侧Hcomm实现细节 |
| 90 | +│ │ ├── common # Hcomm公共定义和工具 | ||
| 91 | +│ │ └── impl # Hcomm协议实现与平台差异代码 | ||
| 92 | +└── tests # asc-comm API UT用例 | ||
| 93 | + └── ut/aicore/hcomm # AICore Hcomm UT工程 | ||
| 17 | ``` | 94 | ``` |
| 18 | 95 | ||
| 19 | -## 许可证 | 96 | +## ⚡️快速入门 |
| 20 | 97 | ||
| 21 | -本仓库源码及相关文档遵循 [CANN Open Software License Agreement Version 2.0](./LICENSE)，免责声明以 LICENSE 中的相关条款为准。 | 98 | +若您希望快速体验项目构建和Hcomm UT验证，请先配置CANN环境： |
| 22 | 99 | ||
| 23 | -## 相关信息 | 100 | +```bash |
| 101 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 102 | +``` | ||
| 24 | 103 | ||
| 104 | +默认构建用于检查基础环境。当前AICore Hcomm代码以头文件形式集成，非UT构建不会生成独立库： | ||
| 105 | + | ||
| 106 | +```bash | ||
| 107 | +bash build.sh | ||
| 108 | +``` | ||
| 109 | + | ||
| 110 | +构建并运行Hcomm UT： | ||
| 111 | + | ||
| 112 | +```bash | ||
| 113 | +bash build.sh -t | ||
| 114 | +``` | ||
| 115 | + | ||
| 116 | +如需直接使用CMake构建UT，可指定CANN三方依赖目录： | ||
| 117 | + | ||
| 118 | +```bash | ||
| 119 | +cmake -S tests/ut -B build/ut-hcomm -DCANN_3RD_LIB_PATH=<third_party_path> | ||
| 120 | +cmake --build build/ut-hcomm | ||
| 121 | +``` | ||
| 122 | + | ||
| 123 | +更多环境准备、Docker、CANN包安装和UT依赖说明请参考[快速开始](./docs/quick_start.md)和[构建与测试](./docs/guide/build_and_test.md)。 | ||
| 124 | + | ||
| 125 | +## 🧰clangd/IDE 支持 | ||
| 126 | + | ||
| 127 | +- 安装clangd，推荐使用15或以上版本。 | ||
| 128 | +- 配置本地IDE时，需要将CANN头文件目录和本仓`include/`目录加入索引路径。 | ||
| 129 | +- 在修改Hcomm Kernel侧代码前，建议先`source /usr/local/Ascend/cann/set_env.sh`，确保CANN相关环境变量已配置。 | ||
| 130 | +- 如果使用VS Code，可结合C/C++、clangd等插件完成代码跳转、语法检查和头文件索引。 | ||
| 131 | + | ||
| 132 | +## 📖相关资源 | ||
| 133 | + | ||
| 134 | +- **文档** | ||
| 135 | + | ||
| 136 | + | 文档 | 说明 | | ||
| 137 | + | --- | --- | | ||
| 138 | + | [文档入口](./docs/README.md) | asc-comm文档总入口。 | | ||
| 139 | + | [快速开始](./docs/quick_start.md) | 环境准备、源码编译和UT验证。 | | ||
| 140 | + | [API参考](./docs/api/README.md) | asc-comm当前公开接口列表。 | | ||
| 141 | + | [Hcomm使用说明](./docs/guide/hcomm_usage.md) | Hcomm点对点通信接口的基本使用流程。 | | ||
| 142 | + | [构建与测试](./docs/guide/build_and_test.md) | CANN环境、构建脚本、UT构建和样例构建说明。 | | ||
| 143 | + | [三方依赖与兼容性](./docs/guide/dependencies.md) | 本仓直接依赖、样例运行依赖、安装配置和集成依赖边界。 | | ||
| 144 | + | [样例目录](./examples/README.md) | asc-comm API样例入口。 | | ||
| 145 | + | ||
| 146 | +- **贡献指南** | ||
| 147 | + | ||
| 148 | + | 文档 | 说明 | | ||
| 149 | + | --- | --- | | ||
| 150 | + | [CANN社区贡献指南](https://gitcode.com/cann/community) | CANN社区Issue、PR等通用处理流程。 | | ||
| 151 | + | [asc-comm贡献指南](./CONTRIBUTING.md) | 本仓Issue、开发、检查和PR提交流程。 | | ||
| 152 | + | [API文档贡献指南](./docs/api_contributing.md) | 新增或修改API文档时的结构、约束和检查要求。 | | ||
| 153 | + | [资料贡献指南](./docs/doc_contributing.md) | README、docs、examples等资料文档的补充规范。 | | ||
| 154 | + | ||
| 155 | +- **其他** | ||
| 156 | + | ||
| 157 | + | 文档 | 说明 | | ||
| 158 | + | --- | --- | | ||
| 159 | + | [更新日志](./CHANGELOG.md) | 版本变更记录。 | | ||
| 160 | + | [安全声明](./SECURITY.md) | 安全问题反馈和处理说明。 | | ||
| 161 | + | [三方开源软件清单](./Third_Party_Open_Source_Software_List.yaml) | 本仓三方开源软件清单。 | | ||
| 162 | + | [三方开源软件声明](./Third_Party_Open_Source_Software_Notice) | 本仓三方开源软件声明。 | | ||
| 163 | + | ||
| 164 | +## 📌相关规划 | ||
| 165 | + | ||
| 166 | +- 持续补充AICore Hcomm端到端样例，覆盖更多协议路径和通信接口。 | ||
| 167 | +- 持续完善不同产品、协议路径下的构建验证和UT覆盖。 | ||
| 168 | +- 持续补充API约束、使用说明和常见问题。 | ||
| 169 | + | ||
| 170 | +## 📝相关信息 | ||
| 171 | + | ||
| 172 | +- [贡献指南](./CONTRIBUTING.md) | ||
| 25 | - [安全声明](./SECURITY.md) | 173 | - [安全声明](./SECURITY.md) |
| 26 | -- [三方开源软件清单](./Third_Party_Open_Source_Software_List.yaml) | 174 | +- [许可证](./LICENSE) |
| 27 | -- [三方开源软件声明](./Third_Party_Open_Source_Software_Notice) | ||
The file is empty
| @@ -0,0 +1,26 @@ | |||
| 1 | +# 项目文档 | ||
| 2 | + | ||
| 3 | +## 目录说明 | ||
| 4 | + | ||
| 5 | +```text | ||
| 6 | +docs/ | ||
| 7 | +├── api/ # API参考文档 | ||
| 8 | +├── guide/ # 使用、构建与测试指南 | ||
| 9 | +├── api_contributing.md # API文档贡献指南 | ||
| 10 | +├── doc_contributing.md # 资料贡献指南 | ||
| 11 | +└── quick_start.md # 快速开始 | ||
| 12 | +``` | ||
| 13 | + | ||
| 14 | +## 文档入口 | ||
| 15 | + | ||
| 16 | +| 文档 | 内容 | | ||
| 17 | +| --- | --- | | ||
| 18 | +| [快速开始](./quick_start.md) | asc-comm环境准备、源码编译和UT验证。 | | ||
| 19 | +| [API参考](./api/README.md) | asc-comm当前公开接口列表。 | | ||
| 20 | +| [Hcomm使用说明](./guide/hcomm_usage.md) | Hcomm点对点通信接口的基本使用流程。 | | ||
| 21 | +| [构建与测试](./guide/build_and_test.md) | CANN环境、构建脚本、UT构建和样例构建说明。 | | ||
| 22 | +| [三方依赖与兼容性](./guide/dependencies.md) | 本仓直接依赖、样例运行依赖、安装配置和集成依赖边界。 | | ||
| 23 | +| [API文档贡献指南](./api_contributing.md) | 新增或修改API文档时的结构、约束和检查要求。 | | ||
| 24 | +| [资料贡献指南](./doc_contributing.md) | README、docs、examples等资料文档的补充规范。 | | ||
| 25 | +| [贡献指南](../CONTRIBUTING.md) | Issue、开发、检查和PR提交流程。 | | ||
| 26 | +| [样例](../examples/README.md) | asc-comm API使用样例入口，包含Hcomm WriteNbi/ReadNbi点对点通信样例。 | | ||
| @@ -0,0 +1,21 @@ | |||
| 1 | +# API参考 | ||
| 2 | + | ||
| 3 | +## AICore Hcomm | ||
| 4 | + | ||
| 5 | +| 文档 | 内容 | | ||
| 6 | +| --- | --- | | ||
| 7 | +| [Hcomm](./aicore/hcomm/Hcomm.md) | AICore侧点对点通信接口模板总览、协议能力和使用约束。 | | ||
| 8 | +| [Init](./aicore/hcomm/Init.md) | 初始化Hcomm临时工作区。 | | ||
| 9 | +| [ReadNbi](./aicore/hcomm/ReadNbi.md) | 通过指定通道提交点对点读任务。 | | ||
| 10 | +| [WriteNbi](./aicore/hcomm/WriteNbi.md) | 通过指定通道提交点对点写任务。 | | ||
| 11 | +| [WriteWithNotifyNbi](./aicore/hcomm/WriteWithNotifyNbi.md) | 提交写任务并写远端通知值。 | | ||
| 12 | +| [AtomicFAA](./aicore/hcomm/AtomicFAA.md) | 提交Fetch-and-add原子操作任务。 | | ||
| 13 | +| [AtomicCAS](./aicore/hcomm/AtomicCAS.md) | 提交Compare-and-swap原子操作任务。 | | ||
| 14 | +| [Commit](./aicore/hcomm/Commit.md) | 显式提交通道上的待执行通信任务。 | | ||
| 15 | +| [Drain](./aicore/hcomm/Drain.md) | 等待通道上的通信任务完成。 | | ||
| 16 | + | ||
| 17 | +## 头文件 | ||
| 18 | + | ||
| 19 | +```cpp | ||
| 20 | +#include "hcomm/hcomm.h" | ||
| 21 | +``` | ||
| @@ -0,0 +1,58 @@ | |||
| 1 | +# AtomicCAS | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +提交Compare-and-swap原子操作任务，比较远端地址`dst`中的值和`compareVal`，相等时将远端值替换为`swapVal`，并将替换前的旧值写入本端`fetchAddr`。 | ||
| 6 | + | ||
| 7 | +当前该接口仅支持`COMM_PROTOCOL_UBC_CTP`路径。 | ||
| 8 | + | ||
| 9 | +## 函数原型 | ||
| 10 | + | ||
| 11 | +```cpp | ||
| 12 | +template < | ||
| 13 | + typename T, | ||
| 14 | + bool commit = true, | ||
| 15 | + pipe_t commitPipe = PIPE_S, | ||
| 16 | + pipe_t reqPipe = PIPE_MTE3, | ||
| 17 | + auto const& config = URMA_DEFAULT_CFG> | ||
| 18 | +__aicore__ inline int32_t AtomicCAS( | ||
| 19 | + AscendC::ChannelHandle channel, | ||
| 20 | + GM_ADDR dst, | ||
| 21 | + GM_ADDR fetchAddr, | ||
| 22 | + T compareVal, | ||
| 23 | + T swapVal); | ||
| 24 | +``` | ||
| 25 | + | ||
| 26 | +## 参数说明 | ||
| 27 | + | ||
| 28 | +| 参数 | 输入/输出 | 说明 | | ||
| 29 | +| --- | --- | --- | | ||
| 30 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 31 | +| `dst` | 输入/输出 | 远端原子操作目标GM地址。 | | ||
| 32 | +| `fetchAddr` | 输出 | 本端GM地址，用于保存远端地址执行CAS前的旧值。 | | ||
| 33 | +| `compareVal` | 输入 | 比较值。 | | ||
| 34 | +| `swapVal` | 输入 | 比较成功时写入远端目标地址的新值。 | | ||
| 35 | + | ||
| 36 | +## 模板参数 | ||
| 37 | + | ||
| 38 | +| 参数 | 说明 | | ||
| 39 | +| --- | --- | | ||
| 40 | +| `T` | 原子操作数据类型，仅支持`int32_t`、`uint32_t`、`int64_t`、`uint64_t`。 | | ||
| 41 | +| `commit` | 是否在提交任务时立即commit。 | | ||
| 42 | +| `commitPipe` | commit使用的pipe，默认`PIPE_S`。 | | ||
| 43 | +| `reqPipe` | 请求使用的pipe，默认`PIPE_MTE3`。 | | ||
| 44 | +| `config` | URMA WQE控制配置，仅URMA路径使用。默认为`URMA_DEFAULT_CFG`（强序 + fence + 使能CQE）。 | | ||
| 45 | + | ||
| 46 | +## 返回值 | ||
| 47 | + | ||
| 48 | +| 返回值 | 说明 | | ||
| 49 | +| --- | --- | | ||
| 50 | +| `0` | 提交成功。 | | ||
| 51 | +| `-1` | 提交失败。 | | ||
| 52 | + | ||
| 53 | +## 约束说明 | ||
| 54 | + | ||
| 55 | +- 调用前通信通道需已完成初始化。 | ||
| 56 | +- 传入的`ChannelHandle`需要对应`COMM_PROTOCOL_UBC_CTP`通道。 | ||
| 57 | +- `dst`需要落在通道注册的远端buffer范围内，`fetchAddr`用于保存旧值，长度为`sizeof(T)`。 | ||
| 58 | +- 单个`AtomicCAS`任务在URMA SQ中占用2个WQE block。 | ||
| @@ -0,0 +1,53 @@ | |||
| 1 | +# AtomicFAA | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +提交Fetch-and-add原子操作任务，对远端地址`dst`中的值执行原子加，并将加法前的旧值写入本端`fetchAddr`。 | ||
| 6 | + | ||
| 7 | +当前该接口仅支持`COMM_PROTOCOL_UBC_CTP`路径。 | ||
| 8 | + | ||
| 9 | +## 函数原型 | ||
| 10 | + | ||
| 11 | +```cpp | ||
| 12 | +template < | ||
| 13 | + typename T, | ||
| 14 | + bool commit = true, | ||
| 15 | + pipe_t commitPipe = PIPE_S, | ||
| 16 | + pipe_t reqPipe = PIPE_MTE3, | ||
| 17 | + auto const& config = URMA_DEFAULT_CFG> | ||
| 18 | +__aicore__ inline int32_t AtomicFAA( | ||
| 19 | + AscendC::ChannelHandle channel, GM_ADDR dst, GM_ADDR fetchAddr, T addVal); | ||
| 20 | +``` | ||
| 21 | + | ||
| 22 | +## 参数说明 | ||
| 23 | + | ||
| 24 | +| 参数 | 输入/输出 | 说明 | | ||
| 25 | +| --- | --- | --- | | ||
| 26 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 27 | +| `dst` | 输入/输出 | 远端原子操作目标GM地址。 | | ||
| 28 | +| `fetchAddr` | 输出 | 本端GM地址，用于保存远端地址执行原子加前的旧值。 | | ||
| 29 | +| `addVal` | 输入 | 加到远端目标地址的值。 | | ||
| 30 | + | ||
| 31 | +## 模板参数 | ||
| 32 | + | ||
| 33 | +| 参数 | 说明 | | ||
| 34 | +| --- | --- | | ||
| 35 | +| `T` | 原子操作数据类型，仅支持`int32_t`、`uint32_t`、`int64_t`、`uint64_t`。 | | ||
| 36 | +| `commit` | 是否在提交任务时立即commit。 | | ||
| 37 | +| `commitPipe` | commit使用的pipe，默认`PIPE_S`。 | | ||
| 38 | +| `reqPipe` | 请求使用的pipe，默认`PIPE_MTE3`。 | | ||
| 39 | +| `config` | URMA WQE控制配置，仅URMA路径使用。默认为`URMA_DEFAULT_CFG`（强序 + fence + 使能CQE）。 | | ||
| 40 | + | ||
| 41 | +## 返回值 | ||
| 42 | + | ||
| 43 | +| 返回值 | 说明 | | ||
| 44 | +| --- | --- | | ||
| 45 | +| `0` | 提交成功。 | | ||
| 46 | +| `-1` | 提交失败。 | | ||
| 47 | + | ||
| 48 | +## 约束说明 | ||
| 49 | + | ||
| 50 | +- 调用前通信通道需已完成初始化。 | ||
| 51 | +- 传入的`ChannelHandle`需要对应`COMM_PROTOCOL_UBC_CTP`通道。 | ||
| 52 | +- `dst`需要落在通道注册的远端buffer范围内，`fetchAddr`用于保存旧值，长度为`sizeof(T)`。 | ||
| 53 | +- 单个`AtomicFAA`任务在URMA SQ中占用2个WQE block。 | ||
| @@ -0,0 +1,31 @@ | |||
| 1 | +# Commit | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +通知指定通道上已提交的通信任务可以开始执行。通常在调用`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`或`AtomicCAS`时将`commit`设置为`false`后显式调用。 | ||
| 6 | + | ||
| 7 | +## 函数原型 | ||
| 8 | + | ||
| 9 | +```cpp | ||
| 10 | +template <pipe_t pipe = PIPE_S> | ||
| 11 | +__aicore__ inline int32_t Commit(AscendC::ChannelHandle channel); | ||
| 12 | +``` | ||
| 13 | + | ||
| 14 | +## 参数说明 | ||
| 15 | + | ||
| 16 | +| 参数 | 输入/输出 | 说明 | | ||
| 17 | +| --- | --- | --- | | ||
| 18 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 19 | + | ||
| 20 | +## 模板参数 | ||
| 21 | + | ||
| 22 | +| 参数 | 说明 | | ||
| 23 | +| --- | --- | | ||
| 24 | +| `pipe` | commit使用的pipe，默认`PIPE_S`。 | | ||
| 25 | + | ||
| 26 | +## 返回值 | ||
| 27 | + | ||
| 28 | +| 返回值 | 说明 | | ||
| 29 | +| --- | --- | | ||
| 30 | +| `0` | 提交成功。 | | ||
| 31 | +| `-1` | 提交失败。 | | ||
| @@ -0,0 +1,31 @@ | |||
| 1 | +# Drain | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +阻塞等待指定通道上的通信任务执行完成。 | ||
| 6 | + | ||
| 7 | +## 函数原型 | ||
| 8 | + | ||
| 9 | +```cpp | ||
| 10 | +template <pipe_t pipe = PIPE_MTE3> | ||
| 11 | +__aicore__ inline int32_t Drain(AscendC::ChannelHandle channel); | ||
| 12 | +``` | ||
| 13 | + | ||
| 14 | +## 参数说明 | ||
| 15 | + | ||
| 16 | +| 参数 | 输入/输出 | 说明 | | ||
| 17 | +| --- | --- | --- | | ||
| 18 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 19 | + | ||
| 20 | +## 模板参数 | ||
| 21 | + | ||
| 22 | +| 参数 | 说明 | | ||
| 23 | +| --- | --- | | ||
| 24 | +| `pipe` | drain使用的pipe，默认`PIPE_MTE3`。 | | ||
| 25 | + | ||
| 26 | +## 返回值 | ||
| 27 | + | ||
| 28 | +| 返回值 | 说明 | | ||
| 29 | +| --- | --- | | ||
| 30 | +| `0` | 等待成功。 | | ||
| 31 | +| `-1` | 等待失败。 | | ||
| @@ -0,0 +1,63 @@ | |||
| 1 | +# Hcomm | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +头文件为： | ||
| 6 | + | ||
| 7 | +```cpp | ||
| 8 | +#include "hcomm/hcomm.h" | ||
| 9 | +``` | ||
| 10 | + | ||
| 11 | +`AscendC::Hcomm`是AICore侧点对点通信接口模板，支持通过通信通道提交读、写、写通知和原子操作任务，并通过`Commit`和`Drain`控制任务提交与完成等待。本文档说明RoCE和UBC_CTP/URMA路径的协议能力和使用约束。 | ||
| 12 | + | ||
| 13 | +## 模板参数 | ||
| 14 | + | ||
| 15 | +```cpp | ||
| 16 | +template <AscendC::CommProtocol commProtocol = AscendC::COMM_PROTOCOL_UBC_CTP> | ||
| 17 | +class Hcomm; | ||
| 18 | +``` | ||
| 19 | + | ||
| 20 | +| 参数 | 说明 | | ||
| 21 | +| --- | --- | | ||
| 22 | +| `commProtocol` | 通信协议类型，支持`COMM_PROTOCOL_ROCE`和`COMM_PROTOCOL_UBC_CTP`，默认值为`COMM_PROTOCOL_UBC_CTP`。 | | ||
| 23 | + | ||
| 24 | +## 协议能力 | ||
| 25 | + | ||
| 26 | +| 接口 | `COMM_PROTOCOL_ROCE` | `COMM_PROTOCOL_UBC_CTP` | | ||
| 27 | +| --- | --- | --- | | ||
| 28 | +| `Init` | 支持，需要提供UB临时工作区。 | 支持，需要提供URMA临时工作区。 | | ||
| 29 | +| `ReadNbi` | 支持。 | 支持。 | | ||
| 30 | +| `WriteNbi` | 支持。 | 支持。 | | ||
| 31 | +| `WriteWithNotifyNbi` | 不支持，调用会返回失败。 | 支持。 | | ||
| 32 | +| `AtomicFAA` | 不支持。 | 支持。 | | ||
| 33 | +| `AtomicCAS` | 不支持。 | 支持。 | | ||
| 34 | +| `Commit` | 支持。 | 支持。 | | ||
| 35 | +| `Drain` | 支持。 | 支持。 | | ||
| 36 | + | ||
| 37 | +## 常用接口 | ||
| 38 | + | ||
| 39 | +| 接口 | 说明 | | ||
| 40 | +| --- | --- | | ||
| 41 | +| [Init](./Init.md) | 初始化Hcomm临时工作区。 | | ||
| 42 | +| [ReadNbi](./ReadNbi.md) | 通过指定通道提交读任务。 | | ||
| 43 | +| [WriteNbi](./WriteNbi.md) | 通过指定通道提交写任务。 | | ||
| 44 | +| [WriteWithNotifyNbi](./WriteWithNotifyNbi.md) | 提交写任务并写通知值。 | | ||
| 45 | +| [AtomicFAA](./AtomicFAA.md) | 提交Fetch-and-add原子操作任务。 | | ||
| 46 | +| [AtomicCAS](./AtomicCAS.md) | 提交Compare-and-swap原子操作任务。 | | ||
| 47 | +| [Commit](./Commit.md) | 显式提交通道上的待执行任务。 | | ||
| 48 | +| [Drain](./Drain.md) | 等待通道上的通信任务完成。 | | ||
| 49 | + | ||
| 50 | +## 返回值 | ||
| 51 | + | ||
| 52 | +| 返回值 | 说明 | | ||
| 53 | +| --- | --- | | ||
| 54 | +| `0` | 执行成功。 | | ||
| 55 | +| `-1` | 执行失败。 | | ||
| 56 | + | ||
| 57 | +## 使用约束 | ||
| 58 | + | ||
| 59 | +- 调用通信接口前，通信通道需要由调用方完成初始化。 | ||
| 60 | +- `COMM_PROTOCOL_ROCE`和`COMM_PROTOCOL_UBC_CTP`路径均需要通过`Init`提供临时工作区，不同协议使用的临时工作区布局不同。 | ||
| 61 | +- `WriteWithNotifyNbi`仅支持`COMM_PROTOCOL_UBC_CTP`路径，`COMM_PROTOCOL_ROCE`路径会返回失败。 | ||
| 62 | +- `AtomicFAA`和`AtomicCAS`仅支持`COMM_PROTOCOL_UBC_CTP`路径，数据类型仅支持`int32_t`、`uint32_t`、`int64_t`、`uint64_t`。 | ||
| 63 | +- 传入的`ChannelHandle`需要指向与协议匹配的通道实体。 | ||
| @@ -0,0 +1,35 @@ | |||
| 1 | +# Init | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +初始化Hcomm临时工作区。`COMM_PROTOCOL_ROCE`和`COMM_PROTOCOL_UBC_CTP`路径均需要在提交通信任务前调用该接口完成临时工作区配置。 | ||
| 6 | + | ||
| 7 | +## 函数原型 | ||
| 8 | + | ||
| 9 | +```cpp | ||
| 10 | +__aicore__ inline int32_t Init(__ubuf__ uint8_t* buff, uint32_t len); | ||
| 11 | + | ||
| 12 | +template <typename T> | ||
| 13 | +__aicore__ inline int32_t Init(const AscendC::LocalTensor<T>& buff, uint32_t len); | ||
| 14 | +``` | ||
| 15 | + | ||
| 16 | +## 参数说明 | ||
| 17 | + | ||
| 18 | +| 参数 | 输入/输出 | 说明 | | ||
| 19 | +| --- | --- | --- | | ||
| 20 | +| `buff` | 输入 | 调用方提供的UB缓冲区或`LocalTensor`。 | | ||
| 21 | +| `len` | 输入 | 缓冲区长度，单位为字节。 | | ||
| 22 | + | ||
| 23 | +## 返回值 | ||
| 24 | + | ||
| 25 | +| 返回值 | 说明 | | ||
| 26 | +| --- | --- | | ||
| 27 | +| `0` | 初始化成功。 | | ||
| 28 | +| `-1` | 初始化失败。 | | ||
| 29 | + | ||
| 30 | +## 约束说明 | ||
| 31 | + | ||
| 32 | +- RoCE路径会使用`buff`作为WQE/CQE/doorbell临时工作区，当前最小工作区大小为512字节。 | ||
| 33 | +- UBC_CTP/URMA路径会使用`buff`作为WQE/CQE临时工作区，当前最小工作区大小为512字节。 | ||
| 34 | +- 使用`__ubuf__ uint8_t*`初始化时，实现会对`buff`按32字节对齐后的地址作为临时工作区起始地址。 | ||
| 35 | +- 使用`LocalTensor`初始化时，实现直接使用传入tensor。RoCE路径要求`len`和`buff.GetSize()`均不小于512字节；UBC_CTP/URMA路径要求`len`不小于512字节且不能超过`buff.GetSize()`。 | ||
| @@ -0,0 +1,47 @@ | |||
| 1 | +# ReadNbi | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +通过指定通信通道提交点对点读任务，将数据从`src`读取到`dst`。 | ||
| 6 | + | ||
| 7 | +## 函数原型 | ||
| 8 | + | ||
| 9 | +```cpp | ||
| 10 | +template < | ||
| 11 | + bool commit = true, | ||
| 12 | + pipe_t commitPipe = PIPE_S, | ||
| 13 | + pipe_t reqPipe = PIPE_MTE3, | ||
| 14 | + auto const& config = URMA_DEFAULT_CFG> | ||
| 15 | +__aicore__ inline int32_t ReadNbi( | ||
| 16 | + AscendC::ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len); | ||
| 17 | +``` | ||
| 18 | + | ||
| 19 | +## 参数说明 | ||
| 20 | + | ||
| 21 | +| 参数 | 输入/输出 | 说明 | | ||
| 22 | +| --- | --- | --- | | ||
| 23 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 24 | +| `dst` | 输出 | 目的GM地址。 | | ||
| 25 | +| `src` | 输入 | 源GM地址。 | | ||
| 26 | +| `len` | 输入 | 读取长度，单位为字节。 | | ||
| 27 | + | ||
| 28 | +## 模板参数 | ||
| 29 | + | ||
| 30 | +| 参数 | 说明 | | ||
| 31 | +| --- | --- | | ||
| 32 | +| `commit` | 是否在提交任务时立即commit。 | | ||
| 33 | +| `commitPipe` | commit使用的pipe，默认`PIPE_S`。 | | ||
| 34 | +| `reqPipe` | 请求使用的pipe，默认`PIPE_MTE3`。 | | ||
| 35 | +| `config` | URMA WQE控制配置，仅URMA路径使用。默认为`URMA_DEFAULT_CFG`（强序 + fence + 使能CQE）。 | | ||
| 36 | + | ||
| 37 | +## 返回值 | ||
| 38 | + | ||
| 39 | +| 返回值 | 说明 | | ||
| 40 | +| --- | --- | | ||
| 41 | +| `0` | 提交成功。 | | ||
| 42 | +| `-1` | 提交失败。 | | ||
| 43 | + | ||
| 44 | +## 约束说明 | ||
| 45 | + | ||
| 46 | +- 调用前通信通道需已完成初始化。 | ||
| 47 | +- `COMM_PROTOCOL_UBC_CTP`路径下，`src`需要落在通道注册的远端buffer范围内，`dst`为本端目标地址。 | ||
| @@ -0,0 +1,47 @@ | |||
| 1 | +# WriteNbi | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +通过指定通信通道提交点对点写任务，将数据从`src`写入到`dst`。 | ||
| 6 | + | ||
| 7 | +## 函数原型 | ||
| 8 | + | ||
| 9 | +```cpp | ||
| 10 | +template < | ||
| 11 | + bool commit = true, | ||
| 12 | + pipe_t commitPipe = PIPE_S, | ||
| 13 | + pipe_t reqPipe = PIPE_MTE3, | ||
| 14 | + auto const& config = URMA_DEFAULT_CFG> | ||
| 15 | +__aicore__ inline int32_t WriteNbi( | ||
| 16 | + AscendC::ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len); | ||
| 17 | +``` | ||
| 18 | + | ||
| 19 | +## 参数说明 | ||
| 20 | + | ||
| 21 | +| 参数 | 输入/输出 | 说明 | | ||
| 22 | +| --- | --- | --- | | ||
| 23 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 24 | +| `dst` | 输出 | 目的GM地址。 | | ||
| 25 | +| `src` | 输入 | 源GM地址。 | | ||
| 26 | +| `len` | 输入 | 写入长度，单位为字节。 | | ||
| 27 | + | ||
| 28 | +## 模板参数 | ||
| 29 | + | ||
| 30 | +| 参数 | 说明 | | ||
| 31 | +| --- | --- | | ||
| 32 | +| `commit` | 是否在提交任务时立即commit。 | | ||
| 33 | +| `commitPipe` | commit使用的pipe，默认`PIPE_S`。 | | ||
| 34 | +| `reqPipe` | 请求使用的pipe，默认`PIPE_MTE3`。 | | ||
| 35 | +| `config` | URMA WQE控制配置，仅URMA路径使用。默认为`URMA_DEFAULT_CFG`（强序 + fence + 使能CQE）。 | | ||
| 36 | + | ||
| 37 | +## 返回值 | ||
| 38 | + | ||
| 39 | +| 返回值 | 说明 | | ||
| 40 | +| --- | --- | | ||
| 41 | +| `0` | 提交成功。 | | ||
| 42 | +| `-1` | 提交失败。 | | ||
| 43 | + | ||
| 44 | +## 约束说明 | ||
| 45 | + | ||
| 46 | +- 调用前通信通道需已完成初始化。 | ||
| 47 | +- `COMM_PROTOCOL_UBC_CTP`路径下，`dst`需要落在通道注册的远端buffer范围内，`src`为本端源地址。 | ||
| @@ -0,0 +1,58 @@ | |||
| 1 | +# WriteWithNotifyNbi | ||
| 2 | + | ||
| 3 | +## 功能说明 | ||
| 4 | + | ||
| 5 | +提交点对点写任务，并在写任务中携带远端通知地址和通知值。 | ||
| 6 | + | ||
| 7 | +当前该接口仅支持`COMM_PROTOCOL_UBC_CTP`路径。`COMM_PROTOCOL_ROCE`路径保留同名接口，但实现会返回失败。 | ||
| 8 | + | ||
| 9 | +## 函数原型 | ||
| 10 | + | ||
| 11 | +```cpp | ||
| 12 | +template < | ||
| 13 | + bool commit = true, | ||
| 14 | + pipe_t commitPipe = PIPE_S, | ||
| 15 | + pipe_t reqPipe = PIPE_MTE3, | ||
| 16 | + auto const& config = URMA_DEFAULT_CFG> | ||
| 17 | +__aicore__ inline int32_t WriteWithNotifyNbi( | ||
| 18 | + AscendC::ChannelHandle channel, | ||
| 19 | + GM_ADDR dst, | ||
| 20 | + GM_ADDR src, | ||
| 21 | + uint64_t len, | ||
| 22 | + GM_ADDR notifyAddr, | ||
| 23 | + uint64_t notifyVal); | ||
| 24 | +``` | ||
| 25 | + | ||
| 26 | +## 参数说明 | ||
| 27 | + | ||
| 28 | +| 参数 | 输入/输出 | 说明 | | ||
| 29 | +| --- | --- | --- | | ||
| 30 | +| `channel` | 输入 | 通信通道句柄。 | | ||
| 31 | +| `dst` | 输出 | 目的GM地址。 | | ||
| 32 | +| `src` | 输入 | 源GM地址。 | | ||
| 33 | +| `len` | 输入 | 写入长度，单位为字节。 | | ||
| 34 | +| `notifyAddr` | 输入 | 远端通知地址。 | | ||
| 35 | +| `notifyVal` | 输入 | 远端通知值。 | | ||
| 36 | + | ||
| 37 | +## 模板参数 | ||
| 38 | + | ||
| 39 | +| 参数 | 说明 | | ||
| 40 | +| --- | --- | | ||
| 41 | +| `commit` | 是否在提交任务时立即commit。 | | ||
| 42 | +| `commitPipe` | commit使用的pipe，默认`PIPE_S`。 | | ||
| 43 | +| `reqPipe` | 请求使用的pipe，默认`PIPE_MTE3`。 | | ||
| 44 | +| `config` | URMA WQE控制配置，仅URMA路径使用。默认为`URMA_DEFAULT_CFG`（强序 + fence + 使能CQE）。 | | ||
| 45 | + | ||
| 46 | +## 返回值 | ||
| 47 | + | ||
| 48 | +| 返回值 | 说明 | | ||
| 49 | +| --- | --- | | ||
| 50 | +| `0` | 提交成功。 | | ||
| 51 | +| `-1` | 提交失败。 | | ||
| 52 | + | ||
| 53 | +## 约束说明 | ||
| 54 | + | ||
| 55 | +- 调用前通信通道需已完成初始化。 | ||
| 56 | +- 传入的`ChannelHandle`需要对应`COMM_PROTOCOL_UBC_CTP`通道。 | ||
| 57 | +- `COMM_PROTOCOL_ROCE`路径不支持该接口，调用会返回`-1`。 | ||
| 58 | +- 单个`WriteWithNotifyNbi`任务在URMA SQ中占用2个WQE block。 | ||
| @@ -0,0 +1,35 @@ | |||
| 1 | +# API文档贡献指南 | ||
| 2 | + | ||
| 3 | +本文档说明asc-comm API文档的补充和修改要求。新增或修改公开API时，应同步更新对应API文档、使用说明、样例和测试说明。 | ||
| 4 | + | ||
| 5 | +## 适用范围 | ||
| 6 | + | ||
| 7 | +适用于`docs/api/`下的公开API参考文档，以及与API行为直接相关的使用说明和样例文档。 | ||
| 8 | + | ||
| 9 | +## 文档结构 | ||
| 10 | + | ||
| 11 | +新增API文档建议包含以下章节： | ||
| 12 | + | ||
| 13 | +- 功能说明：说明API的用途、使用场景和适用范围。 | ||
| 14 | +- 函数原型：保持与公开头文件中的声明一致。 | ||
| 15 | +- 参数说明：列出参数名称、输入输出属性、单位和约束。 | ||
| 16 | +- 模板参数：模板接口需说明默认值、适用协议或平台差异。 | ||
| 17 | +- 返回值：说明成功、失败和常见失败条件。 | ||
| 18 | +- 约束说明：说明调用顺序、地址要求、对齐要求、协议支持范围和配套资源依赖。 | ||
| 19 | + | ||
| 20 | +## 编写要求 | ||
| 21 | + | ||
| 22 | +- API名称、参数名称、默认模板参数和返回值必须以代码为准。 | ||
| 23 | +- 文档中涉及协议能力时，需要明确支持范围，例如`COMM_PROTOCOL_ROCE`、`COMM_PROTOCOL_UBC_CTP`。 | ||
| 24 | +- 若接口涉及通信通道、注册内存或算子工程能力，需要在约束说明中写清资源准备要求。 | ||
| 25 | +- 示例代码应能反映可验证的调用方式；无法独立运行的片段需要明确说明前置条件。 | ||
| 26 | +- 修改API行为时，应同步更新`docs/api/README.md`、相关guide、examples和UT说明。 | ||
| 27 | + | ||
| 28 | +## 检查建议 | ||
| 29 | + | ||
| 30 | +提交前建议完成以下检查： | ||
| 31 | + | ||
| 32 | +- 确认文档中的函数原型与`include/`下公开头文件一致。 | ||
| 33 | +- 确认链接路径可从当前文档位置正确跳转。 | ||
| 34 | +- 确认新增约束与现有UT或实现逻辑一致。 | ||
| 35 | +- 若新增三方依赖或直接打包内容，同步检查三方软件清单和Notice。 | ||
| @@ -0,0 +1,114 @@ | |||
| 1 | +# 资料贡献指南 | ||
| 2 | + | ||
| 3 | +## 概述 | ||
| 4 | + | ||
| 5 | +asc-comm资料体系包含仓库入口、快速开始、构建测试、API参考、使用指南和样例说明等内容。开发者可以通过提交PR对资料进行纠错、补充和优化。 | ||
| 6 | + | ||
| 7 | +| 文档类型 | 内容 | 文档目录 | | ||
| 8 | +| --- | --- | --- | | ||
| 9 | +| 仓库入口 | 项目概述、目录结构、常用文档入口 | `README.md` | | ||
| 10 | +| 快速开始 | 环境准备、源码下载、构建和UT验证 | `docs/quick_start.md` | | ||
| 11 | +| 构建与依赖 | 构建脚本、CMake入口、三方依赖说明 | `docs/guide/` | | ||
| 12 | +| API参考 | 接口功能、原型、参数、返回值和约束 | `docs/api/` | | ||
| 13 | +| 使用指南 | API调用流程、协议能力和注意事项 | `docs/guide/` | | ||
| 14 | +| 样例说明 | 样例目录入口、运行边界和验证说明 | `examples/` | | ||
| 15 | + | ||
| 16 | +## 贡献场景 | ||
| 17 | + | ||
| 18 | +### 文档纠错 | ||
| 19 | + | ||
| 20 | +如果发现资料中存在链接失效、路径错误、命令不可执行、参数值不准确或约束遗漏等问题： | ||
| 21 | + | ||
| 22 | +1. 新建 `Documentation | 文档反馈` 类Issue，说明问题位置和期望修改。 | ||
| 23 | +2. 在评论框中输入 `/assign` 或 `/assign @yourself` 将该Issue分配给您。 | ||
| 24 | +3. 修复后提交PR，并在PR中说明验证方式。 | ||
| 25 | + | ||
| 26 | +### 文档补充 | ||
| 27 | + | ||
| 28 | +如果需要补充接口说明、构建说明、依赖说明、样例说明或常见问题： | ||
| 29 | + | ||
| 30 | +1. 新建 `Requirement | 需求建议` 类Issue，描述补充内容和适用场景。 | ||
| 31 | +2. 按照本文档的编写规范补充资料。 | ||
| 32 | +3. 同步更新相关入口文档，避免新增内容成为孤立页面。 | ||
| 33 | + | ||
| 34 | +### 样例资料补充 | ||
| 35 | + | ||
| 36 | +新增样例时，需要同步补充样例说明，明确样例用途、前置条件、构建运行方式和验证命令。 | ||
| 37 | + | ||
| 38 | +## 编写规范 | ||
| 39 | + | ||
| 40 | +### 通用规范 | ||
| 41 | + | ||
| 42 | +| 规则 | 要求 | | ||
| 43 | +| --- | --- | | ||
| 44 | +| 与代码一致 | 目录结构、构建命令、接口名称、函数原型和返回值应与当前仓库实际内容一致。 | | ||
| 45 | +| 适用范围清晰 | 对需要前置条件或仅适用于特定场景的能力，需要明确说明适用范围和验证方式。 | | ||
| 46 | +| 面向验证 | 快速开始、构建测试和样例文档应说明用户能直接执行什么、不能直接执行什么。 | | ||
| 47 | +| 链接有效 | 新增文档后需要同步更新入口页面，并检查相对链接和图片路径有效。 | | ||
| 48 | +| 首次引入加链接 | 首次提到其他文档负责的概念或API时添加链接，后续重复出现不必反复链接。 | | ||
| 49 | + | ||
| 50 | +### 文档结构 | ||
| 51 | + | ||
| 52 | +新增专题文档建议包含以下内容： | ||
| 53 | + | ||
| 54 | +1. 背景或适用范围。 | ||
| 55 | +2. 前置条件和依赖。 | ||
| 56 | +3. 操作步骤或接口使用流程。 | ||
| 57 | +4. 约束说明和适用范围。 | ||
| 58 | +5. 验证方式。 | ||
| 59 | +6. 相关文档链接。 | ||
| 60 | + | ||
| 61 | +### API相关资料 | ||
| 62 | + | ||
| 63 | +API行为相关资料应与 `include/` 下公开头文件保持一致。涉及协议能力、地址约束、对齐要求、调用顺序或产品差异时，需要在API参考和使用指南中同步说明。 | ||
| 64 | + | ||
| 65 | +### 样例相关资料 | ||
| 66 | + | ||
| 67 | +样例资料应说明： | ||
| 68 | + | ||
| 69 | +- 样例用途和覆盖的API。 | ||
| 70 | +- 样例文件结构。 | ||
| 71 | +- 是否可独立构建或运行。 | ||
| 72 | +- 构建、运行和验证命令。 | ||
| 73 | +- 必要的前置条件和资源准备。 | ||
| 74 | + | ||
| 75 | +## 常见修改场景 | ||
| 76 | + | ||
| 77 | +- 新增接口：同步更新API参考、使用指南、样例说明和文档入口。 | ||
| 78 | +- 新增样例：同步更新`examples/README.md`，说明样例用途、前置条件和验证方式。 | ||
| 79 | +- 修改构建流程：同步更新快速开始、构建与测试文档。 | ||
| 80 | +- 新增依赖：同步更新三方依赖说明、三方软件清单和Notice。 | ||
| 81 | +- 调整目录结构：同步更新README、docs入口和相关相对链接。 | ||
| 82 | + | ||
| 83 | +## 提交PR前的自检清单 | ||
| 84 | + | ||
| 85 | +- [ ] 文档描述与当前仓库代码、目录和脚本一致。 | ||
| 86 | +- [ ] Markdown表格、代码块、标题层级和列表格式正确。 | ||
| 87 | +- [ ] 相对链接和图片路径可以从当前文档位置正确跳转。 | ||
| 88 | +- [ ] 命令示例可执行，或已明确说明仅作为示例。 | ||
| 89 | +- [ ] API函数原型、参数、返回值与公开头文件一致。 | ||
| 90 | +- [ ] 样例说明已标注前置条件、构建运行方式和验证方式。 | ||
| 91 | +- [ ] 新增依赖已同步更新三方软件清单和Notice。 | ||
| 92 | +- [ ] 尚未合入代码或文档的能力，没有提前描述为已支持能力。 | ||
| 93 | + | ||
| 94 | +## 资料体系与链接关系 | ||
| 95 | + | ||
| 96 | +asc-comm资料通过入口文档、指南、API参考和样例说明形成导航关系。新增或修改文档时，应遵循“谁提到其他文档负责的内容，谁添加链接”的原则。 | ||
| 97 | + | ||
| 98 | +```text | ||
| 99 | +README.md ──快速上手──→ docs/quick_start.md | ||
| 100 | + ──构建测试──→ docs/guide/build_and_test.md | ||
| 101 | + ──API参考───→ docs/api/ | ||
| 102 | + ──样例入口──→ examples/README.md | ||
| 103 | + | ||
| 104 | +docs/guide/ ──首次引入API──→ docs/api/ | ||
| 105 | + ──涉及样例────→ examples/README.md | ||
| 106 | + | ||
| 107 | +docs/api/ ───使用流程──→ docs/guide/ | ||
| 108 | + ───调用示例──→ examples/README.md | ||
| 109 | +``` | ||
| 110 | + | ||
| 111 | +## 更多信息 | ||
| 112 | + | ||
| 113 | +- API文档贡献指南：[api_contributing.md](./api_contributing.md) | ||
| 114 | +- asc-comm贡献指南：[CONTRIBUTING.md](../CONTRIBUTING.md) | ||
| @@ -0,0 +1,94 @@ | |||
| 1 | +# 构建与测试 | ||
| 2 | + | ||
| 3 | +## 环境准备 | ||
| 4 | + | ||
| 5 | +执行`build.sh`前，需要先source CANN环境脚本设置环境变量（`build.sh`会检查`ASCEND_HOME_PATH`，无论是否构建UT均需配置）。 | ||
| 6 | + | ||
| 7 | +```bash | ||
| 8 | +# 默认路径安装，以root用户为例（非root用户，将/usr/local替换为${HOME}） | ||
| 9 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 10 | +# 指定路径安装 | ||
| 11 | +# source ${install_path}/cann/set_env.sh | ||
| 12 | +``` | ||
| 13 | + | ||
| 14 | +基础环境和三方依赖清单见[三方依赖与兼容性](./dependencies.md)。 | ||
| 15 | + | ||
| 16 | +## 构建说明 | ||
| 17 | + | ||
| 18 | +直接执行构建脚本时，脚本会完成基础环境检查。UT需要通过测试构建参数单独触发。 | ||
| 19 | + | ||
| 20 | +```bash | ||
| 21 | +bash build.sh | ||
| 22 | +``` | ||
| 23 | + | ||
| 24 | +## 构建UT | ||
| 25 | + | ||
| 26 | +使用`-t`或`--test`构建Hcomm UT。 | ||
| 27 | + | ||
| 28 | +```bash | ||
| 29 | +bash build.sh -t | ||
| 30 | +``` | ||
| 31 | + | ||
| 32 | +构建目录默认为： | ||
| 33 | + | ||
| 34 | +```text | ||
| 35 | +build/ut-hcomm | ||
| 36 | +``` | ||
| 37 | + | ||
| 38 | +## CMake入口 | ||
| 39 | + | ||
| 40 | +UT的CMake入口为： | ||
| 41 | + | ||
| 42 | +```text | ||
| 43 | +tests/ut/CMakeLists.txt | ||
| 44 | +``` | ||
| 45 | + | ||
| 46 | +常用CMake变量： | ||
| 47 | + | ||
| 48 | +| 变量 | 说明 | | ||
| 49 | +| --- | --- | | ||
| 50 | +| `ASCEND_CANN_PACKAGE_PATH` | CANN包路径。未显式指定时，优先从环境变量推导。 | | ||
| 51 | +| `PRODUCT_TYPE_LIST` | 要构建的产品类型列表，默认包含`ascend950pr_9599_AIV`和`ascend910B1_AIC`。 | | ||
| 52 | +| `TEST_MOD` | 要运行的UT目标过滤项，默认`all`。 | | ||
| 53 | +| `ASCCOMM_UT_RUN_AFTER_BUILD` | 是否在构建后运行UT，默认`ON`。如只需构建不运行，可设置为`OFF`。 | | ||
| 54 | + | ||
| 55 | +## GTest依赖 | ||
| 56 | + | ||
| 57 | +UT会优先查找系统GTest。若系统中没有GTest，可以通过`CANN_3RD_LIB_PATH`指向CANN third_party目录。 | ||
| 58 | + | ||
| 59 | +```bash | ||
| 60 | +cmake -S tests/ut -B build/ut-hcomm -DCANN_3RD_LIB_PATH=<path-to-third-party> | ||
| 61 | +``` | ||
| 62 | + | ||
| 63 | +`build.sh -t`当前未暴露`CANN_3RD_LIB_PATH`参数；需要指定离线GTest路径时，建议直接使用上述CMake命令构建UT。 | ||
| 64 | + | ||
| 65 | +## 样例构建与运行 | ||
| 66 | + | ||
| 67 | +`examples/hcomm_write_read_nbi`提供Hcomm `WriteNbi`和`ReadNbi`点对点通信样例。该样例使用独立CMake工程构建： | ||
| 68 | + | ||
| 69 | +```bash | ||
| 70 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 71 | +cd examples/hcomm_write_read_nbi | ||
| 72 | +mkdir -p build | ||
| 73 | +cd build | ||
| 74 | +cmake -DCMAKE_ASC_ARCHITECTURES=dav-3510 .. | ||
| 75 | +make -j | ||
| 76 | +``` | ||
| 77 | + | ||
| 78 | +样例默认可直接启动两个rank： | ||
| 79 | + | ||
| 80 | +```bash | ||
| 81 | +./demo | ||
| 82 | +``` | ||
| 83 | + | ||
| 84 | +也可以手动指定rank运行： | ||
| 85 | + | ||
| 86 | +```bash | ||
| 87 | +# 终端1：rank 0 | ||
| 88 | +./demo 0 2 tcp://127.0.0.1:29621 | ||
| 89 | + | ||
| 90 | +# 终端2：rank 1 | ||
| 91 | +./demo 1 2 tcp://127.0.0.1:29621 | ||
| 92 | +``` | ||
| 93 | + | ||
| 94 | +样例支持Ascend 950PR/Ascend 950DT，要求CANN 9.1.0或以上版本。运行样例需要至少2张NPU；单卡环境仅支持编译验证。 | ||
| @@ -0,0 +1,64 @@ | |||
| 1 | +# 三方依赖与兼容性 | ||
| 2 | + | ||
| 3 | +## 适用范围 | ||
| 4 | + | ||
| 5 | +asc-comm当前仓内构建主要用于环境检查、AICore Hcomm接口UT验证和Hcomm样例验证。本文档说明当前已纳入仓内构建、验证和样例流程的直接依赖。 | ||
| 6 | + | ||
| 7 | +## 基础环境 | ||
| 8 | + | ||
| 9 | +| 依赖 | 要求 | 说明 | | ||
| 10 | +| --- | --- | --- | | ||
| 11 | +| CANN Toolkit | 与当前分支或Tag配套 | 执行`build.sh`前必须先`source ${install_path}/cann/set_env.sh`，脚本会检查`ASCEND_HOME_PATH`。 | | ||
| 12 | +| CANN Runtime/HCCL/Hcomm | CANN 9.1.0或以上 | `hcomm_write_read_nbi`样例需要通信域创建、内存注册和P2P通道创建能力，并在链接阶段依赖`hcomm`库。 | | ||
| 13 | +| CMake | >= 3.16 | UT CMake入口为`tests/ut/CMakeLists.txt`。 | | ||
| 14 | +| C++ 编译器 | 支持C++17 | UT目标使用`CMAKE_CXX_STANDARD 17`，建议`gcc/g++ >= 7.3.0`且版本一致。 | | ||
| 15 | +| Python | Python 3 | UT可用于生成tiling头文件；OAT钩子要求Python 3.7+。源码和examples环境建议Python >= 3.9.0。 | | ||
| 16 | + | ||
| 17 | +## 本仓直接三方依赖 | ||
| 18 | + | ||
| 19 | +| 场景 | 依赖 | 版本 | 获取或配置方式 | | ||
| 20 | +| --- | --- | --- | --- | | ||
| 21 | +| UT | googletest | 1.14.0 | 优先使用系统GTest；没有系统GTest时，通过`CANN_3RD_LIB_PATH`指向CANN third_party目录。 | | ||
| 22 | +| 样例 | CANN ASC CMake能力和hcomm库 | CANN 9.1.0或以上 | 执行`source ${install_path}/cann/set_env.sh`后，在`examples/hcomm_write_read_nbi`目录下使用CMake构建。 | | ||
| 23 | +| 代码格式化 | clang-format | v18.1.8 | `pre-commit-config.yaml`从`pre-commit-clang/mirrors-clang-format`拉取。 | | ||
| 24 | +| 开源合规检查 | oat-py | >= 1.0.1 | `scripts/oat_check.sh`会尝试自动安装；失败时手动执行`pip install oat-py>=1.0.1`。 | | ||
| 25 | + | ||
| 26 | +仓根目录的`Third_Party_Open_Source_Software_List.yaml`当前只登记本仓测试直接使用的`googletest`。如果后续新增直接链接或打包的三方库，需要同步更新该清单和Notice。 | ||
| 27 | + | ||
| 28 | +## 样例运行依赖 | ||
| 29 | + | ||
| 30 | +`examples/hcomm_write_read_nbi`样例支持Ascend 950PR/Ascend 950DT，运行时需要至少2张NPU。单卡环境可完成编译验证，但无法完成两卡点对点通信运行验证。 | ||
| 31 | + | ||
| 32 | +样例编译命令如下： | ||
| 33 | + | ||
| 34 | +```bash | ||
| 35 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 36 | +cd examples/hcomm_write_read_nbi | ||
| 37 | +mkdir -p build | ||
| 38 | +cd build | ||
| 39 | +cmake -DCMAKE_ASC_ARCHITECTURES=dav-3510 .. | ||
| 40 | +make -j | ||
| 41 | +``` | ||
| 42 | + | ||
| 43 | +## GTest安装与配置 | ||
| 44 | + | ||
| 45 | +若系统没有安装GTest，可准备如下目录结构： | ||
| 46 | + | ||
| 47 | +```text | ||
| 48 | +<third_party>/gtest/include/gtest/gtest.h | ||
| 49 | +<third_party>/gtest/lib64/libgtest.a | ||
| 50 | +``` | ||
| 51 | + | ||
| 52 | +然后执行： | ||
| 53 | + | ||
| 54 | +```bash | ||
| 55 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 56 | +cmake -S tests/ut -B build/ut-hcomm -DCANN_3RD_LIB_PATH=<third_party> | ||
| 57 | +cmake --build build/ut-hcomm | ||
| 58 | +``` | ||
| 59 | + | ||
| 60 | +`build.sh -t`会调用UT构建，但当前脚本未暴露`CANN_3RD_LIB_PATH`参数；需要指定离线GTest路径时，建议直接使用上面的CMake命令。 | ||
| 61 | + | ||
| 62 | +## 集成依赖边界 | ||
| 63 | + | ||
| 64 | +后续新增模块、样例或端到端流程引入新的三方组件时，需要同步更新本节、本仓三方开源软件清单和Notice。 | ||
| @@ -0,0 +1,80 @@ | |||
| 1 | +# Hcomm使用说明 | ||
| 2 | + | ||
| 3 | +## 概述 | ||
| 4 | + | ||
| 5 | +Hcomm是asc-comm当前提供的AICore侧点对点通信接口。使用方通过`AscendC::Hcomm`模板选择通信协议，并通过`ChannelHandle`指定通信通道。当前主要覆盖RoCE和UBC_CTP/URMA两条路径。 | ||
| 6 | + | ||
| 7 | +## 基本流程 | ||
| 8 | + | ||
| 9 | +1. 包含头文件。 | ||
| 10 | + | ||
| 11 | +```cpp | ||
| 12 | +#include "hcomm/hcomm.h" | ||
| 13 | +``` | ||
| 14 | + | ||
| 15 | +2. 创建Hcomm对象。 | ||
| 16 | + | ||
| 17 | +```cpp | ||
| 18 | +AscendC::Hcomm<AscendC::COMM_PROTOCOL_UBC_CTP> hcomm; | ||
| 19 | +``` | ||
| 20 | + | ||
| 21 | +3. 调用`Init`初始化临时工作区。 | ||
| 22 | + | ||
| 23 | +```cpp | ||
| 24 | +int32_t ret = hcomm.Init(tmpBuf, tmpLen); | ||
| 25 | +``` | ||
| 26 | + | ||
| 27 | +4. 提交通信任务。 | ||
| 28 | + | ||
| 29 | +```cpp | ||
| 30 | +ret = hcomm.WriteNbi(channel, dst, src, len); | ||
| 31 | +ret = hcomm.ReadNbi(channel, dst, src, len); | ||
| 32 | +ret = hcomm.WriteWithNotifyNbi(channel, dst, src, len, notifyAddr, notifyVal); | ||
| 33 | +ret = hcomm.AtomicFAA<uint64_t>(channel, remoteCounter, fetchAddr, addVal); | ||
| 34 | +ret = hcomm.AtomicCAS<uint64_t>(channel, remoteValue, fetchAddr, compareVal, swapVal); | ||
| 35 | +``` | ||
| 36 | + | ||
| 37 | +5. 如果提交任务时设置`commit = false`，需要显式调用`Commit`。 | ||
| 38 | + | ||
| 39 | +```cpp | ||
| 40 | +ret = hcomm.WriteNbi<false>(channel, dst, src, len); | ||
| 41 | +ret = hcomm.Commit(channel); | ||
| 42 | +``` | ||
| 43 | + | ||
| 44 | +6. 调用`Drain`等待任务完成。 | ||
| 45 | + | ||
| 46 | +```cpp | ||
| 47 | +ret = hcomm.Drain(channel); | ||
| 48 | +``` | ||
| 49 | + | ||
| 50 | +## 协议说明 | ||
| 51 | + | ||
| 52 | +| 协议 | 说明 | | ||
| 53 | +| --- | --- | | ||
| 54 | +| `COMM_PROTOCOL_ROCE` | RoCE点对点通信路径，支持`ReadNbi`、`WriteNbi`、`Commit`、`Drain`，不支持`WriteWithNotifyNbi`。 | | ||
| 55 | +| `COMM_PROTOCOL_UBC_CTP` | UBC CTP/URMA路径，支持`ReadNbi`、`WriteNbi`、`WriteWithNotifyNbi`、`AtomicFAA`、`AtomicCAS`、`Commit`、`Drain`。 | | ||
| 56 | + | ||
| 57 | +## 注意事项 | ||
| 58 | + | ||
| 59 | +- `COMM_PROTOCOL_ROCE`和`COMM_PROTOCOL_UBC_CTP`路径均需要先调用`Init`提供临时工作区，当前最小工作区大小为512字节。 | ||
| 60 | +- 使用`__ubuf__ uint8_t*`初始化时，实现会对临时工作区起始地址做32字节对齐；使用`LocalTensor`初始化时，调用方需要保证tensor容量满足工作区要求。 | ||
| 61 | +- `ChannelHandle`指向的通道实体由调用方负责初始化和维护。 | ||
| 62 | +- 源地址、目的地址和长度需要满足底层协议和硬件要求。 | ||
| 63 | +- `WriteWithNotifyNbi`、`AtomicFAA`和`AtomicCAS`仅支持`COMM_PROTOCOL_UBC_CTP`路径。 | ||
| 64 | +- 原子操作的数据类型仅支持`int32_t`、`uint32_t`、`int64_t`和`uint64_t`。 | ||
| 65 | +- `COMM_PROTOCOL_UBC_CTP`路径中，`WriteWithNotifyNbi`、`AtomicFAA`和`AtomicCAS`单任务占用2个WQE block，普通`ReadNbi`/`WriteNbi`占用1个WQE block。 | ||
| 66 | +- 返回值为`0`表示成功，`-1`表示失败。 | ||
| 67 | + | ||
| 68 | +## 样例 | ||
| 69 | + | ||
| 70 | +可参考[asc-comm样例](../../examples/README.md)中的`hcomm_write_read_nbi`了解Kernel侧接口调用方式和Host侧通信资源创建流程。 | ||
| 71 | + | ||
| 72 | +该样例在两卡场景下对称执行`WriteNbi`和`ReadNbi`： | ||
| 73 | + | ||
| 74 | +```cpp | ||
| 75 | +hcomm.WriteNbi(channel, remoteBuf + DATA_SIZE, localBuf, DATA_SIZE); | ||
| 76 | +hcomm.ReadNbi(channel, localBuf + 2 * DATA_SIZE, remoteBuf, DATA_SIZE); | ||
| 77 | +hcomm.Drain(channel); | ||
| 78 | +``` | ||
| 79 | + | ||
| 80 | +样例运行依赖Ascend 950PR/Ascend 950DT和至少2张NPU；单卡环境仅支持编译验证。 | ||
| @@ -0,0 +1,274 @@ | |||
| 1 | +# 快速开始 | ||
| 2 | + | ||
| 3 | +## 🛠️ 环境准备<a name="prepare&install"></a> | ||
| 4 | + | ||
| 5 | +根据**本地是否有NPU设备**和**使用目标**选择对应的环境准备方式： | ||
| 6 | + | ||
| 7 | +<table> | ||
| 8 | + <thead> | ||
| 9 | + <tr> | ||
| 10 | + <th align="center">环境准备</th> | ||
| 11 | + <th align="center">社区体验/算子开发(CANN商用/社区版)</th> | ||
| 12 | + <th align="center">生态开发者贡献(CANN master)</th> | ||
| 13 | + </tr> | ||
| 14 | + </thead> | ||
| 15 | + <tbody> | ||
| 16 | + <tr> | ||
| 17 | + <td align="center"><strong>无NPU设备</strong></td> | ||
| 18 | + <td align="center" colspan="2"><a href="#cloud-dev-env">云开发环境</a>+<a href="#cann-install">手动下载安装CANN包</a></td> | ||
| 19 | + </tr> | ||
| 20 | + <tr> | ||
| 21 | + <td align="center"><strong>有NPU设备</strong></td> | ||
| 22 | + <td align="center"><a href="#cann-docker-image">CANN官方Docker镜像</a></td> | ||
| 23 | + <td align="center"><a href="#cann-install">手动下载安装CANN包</a></td> | ||
| 24 | + </tr> | ||
| 25 | + </tbody> | ||
| 26 | +</table> | ||
| 27 | + | ||
| 28 | +> [!TIP]选择建议 | ||
| 29 | +> | ||
| 30 | +> - 为了保障开发体验环境的质量，推荐用户基于**容器化技术**完成**环境准备**。 | ||
| 31 | +> - 如不希望使用容器，也可在带NPU设备的主机上完成**环境准备**，请参考[CANN软件安装指南 - 在物理机上安装](https://www.hiascend.com/cann/download)。 | ||
| 32 | +> - 针对仅体验"编译本开源仓 + 编译验证样例"的用户，不要求主机带NPU设备，可跳过安装NPU驱动和固件，直接安装CANN包，请参考[下载安装CANN包](#cann-install)。`hcomm_write_read_nbi`样例运行需要至少2张NPU。 | ||
| 33 | + | ||
| 34 | +### 1️⃣ 云开发环境<a name="cloud-dev-env"></a> | ||
| 35 | + | ||
| 36 | +对于无NPU设备的用户，可直接使用**CANNLab云开发环境**，即"**一站式开发平台**"，该平台提供了在线可直接运行的昇腾ARM架构环境，环境中已安装必备的驱动固件、软件包和依赖，无需手动安装。该平台目前仅适用于Atlas A2系列产品，提供两种接入方式： | ||
| 37 | + | ||
| 38 | +- **WebIDE**，提供网页版的便携开发体验。 | ||
| 39 | +- **VSCode IDE**，支持远程连接**云开发环境**，提供VSCode强大插件市场的支持。 | ||
| 40 | + | ||
| 41 | +1. 进入开源仓Gitcode页面，单击"`CANNLab > 云开发`"按钮，使用已认证过的华为云账号登录。若未注册或认证，请根据页面提示进行注册和认证。 | ||
| 42 | + | ||
| 43 | + <p align="center"><img src="./figures/cloudIDE.png" alt="云平台" width="750px" height="90px"></p> | ||
| 44 | + | ||
| 45 | +2. 根据页面提示创建NPU环境并配置规格，启动云开发环境后，单击"`连接 > WebIDE 或 Visual Studio Code`"进入一站式开发平台。开源项目的资源默认在`/mnt/workspace`目录下。 | ||
| 46 | + | ||
| 47 | + <p align="center"><img src="./figures/webIDE.png" alt="云平台" width="1000px" height="150px"></p> | ||
| 48 | + | ||
| 49 | +> [!NOTE] 使用说明 | ||
| 50 | +> | ||
| 51 | +> - 环境默认安装了最新的商用版NPU驱动和固件、CANN包，源码下载时注意与软件配套。 | ||
| 52 | +> - 如需下载特定版本的CANN包，请参考[下载安装CANN包](#cann-install)。 | ||
| 53 | +> - 更多关于**CANNLab云开发环境**的介绍，请参考[CANNLab指导](https://gitcode.com/org/cann/discussions/54)。 | ||
| 54 | +> - [Huawei Developer Space插件](https://marketplace.visualstudio.com/items?itemName=HuaweiCloud.developerspace)为VSCode IDE接入**云开发环境**提供技术支持。 | ||
| 55 | + | ||
| 56 | +### 2️⃣ CANN官方Docker镜像<a name="cann-docker-image"></a> | ||
| 57 | + | ||
| 58 | +对于有NPU设备的用户，可使用CANN官方Docker镜像进行开发体验。 | ||
| 59 | + | ||
| 60 | +1. 确认主机环境 | ||
| 61 | + | ||
| 62 | + - 是否已安装NPU驱动和固件，使用`npu-smi info`能够输出NPU相关信息，如没有安装，请参考《[CANN软件安装指南](https://www.hiascend.com/document/redirect/CannCommunityInstWizard)》中“准备软件包”和“安装NPU驱动和固件”章节。驱动与固件是运行态依赖，若仅编译本项目源码，可以不安装。 | ||
| 63 | + - 是否已安装Docker，使用`docker --version`能够输出Docker版本信息，如没有安装，请参考[Docker官方安装指南](https://docs.docker.com/engine/install/)。 | ||
| 64 | + | ||
| 65 | +2. 下载CANN镜像 | ||
| 66 | + | ||
| 67 | + 从[昇腾镜像仓库](https://www.hiascend.com/developer/ascendhub/detail/17da20d1c2b6493cb38765adeba85884)拉取已预集成CANN镜像： | ||
| 68 | + | ||
| 69 | + ```bash | ||
| 70 | + # 示例：ascend/cann:tag为9.0.0-beta.2的CANN社区包 | ||
| 71 | + # docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-beta.2-910b-ubuntu22.04-py3.11 | ||
| 72 | + docker pull <ascend/cann:tag> | ||
| 73 | + ``` | ||
| 74 | + | ||
| 75 | + > [!NOTE] 使用说明 | ||
| 76 | + > - 镜像默认安装了对应版本的CANN包，源码下载时注意与软件配套。 | ||
| 77 | + > - 镜像文件比较大，正常网速下，下载时间约为5～10分钟，请您耐心等待。 | ||
| 78 | + | ||
| 79 | +3. 运行Docker | ||
| 80 | + | ||
| 81 | + 拉取镜像后，需要以特定参数启动，以便容器内能访问宿主机的NPU设备。 | ||
| 82 | + | ||
| 83 | + ```bash | ||
| 84 | + docker run --name <cann_container> \ | ||
| 85 | + --ipc=host --net=host --privileged \ | ||
| 86 | + --device /dev/davinci0 \ | ||
| 87 | + --device /dev/davinci_manager \ | ||
| 88 | + --device /dev/devmm_svm \ | ||
| 89 | + --device /dev/hisi_hdc \ | ||
| 90 | + -v /usr/local/dcmi:/usr/local/dcmi \ | ||
| 91 | + -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ | ||
| 92 | + -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ | ||
| 93 | + -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ | ||
| 94 | + -v /etc/ascend_install.info:/etc/ascend_install.info \ | ||
| 95 | + -v </home/your_host_dir>:</home/your_container_dir> \ | ||
| 96 | + -it <ascend/cann:tag> bash | ||
| 97 | + ``` | ||
| 98 | + | ||
| 99 | + | 参数 | 说明 | 注意事项 | | ||
| 100 | + | :--- | :--- | :--- | | ||
| 101 | + | `--name <cann_container>` | 为容器指定名称，便于管理 | 自定义 | | ||
| 102 | + | `--ipc=host` | 与宿主机共享IPC命名空间，NPU进程间通信（共享内存、信号量）所需 | - | | ||
| 103 | + | `--net=host` | 使用宿主机网络栈，避免容器网络转发带来的通信延迟 | - | | ||
| 104 | + | `--privileged` | 赋予容器完整设备访问权限，NPU驱动正常工作所需 | - | | ||
| 105 | + | `--device /dev/davinci0` | 将宿主机的NPU设备卡映射到容器内，可指定映射多张NPU设备卡 | 必须根据实际情况调整：`davinci0`对应系统中的第0张NPU卡。请先在宿主机执行`npu-smi info`命令，根据输出显示的设备号（如`NPU 0`, `NPU 1`）来修改此编号 | | ||
| 106 | + | `--device /dev/davinci_manager` | 映射NPU设备管理接口 | - | | ||
| 107 | + | `--device /dev/devmm_svm` | 映射设备内存管理接口 | - | | ||
| 108 | + | `--device /dev/hisi_hdc` | 映射主机与设备间的通信接口 | - | | ||
| 109 | + | `-v /usr/local/dcmi:/usr/local/dcmi` | 挂载设备容器管理接口（DCMI）相关工具和库 | - | | ||
| 110 | + | `-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi` | 挂载`npu-smi`工具 | 使容器内可以直接运行此命令来查询NPU状态和性能信息 | | ||
| 111 | + | `-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/` | 将宿主机的NPU驱动库映射到容器内 | - | | ||
| 112 | + | `-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info` | 挂载驱动版本信息文件 | - | | ||
| 113 | + | `-v /etc/ascend_install.info:/etc/ascend_install.info` | 挂载CANN软件安装信息文件 | - | | ||
| 114 | + | `-v </home/your_host_dir>:</home/your_container_dir>` | 挂载宿主机的一个路径到容器中 | 自定义 | | ||
| 115 | + | `-it` | `-i`（交互式）和`-t`（分配伪终端）的组合参数 | - | | ||
| 116 | + | `<ascend/cann:tag>` | 指定要运行的Docker镜像 | 请确保此镜像名和标签（tag）与您通过`docker pull`拉取的镜像完全一致 | | ||
| 117 | + | `bash` | 容器启动后立即执行的命令 | - | | ||
| 118 | + | ||
| 119 | +### 📥 下载安装CANN包<a name="cann-install"></a> | ||
| 120 | + | ||
| 121 | +CANN包分为CANN toolkit包和CANN ops包。 | ||
| 122 | + | ||
| 123 | +#### 下载CANN包 | ||
| 124 | + | ||
| 125 | +1. <a name="下载-cann-商用社区版"></a>下载CANN商用/社区版 | ||
| 126 | + | ||
| 127 | + 如果您想体验**官网正式发布的CANN包**，请访问[CANN安装部署-昇腾社区](https://www.hiascend.com/cann/download)获取对应版本CANN包。 | ||
| 128 | + | ||
| 129 | +2. <a name="下载-cann-master"></a>下载CANN master | ||
| 130 | + | ||
| 131 | + 如果您想体验**CANN master**，请访问[CANN master obs镜像网站](https://ascend.devcloud.huaweicloud.com/artifactory/cann-run-mirror/software/master)，下载**日期最新**的CANN包。 | ||
| 132 | + | ||
| 133 | +#### 安装CANN包 | ||
| 134 | + | ||
| 135 | +1. 安装CANN toolkit包 (必选) | ||
| 136 | + | ||
| 137 | + ```bash | ||
| 138 | + chmod +x Ascend-cann-toolkit_${cann_version}_linux-$(uname -m).run | ||
| 139 | + ./Ascend-cann-toolkit_${cann_version}_linux-$(uname -m).run --install --install-path=${install_path} | ||
| 140 | + ``` | ||
| 141 | + | ||
| 142 | +2. 安装CANN ops包 (可选) | ||
| 143 | + | ||
| 144 | + ```bash | ||
| 145 | + chmod +x Ascend-cann-${soc_name}-ops_${cann_version}_linux-$(uname -m).run | ||
| 146 | + ./Ascend-cann-${soc_name}-ops_${cann_version}_linux-$(uname -m).run --install --install-path=${install_path} | ||
| 147 | + ``` | ||
| 148 | + | ||
| 149 | + > [!IMPORTANT] 安装说明 | ||
| 150 | + > [examples](../examples)中部分算子样例的编译运行依赖本包，若想完整体验样例编译运行流程，建议安装此包。 | ||
| 151 | + | ||
| 152 | +| 参数 | 说明 | | ||
| 153 | +| :--- | :--- | | ||
| 154 | +| `${cann_version}` | CANN包版本号 | | ||
| 155 | +| `${soc_name}` | NPU型号，如`910b` | | ||
| 156 | +| `${install_path}` | 安装路径，toolkit包和ops包需相同。默认：root用户`/usr/local/Ascend`，非root用户`$HOME/Ascend` | | ||
| 157 | + | ||
| 158 | +## ✅ 环境验证<a name="cann-verify"></a> | ||
| 159 | + | ||
| 160 | +> [!NOTE] 使用前须知 | ||
| 161 | +> 云开发环境和CANN官方Docker镜像已预装CANN包，可直接执行以下命令验证。 | ||
| 162 | + | ||
| 163 | +验证环境和驱动是否正常： | ||
| 164 | + | ||
| 165 | +- **检查NPU设备**： | ||
| 166 | + | ||
| 167 | + ```bash | ||
| 168 | + # 运行npu-smi，若能正常显示设备信息，则驱动正常 | ||
| 169 | + npu-smi info | ||
| 170 | + ``` | ||
| 171 | + | ||
| 172 | +- **检查CANN包安装**： | ||
| 173 | + | ||
| 174 | + ```bash | ||
| 175 | + # 查看CANN Toolkit的version字段提供的版本信息（默认路径安装） | ||
| 176 | + cat /usr/local/Ascend/cann/$(uname -m)-linux/ascend_toolkit_install.info | ||
| 177 | + ``` | ||
| 178 | + | ||
| 179 | +## ⚙️ 环境变量配置<a name="cann-env-setup"></a> | ||
| 180 | + | ||
| 181 | +> [!NOTE] 使用前须知 | ||
| 182 | +> 云开发环境和CANN官方Docker镜像已自动配置环境变量，可跳过此步骤。 | ||
| 183 | + | ||
| 184 | +按需选择合适的命令使环境变量生效： | ||
| 185 | + | ||
| 186 | +```bash | ||
| 187 | +# 默认路径安装，以root用户为例（非root用户，将/usr/local替换为${HOME}） | ||
| 188 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 189 | +# 指定路径安装 | ||
| 190 | +# source ${install_path}/cann/set_env.sh | ||
| 191 | +``` | ||
| 192 | + | ||
| 193 | +## 🔨 源码编译步骤<a name="source-build"></a> | ||
| 194 | + | ||
| 195 | +### 📥 下载源码<a name="source-download"></a> | ||
| 196 | + | ||
| 197 | +开发者可通过如下命令下载本仓源码： | ||
| 198 | + | ||
| 199 | +```bash | ||
| 200 | +git clone https://gitcode.com/cann/asc-comm.git | ||
| 201 | +cd asc-comm | ||
| 202 | +``` | ||
| 203 | + | ||
| 204 | +### 📦 依赖检查<a name="dependency-check"></a> | ||
| 205 | + | ||
| 206 | +> [!NOTE] 使用前须知 | ||
| 207 | +> 如您使用**容器化技术**，容器中已为您安装好依赖，可跳过此步骤。 | ||
| 208 | + | ||
| 209 | +以下为本开源仓源码编译和UT验证的基础依赖条件： | ||
| 210 | + | ||
| 211 | +- python >= 3.7.0 | ||
| 212 | +- gcc/g++支持C++17 | ||
| 213 | +- cmake >= 3.16.0 | ||
| 214 | + | ||
| 215 | +### ⚡ 编译安装<a name="compile&install"></a> | ||
| 216 | + | ||
| 217 | +进入本开源仓代码根目录，执行如下命令： | ||
| 218 | + | ||
| 219 | +```bash | ||
| 220 | +bash build.sh | ||
| 221 | +``` | ||
| 222 | + | ||
| 223 | +### 🧪 UT验证<a name="ut-verify"></a> | ||
| 224 | + | ||
| 225 | +#### 安装依赖 | ||
| 226 | + | ||
| 227 | +UT依赖googletest。若系统中没有GTest，可以通过`CANN_3RD_LIB_PATH`指向CANN third_party目录。 | ||
| 228 | + | ||
| 229 | +#### UT执行 | ||
| 230 | + | ||
| 231 | +方式一：在开源仓根目录执行下列命令，构建Hcomm UT。 | ||
| 232 | + | ||
| 233 | +```bash | ||
| 234 | +bash build.sh -t | ||
| 235 | +``` | ||
| 236 | + | ||
| 237 | +方式二：用户也可直接使用CMake命令指定离线GTest路径。 | ||
| 238 | + | ||
| 239 | +```bash | ||
| 240 | +cmake -S tests/ut -B build/ut-hcomm -DCANN_3RD_LIB_PATH=<path-to-third-party> | ||
| 241 | +cmake --build build/ut-hcomm | ||
| 242 | +``` | ||
| 243 | + | ||
| 244 | +#### 开源第三方软件依赖 | ||
| 245 | + | ||
| 246 | +在执行UT时，依赖的第三方开源软件列表如下： | ||
| 247 | + | ||
| 248 | +| 开源软件 | 版本 | | ||
| 249 | +| :---: | :---: | | ||
| 250 | +| googletest | 1.14.0 | | ||
| 251 | + | ||
| 252 | +### 🧩 样例验证<a name="sample-verify"></a> | ||
| 253 | + | ||
| 254 | +`examples/hcomm_write_read_nbi`提供Hcomm `WriteNbi`和`ReadNbi`点对点通信样例。样例支持Ascend 950PR/Ascend 950DT，要求CANN 9.1.0或以上版本。运行样例需要至少2张NPU；单卡环境仅支持编译验证。 | ||
| 255 | + | ||
| 256 | +进入样例目录后执行： | ||
| 257 | + | ||
| 258 | +```bash | ||
| 259 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 260 | +cd examples/hcomm_write_read_nbi | ||
| 261 | +mkdir -p build | ||
| 262 | +cd build | ||
| 263 | +cmake -DCMAKE_ASC_ARCHITECTURES=dav-3510 .. | ||
| 264 | +make -j | ||
| 265 | +./demo | ||
| 266 | +``` | ||
| 267 | + | ||
| 268 | +执行成功时输出如下： | ||
| 269 | + | ||
| 270 | +```text | ||
| 271 | +rank 0 test pass! | ||
| 272 | +rank 1 test pass! | ||
| 273 | +test pass! | ||
| 274 | +``` | ||
The file is empty
| @@ -0,0 +1,50 @@ | |||
| 1 | +# asc-comm样例 | ||
| 2 | + | ||
| 3 | +本目录提供asc-comm API的使用样例。 | ||
| 4 | + | ||
| 5 | +## 样例列表 | ||
| 6 | + | ||
| 7 | +| 样例 | 说明 | 支持产品 | | ||
| 8 | +| --- | --- | --- | | ||
| 9 | +| [hcomm_write_read_nbi](./hcomm_write_read_nbi/README.md) | 演示两卡场景下使用`Hcomm::WriteNbi`和`Hcomm::ReadNbi`完成点对点通信，并校验通信结果。 | Ascend 950PR/Ascend 950DT | | ||
| 10 | + | ||
| 11 | +## hcomm_write_read_nbi | ||
| 12 | + | ||
| 13 | +`hcomm_write_read_nbi`展示完整的Hcomm点对点通信流程，包括Host侧通信域创建、通信内存注册、P2P通道创建、Kernel侧`Init`、`WriteNbi`、`ReadNbi`和`Drain`调用。 | ||
| 14 | + | ||
| 15 | +样例采用两卡对称执行方式： | ||
| 16 | + | ||
| 17 | +1. Host侧为每个rank创建通信域并注册通信buffer。 | ||
| 18 | +2. 通过`HcclChannelAcquire`创建到对端rank的P2P通道。 | ||
| 19 | +3. Kernel侧将本卡数据通过`WriteNbi`写入对端buffer。 | ||
| 20 | +4. Kernel侧通过`ReadNbi`从对端buffer读回数据。 | ||
| 21 | +5. Host侧回读校验结果，两个rank均通过时输出`test pass!`。 | ||
| 22 | + | ||
| 23 | +## 编译运行 | ||
| 24 | + | ||
| 25 | +进入样例目录后执行： | ||
| 26 | + | ||
| 27 | +```bash | ||
| 28 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 29 | +mkdir -p build | ||
| 30 | +cd build | ||
| 31 | +cmake -DCMAKE_ASC_ARCHITECTURES=dav-3510 .. | ||
| 32 | +make -j | ||
| 33 | +./demo | ||
| 34 | +``` | ||
| 35 | + | ||
| 36 | +也可以手动启动两个rank： | ||
| 37 | + | ||
| 38 | +```bash | ||
| 39 | +# 终端1：rank 0 | ||
| 40 | +./demo 0 2 tcp://127.0.0.1:29621 | ||
| 41 | + | ||
| 42 | +# 终端2：rank 1 | ||
| 43 | +./demo 1 2 tcp://127.0.0.1:29621 | ||
| 44 | +``` | ||
| 45 | + | ||
| 46 | +## 运行约束 | ||
| 47 | + | ||
| 48 | +- 样例支持Ascend 950PR/Ascend 950DT，CANN软件版本要求为9.1.0或以上。 | ||
| 49 | +- 样例运行需要至少2张NPU；单卡环境仅支持编译验证。 | ||
| 50 | +- 样例编译依赖CANN ASC CMake能力，并在链接阶段依赖`hcomm`库。 | ||
The file is empty
| @@ -1,89 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_base.h | ||
| 13 | - * \brief Hcomm base class | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/common/hcomm_base.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | -namespace AscendC { | ||
| 29 | -template <CommProtocol commProtocol> | ||
| 30 | -class HcommImpl { | ||
| 31 | -public: | ||
| 32 | - __aicore__ inline HcommImpl(){}; | ||
| 33 | - __aicore__ inline ~HcommImpl(){}; | ||
| 34 | - __aicore__ inline int32_t Init(__ubuf__ uint8_t* buff, uint32_t len) | ||
| 35 | - { | ||
| 36 | - (void)buff; | ||
| 37 | - (void)len; | ||
| 38 | - return HCOMM_FAILED; | ||
| 39 | - } | ||
| 40 | - | ||
| 41 | - template <typename T> | ||
| 42 | - __aicore__ inline int32_t Init(const LocalTensor<T>& buff, uint32_t len) | ||
| 43 | - { | ||
| 44 | - (void)buff; | ||
| 45 | - (void)len; | ||
| 46 | - return HCOMM_FAILED; | ||
| 47 | - } | ||
| 48 | - | ||
| 49 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 50 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 51 | - __aicore__ inline int32_t WriteNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 52 | - { | ||
| 53 | - return HCOMM_FAILED; | ||
| 54 | - } | ||
| 55 | - | ||
| 56 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 57 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 58 | - __aicore__ inline int32_t ReadNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 59 | - { | ||
| 60 | - return HCOMM_FAILED; | ||
| 61 | - } | ||
| 62 | - | ||
| 63 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 64 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 65 | - __aicore__ inline int32_t WriteWithNotifyNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, | ||
| 66 | - uint64_t len, GM_ADDR notifyAddr, uint64_t notifyVal) | ||
| 67 | - { | ||
| 68 | - return HCOMM_FAILED; | ||
| 69 | - } | ||
| 70 | - | ||
| 71 | - template <pipe_t pipe = PIPE_S> | ||
| 72 | - __aicore__ inline int32_t Commit(ChannelHandle channel) | ||
| 73 | - { | ||
| 74 | - return HCOMM_FAILED; | ||
| 75 | - } | ||
| 76 | - | ||
| 77 | - template <pipe_t pipe = PIPE_MTE3> | ||
| 78 | - __aicore__ inline int32_t Drain(ChannelHandle channel) | ||
| 79 | - { | ||
| 80 | - return HCOMM_FAILED; | ||
| 81 | - } | ||
| 82 | -}; | ||
| 83 | -} // namespace AscendC | ||
| 84 | - | ||
| 85 | - | ||
| 86 | - | ||
| 87 | - | ||
| 88 | - | ||
| 89 | - | ||
| @@ -1,427 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_inner_def.h | ||
| 13 | - * \brief Hcomm inner definition | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/common/hcomm_inner_def.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | -namespace AscendC { | ||
| 29 | - | ||
| 30 | -constexpr int32_t HCOMM_FAILED = -1; | ||
| 31 | -constexpr int32_t HCOMM_SUCCESS = 0; | ||
| 32 | -constexpr uint32_t HCOMM_UB_BUF_SIZE = 512; | ||
| 33 | -constexpr uint32_t HCOMM_POLLCQ_MAX_RETRY_TIMES = 1000000; | ||
| 34 | - | ||
| 35 | - | ||
| 36 | -struct UrmaWqeEntry { | ||
| 37 | - uint32_t odr : 3; // ordering, request ordering strength (0=unordered, 5=strongly ordered) | ||
| 38 | - uint32_t fence : 1; // fence, 1=current WQE waits for all prior WQEs to complete | ||
| 39 | - uint32_t se : 1; // solicited event, 1=trigger remote event notification on completion | ||
| 40 | - uint32_t cqe : 1; // completion queue entry, 1=generate CQE on completion | ||
| 41 | - uint32_t inlineEn : 1; // inline enable, 1=data inlined in WQE instead of SGE reference | ||
| 42 | -} __attribute__((packed)); | ||
| 43 | - | ||
| 44 | - | ||
| 45 | -static constexpr UrmaWqeEntry URMA_DEFAULT_CFG = { | ||
| 46 | - .odr = 5, | ||
| 47 | - .fence = 1, | ||
| 48 | - .se = 0, | ||
| 49 | - .cqe = 1, | ||
| 50 | - .inlineEn = 0, | ||
| 51 | -}; | ||
| 52 | - | ||
| 53 | -typedef enum { | ||
| 54 | - COMM_ENGINE_RESERVED = -1, | ||
| 55 | - COMM_ENGINE_CPU = 0, | ||
| 56 | - COMM_ENGINE_CPU_TS = 1, | ||
| 57 | - COMM_ENGINE_AICPU = 2, | ||
| 58 | - COMM_ENGINE_AICPU_TS = 3, | ||
| 59 | - COMM_ENGINE_AIV = 4, | ||
| 60 | - COMM_ENGINE_CCU = 5, | ||
| 61 | -} CommEngineType; | ||
| 62 | - | ||
| 63 | -typedef enum { | ||
| 64 | - PROTECTION_TYPE_INVALID = -1, | ||
| 65 | - PROTECTION_TYPE_ROCE = 0, | ||
| 66 | - PROTECTION_TYPE_UB = 1, | ||
| 67 | -} ProtectionType; | ||
| 68 | - | ||
| 69 | -typedef enum { | ||
| 70 | - SQ_CONTEXT_TYPE_INVALID = -1, | ||
| 71 | - SQ_CONTEXT_TYPE_UB_JFS = 0, | ||
| 72 | - SQ_CONTEXT_TYPE_ROCE = 1, | ||
| 73 | -} SqContextType; | ||
| 74 | - | ||
| 75 | -typedef enum { | ||
| 76 | - CQ_CONTEXT_TYPE_INVALID = -1, | ||
| 77 | - CQ_CONTEXT_TYPE_UB_JFC = 0, | ||
| 78 | - CQ_CONTEXT_TYPE_ROCE = 1, | ||
| 79 | -} CqContextType; | ||
| 80 | - | ||
| 81 | -typedef enum { | ||
| 82 | - REGED_NOTIFY_INVALID = -1, | ||
| 83 | - REGED_NOTIFY_IPC_RT = 0, | ||
| 84 | - REGED_NOTIFY_IPC_MEM = 1, | ||
| 85 | - REGED_NOTIFY_RMA_RT = 2, | ||
| 86 | - REGED_NOTIFY_RMA_MEM = 3, | ||
| 87 | -} RegedNotifyType; | ||
| 88 | - | ||
| 89 | -typedef enum { | ||
| 90 | - REGED_BUFFER_INVALID = -1, | ||
| 91 | - REGED_BUFFER_IPC = 0, | ||
| 92 | - REGED_BUFFER_RMA = 1, | ||
| 93 | -} RegedBufferType; | ||
| 94 | - | ||
| 95 | -typedef struct { | ||
| 96 | - uint32_t version; | ||
| 97 | - uint32_t magicWord; | ||
| 98 | - uint32_t size; | ||
| 99 | - uint32_t reserved; | ||
| 100 | -} CommAbiHeader; | ||
| 101 | - | ||
| 102 | -typedef struct { | ||
| 103 | - ProtectionType type; | ||
| 104 | - union { | ||
| 105 | - struct { | ||
| 106 | - uint32_t lkey; | ||
| 107 | - uint32_t rkey; | ||
| 108 | - } roce; | ||
| 109 | - struct { | ||
| 110 | - uint32_t tokenId; | ||
| 111 | - uint32_t tokenValue; | ||
| 112 | - } ub; | ||
| 113 | - uint8_t raws[24]; | ||
| 114 | - } memInfo; | ||
| 115 | -} ProtectionInfo; | ||
| 116 | - | ||
| 117 | -typedef struct { | ||
| 118 | - SqContextType type; | ||
| 119 | - union { | ||
| 120 | - struct { | ||
| 121 | - uint64_t sqVa; | ||
| 122 | - uint64_t headAddr; | ||
| 123 | - uint64_t tailAddr; | ||
| 124 | - uint64_t dbVa; | ||
| 125 | - uint32_t jfsID; | ||
| 126 | - uint32_t wqeSize; | ||
| 127 | - uint32_t sqDepth; | ||
| 128 | - uint32_t tpID; | ||
| 129 | - uint8_t remoteEID[16]; | ||
| 130 | - } ubJfs; | ||
| 131 | - struct { | ||
| 132 | - uint64_t sqVa; | ||
| 133 | - uint64_t headAddr; | ||
| 134 | - uint64_t tailAddr; | ||
| 135 | - uint64_t dbVa; | ||
| 136 | - uint32_t qpn; | ||
| 137 | - uint32_t wqeSize; | ||
| 138 | - uint32_t depth; | ||
| 139 | - int8_t dbMode; | ||
| 140 | - uint8_t sl; | ||
| 141 | - } roceSq; | ||
| 142 | - uint8_t raws[120]; | ||
| 143 | - } contextInfo; | ||
| 144 | -} SqContext; | ||
| 145 | - | ||
| 146 | -typedef struct { | ||
| 147 | - CqContextType type; | ||
| 148 | - union { | ||
| 149 | - struct { | ||
| 150 | - uint64_t scqVa; | ||
| 151 | - uint64_t headAddr; | ||
| 152 | - uint64_t tailAddr; | ||
| 153 | - uint64_t dbVa; | ||
| 154 | - uint32_t jfcID; | ||
| 155 | - uint32_t cqeSize; | ||
| 156 | - uint32_t cqDepth; | ||
| 157 | - } ubJfc; | ||
| 158 | - struct { | ||
| 159 | - uint64_t cqVa; | ||
| 160 | - uint64_t headAddr; | ||
| 161 | - uint64_t tailAddr; | ||
| 162 | - uint64_t dbVa; | ||
| 163 | - uint32_t cqn; | ||
| 164 | - uint32_t cqeSize; | ||
| 165 | - uint32_t cqDepth; | ||
| 166 | - int8_t dbMode; | ||
| 167 | - } roceCq; | ||
| 168 | - uint8_t raws[120]; | ||
| 169 | - } contextInfo; | ||
| 170 | -} CqContext; | ||
| 171 | - | ||
| 172 | -typedef struct { | ||
| 173 | - RegedBufferType type; | ||
| 174 | - union { | ||
| 175 | - struct { | ||
| 176 | - uint64_t addr; | ||
| 177 | - uint64_t size; | ||
| 178 | - } ipc; | ||
| 179 | - struct { | ||
| 180 | - uint64_t addr; | ||
| 181 | - uint64_t size; | ||
| 182 | - ProtectionInfo protectionInfo; | ||
| 183 | - } rma; | ||
| 184 | - uint8_t raws[56]; | ||
| 185 | - } bufferInfo; | ||
| 186 | -} RegedBufferEntity; | ||
| 187 | - | ||
| 188 | -typedef struct { | ||
| 189 | - RegedNotifyType type; | ||
| 190 | - union { | ||
| 191 | - struct { | ||
| 192 | - uint64_t addr; | ||
| 193 | - uint32_t size; | ||
| 194 | - int32_t notifyId; | ||
| 195 | - } ipcRt; | ||
| 196 | - struct { | ||
| 197 | - uint64_t addr; | ||
| 198 | - uint32_t size; | ||
| 199 | - } ipcMem; | ||
| 200 | - struct { | ||
| 201 | - uint64_t addr; | ||
| 202 | - uint32_t size; | ||
| 203 | - int32_t notifyId; | ||
| 204 | - ProtectionInfo protectionInfo; | ||
| 205 | - } rmaRt; | ||
| 206 | - struct { | ||
| 207 | - uint64_t addr; | ||
| 208 | - uint32_t size; | ||
| 209 | - ProtectionInfo protectionInfo; | ||
| 210 | - } rmaMem; | ||
| 211 | - uint8_t raws[56]; | ||
| 212 | - } notifyInfo; | ||
| 213 | -} RegedNotifyEntity; | ||
| 214 | - | ||
| 215 | -typedef struct { | ||
| 216 | - CommAbiHeader abiHeader; | ||
| 217 | - CommEngineType engine; | ||
| 218 | - int32_t protocol; | ||
| 219 | - uint32_t localNotifyNum; | ||
| 220 | - uint32_t remoteNotifyNum; | ||
| 221 | - uint32_t localBufferNum; | ||
| 222 | - uint32_t remoteBufferNum; | ||
| 223 | - uint32_t sqNum; | ||
| 224 | - uint32_t cqNum; | ||
| 225 | - RegedNotifyEntity* localNotifyAddr; | ||
| 226 | - RegedNotifyEntity* remoteNotifyAddr; | ||
| 227 | - RegedBufferEntity* localBufferAddr; | ||
| 228 | - RegedBufferEntity* remoteBufferAddr; | ||
| 229 | - SqContext* sqContextAddr; | ||
| 230 | - CqContext* cqContextAddr; | ||
| 231 | - uint8_t reserve[160]; | ||
| 232 | -} ChannelEntity; | ||
| 233 | - | ||
| 234 | -static_assert(sizeof(ChannelEntity) == 256, "ChannelEntity size must keep aligned with hcomm"); | ||
| 235 | - | ||
| 236 | -// RoCE WQE, CQE, DB struct | ||
| 237 | -typedef struct { | ||
| 238 | - // Control Segment | ||
| 239 | - union { | ||
| 240 | - struct { | ||
| 241 | - uint32_t o : 1; // Owner | ||
| 242 | - uint32_t ctrlSl : 2; | ||
| 243 | - uint32_t csl : 2; | ||
| 244 | - uint32_t difSl : 3; | ||
| 245 | - uint32_t cr : 1; | ||
| 246 | - uint32_t df : 1; | ||
| 247 | - uint32_t va : 1; | ||
| 248 | - uint32_t tsl : 5; | ||
| 249 | - uint32_t cf : 1; | ||
| 250 | - uint32_t wf : 1; | ||
| 251 | - uint32_t rsvd0 : 4; | ||
| 252 | - uint32_t rrvSl : 2; | ||
| 253 | - uint32_t bdsLen : 8; | ||
| 254 | - } bs; | ||
| 255 | - uint32_t value; | ||
| 256 | - } dw0; | ||
| 257 | - union { | ||
| 258 | - struct { | ||
| 259 | - uint32_t cl : 4; | ||
| 260 | - uint32_t rsvd1 : 8; | ||
| 261 | - uint32_t maskPi : 20; | ||
| 262 | - } bs; | ||
| 263 | - uint32_t value; | ||
| 264 | - } dw1; | ||
| 265 | -} RoceWqeCtrlSeg; | ||
| 266 | - | ||
| 267 | -typedef struct { | ||
| 268 | - // Task Segment | ||
| 269 | - union { | ||
| 270 | - struct { | ||
| 271 | - uint32_t se : 1; | ||
| 272 | - uint32_t f : 1; | ||
| 273 | - uint32_t c : 1; | ||
| 274 | - uint32_t opType : 5; | ||
| 275 | - uint32_t so : 1; | ||
| 276 | - uint32_t rsvd0 : 3; | ||
| 277 | - uint32_t dif : 1; | ||
| 278 | - uint32_t ext : 1; | ||
| 279 | - uint32_t xrcSrqn : 18; | ||
| 280 | - } bs; | ||
| 281 | - uint32_t value; | ||
| 282 | - } dw0; | ||
| 283 | -} RoceWqeTaskSeg; | ||
| 284 | - | ||
| 285 | -typedef struct { | ||
| 286 | - uint64_t vaLocal; | ||
| 287 | - uint32_t rLen; | ||
| 288 | - uint32_t leKey; | ||
| 289 | -} RoceWqeDataSeg; | ||
| 290 | - | ||
| 291 | -typedef struct { | ||
| 292 | - RoceWqeCtrlSeg ctrl; | ||
| 293 | - uint64_t doorbell; | ||
| 294 | - RoceWqeTaskSeg task; | ||
| 295 | - uint32_t dataLen; | ||
| 296 | - uint32_t immeData; | ||
| 297 | - uint32_t firstLast : 1; | ||
| 298 | - uint32_t nxtEthHdr : 7; | ||
| 299 | - uint32_t cmdLen : 8; | ||
| 300 | - uint32_t rsvd0 : 8; | ||
| 301 | - uint32_t lastExtLen : 8; | ||
| 302 | - uint64_t vaRemote; | ||
| 303 | - uint32_t rKey; | ||
| 304 | - uint32_t rsvd1; | ||
| 305 | - RoceWqeDataSeg data; | ||
| 306 | -} RoceWqeEntry; | ||
| 307 | - | ||
| 308 | -typedef struct { | ||
| 309 | - uint32_t cqe0; | ||
| 310 | - uint32_t cqe1; | ||
| 311 | - uint32_t cqe2; | ||
| 312 | - uint32_t cqe3; | ||
| 313 | - uint32_t cqe4; | ||
| 314 | - uint32_t cqe5; | ||
| 315 | - uint32_t cqe6; | ||
| 316 | - uint32_t cqe7; | ||
| 317 | -} RoceCqeEntry; | ||
| 318 | - | ||
| 319 | -typedef struct { | ||
| 320 | - union { | ||
| 321 | - struct { | ||
| 322 | - uint64_t qpn: 20; | ||
| 323 | - uint64_t ctxSize: 2; | ||
| 324 | - uint64_t r: 1; | ||
| 325 | - uint64_t c: 1; | ||
| 326 | - uint64_t cos: 3; | ||
| 327 | - uint64_t type: 5; | ||
| 328 | - | ||
| 329 | - uint64_t pi: 8; | ||
| 330 | - uint64_t resv: 8; | ||
| 331 | - uint64_t xrcVld: 1; | ||
| 332 | - uint64_t rsvd: 1; | ||
| 333 | - uint64_t mtuShift: 3; | ||
| 334 | - uint64_t sgidIdx: 7; | ||
| 335 | - uint64_t subType: 4; | ||
| 336 | - } bs; | ||
| 337 | - uint64_t value; | ||
| 338 | - } dw0; | ||
| 339 | -} RoceDbEntry; | ||
| 340 | - | ||
| 341 | -// URMA struct | ||
| 342 | -typedef struct { | ||
| 343 | - uint32_t sqeBbIdx : 16; | ||
| 344 | - uint32_t flag : 8; | ||
| 345 | - uint32_t rsv0 : 3; | ||
| 346 | - uint32_t nf : 1; | ||
| 347 | - uint32_t tokenEn : 1; | ||
| 348 | - uint32_t rmtJettyType : 2; | ||
| 349 | - uint32_t owner : 1; | ||
| 350 | - uint32_t targetHint : 8; | ||
| 351 | - uint32_t opcode : 8; | ||
| 352 | - uint32_t rsv1 : 6; | ||
| 353 | - uint32_t inlineMsgLen : 10; | ||
| 354 | - uint32_t tpId : 24; | ||
| 355 | - uint32_t sgeNum : 8; | ||
| 356 | - uint32_t rmtJettyOrSegId : 20; | ||
| 357 | - uint32_t rsv2 : 12; | ||
| 358 | - uint64_t rmtEidL; | ||
| 359 | - uint64_t rmtEidH; | ||
| 360 | - uint32_t rmtTokenValue; | ||
| 361 | - uint32_t udfType : 8; | ||
| 362 | - uint32_t reduceDataType : 4; | ||
| 363 | - uint32_t reduceOpcode : 4; | ||
| 364 | - uint32_t rsv3 : 16; | ||
| 365 | - uint32_t rmtAddrLOrTokenId; | ||
| 366 | - uint32_t rmtAddrHOrTokenValue; | ||
| 367 | -} HcommUrmaSqeCtx; | ||
| 368 | - | ||
| 369 | -typedef struct { | ||
| 370 | - uint32_t len; | ||
| 371 | - uint32_t tokenId; | ||
| 372 | - uint64_t va; | ||
| 373 | -} HcommUrmaSgeCtx; | ||
| 374 | - | ||
| 375 | -typedef struct { | ||
| 376 | - uint32_t notifyTokenId : 20; | ||
| 377 | - uint32_t rsv : 12; | ||
| 378 | - uint32_t notifyTokenValue; | ||
| 379 | - uint32_t notifyAddrL; | ||
| 380 | - uint32_t notifyAddrH; | ||
| 381 | - uint32_t notifyDataL; | ||
| 382 | - uint32_t notifyDataH; | ||
| 383 | - uint32_t rsv2[2]; | ||
| 384 | -} HcommUrmaNotifyCtx; | ||
| 385 | - | ||
| 386 | -typedef struct { | ||
| 387 | - uint32_t sR : 1; | ||
| 388 | - uint32_t isJetty : 1; | ||
| 389 | - uint32_t owner : 1; | ||
| 390 | - uint32_t inlineEn : 1; | ||
| 391 | - uint32_t opcode : 3; | ||
| 392 | - uint32_t fd : 1; | ||
| 393 | - uint32_t rsv : 8; | ||
| 394 | - uint32_t substatus : 8; | ||
| 395 | - uint32_t status : 8; | ||
| 396 | - uint32_t entryIdx : 16; | ||
| 397 | - uint32_t localNumL : 16; | ||
| 398 | - uint32_t localNumH : 4; | ||
| 399 | - uint32_t rmtIdx : 20; | ||
| 400 | - uint32_t rsv1 : 8; | ||
| 401 | - uint32_t tpn : 24; | ||
| 402 | - uint32_t rsv2 : 8; | ||
| 403 | - uint32_t byteCnt; | ||
| 404 | - uint32_t userDataL; | ||
| 405 | - uint32_t userDataH; | ||
| 406 | - uint32_t rmtEid[4]; | ||
| 407 | - uint32_t dataL; | ||
| 408 | - uint32_t dataH; | ||
| 409 | - uint32_t inlineData[3]; | ||
| 410 | -} HcommUrmaJfcCqeCtx; | ||
| 411 | - | ||
| 412 | - | ||
| 413 | - | ||
| 414 | - | ||
| 415 | - | ||
| 416 | - | ||
| 417 | - | ||
| 418 | - | ||
| 419 | - | ||
| 420 | - | ||
| 421 | - | ||
| 422 | -} // namespace AscendC | ||
| 423 | - | ||
| 424 | - | ||
| 425 | - | ||
| 426 | - | ||
| 427 | - | ||
| @@ -1,149 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2025 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_util.h | ||
| 13 | - * \brief Hcomm utils | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/common/hcomm_utils.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | -namespace AscendC { | ||
| 29 | - | ||
| 30 | -__aicore__ inline uint32_t HtoNL(uint32_t x) | ||
| 31 | -{ | ||
| 32 | - constexpr uint32_t byte0Mask = 0x000000ffU; | ||
| 33 | - constexpr uint32_t byte1Mask = 0x0000ff00U; | ||
| 34 | - constexpr uint32_t byte2Mask = 0x00ff0000U; | ||
| 35 | - constexpr uint32_t byte3Mask = 0xff000000U; | ||
| 36 | - constexpr uint32_t byteShift = 8; | ||
| 37 | - constexpr uint32_t wordShift = 24; | ||
| 38 | - | ||
| 39 | - return (((x & byte3Mask) >> wordShift) | ((x & byte2Mask) >> byteShift) | ((x & byte1Mask) << byteShift) | | ||
| 40 | - ((x & byte0Mask) << wordShift)); | ||
| 41 | -} | ||
| 42 | - | ||
| 43 | -__aicore__ inline uint64_t HtoNLL(uint64_t x) | ||
| 44 | -{ | ||
| 45 | - constexpr uint64_t byte0Mask = 0x00000000000000ffU; | ||
| 46 | - constexpr uint64_t byte1Mask = 0x000000000000ff00U; | ||
| 47 | - constexpr uint64_t byte2Mask = 0x0000000000ff0000U; | ||
| 48 | - constexpr uint64_t byte3Mask = 0x00000000ff000000U; | ||
| 49 | - constexpr uint64_t byte4Mask = 0x000000ff00000000U; | ||
| 50 | - constexpr uint64_t byte5Mask = 0x0000ff0000000000U; | ||
| 51 | - constexpr uint64_t byte6Mask = 0x00ff000000000000U; | ||
| 52 | - constexpr uint64_t byte7Mask = 0xff00000000000000U; | ||
| 53 | - | ||
| 54 | - constexpr uint64_t shift8 = 8; | ||
| 55 | - constexpr uint64_t shift24 = 24; | ||
| 56 | - constexpr uint64_t shift40 = 40; | ||
| 57 | - constexpr uint64_t shift56 = 56; | ||
| 58 | - | ||
| 59 | - return (((x & byte0Mask) << shift56) | ((x & byte1Mask) << shift40) | | ||
| 60 | - ((x & byte2Mask) << shift24) | ((x & byte3Mask) << shift8) | | ||
| 61 | - ((x & byte4Mask) >> shift8) | ((x & byte5Mask) >> shift24) | | ||
| 62 | - ((x & byte6Mask) >> shift40) | ((x & byte7Mask) >> shift56)); | ||
| 63 | -} | ||
| 64 | - | ||
| 65 | -template<HardEvent event> | ||
| 66 | -__aicore__ inline void SyncAction() | ||
| 67 | -{ | ||
| 68 | - TEventID eventID = GetTPipePtr()->FetchEventID(event); | ||
| 69 | - SetFlag<event>(eventID); | ||
| 70 | - WaitFlag<event>(eventID); | ||
| 71 | -} | ||
| 72 | -__aicore__ inline __ubuf__ uint8_t* AlignAddrTo32Bytes(__ubuf__ uint8_t* buff) | ||
| 73 | -{ | ||
| 74 | - uintptr_t addr = reinterpret_cast<uintptr_t>(buff); | ||
| 75 | - const uintptr_t alignment = 32; | ||
| 76 | - uintptr_t alignedAddr = (addr + alignment - 1) & ~(alignment - 1); | ||
| 77 | - return reinterpret_cast<__ubuf__ uint8_t*>(alignedAddr); | ||
| 78 | -} | ||
| 79 | - | ||
| 80 | -template <typename T> | ||
| 81 | -__aicore__ inline void Gm2Ub( | ||
| 82 | - const AscendC::LocalTensor<T>& dstLocal, const AscendC::GlobalTensor<T>& srcGlobal, int32_t length) | ||
| 83 | -{ | ||
| 84 | - AscendC::DataCopyExtParams copyParams{1, (uint32_t)(length * sizeof(T)), 0, 0, 0}; | ||
| 85 | - AscendC::DataCopyPadExtParams<T> padParams{true, 0, (ONE_BLK_SIZE - sizeof(T)) / sizeof(T), 0}; | ||
| 86 | - | ||
| 87 | - PipeBarrier<PIPE_ALL>(); | ||
| 88 | - AscendC::DataCopyPad(dstLocal, srcGlobal, copyParams, padParams); | ||
| 89 | - PipeBarrier<PIPE_ALL>(); | ||
| 90 | -} | ||
| 91 | - | ||
| 92 | -template <typename T> | ||
| 93 | -__aicore__ inline void Ub2Gm( | ||
| 94 | - const AscendC::GlobalTensor<T>& dstGlobal, const AscendC::LocalTensor<T>& srcLocal, int32_t length) | ||
| 95 | -{ | ||
| 96 | - AscendC::DataCopyExtParams copyParams{1, (uint32_t)(length * sizeof(T)), 0, 0, 0}; | ||
| 97 | - PipeBarrier<PIPE_ALL>(); | ||
| 98 | - AscendC::DataCopyPad(dstGlobal, srcLocal, copyParams); | ||
| 99 | - PipeBarrier<PIPE_ALL>(); | ||
| 100 | -} | ||
| 101 | - | ||
| 102 | -__aicore__ inline int32_t HcommFindBufferIdx( | ||
| 103 | - RegedBufferEntity* bufferAddr, uint32_t bufferNum, GM_ADDR addr, uint64_t len) | ||
| 104 | -{ | ||
| 105 | - if (bufferAddr == nullptr) { | ||
| 106 | - KERNEL_LOG(KERNEL_ERROR, "HcommFindBufferIdx failed with null bufferAddr, bufferNum=%u\n", bufferNum); | ||
| 107 | - return HCOMM_FAILED; | ||
| 108 | - } | ||
| 109 | - uint64_t targetAddr = reinterpret_cast<uint64_t>(addr); | ||
| 110 | - for (uint32_t i = 0; i < bufferNum; i++) { | ||
| 111 | - uint64_t baseAddr = bufferAddr[i].bufferInfo.rma.addr; | ||
| 112 | - uint64_t bufferSize = bufferAddr[i].bufferInfo.rma.size; | ||
| 113 | - if (targetAddr < baseAddr) { | ||
| 114 | - continue; | ||
| 115 | - } | ||
| 116 | - uint64_t offset = targetAddr - baseAddr; | ||
| 117 | - if (offset <= bufferSize && len <= bufferSize - offset) { | ||
| 118 | - KERNEL_LOG(KERNEL_INFO, "HcommFindBufferIdx hit idx=%u addr=%llu len=%llu base=%llu size=%llu\n", i, | ||
| 119 | - static_cast<uint64_t>(targetAddr), static_cast<uint64_t>(len), | ||
| 120 | - static_cast<uint64_t>(baseAddr), static_cast<uint64_t>(bufferSize)); | ||
| 121 | - return static_cast<int32_t>(i); | ||
| 122 | - } | ||
| 123 | - } | ||
| 124 | - KERNEL_LOG(KERNEL_ERROR, "HcommFindBufferIdx failed addr=%llu len=%llu bufferNum=%u\n", | ||
| 125 | - static_cast<uint64_t>(targetAddr), static_cast<uint64_t>(len), bufferNum); | ||
| 126 | - return HCOMM_FAILED; | ||
| 127 | -} | ||
| 128 | - | ||
| 129 | -template <typename T> | ||
| 130 | -__aicore__ inline void CacheWriteThrough(__gm__ T* sourceAddr, uint64_t length) | ||
| 131 | -{ | ||
| 132 | - if (length == 0) { | ||
| 133 | - return; | ||
| 134 | - } | ||
| 135 | - __gm__ T* start = (__gm__ T*)((uint64_t)sourceAddr / CACHE_LINE_SIZE * CACHE_LINE_SIZE); | ||
| 136 | - __gm__ T* end = (__gm__ T*)(((uint64_t)sourceAddr + length) / CACHE_LINE_SIZE * CACHE_LINE_SIZE); | ||
| 137 | - GlobalTensor<T> global; | ||
| 138 | - global.SetGlobalBuffer(start); | ||
| 139 | - for (uint32_t i = 0; i <= end - start; i += CACHE_LINE_SIZE) { | ||
| 140 | - DataCacheCleanAndInvalid<T, CacheLine::SINGLE_CACHE_LINE, DcciDst::CACHELINE_OUT>(global[i]); | ||
| 141 | - } | ||
| 142 | -} | ||
| 143 | -} // namespace AscendC | ||
| 144 | - | ||
| 145 | - | ||
| 146 | - | ||
| 147 | - | ||
| 148 | - | ||
| 149 | - | ||
| @@ -1,92 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_impl.h | ||
| 13 | - * \brief Hcomm implementation | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/hcomm_impl.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| 33 | -namespace AscendC { | ||
| 34 | - | ||
| 35 | -template <CommProtocol commProtocol> | ||
| 36 | -__aicore__ inline int32_t Hcomm<commProtocol>::Init(__ubuf__ uint8_t* buff, uint32_t len) | ||
| 37 | -{ | ||
| 38 | - return impl_.Init(buff, len); | ||
| 39 | -} | ||
| 40 | - | ||
| 41 | -template <CommProtocol commProtocol> | ||
| 42 | -template <typename T> | ||
| 43 | -__aicore__ inline int32_t Hcomm<commProtocol>::Init(const LocalTensor<T>& buff, uint32_t len) | ||
| 44 | -{ | ||
| 45 | - return impl_.Init(buff, len); | ||
| 46 | -} | ||
| 47 | - | ||
| 48 | -template <CommProtocol commProtocol> | ||
| 49 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 50 | -__aicore__ inline int32_t Hcomm<commProtocol>::WriteNbi( | ||
| 51 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 52 | -{ | ||
| 53 | - return impl_.template WriteNbi<commit, commitPipe, reqPipe, config>(channel, dst, src, len); | ||
| 54 | -} | ||
| 55 | - | ||
| 56 | -template <CommProtocol commProtocol> | ||
| 57 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 58 | -__aicore__ inline int32_t Hcomm<commProtocol>::WriteWithNotifyNbi( | ||
| 59 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len, GM_ADDR notifyAddr, uint64_t notifyVal) | ||
| 60 | -{ | ||
| 61 | - return impl_.template WriteWithNotifyNbi<commit, commitPipe, reqPipe, config>( | ||
| 62 | - channel, dst, src, len, notifyAddr, notifyVal); | ||
| 63 | -} | ||
| 64 | - | ||
| 65 | -template <CommProtocol commProtocol> | ||
| 66 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 67 | -__aicore__ inline int32_t Hcomm<commProtocol>::ReadNbi( | ||
| 68 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 69 | -{ | ||
| 70 | - return impl_.template ReadNbi<commit, commitPipe, reqPipe, config>(channel, dst, src, len); | ||
| 71 | -} | ||
| 72 | - | ||
| 73 | -template <CommProtocol commProtocol> | ||
| 74 | -template <pipe_t pipe> | ||
| 75 | -__aicore__ inline int32_t Hcomm<commProtocol>::Commit(ChannelHandle channel) | ||
| 76 | -{ | ||
| 77 | - return impl_.template Commit<pipe>(channel); | ||
| 78 | -} | ||
| 79 | - | ||
| 80 | -template <CommProtocol commProtocol> | ||
| 81 | -template <pipe_t pipe> | ||
| 82 | -__aicore__ inline int32_t Hcomm<commProtocol>::Drain(ChannelHandle channel) | ||
| 83 | -{ | ||
| 84 | - return impl_.template Drain<pipe>(channel); | ||
| 85 | -} | ||
| 86 | -} // namespace AscendC | ||
| 87 | - | ||
| 88 | - | ||
| 89 | - | ||
| 90 | - | ||
| 91 | - | ||
| 92 | - | ||
| @@ -1,40 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_impl_def.h | ||
| 13 | - * \brief Hcomm implementation definition | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/hcomm_impl_def.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| 33 | - | ||
| 34 | - | ||
| 35 | - | ||
| 36 | - | ||
| 37 | - | ||
| 38 | - | ||
| 39 | - | ||
| 40 | - | ||
| @@ -1,32 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_v220_impl.h | ||
| 13 | - * \brief Hcomm implementation for V220 | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/hcomm_v220_impl.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| @@ -1,33 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_v310_impl.h | ||
| 13 | - * \brief Hcomm implementation for V310 | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/hcomm_v310_impl.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| 33 | - | ||
| @@ -1,182 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_aiv.h | ||
| 13 | - * \brief Hcomm AIV implementation for V220 | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/platform_v220/hcomm_aiv.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | -namespace AscendC { | ||
| 31 | - | ||
| 32 | -__aicore__ inline HcommImpl<COMM_PROTOCOL_ROCE>::HcommImpl() | ||
| 33 | -{ | ||
| 34 | - TBuf<TPosition::VECOUT> rdmaInBuf; | ||
| 35 | - GetTPipePtr()->InitBuffer(rdmaInBuf, ONE_BLK_SIZE); | ||
| 36 | - ubLocal_ = rdmaInBuf.Get<uint64_t>(); | ||
| 37 | - | ||
| 38 | - TBuf<TPosition::VECOUT> rdmaInBuf2; | ||
| 39 | - GetTPipePtr()->InitBuffer(rdmaInBuf2, ONE_BLK_SIZE); | ||
| 40 | - ubLocalHead_ = rdmaInBuf2.Get<uint32_t>(); | ||
| 41 | -} | ||
| 42 | - | ||
| 43 | -__aicore__ inline HcommImpl<COMM_PROTOCOL_ROCE>::~HcommImpl() {} | ||
| 44 | - | ||
| 45 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 46 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::WriteNbi( | ||
| 47 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 48 | -{ | ||
| 49 | - (void)config; | ||
| 50 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Write channel:%llu, dst:%p, src:%p, len:%llu", channel, dst, src, len); | ||
| 51 | - PostSend(channel, dst, src, len, false); | ||
| 52 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Write complete"); | ||
| 53 | - return HCOMM_SUCCESS; | ||
| 54 | -} | ||
| 55 | - | ||
| 56 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 57 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::ReadNbi( | ||
| 58 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 59 | -{ | ||
| 60 | - (void)config; | ||
| 61 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Read channel:%llu, dst:%p, src:%p, len:%llu", channel, dst, src, len); | ||
| 62 | - PostSend(channel, dst, src, len, true); | ||
| 63 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Read complete"); | ||
| 64 | - return HCOMM_SUCCESS; | ||
| 65 | -} | ||
| 66 | - | ||
| 67 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 68 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::WriteWithNotifyNbi( | ||
| 69 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len, GM_ADDR notifyAddr, uint64_t notifyVal) | ||
| 70 | -{ | ||
| 71 | - (void)commit; | ||
| 72 | - (void)commitPipe; | ||
| 73 | - (void)reqPipe; | ||
| 74 | - (void)config; | ||
| 75 | - (void)channel; | ||
| 76 | - (void)dst; | ||
| 77 | - (void)src; | ||
| 78 | - (void)len; | ||
| 79 | - (void)notifyAddr; | ||
| 80 | - (void)notifyVal; | ||
| 81 | - KERNEL_LOG(KERNEL_ERROR, "Hcomm ROCE WriteWithNotifyNbi is not supported"); | ||
| 82 | - return HCOMM_FAILED; | ||
| 83 | -} | ||
| 84 | - | ||
| 85 | -__aicore__ inline void HcommImpl<COMM_PROTOCOL_ROCE>::doorBell( | ||
| 86 | - __gm__ ChannelEntity* channel, uint64_t curHead) | ||
| 87 | -{ | ||
| 88 | - uint64_t doorBellInfo = 0; | ||
| 89 | - doorBellInfo |= channel->sqContextAddr[0].contextInfo.roceSq.qpn; // [0:23] DB_TAG (qp_num) | ||
| 90 | - doorBellInfo |= 0UL << 24UL; // [24:27] DB_CMD = HNS_ROCE_V2_SQ_DB (0) | ||
| 91 | - doorBellInfo |= (curHead % 65536UL) << 32UL; // [32:47] DB_PI = sq.head | ||
| 92 | - doorBellInfo |= (uint64_t)(channel->sqContextAddr[0].contextInfo.roceSq.sl) << 48UL; // [48:50] DB_SL = qp.sl | ||
| 93 | - | ||
| 94 | - __gm__ uint64_t* doorBellAddr = (__gm__ uint64_t*)(channel->sqContextAddr[0].contextInfo.roceSq.dbVa); | ||
| 95 | - KERNEL_LOG(KERNEL_INFO, "Hcomm doorBell doorBellAddr:%p, doorBellInfo:%llu", doorBellAddr, doorBellInfo); | ||
| 96 | - | ||
| 97 | - ubLocal_.SetValue(0, doorBellInfo); | ||
| 98 | - AscendC::GlobalTensor<uint64_t> DBGlobalTensor; | ||
| 99 | - DBGlobalTensor.SetGlobalBuffer(doorBellAddr); | ||
| 100 | - AscendC::DataCopyExtParams copyParams{1, 1 * sizeof(uint64_t), 0, 0, 0}; | ||
| 101 | - AscendC::DataCopyPad(DBGlobalTensor, ubLocal_, copyParams); | ||
| 102 | -} | ||
| 103 | - | ||
| 104 | -__aicore__ inline void HcommImpl<COMM_PROTOCOL_ROCE>::PostSend( | ||
| 105 | - ChannelHandle channelHandle, GM_ADDR dst, GM_ADDR src, uint64_t len, bool isRead) | ||
| 106 | -{ | ||
| 107 | - __gm__ ChannelEntity* channel = (__gm__ ChannelEntity*)channelHandle; | ||
| 108 | - auto qpNum = channel->sqContextAddr[0].contextInfo.roceSq.qpn; | ||
| 109 | - auto sqBaseAddr = channel->sqContextAddr[0].contextInfo.roceSq.sqVa; | ||
| 110 | - auto wqeSize = channel->sqContextAddr[0].contextInfo.roceSq.wqeSize; | ||
| 111 | - auto curHardwareHead = channel->sqContextAddr[0].contextInfo.roceSq.headAddr; | ||
| 112 | - CacheWriteThrough(reinterpret_cast<__gm__ uint8_t*>(curHardwareHead), 8); | ||
| 113 | - uint64_t curHead = *(__gm__ uint32_t*)(curHardwareHead); | ||
| 114 | - | ||
| 115 | - auto curHardwareTailAddr = channel->sqContextAddr[0].contextInfo.roceSq.tailAddr; | ||
| 116 | - uint64_t shift = 15U; | ||
| 117 | - auto qpDepth = channel->sqContextAddr[0].contextInfo.roceSq.depth; | ||
| 118 | - | ||
| 119 | - KERNEL_LOG( | ||
| 120 | - KERNEL_INFO, "Hcomm doorBell qpNum:%d, sqBaseAddr:%p, wqeSize:%d, curHead:%d, qpDepth:%d", qpNum, sqBaseAddr, | ||
| 121 | - wqeSize, curHead, qpDepth); | ||
| 122 | - | ||
| 123 | - // Make sure we don't overflow the SQ in an infinite loop - no need to mitigate endless loop as the host | ||
| 124 | - // will timeout and kill the kernel, same as all2all kernel if it fails to complete (e.g. in case of link loss) | ||
| 125 | - while (1) { | ||
| 126 | - CacheWriteThrough((__gm__ uint8_t*)curHardwareTailAddr, 8); | ||
| 127 | - if ((curHead - *(__gm__ uint32_t*)(curHardwareTailAddr)) < qpDepth - 1) { | ||
| 128 | - break; | ||
| 129 | - } | ||
| 130 | - } | ||
| 131 | - | ||
| 132 | - __gm__ uint8_t* wqeAddr = (__gm__ uint8_t*)(sqBaseAddr + wqeSize * (curHead % qpDepth)); | ||
| 133 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend wqeAddr:%p", wqeAddr); | ||
| 134 | - | ||
| 135 | - // Write the WQE to GM | ||
| 136 | - uint64_t ownBit = (curHead >> shift) & 1U; | ||
| 137 | - uint32_t byte_4 = isRead ? | ||
| 138 | - static_cast<uint32_t>(HCOMM_OP_TYPE::READ) : | ||
| 139 | - static_cast<uint32_t>(HCOMM_OP_TYPE::WRITE); // [0:4] opcode=0x3(RDMA_WRITE), 0x5(RDMA_READ) | ||
| 140 | - byte_4 |= ((~ownBit) << 7U) & (1U << 7U); // [7] owner_bit | ||
| 141 | - byte_4 |= 1U << 8U; // [8:8] IBV_SEND_SIGNALED | ||
| 142 | - | ||
| 143 | - *(__gm__ uint32_t*)(wqeAddr) = byte_4; // Control set by local parameter see above lines | ||
| 144 | - *(__gm__ uint32_t*)(wqeAddr + 4) = len; // message size | ||
| 145 | - *(__gm__ uint32_t*)(wqeAddr + 8) = 0; // immtdata is always 0 till we provide poll CQ flow in AIV | ||
| 146 | - *(__gm__ uint32_t*)(wqeAddr + 12) = 1U << 24U; // [120:127] num_sge = 1 | ||
| 147 | - *(__gm__ uint32_t*)(wqeAddr + 16) = 0; // [128:151] start_sge_idx = 0; | ||
| 148 | - *(__gm__ uint32_t*)(wqeAddr + 20) = | ||
| 149 | - channel->remoteBufferAddr[0].bufferInfo.rma.protectionInfo.memInfo.roce.rkey; | ||
| 150 | - *(__gm__ uint64_t*)(wqeAddr + 24) = (uint64_t)dst; // destination VA | ||
| 151 | - | ||
| 152 | - constexpr uint32_t sgeAddrOffset = 32; | ||
| 153 | - __gm__ uint8_t* sgeAddr = wqeAddr + sgeAddrOffset; | ||
| 154 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend sgeAddr:%p", sgeAddr); | ||
| 155 | - *(__gm__ uint32_t*)(sgeAddr) = len; | ||
| 156 | - *(__gm__ uint32_t*)(sgeAddr + sizeof(uint32_t)) = | ||
| 157 | - channel->localBufferAddr[0].bufferInfo.rma.protectionInfo.memInfo.roce.lkey; | ||
| 158 | - *(__gm__ uint64_t*)(sgeAddr + 2 * sizeof(uint32_t)) = (uint64_t)src; // src VA addr memory registered by RNIC | ||
| 159 | - | ||
| 160 | - constexpr uint32_t wqeAddrWriteLength = 48; | ||
| 161 | - CacheWriteThrough(wqeAddr, wqeAddrWriteLength); | ||
| 162 | - | ||
| 163 | - curHead++; | ||
| 164 | - | ||
| 165 | - // Post doorbell | ||
| 166 | - doorBell(channel, curHead); | ||
| 167 | - | ||
| 168 | - ubLocalHead_.SetValue(0, (uint32_t)curHead); | ||
| 169 | - AscendC::GlobalTensor<uint32_t> HeadGlobalTensor; | ||
| 170 | - HeadGlobalTensor.SetGlobalBuffer((__gm__ uint32_t*)curHardwareHead); | ||
| 171 | - AscendC::DataCopyExtParams copyParamsHead{1, 1 * sizeof(uint32_t), 0, 0, 0}; | ||
| 172 | - AscendC::DataCopyPad(HeadGlobalTensor, ubLocalHead_, copyParamsHead); | ||
| 173 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend finish"); | ||
| 174 | -} | ||
| 175 | - | ||
| 176 | -} // namespace AscendC | ||
| 177 | - | ||
| 178 | - | ||
| 179 | - | ||
| 180 | - | ||
| 181 | - | ||
| 182 | - | ||
| @@ -1,87 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_aiv_def.h | ||
| 13 | - * \brief Hcomm AIV definition for V220 | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/platform_v220/hcomm_aiv_def.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | -namespace AscendC { | ||
| 29 | - | ||
| 30 | -enum class HCOMM_OP_TYPE : uint32_t { WRITE = 3U, READ = 5U }; | ||
| 31 | - | ||
| 32 | -template <> | ||
| 33 | -class HcommImpl<COMM_PROTOCOL_ROCE> { | ||
| 34 | -public: | ||
| 35 | - __aicore__ inline HcommImpl(); | ||
| 36 | - | ||
| 37 | - __aicore__ inline ~HcommImpl(); | ||
| 38 | - | ||
| 39 | - __aicore__ inline int32_t Init(__ubuf__ uint8_t* buff, uint32_t len) | ||
| 40 | - { | ||
| 41 | - (void)buff; | ||
| 42 | - (void)len; | ||
| 43 | - return HCOMM_FAILED; | ||
| 44 | - } | ||
| 45 | - | ||
| 46 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 47 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 48 | - __aicore__ inline int32_t WriteNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len); | ||
| 49 | - | ||
| 50 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 51 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 52 | - __aicore__ inline int32_t ReadNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len); | ||
| 53 | - | ||
| 54 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 55 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 56 | - __aicore__ inline int32_t WriteWithNotifyNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, | ||
| 57 | - uint64_t len, GM_ADDR notifyAddr, uint64_t notifyVal); | ||
| 58 | - | ||
| 59 | - template <pipe_t pipe = PIPE_S> | ||
| 60 | - __aicore__ inline int32_t Commit(ChannelHandle channel) | ||
| 61 | - { | ||
| 62 | - return HCOMM_FAILED; | ||
| 63 | - } | ||
| 64 | - | ||
| 65 | - template <pipe_t pipe = PIPE_MTE3> | ||
| 66 | - __aicore__ inline int32_t Drain(ChannelHandle channel) | ||
| 67 | - { | ||
| 68 | - return HCOMM_FAILED; | ||
| 69 | - } | ||
| 70 | - | ||
| 71 | -private: | ||
| 72 | - __aicore__ inline void PostSend(ChannelHandle channelHandle, GM_ADDR dst, GM_ADDR src, uint64_t len, bool isRead); | ||
| 73 | - | ||
| 74 | - __aicore__ inline void doorBell(__gm__ ChannelEntity* channel, uint64_t curHead); | ||
| 75 | - | ||
| 76 | -private: | ||
| 77 | - LocalTensor<uint64_t> ubLocal_; | ||
| 78 | - LocalTensor<uint32_t> ubLocalHead_; | ||
| 79 | -}; | ||
| 80 | - | ||
| 81 | -} // namespace AscendC | ||
| 82 | - | ||
| 83 | - | ||
| 84 | - | ||
| 85 | - | ||
| 86 | - | ||
| 87 | - | ||
| @@ -1,339 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_aiv_roce.h | ||
| 13 | - * \brief Hcomm AIV implementation for V310 | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/platform_v310/hcomm_aiv_roce.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | -namespace AscendC { | ||
| 31 | - | ||
| 32 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::Init(__ubuf__ uint8_t *buff, uint32_t len) | ||
| 33 | -{ | ||
| 34 | - if (len < HCOMM_UB_BUF_SIZE) { | ||
| 35 | - return HCOMM_FAILED; | ||
| 36 | - } | ||
| 37 | - __ubuf__ uint8_t *alignedAddr = AlignAddrTo32Bytes(buff); | ||
| 38 | - TBuffAddr addr; | ||
| 39 | - addr.logicPos = static_cast<uint8_t>(TPosition::VECOUT); | ||
| 40 | - addr.dataLen = len; | ||
| 41 | - addr.bufferAddr = static_cast<uint32_t>(reinterpret_cast<uintptr_t>(alignedAddr)); | ||
| 42 | - | ||
| 43 | - addr.absAddr = reinterpret_cast<uint8_t *>(alignedAddr); | ||
| 44 | - | ||
| 45 | - | ||
| 46 | - wqeUB_.SetAddr(addr); | ||
| 47 | - cqeUB_ = wqeUB_[ROCE_CQE_POS]; | ||
| 48 | - dbUB_ = wqeUB_[ROCE_DB_POS]; | ||
| 49 | - wqeAddr_ = alignedAddr; | ||
| 50 | - cqeAddr_ = alignedAddr + ROCE_CQE_POS; | ||
| 51 | - dbAddr_ = alignedAddr + ROCE_DB_POS; | ||
| 52 | - return HCOMM_SUCCESS; | ||
| 53 | -} | ||
| 54 | - | ||
| 55 | -template <typename T> | ||
| 56 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::Init(const LocalTensor<T>& buff, uint32_t len) | ||
| 57 | -{ | ||
| 58 | - if (len < HCOMM_UB_BUF_SIZE || buff.GetSize() < HCOMM_UB_BUF_SIZE) { | ||
| 59 | - return HCOMM_FAILED; | ||
| 60 | - } | ||
| 61 | - LocalTensor<uint8_t> buffTensor = buff.template ReinterpretCast<uint8_t>(); | ||
| 62 | - wqeUB_ = buffTensor[0]; | ||
| 63 | - cqeUB_ = buffTensor[ROCE_CQE_POS]; | ||
| 64 | - dbUB_ = buffTensor[ROCE_DB_POS]; | ||
| 65 | - wqeAddr_ = (__ubuf__ uint8_t *)wqeUB_.GetPhyAddr(); | ||
| 66 | - cqeAddr_ = (__ubuf__ uint8_t *)cqeUB_.GetPhyAddr(); | ||
| 67 | - dbAddr_ = (__ubuf__ uint8_t *)dbUB_.GetPhyAddr(); | ||
| 68 | - return HCOMM_SUCCESS; | ||
| 69 | -} | ||
| 70 | - | ||
| 71 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::MakeWqe(__gm__ ChannelEntity* chnlPtr, | ||
| 72 | - GM_ADDR dst, GM_ADDR src, uint64_t len, uint32_t opType, uint32_t sqHead, uint32_t sqDepth) | ||
| 73 | -{ | ||
| 74 | - int32_t remoteIdx = HcommFindBufferIdx(chnlPtr->remoteBufferAddr, chnlPtr->remoteBufferNum, dst, len); | ||
| 75 | - if (remoteIdx < 0) { | ||
| 76 | - KERNEL_LOG(KERNEL_INFO, "Hcomm MakeWqe: failed with invalid remote buffer addr %llu\n", dst); | ||
| 77 | - return HCOMM_FAILED; | ||
| 78 | - } | ||
| 79 | - int32_t localIdx = HcommFindBufferIdx(chnlPtr->localBufferAddr, chnlPtr->localBufferNum, src, len); | ||
| 80 | - if (localIdx < 0) { | ||
| 81 | - KERNEL_LOG(KERNEL_INFO, "Hcomm MakeWqe: failed with invalid local buffer addr %llu\n", src); | ||
| 82 | - return HCOMM_FAILED; | ||
| 83 | - } | ||
| 84 | - __gm__ uint8_t* sqBaseAddr = (__gm__ uint8_t*)(chnlPtr->sqContextAddr->contextInfo.roceSq.sqVa); | ||
| 85 | - uint32_t wqeSize = chnlPtr->sqContextAddr->contextInfo.roceSq.wqeSize; | ||
| 86 | - __gm__ uint8_t* sqAddr = (__gm__ uint8_t*)(sqBaseAddr + (sqHead % sqDepth) * wqeSize); | ||
| 87 | - GlobalTensor<uint8_t> sqGlobal; | ||
| 88 | - sqGlobal.SetGlobalBuffer(sqAddr); | ||
| 89 | - __ubuf__ RoceWqeEntry *wqePtr = (__ubuf__ RoceWqeEntry*)(wqeAddr_); | ||
| 90 | - | ||
| 91 | - uint8_t owner = (sqHead & sqDepth) == 0 ? 0 : 1; | ||
| 92 | - wqePtr->ctrl.dw0.value = HtoNL( | ||
| 93 | - owner << HCOMM_WQE_OWNER_OFFSET | 2U << HCOMM_WQE_CTRLSL_OFFSET | 1U << HCOMM_WQE_CR_OFFSET | | ||
| 94 | - 1U << HCOMM_WQE_VA_OFFSET | 4U << HCOMM_WQE_TSL_OFFSET | 2U << HCOMM_WQE_BDSL_OFFSET); | ||
| 95 | - wqePtr->ctrl.dw1.value = HtoNL(1U << HCOMM_WQE_CL_OFFSET); | ||
| 96 | - wqePtr->doorbell = 0; | ||
| 97 | - wqePtr->task.dw0.value = HtoNL(opType << HCOMM_WQE_OP_TYPE_OFFSET | 1U << HCOMM_WQE_C_OFFSET); | ||
| 98 | - wqePtr->dataLen = HtoNL(len); | ||
| 99 | - wqePtr->immeData = 0; | ||
| 100 | - wqePtr->firstLast = 0; | ||
| 101 | - wqePtr->nxtEthHdr = 0; | ||
| 102 | - wqePtr->cmdLen = 0; | ||
| 103 | - wqePtr->rsvd0 = 0; | ||
| 104 | - wqePtr->lastExtLen = 0; | ||
| 105 | - wqePtr->vaRemote = HtoNLL((uint64_t)dst); | ||
| 106 | - uint32_t rKey = chnlPtr->remoteBufferAddr[remoteIdx].bufferInfo.rma.protectionInfo.memInfo.roce.rkey; | ||
| 107 | - wqePtr->rKey = HtoNL(rKey); | ||
| 108 | - wqePtr->rsvd1 = 0; | ||
| 109 | - wqePtr->data.vaLocal = HtoNLL((uint64_t)src); | ||
| 110 | - wqePtr->data.rLen = HtoNL((uint32_t)len); | ||
| 111 | - uint32_t lKey = chnlPtr->localBufferAddr[localIdx].bufferInfo.rma.protectionInfo.memInfo.roce.lkey; | ||
| 112 | - wqePtr->data.leKey = HtoNL(lKey & (~(1U << 31))); | ||
| 113 | - KERNEL_LOG(KERNEL_INFO, "Hcomm MakeWqe: make wqe ok\n"); | ||
| 114 | - | ||
| 115 | - SyncAction<HardEvent::S_MTE3>(); | ||
| 116 | - DataCopy(sqGlobal, wqeUB_, sizeof(RoceWqeEntry)); | ||
| 117 | - SyncAction<HardEvent::MTE3_S>(); | ||
| 118 | - | ||
| 119 | - KERNEL_LOG(KERNEL_INFO, "Hcomm MakeWqe: set wqe to qp ok\n"); | ||
| 120 | - return HCOMM_SUCCESS; | ||
| 121 | -} | ||
| 122 | - | ||
| 123 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe> | ||
| 124 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::PostSend( | ||
| 125 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len, uint32_t opType) | ||
| 126 | -{ | ||
| 127 | - __gm__ ChannelEntity* chnlPtr = (__gm__ ChannelEntity*)(channel); | ||
| 128 | - if (chnlPtr == nullptr || chnlPtr->sqNum == 0 || chnlPtr->cqNum == 0 || | ||
| 129 | - chnlPtr->sqContextAddr == nullptr || chnlPtr->cqContextAddr == nullptr) { | ||
| 130 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: channel sqNum = %u, cqNum = %u.\n", chnlPtr->sqNum, chnlPtr->cqNum); | ||
| 131 | - return HCOMM_FAILED; | ||
| 132 | - } | ||
| 133 | - auto sqPIAddr = chnlPtr->sqContextAddr[0].contextInfo.roceSq.headAddr; | ||
| 134 | - CacheWriteThrough((__gm__ uint8_t*)(sqPIAddr), sizeof(uint32_t)); | ||
| 135 | - uint32_t sqHead = *(__gm__ uint32_t*)(sqPIAddr); | ||
| 136 | - uint32_t sqDepth = chnlPtr->sqContextAddr[0].contextInfo.roceSq.depth; | ||
| 137 | - uint32_t cqDepth = chnlPtr->cqContextAddr[0].contextInfo.roceCq.cqDepth; | ||
| 138 | - auto sqCIAddr = chnlPtr->sqContextAddr[0].contextInfo.roceSq.tailAddr; | ||
| 139 | - CacheWriteThrough((__gm__ uint8_t*)(sqCIAddr), sizeof(uint32_t)); | ||
| 140 | - uint32_t sqTail = *(__gm__ uint32_t*)(sqCIAddr); | ||
| 141 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: opType = %u, sqHead = %u, sqTail = %u, sqDepth = %u, cqDepth = %u.\n", | ||
| 142 | - opType, sqHead, sqTail, sqDepth, cqDepth); | ||
| 143 | - | ||
| 144 | - constexpr uint32_t POLL_CQ_THRESHOLD = 10; | ||
| 145 | - constexpr uint32_t NUM_CQE_PER_POLL_CQ = 100; | ||
| 146 | - if ((sqHead + POLL_CQ_THRESHOLD) % cqDepth == sqTail % cqDepth) { | ||
| 147 | - uint32_t idx = (sqTail + NUM_CQE_PER_POLL_CQ) > sqHead ? sqHead : (sqTail + NUM_CQE_PER_POLL_CQ); | ||
| 148 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: RoCE SQ overflow sqHead=%u sqTail=%u idx=%u cqDepth=%u\n", sqHead, | ||
| 149 | - sqTail, idx, cqDepth); | ||
| 150 | - if (PollCq(chnlPtr, idx) != HCOMM_SUCCESS) { | ||
| 151 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: RoCE SQ overflow, PollCq failed.\n"); | ||
| 152 | - return HCOMM_FAILED; | ||
| 153 | - } | ||
| 154 | - } | ||
| 155 | - | ||
| 156 | - if (MakeWqe(chnlPtr, dst, src, len, opType, sqHead, sqDepth) != HCOMM_SUCCESS) { | ||
| 157 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: MakeWqe failed.\n"); | ||
| 158 | - return HCOMM_FAILED; | ||
| 159 | - } | ||
| 160 | - | ||
| 161 | - sqHead++; | ||
| 162 | - *(__gm__ uint32_t*)(sqPIAddr) = sqHead; | ||
| 163 | - CacheWriteThrough((__gm__ uint8_t*)(sqPIAddr), sizeof(uint32_t)); | ||
| 164 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: update PI to GM sqHead = %u\n", sqHead); | ||
| 165 | - | ||
| 166 | - if constexpr (commit) { | ||
| 167 | - KnockDoorBell<commitPipe>(chnlPtr, sqHead); | ||
| 168 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PostSend: Commit ok.\n"); | ||
| 169 | - } | ||
| 170 | - return HCOMM_SUCCESS; | ||
| 171 | -} | ||
| 172 | - | ||
| 173 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 174 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::WriteNbi( | ||
| 175 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 176 | -{ | ||
| 177 | - (void)config; | ||
| 178 | - return PostSend<commit, commitPipe, reqPipe>(channel, dst, src, len, | ||
| 179 | - static_cast<uint32_t>(HCOMM_ROCE_OP_TYPE::WRITE)); | ||
| 180 | -} | ||
| 181 | - | ||
| 182 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 183 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::ReadNbi( | ||
| 184 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len) | ||
| 185 | -{ | ||
| 186 | - (void)config; | ||
| 187 | - return PostSend<commit, commitPipe, reqPipe>(channel, src, dst, len, | ||
| 188 | - static_cast<uint32_t>(HCOMM_ROCE_OP_TYPE::READ)); | ||
| 189 | -} | ||
| 190 | - | ||
| 191 | -template <bool commit, pipe_t commitPipe, pipe_t reqPipe, auto const &config> | ||
| 192 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::WriteWithNotifyNbi( | ||
| 193 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len, GM_ADDR notifyAddr, uint64_t notifyVal) | ||
| 194 | -{ | ||
| 195 | - (void)commit; | ||
| 196 | - (void)commitPipe; | ||
| 197 | - (void)reqPipe; | ||
| 198 | - (void)config; | ||
| 199 | - (void)channel; | ||
| 200 | - (void)dst; | ||
| 201 | - (void)src; | ||
| 202 | - (void)len; | ||
| 203 | - (void)notifyAddr; | ||
| 204 | - (void)notifyVal; | ||
| 205 | - KERNEL_LOG(KERNEL_ERROR, "Hcomm ROCE WriteWithNotifyNbi is not supported"); | ||
| 206 | - return HCOMM_FAILED; | ||
| 207 | -} | ||
| 208 | - | ||
| 209 | -__aicore__ inline uint64_t HcommImpl<COMM_PROTOCOL_ROCE>::GetDbValue(uint32_t qpn) | ||
| 210 | -{ | ||
| 211 | - RoceDbEntry dbEntry; | ||
| 212 | - dbEntry.dw0.value = 0; | ||
| 213 | - dbEntry.dw0.bs.c = 0; | ||
| 214 | - dbEntry.dw0.bs.r = 0; | ||
| 215 | - dbEntry.dw0.bs.ctxSize = 1; | ||
| 216 | - dbEntry.dw0.bs.qpn = qpn; | ||
| 217 | - dbEntry.dw0.bs.subType = 0; | ||
| 218 | - dbEntry.dw0.bs.resv = 0; | ||
| 219 | - dbEntry.dw0.bs.pi = 0; | ||
| 220 | - dbEntry.dw0.bs.sgidIdx = ROCE_INIT_SQ_DB_SGIT_IDX; | ||
| 221 | - dbEntry.dw0.bs.type = ROCE_SQ_DOORBELL_TYPE; | ||
| 222 | - dbEntry.dw0.bs.mtuShift = 0; | ||
| 223 | - dbEntry.dw0.bs.cos = 0x7; | ||
| 224 | - dbEntry.dw0.bs.xrcVld = 0; | ||
| 225 | - return dbEntry.dw0.value; | ||
| 226 | -} | ||
| 227 | - | ||
| 228 | -template <pipe_t pipe> | ||
| 229 | -__aicore__ inline void HcommImpl<COMM_PROTOCOL_ROCE>::KnockDoorBell(__gm__ ChannelEntity* chnlPtr, | ||
| 230 | - uint32_t sqHead) | ||
| 231 | -{ | ||
| 232 | - uint64_t dbValue = GetDbValue(chnlPtr->sqContextAddr->contextInfo.roceSq.qpn); | ||
| 233 | - KERNEL_LOG(KERNEL_INFO, "Hcomm KnockDoorBell: dbValue = %llu\n", dbValue); | ||
| 234 | - if constexpr (pipe == PIPE_MTE3) { | ||
| 235 | - GlobalTensor<uint8_t> dbGlobalTensor; | ||
| 236 | - __gm__ uint8_t* doorBellAddr = (__gm__ uint8_t*)(chnlPtr->sqContextAddr->contextInfo.roceSq.dbVa); | ||
| 237 | - dbGlobalTensor.SetGlobalBuffer(doorBellAddr); | ||
| 238 | - __ubuf__ uint32_t* dbUBPtr = (__ubuf__ uint32_t*)(dbAddr_); | ||
| 239 | - *(dbUBPtr + 0) = (dbValue & 0xffffffff); | ||
| 240 | - *(dbUBPtr + 1) = (dbValue >> 32) | ((sqHead >> 8) & 0xff); | ||
| 241 | - Ub2Gm<uint8_t>(dbGlobalTensor, dbUB_, sizeof(uint64_t)); | ||
| 242 | - } else { | ||
| 243 | - __gm__ uint64_t* dbAddr = (__gm__ uint64_t*)(chnlPtr->sqContextAddr->contextInfo.roceSq.dbVa); | ||
| 244 | - uint64_t dbFinalVal = dbValue | ((((uint64_t)(sqHead) >> 8) & 0xff) << 32); | ||
| 245 | - st_dev(dbFinalVal, dbAddr, 0); | ||
| 246 | - } | ||
| 247 | -} | ||
| 248 | - | ||
| 249 | -template <pipe_t pipe> | ||
| 250 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::Commit(ChannelHandle channel) | ||
| 251 | -{ | ||
| 252 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Commit: Enter\n"); | ||
| 253 | - __gm__ ChannelEntity* chnlPtr = (__gm__ ChannelEntity*)(channel); | ||
| 254 | - auto sqPIAddr = chnlPtr->sqContextAddr->contextInfo.roceSq.headAddr; | ||
| 255 | - CacheWriteThrough((__gm__ uint8_t*)(sqPIAddr), sizeof(uint32_t)); | ||
| 256 | - uint32_t sqHead = *(__gm__ uint32_t*)(sqPIAddr); | ||
| 257 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Commit: sqHead = %u\n", sqHead); | ||
| 258 | - | ||
| 259 | - KnockDoorBell<pipe>(chnlPtr, sqHead); | ||
| 260 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Commit: Exit ok\n"); | ||
| 261 | - return HCOMM_SUCCESS; | ||
| 262 | -} | ||
| 263 | - | ||
| 264 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::PollCq(__gm__ ChannelEntity* chnlPtr, uint32_t expectIdx) | ||
| 265 | -{ | ||
| 266 | - if (expectIdx == 0) { | ||
| 267 | - return HCOMM_SUCCESS; | ||
| 268 | - } | ||
| 269 | - uint32_t cqeSize = chnlPtr->cqContextAddr[0].contextInfo.roceCq.cqeSize; | ||
| 270 | - uint32_t cqDepth = chnlPtr->cqContextAddr[0].contextInfo.roceCq.cqDepth; | ||
| 271 | - uint32_t qpn = chnlPtr->sqContextAddr[0].contextInfo.roceSq.qpn; | ||
| 272 | - auto sqCIAddr = chnlPtr->sqContextAddr[0].contextInfo.roceSq.tailAddr; | ||
| 273 | - auto cqCIAddr = chnlPtr->cqContextAddr[0].contextInfo.roceCq.tailAddr; | ||
| 274 | - CacheWriteThrough((__gm__ uint8_t*)(cqCIAddr), sizeof(uint32_t)); | ||
| 275 | - uint32_t cqTail = *(__gm__ uint32_t*)(cqCIAddr); | ||
| 276 | - CacheWriteThrough((__gm__ uint8_t*)(sqCIAddr), sizeof(uint32_t)); | ||
| 277 | - uint32_t sqTail = *(__gm__ uint32_t*)(sqCIAddr); | ||
| 278 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PollCq: cqeSize = %u cqDepth = %u qpn = %u cqTail= %u sqTail = %u expectIdx = %u\n", | ||
| 279 | - cqeSize, cqDepth, qpn, cqTail, sqTail, expectIdx); | ||
| 280 | - | ||
| 281 | - __ubuf__ RoceCqeEntry* cqePtr = (__ubuf__ RoceCqeEntry*)(cqeAddr_); | ||
| 282 | - __gm__ uint8_t* cqBaseBuf = (__gm__ uint8_t*)(chnlPtr->cqContextAddr[0].contextInfo.roceCq.cqVa); | ||
| 283 | - AscendC::GlobalTensor<uint8_t> cqeGlobalTensor; | ||
| 284 | - while (cqTail < expectIdx) { | ||
| 285 | - __gm__ uint8_t* cqeAddr = (__gm__ uint8_t*)(cqBaseBuf + cqeSize * (cqTail % cqDepth)); | ||
| 286 | - cqeGlobalTensor.SetGlobalBuffer(cqeAddr); | ||
| 287 | - uint32_t loop = 0; | ||
| 288 | - for (; loop < HCOMM_POLLCQ_MAX_RETRY_TIMES; loop ++) { | ||
| 289 | - SyncAction<HardEvent::S_MTE2>(); | ||
| 290 | - DataCopy(cqeUB_, cqeGlobalTensor, sizeof(RoceCqeEntry)); | ||
| 291 | - SyncAction<HardEvent::MTE2_S>(); | ||
| 292 | - | ||
| 293 | - cqTail = expectIdx; | ||
| 294 | - break; | ||
| 295 | - | ||
| 296 | - uint8_t owner = (HtoNL(cqTail) & (cqDepth + 1)) == 0 ? 0 : 1; | ||
| 297 | - if (((((cqePtr->cqe0) >> 31) & 0x1) == owner) && ((cqePtr->cqe0 & 0xfffff) == qpn)) { | ||
| 298 | - break; | ||
| 299 | - } | ||
| 300 | - | ||
| 301 | - } | ||
| 302 | - if (loop >= HCOMM_POLLCQ_MAX_RETRY_TIMES) { | ||
| 303 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PollCq: Overtime exit\n"); | ||
| 304 | - return HCOMM_FAILED; | ||
| 305 | - } | ||
| 306 | - cqTail += 1; | ||
| 307 | - sqTail += 1; | ||
| 308 | - KERNEL_LOG(KERNEL_INFO, "Hcomm PollCq: cqTail = %u, sqTail = %u.\n", cqTail, sqTail); | ||
| 309 | - } | ||
| 310 | - *(__gm__ uint32_t*)(cqCIAddr) = cqTail; | ||
| 311 | - CacheWriteThrough((__gm__ uint8_t*)(cqCIAddr), sizeof(uint32_t)); | ||
| 312 | - *(__gm__ uint32_t*)(sqCIAddr) = sqTail; | ||
| 313 | - CacheWriteThrough((__gm__ uint8_t*)(sqCIAddr), sizeof(uint32_t)); | ||
| 314 | - return HCOMM_SUCCESS; | ||
| 315 | -} | ||
| 316 | - | ||
| 317 | -template <pipe_t pipe> | ||
| 318 | -__aicore__ inline int32_t HcommImpl<COMM_PROTOCOL_ROCE>::Drain(ChannelHandle channel) | ||
| 319 | -{ | ||
| 320 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Drain: Enter\n"); | ||
| 321 | - __gm__ ChannelEntity* chnlPtr = (__gm__ ChannelEntity*)(channel); | ||
| 322 | - auto sqPIAddr = chnlPtr->sqContextAddr[0].contextInfo.roceSq.headAddr; | ||
| 323 | - CacheWriteThrough((__gm__ uint8_t*)(sqPIAddr), sizeof(uint32_t)); | ||
| 324 | - uint32_t sqHead = *(__gm__ uint32_t*)(sqPIAddr); | ||
| 325 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Drain: sqHead = %u\n", sqHead); | ||
| 326 | - if (PollCq(chnlPtr, sqHead) != HCOMM_SUCCESS) { | ||
| 327 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Drain: PollCq failed.\n"); | ||
| 328 | - return HCOMM_FAILED; | ||
| 329 | - } | ||
| 330 | - KERNEL_LOG(KERNEL_INFO, "Hcomm Drain: Exit ok\n"); | ||
| 331 | - return HCOMM_SUCCESS; | ||
| 332 | -} | ||
| 333 | -} // namespace AscendC | ||
| 334 | - | ||
| 335 | - | ||
| 336 | - | ||
| 337 | - | ||
| 338 | - | ||
| 339 | - | ||
| @@ -1,83 +0,0 @@ | |||
| 1 | -/** | ||
| 2 | - * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | - * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | - * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | - * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | - */ | ||
| 10 | - | ||
| 11 | -/*! | ||
| 12 | - * \file hcomm_aiv_roce_def.h | ||
| 13 | - * \brief Hcomm AIV RoCE definition for V310 | ||
| 14 | - */ | ||
| 15 | - | ||
| 16 | - | ||
| 17 | - | ||
| 18 | - "impl/adv_api/detail/hcomm/impl/platform_v310/hcomm_aiv_roce_def.h is an internal header file and must not be used directly. Functions or variables defined in this file may be removed in the future. Please use \"#include \"adv_api/activation/simplesoftmax.h\"\" and use public functions or variables defined in interface headers files.") | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | -namespace AscendC { | ||
| 29 | -constexpr uint32_t ROCE_SQ_DOORBELL_TYPE = 2; | ||
| 30 | -constexpr uint32_t ROCE_INIT_SQ_DB_SGIT_IDX = 1; | ||
| 31 | -constexpr uint32_t ROCE_CQE_POS = 128; | ||
| 32 | -constexpr uint32_t ROCE_DB_POS = 192; | ||
| 33 | -enum class HCOMM_ROCE_OP_TYPE : uint32_t { WRITE = 4U, READ = 8U }; | ||
| 34 | - | ||
| 35 | -template <> | ||
| 36 | -class HcommImpl<COMM_PROTOCOL_ROCE> { | ||
| 37 | -public: | ||
| 38 | - __aicore__ inline HcommImpl() {} | ||
| 39 | - __aicore__ inline ~HcommImpl() {} | ||
| 40 | - __aicore__ inline int32_t Init(__ubuf__ uint8_t* buff, uint32_t len); | ||
| 41 | - template <typename T> | ||
| 42 | - __aicore__ inline int32_t Init(const LocalTensor<T>& buff, uint32_t len); | ||
| 43 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 44 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 45 | - __aicore__ inline int32_t WriteNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len); | ||
| 46 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 47 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 48 | - __aicore__ inline int32_t ReadNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len); | ||
| 49 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3, | ||
| 50 | - auto const &config = URMA_DEFAULT_CFG> | ||
| 51 | - __aicore__ inline int32_t WriteWithNotifyNbi(ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len, | ||
| 52 | - GM_ADDR notifyAddr, uint64_t notifyVal); | ||
| 53 | - template <pipe_t pipe = PIPE_S> | ||
| 54 | - __aicore__ inline int32_t Commit(ChannelHandle channel); | ||
| 55 | - template <pipe_t pipe = PIPE_MTE3> | ||
| 56 | - __aicore__ inline int32_t Drain(ChannelHandle channel); | ||
| 57 | - | ||
| 58 | -private: | ||
| 59 | - template <bool commit = true, pipe_t commitPipe = PIPE_S, pipe_t reqPipe = PIPE_MTE3> | ||
| 60 | - __aicore__ inline int32_t PostSend( | ||
| 61 | - ChannelHandle channel, GM_ADDR dst, GM_ADDR src, uint64_t len, uint32_t opType); | ||
| 62 | - template <pipe_t pipe> | ||
| 63 | - __aicore__ inline void KnockDoorBell(__gm__ ChannelEntity* chnlPtr, uint32_t sqHead); | ||
| 64 | - __aicore__ inline int32_t PollCq(__gm__ ChannelEntity* chnlPtr, uint32_t expectIdx); | ||
| 65 | - __aicore__ inline int32_t MakeWqe(__gm__ ChannelEntity* chnlPtr, GM_ADDR dst, GM_ADDR src, uint64_t len, | ||
| 66 | - uint32_t opType, uint32_t sqHead, uint32_t sqDepth); | ||
| 67 | - __aicore__ inline uint64_t GetDbValue(uint32_t qpn); | ||
| 68 | - | ||
| 69 | -private: | ||
| 70 | - __ubuf__ uint8_t *wqeAddr_; | ||
| 71 | - __ubuf__ uint8_t *cqeAddr_; | ||
| 72 | - __ubuf__ uint8_t *dbAddr_; | ||
| 73 | - LocalTensor<uint8_t> wqeUB_; | ||
| 74 | - LocalTensor<uint8_t> cqeUB_; | ||
| 75 | - LocalTensor<uint8_t> dbUB_; | ||
| 76 | -}; | ||
| 77 | -} // namespace AscendC | ||
| 78 | - | ||
| 79 | - | ||
| 80 | - | ||
| 81 | - | ||
| 82 | - | ||
| 83 | - | ||