Star441
976
代码
代码
Issues87
Pull Requests260
Actions
项目讨论
Wiki
分析
Star441
976
已合并
同步 CheckerL2 最新代码到 hccl_vm #3792
git_qk创建于 7月16日
同步 CheckerL2 最新代码到 hccl_vm #3792
已合并
git_qk创建于 7月16日
从git_qk/hcommmaster合入到cann/hcommmaster
共 42 个文件变更+442-10219
Mtest/hccl_vm/README.md+24-19
@@ -26,6 +26,7 @@ chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run
26./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend26./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend
27```27```
28 28 
29+ 
29### 2.2 hccl_test编译30### 2.2 hccl_test编译
30 31 
31hccl_test是昇腾官方提供的HCCL性能测试工具，详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html)，HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。32hccl_test是昇腾官方提供的HCCL性能测试工具，详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html)，HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。
@@ -34,15 +35,19 @@ hccl_test是昇腾官方提供的HCCL性能测试工具，详见[HCCL性能测
34 35 
35---36---
36 37 
37-## 一键安装38+## 3. 快速上手
38 39 
39-一行完成依赖安装、源码拉取、CANN 检测与编译（默认 `campus-2026` 配套方案）：40+### 3.1 一键安装
41+ 
42+一行完成依赖安装、源码拉取、CANN 检测与编译（默认 `campus-2026` 配套方案）。工作目录与手动安装保持一致，用 `/home/workspace`（后文示例路径均以此为准）：
40 43 
41```bash44```bash
45+# 创建并进入工作目录（脚本默认装到当前目录）
46+mkdir -p /home/workspace && cd /home/workspace
42curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/competition%2Fcampus-2026/test/hccl_vm/hccl_vm_installer | bash47curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/competition%2Fcampus-2026/test/hccl_vm/hccl_vm_installer | bash
43```48```
44 49 
45-也可下载后本地运行（便于先审阅或离线分发）：`bash hccl_vm_installer`；追加参数用 `... | bash -s -- --workspace /root/hvm`。50+也可下载后本地运行（便于先审阅或离线分发）：`bash hccl_vm_installer`；或用 `--workspace` 显式指定：`... | bash -s -- --workspace /home/workspace`。
46 51 
47**前提**：x86_64 Linux；工具链需满足 hcomm build.md 要求——gcc/g++ 7.3.0–13.3.x、cmake ≥ 3.16.0（同时约束宿主与 aarch64 交叉编译器）。Ubuntu 22.04 / 24.04 开箱即用；更高版本默认 gcc（14/15）超范围，脚本会告警并继续尝试，建议在满足范围的环境编译。52**前提**：x86_64 Linux；工具链需满足 hcomm build.md 要求——gcc/g++ 7.3.0–13.3.x、cmake ≥ 3.16.0（同时约束宿主与 aarch64 交叉编译器）。Ubuntu 22.04 / 24.04 开箱即用；更高版本默认 gcc（14/15）超范围，脚本会告警并继续尝试，建议在满足范围的环境编译。
48 53 
@@ -59,13 +64,11 @@ curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/competition%2Fcampus-2026/test
59- `--skip-hccl-test`：跳过 hccl_test 编译64- `--skip-hccl-test`：跳过 hccl_test 编译
60- `-h`：完整帮助65- `-h`：完整帮助
61 66 
62-完成后工具位于 `<工作目录>/hcomm/test/hccl_vm/hccl_vm_install/bin/hccl-vm`。删除工作目录即可清理本工具产物（apt 安装的系统依赖如需卸载请自行 `apt remove`），本工具不改动 CANN。67+完成后工具位于 `/home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin/hccl-vm`。删除工作目录即可清理本工具产物（apt 安装的系统依赖如需卸载请自行 `apt remove`），本工具不改动 CANN。（若用 `--workspace` 指定了别的目录，后文示例中的 `/home/workspace` 请相应替换。）
63 68 
64----69+> 一键安装已自动完成 `build.sh` 编译与 `build_pkg.sh` 子包安装（含 AICPU/AIV 所需的设备侧符号），装完后 [使用示例](#33-使用示例) 中 CCU/AICPU/AIV 各模式均可直接运行，无需再单独执行 `build_pkg.sh`。
65 70 
66-## 3. 快速上手71+### 3.2 手动构建&安装
67- 
68-### 3.1 工具构建&安装
69 72 
70```bash73```bash
71# 1. 创建工作目录74# 1. 创建工作目录
@@ -78,7 +81,7 @@ git clone https://gitcode.com/cann/hcomm.git
78 81 
79# 3. 安装第三方依赖82# 3. 安装第三方依赖
80sudo apt-get update83sudo apt-get update
81-sudo apt install build-essential cmake libsqlite3-dev rdma-core libibverbs-dev pkg-config gcc-aarch64-linux-gnu g++-aarch64-linux-gnu qemu-user-static binfmt-support84+sudo apt install build-essential cmake libsqlite3-dev libboost-all-dev rdma-core libibverbs-dev pkg-config gcc-aarch64-linux-gnu g++-aarch64-linux-gnu qemu-user-static binfmt-support
82 85 
83# 4. 编译HCCL-VM工具，下载hcomm代码之后，工具源码所在路径：/home/workspace/hcomm/test/hccl_vm86# 4. 编译HCCL-VM工具，下载hcomm代码之后，工具源码所在路径：/home/workspace/hcomm/test/hccl_vm
84cd /home/workspace/hcomm/test/hccl_vm87cd /home/workspace/hcomm/test/hccl_vm
@@ -88,13 +91,13 @@ export HCOMM_CODE_HOME=/home/workspace/hcomm
88bash ./build.sh --full91bash ./build.sh --full
89```92```
90 93 
91-### 3.2 使用示例94+### 3.3 使用示例
92 95 
93-#### 3.2.1 环境配置96+#### 3.3.1 环境配置
94 97 
95请参照[hccl_rootinfo文件内容](#47-hccl_rootinfojson文件)，创建并配置hccl_rootinfo.json文件。98请参照[hccl_rootinfo文件内容](#47-hccl_rootinfojson文件)，创建并配置hccl_rootinfo.json文件。
96 99 
97-#### 3.2.2 CCU模式100+#### 3.3.2 CCU模式
98 101 
991. 环境变量配置。1021. 环境变量配置。
100 103 
@@ -109,6 +112,7 @@ export HCCL_OP_EXPANSION_MODE="CCU_SCHED"
109 112 
1102. 执行1132. 执行
111 114 
115+ 
112```bash116```bash
113# 需要进入到新的bin文件目录下执行hccl-vm117# 需要进入到新的bin文件目录下执行hccl-vm
114cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin118cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
@@ -131,17 +135,16 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
131```135```
132 136 
1333. 验证hccl_test用例运行结果1373. 验证hccl_test用例运行结果
134- 
135[Runner结果查看](#491-runner插件结果) 138[Runner结果查看](#491-runner插件结果)
136[Checker结果查看](#492-checker插件结果)139[Checker结果查看](#492-checker插件结果)
137 140 
138-#### 3.2.3 AICPU模式141+#### 3.3.3 AICPU模式
139 142 
140AICPU展开模式需要将算法展开步骤放到设备侧执行，因此hccl-vm工具需要将HCCL的设备侧的符号编译并模拟执行。由于设备侧符号是ARM架构的，因此在X86环境上编译时需要借助交叉编译器，运行时需要借助QEMU实现AICPU模式的模拟运行。143AICPU展开模式需要将算法展开步骤放到设备侧执行，因此hccl-vm工具需要将HCCL的设备侧的符号编译并模拟执行。由于设备侧符号是ARM架构的，因此在X86环境上编译时需要借助交叉编译器，运行时需要借助QEMU实现AICPU模式的模拟运行。
141 144 
142设备侧符号使用hccl和hcomm的源码编译，为了保证Host与Device通信协议正确，需要同时编译Host侧的安装包并进行替换安装。145设备侧符号使用hccl和hcomm的源码编译，为了保证Host与Device通信协议正确，需要同时编译Host侧的安装包并进行替换安装。
143 146 
144-1. HCCL设备侧符号编译、安装、拷贝等。147+1. HCCL设备侧符号编译、安装、拷贝等（若已用 [3.1 一键安装](#31-一键安装)，此步已自动完成，可跳过）。
145 148 
146```bash149```bash
147cd /home/workspace/hcomm/test/hccl_vm/150cd /home/workspace/hcomm/test/hccl_vm/
@@ -184,7 +187,7 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
184 187 
1854. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)1884. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)
186 189 
187-#### 3.2.4 AIV模式190+#### 3.3.4 AIV模式
188 191 
1891. 环境变量配置。1921. 环境变量配置。
190 193 
@@ -222,11 +225,11 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
222 225 
2233. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)2263. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)
224 227 
225-### 3.3 Pytorch用例示例228+### 3.4 Pytorch用例示例
226 229 
227暂不支持。230暂不支持。
228 231 
229-### 3.4 hccl代码修改验证示例232+### 3.5 hccl代码修改验证示例
230 233 
231若您修改了CANN的算子包代码，如新增了算法类型，为保证您的修改生效，需要按照如下步骤操作执行。build_pkg.sh脚本帮助用户执行编包、装包、拷贝Device侧依赖符号，执行前需设置环境变量:234若您修改了CANN的算子包代码，如新增了算法类型，为保证您的修改生效，需要按照如下步骤操作执行。build_pkg.sh脚本帮助用户执行编包、装包、拷贝Device侧依赖符号，执行前需设置环境变量:
232 235 
@@ -241,7 +244,7 @@ export HCOMM_CODE_HOME=/home/workspace/hcomm
241 244 
2421. 若您修改了CANN hccl仓代码，请执行bash build_pkg.sh --install hccl。2451. 若您修改了CANN hccl仓代码，请执行bash build_pkg.sh --install hccl。
2432. 若您修改了CANN hcomm仓代码，请执行bash build_pkg.sh --install hcomm。2462. 若您修改了CANN hcomm仓代码，请执行bash build_pkg.sh --install hcomm。
244-3. 参考[使用示例](#32-使用示例)步骤，重新运行用例。247+3. 参考[使用示例](#33-使用示例)步骤，重新运行用例。
245 248 
246---249---
247 250 
@@ -360,6 +363,7 @@ links:
360 device_to_switch_links:363 device_to_switch_links:
361 # 如下示例表示：device0到device15都通过die0的port连接到交换机。结合portGroup可知，device0到device15都通过portGroup[0/4, 0/5, 0/6, 0/7]连接到交换机。364 # 如下示例表示：device0到device15都通过die0的port连接到交换机。结合portGroup可知，device0到device15都通过portGroup[0/4, 0/5, 0/6, 0/7]连接到交换机。
362 - {die_id: 0, devices_range: [0, 15]}365 - {die_id: 0, devices_range: [0, 15]}
366+ 
363```367```
364 368 
365**字段说明**：369**字段说明**：
@@ -396,6 +400,7 @@ server_list:
396 - {super_pod_id: 1, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}400 - {super_pod_id: 1, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
397 - {super_pod_id: 2, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}401 - {super_pod_id: 2, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
398 - {super_pod_id: 3, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}402 - {super_pod_id: 3, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
403+ 
399```404```
400 405 
401上述配置文件描述了一个包含4个超节点，32个Server，共128个NPU卡的集群拓扑。其中，每个Server/Pod采用ascend950_server_topo_normal拓扑类型。406上述配置文件描述了一个包含4个超节点，32个Server，共128个NPU卡的集群拓扑。其中，每个Server/Pod采用ascend950_server_topo_normal拓扑类型。
Dtest/hccl_vm/README_en.md+0-814
@@ -1,814 +0,0 @@
1-# HCCL-VM User Guide
2- 
3-## 1. Overview
4- 
5-HCCL-VM is a virtual execution environment for high-performance collective communication targeting Huawei Ascend NPU cards. This tool enables the development and functional verification of HCCL collective communication operators without real Ascend hardware.
6- 
7-![hccl-vm GIF](docs/hccl-vm.gif)
8- 
9-## 2. Prerequisites
10- 
11-| Dependency | Version Requirement |
12-|------------|---------------------|
13-| System Architecture | x86_64 Ubuntu 22.04 or later |
14-| Specification | Ascend950, for others refer to [Tool Specification Constraints](#45-tool-specification-constraints) |
15- 
16-### 2.1 CANN Package Installation
17- 
18-Install the latest CANN Toolkit development package and CANN ops operator package [Download Link](https://ascend.devcloud.huaweicloud.com/artifactory/cann-run-mirror/software/master/)
19- 
20-```bash
21-# Ensure the installation packages have executable permissions
22-chmod +x Ascend-cann-toolkit_9.1.0_linux-x86_64.run
23-chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run
24-# Installation commands
25-./Ascend-cann-toolkit_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend
26-./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend
27-```
28- 
29-### 2.2 hccl_test Compilation
30- 
31-hccl_test is the official HCCL performance testing tool provided by Ascend. See [HCCL Performance Testing Tool](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html). HCCL-VM supports running hccl_test cases in a virtual environment. Please first refer to the [hccl_test Case Build](#42-hccl-test-case-build) section to compile the test case binary.
32- 
33-Note: Optional. PyTorch test cases will be supported in the future.
34- 
35----
36- 
37-## One-Click Installation
38- 
39-Complete dependency installation, source code retrieval, CANN detection, and compilation in one line (default `campus-2026` profile):
40- 
41-```bash
42-curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/competition%2Fcampus-2026/test/hccl_vm/hccl_vm_installer | bash
43-```
44- 
45-You can also download and run locally (for review or offline distribution): `bash hccl_vm_installer`. Append parameters with `... | bash -s -- --workspace /root/hvm`.
46- 
47-**Prerequisites**: x86_64 Linux; the toolchain must meet hcomm build.md requirements — gcc/g++ 7.3.0–13.3.x, cmake ≥ 3.16.0 (applies to both host and aarch64 cross-compilers). Ubuntu 22.04 / 24.04 work out of the box. Later versions with default gcc (14/15) exceeding the range will trigger a warning; the script continues, but a compliant environment is recommended.
48- 
49-**CANN**: The script only probes for CANN in the working directory `<workspace>/Ascend` (or the path specified by `--ascend-path`). If not found, it automatically downloads and installs the matching version to that location, with consistent behavior for both root and regular users. `--offline` only checks, never downloads. In offline environments without public internet, it falls back to printing instructions for self-provided CANN.
50- 
51-**hccl_test**: By default, OpenMPI and the hccl_test performance testing tool are also compiled. Use `--skip-hccl-test` to disable.
52- 
53-**Common Parameters**:
54-- `--profile <name>`: Profile (default `campus-2026`, use `--list-profiles` to list all)
55-- `--workspace <path>`: Working directory for source code, compilation, and artifacts (default: current directory)
56-- `--ascend-path <path>`: Specify the CANN directory; reuse if exists, install if not
57-- `--reinstall-cann`: Re-download and overwrite existing CANN (use when version mismatch; kept by default)
58-- `--offline`: Use existing CANN only, never download
59-- `--skip-hccl-test`: Skip hccl_test compilation
60-- `-h`: Full help
61- 
62-After completion, the tool is located at `<workspace>/hcomm/test/hccl_vm/hccl_vm_install/bin/hccl-vm`. Delete the working directory to clean up all tool artifacts (system dependencies installed via apt require manual `apt remove`). This tool does not modify CANN.
63- 
64----
65- 
66-## 3. Quick Start
67- 
68-### 3.1 Tool Build & Installation
69- 
70-```bash
71-# 1. Create working directory
72-mkdir -p /home/workspace
73-cd /home/workspace
74- 
75-# 2. Download dependency source code
76-git clone https://gitcode.com/cann/hccl.git
77-git clone https://gitcode.com/cann/hcomm.git
78- 
79-# 3. Install third-party dependencies
80-sudo apt-get update
81-sudo apt install build-essential cmake libsqlite3-dev rdma-core libibverbs-dev pkg-config gcc-aarch64-linux-gnu g++-aarch64-linux-gnu qemu-user-static binfmt-support
82- 
83-# 4. Compile the HCCL-VM tool. After downloading the hcomm code, the tool source is at: /home/workspace/hcomm/test/hccl_vm
84-cd /home/workspace/hcomm/test/hccl_vm
85-source /home/workspace/Ascend/cann/set_env.sh
86-export HCCL_CODE_HOME=/home/workspace/hccl
87-export HCOMM_CODE_HOME=/home/workspace/hcomm
88-bash ./build.sh --full
89-```
90- 
91-### 3.2 Usage Examples
92- 
93-#### 3.2.1 Environment Configuration
94- 
95-Refer to [hccl_rootinfo File Contents](#47-hccl_rootinfojson-file) to create and configure the `hccl_rootinfo.json` file.
96- 
97-#### 3.2.2 CCU Mode
98- 
99-1. Environment variable configuration.
100- 
101-```bash
102-# Enter the tool installation directory
103-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install
104-source /home/workspace/Ascend/cann/set_env.sh
105-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
106-export RANK_TABLE_FILE=$(pwd)/data/ranktable.json
107-export HCCL_OP_EXPANSION_MODE="CCU_SCHED"
108-```
109- 
110-2. Execution
111- 
112-```bash
113-# Navigate to the new bin directory to execute hccl-vm
114-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
115- 
116-# Select the Ascend cluster topology configuration file, start the tool, initialize the cluster environment, and enter the tool command line
117-./hccl-vm start ascend950_cluster_32_server_normal.yaml
118- 
119-# To enable the runner plugin (optional)
120-(hvm)$> hccl-vm plugin install @runner
121- 
122-# Select the communication domain configuration file for this operator execution (run hccl_test in a cluster with 1 super node, 1 Server, 1 NPU)
123-(hvm)$> hccl-vm mock-comm 112
124-(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
125- 
126-# Execute checker validation
127-(hvm)$> hccl-vm plugin run @checker
128- 
129-# Exit the tool terminal
130-(hvm)$> exit
131-```
132- 
133-3. Verify hccl_test case execution results
134- 
135-[Runner Result Viewing](#491-runner-plugin-results)
136-[Checker Result Viewing](#492-checker-plugin-results)
137- 
138-#### 3.2.3 AICPU Mode
139- 
140-AICPU expansion mode requires executing algorithm expansion steps on the device side. Therefore, the hccl-vm tool needs to compile and simulate execution of HCCL's device-side symbols. Since device-side symbols use the ARM architecture, a cross-compiler is needed on x86 environments for compilation, and QEMU is required for runtime simulation of AICPU mode execution.
141- 
142-Device-side symbols are compiled using hccl and hcomm source code. To ensure correct Host-Device communication protocol, the Host-side installation package must also be compiled and replaced.
143- 
144-1. HCCL device-side symbol compilation, installation, copying, etc.
145- 
146-```bash
147-cd /home/workspace/hcomm/test/hccl_vm/
148-bash ./build_pkg.sh
149-```
150- 
151-2. Environment variable configuration.
152- 
153-```bash
154-# Enter the tool installation directory
155-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install
156-source /home/workspace/Ascend/cann/set_env.sh
157-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
158-export RANK_TABLE_FILE=$(pwd)/data/ranktable.json
159-export HCCL_OP_EXPANSION_MODE="AI_CPU"
160-```
161- 
162-3. Execution
163- 
164-```bash
165-# Navigate to the new bin directory to execute hccl-vm
166-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
167- 
168-# Select the Ascend cluster topology configuration file, start the tool, initialize the cluster environment, and enter the tool command line
169-./hccl-vm start ascend950_cluster_32_server_normal.yaml
170- 
171-# To enable the runner plugin (optional)
172-(hvm)$> hccl-vm plugin install @runner
173- 
174-# Select the communication domain configuration file for this operator execution (run hccl_test in a cluster with 1 super node, 1 Server, 1 NPU)
175-(hvm)$> hccl-vm mock-comm 112
176-(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
177- 
178-# Execute checker validation
179-(hvm)$> hccl-vm plugin run @checker
180- 
181-# Exit the tool terminal
182-(hvm)$> exit
183-```
184- 
185-4. Verify hccl_test case execution results [Runner Result Viewing](#491-runner-plugin-results) [Checker Result Viewing](#492-checker-plugin-results)
186- 
187-#### 3.2.4 AIV Mode
188- 
189-1. Environment variable configuration.
190- 
191-```bash
192-# Enter the tool installation directory
193-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install
194-source /home/workspace/Ascend/cann/set_env.sh
195-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
196-export RANK_TABLE_FILE=$(pwd)/data/ranktable.json
197-export HCCL_OP_EXPANSION_MODE="AIV"
198-```
199- 
200-2. Execution
201- 
202-```bash
203-# Navigate to the new bin directory to execute hccl-vm
204-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
205- 
206-# Select the Ascend cluster topology configuration file, start the tool, initialize the cluster environment, and enter the tool command line
207-./hccl-vm start ascend950_cluster_32_server_normal.yaml
208- 
209-# To enable the runner plugin (optional)
210-(hvm)$> hccl-vm plugin install @runner
211- 
212-# Select the communication domain configuration file for this operator execution (run hccl_test in a cluster with 1 super node, 1 Server, 1 NPU)
213-(hvm)$> hccl-vm mock-comm 112
214-(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
215- 
216-# Execute checker validation
217-(hvm)$> hccl-vm plugin run @checker
218- 
219-# Exit the tool terminal
220-(hvm)$> exit
221-```
222- 
223-3. Verify hccl_test case execution results [Runner Result Viewing](#491-runner-plugin-results) [Checker Result Viewing](#492-checker-plugin-results)
224- 
225-### 3.3 PyTorch Example
226- 
227-Not yet supported.
228- 
229-### 3.4 hccl Code Modification Verification Example
230- 
231-If you modify CANN operator package code, such as adding a new algorithm type, follow these steps to apply your changes. The `build_pkg.sh` script helps you build, install, and copy device-side dependency symbols. Set the environment variables before execution:
232- 
233-```bash
234-# Assume your CANN installation directory is: /home/workspace/Ascend
235-source /home/workspace/Ascend/cann/set_env.sh
236-# Configure hccl code repository path
237-export HCCL_CODE_HOME=/home/workspace/hccl
238-# Configure hcomm code repository path
239-export HCOMM_CODE_HOME=/home/workspace/hcomm
240-```
241- 
242-1. If you modified the CANN hccl repository code, run `bash build_pkg.sh --install hccl`.
243-2. If you modified the CANN hcomm repository code, run `bash build_pkg.sh --install hcomm`.
244-3. Refer to the [Usage Examples](#32-usage-examples) section and re-run the test cases.
245- 
246----
247- 
248-## 4. Detailed Guide
249- 
250-### 4.1 Tool Environment Variable Configuration
251- 
252-**HCCL-VM Environment Variables**:
253- 
254-| Environment Variable | Purpose | Example |
255-|----------------------|---------|---------|
256-| `HCCL_CODE_HOME` | Specifies the HCCL source code path for HCCL-VM compilation. Not configured by default. | `export HCCL_CODE_HOME=/home/workspace/hccl` |
257-| `HCOMM_CODE_HOME` | Specifies the HCOMM source code path for HCCL-VM compilation. Not configured by default. | `export HCOMM_CODE_HOME=/home/workspace/hcomm` |
258-| `HCCLVM_ENABLE_DUMP_DATA` | Enables Runner plugin to dump input & output data. When enabled, each operator's input & output data is dumped to the `all_rank_input_output.txt` file during test execution. | `export HCCLVM_ENABLE_DUMP_DATA=1` to enable, `export HCCLVM_ENABLE_DUMP_DATA=0` to disable |
259- 
260-### 4.2 HCCL-Test Case Build
261- 
262-The hccl_test source code is located in the CANN package installation directory. It supports compilation and execution in both OpenMPI and MPICH environments. See [OpenMPI and MPICH Environment Case Execution Differences](#48-differences-in-running-cases-between-openmpi-and-mpich-environments) for runtime differences. This guide uses OpenMPI as an example.
263- 
264-#### 4.2.1 OpenMPI Environment Compilation
265- 
266-1. Install OpenMPI
267- 
268-```bash
269-sudo apt-get update
270-sudo apt install openmpi-bin libopenmpi-dev
271-```
272- 
273-2. Compile hccl_test
274- 
275-```bash
276-# Change CANN installation directory permissions
277-chmod -R 755 /home/workspace/Ascend
278- 
279-# Enter the hccl_test source code directory
280-cd /home/workspace/Ascend/cann/tools/hccl_test
281- 
282-# Set CANN environment variables
283-source /home/workspace/Ascend/cann/set_env.sh
284- 
285-# Temporarily modify the Makefile script
286-if ! grep -q '\-lmpi_cxx' Makefile; then
287- sed -i 's/-lmpi/-lmpi -lmpi_cxx/g' Makefile
288-fi
289- 
290-# Compile hccl_test cases
291-MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH}
292-```
293- 
294-#### 4.2.2 MPICH Environment Compilation
295- 
296-Assume the mpich path is: `/usr/lib/mpich`.
297- 
298-```bash
299-# Enter the hccl_test source code directory
300-cd /home/workspace/Ascend/cann/tools/hccl_test
301- 
302-# Set CANN environment variables
303-source /home/workspace/Ascend/cann/set_env.sh
304- 
305-# Configure environment variables
306-export LD_LIBRARY_PATH=/usr/lib/mpich/lib/:${ASCEND_HOME_PATH}/lib64/:${ASCEND_HOME_PATH}/x86_64-linux/devlib:$LD_LIBRARY_PATH
307- 
308-# Compile hccl_test cases
309-make MPI_HOME=/usr/lib/mpich/ ASCEND_DIR=${ASCEND_HOME_PATH}
310-```
311- 
312-### 4.3 Ascend Cluster Topology Configuration File Description
313- 
314-#### 4.3.1 Server/Pod Topology Configuration File Description
315- 
316-An Ascend cluster topology is composed of one or more Server/Pod sub-topologies combined according to CLOS hierarchical network rules. Therefore, before generating a cluster topology, users need to confirm the topology type of each Server/Pod.
317-Users can either use predefined topology types provided by the HCCL-VM tool or define custom Server/Pod topology types according to the configuration file format requirements.
318- 
319-Describing the topological network relationship of a Server/Pod mainly includes the following aspects:
320- 
321-- **Port Allocation Table**: Describes the physical port configuration of an NPU card, such as NPU-to-NPU direct ports (P2P), NPU out-of-chassis ports (P2NET), etc.
322-- **Link Configuration Table**: Describes the connection relationships among all NPU cards within a Server/Pod, such as full mesh.
323-- **PortBound**: Describes the binding relationship of certain ports on an NPU card, where multiple ports are bound into a PortGroup.
324- 
325-```yaml
326-type: "server_intra_links"
327-name: "ascend950_links_topo_demo"
328-description: "ascend950 chip standard topology connection relationship description file"
329- 
330-soc_version: "Ascend950"
331-device_num: 16
332- 
333-device_ports_allocate_map:
334- # port allocation table: 0: unused, 1: device direct connect, 2: device to switch, 3: d2h port
335- # portId: 0 1 2 3 4 5 6 7 8
336- - {die_id: 0, pin_map: [1, 1, 1, 0, 2, 2, 2, 2, 3]} # die0
337- - {die_id: 1, pin_map: [0, 0, 0, 0, 0, 0, 0, 0, 0]} # die1
338- 
339-# port_group: describes which ports are merged into a portGroup. Ports in the same portGroup share the same IP address.
340-port_group:
341- - {layer: 0, ports: ["0/4", "0/5", "0/6", "0/7"]}
342- 
343-links:
344- # ── Method: every 8 devices form full interconnect ──
345- - link_mode: "fullmesh"
346- connections:
347- # The following example shows that die0 devices 0, 1, 2, 3 are all fully connected via die0 ports
348- - {die_id: 0, devices_range: [0, 3]}
349- - {die_id: 0, devices_range: [4, 7]}
350- - {die_id: 0, devices_range: [8, 11]}
351- - {die_id: 0, devices_range: [12, 15]}
352-
353- #- link_mode: "enum"
354- # device_to_device_links:
355- # The following example shows that device 0 and 1 both connect via die0 ports to devices 1, 3, 5, 7 on die1 ports respectively.
356- # i.e.: device0 connects to device1, device3, device5, device7; device1 connects to device3, device5, device7
357- # - {src_die_id: 1, src_local_id_range: [0, 2], dst_die_id: 1, dst_local_id_range: [1, 3, 5, 7]}
358- 
359- - link_mode: "enum"
360- device_to_switch_links:
361- # The following example shows that devices 0 to 15 all connect to the switch via die0 ports. Combined with portGroup, device0 to device15 all connect to the switch via portGroup[0/4, 0/5, 0/6, 0/7].
362- - {die_id: 0, devices_range: [0, 15]}
363-```
364- 
365-**Field Descriptions**:
366- 
367-- **soc_version**: Chip model, e.g., `Ascend950`.
368-- **device_num**: Total number of devices, determined by chip model and topology type.
369-- **device_ports_allocate_map**: Port allocation table describing each die's port configuration. 1 represents device direct connect ports, 2 represents device-to-switch ports, 3 represents d2h ports.
370-- **port_group**: Describes which ports are merged into a portGroup. Ports in the same portGroup share the same IP address. Ports not configured default to one port per portGroup.
371-- **links**: Link configuration table describing the connection relationships among all NPU cards within a Server/Pod, and between NPUs and switches.
372- - **NPU direct connections**: The tool provides two methods for configuring NPU direct connections:
373- - **link_mode == "fullmesh"**: Indicates all devices are fully connected based on one die's ports. New typical connection modes can be added as new link_mode types, such as "ring".
374- - **link_mode == "enum"**: Enumeration method. When the NPU connection method within a Server/Pod is complex, all link relationships can be described through enumeration.
375- - **NPU-to-switch connections**: Users can configure NPU-to-switch connection relationships using the enumeration method.
376-- **device_to_device_links**: Describes NPU-to-NPU connection relationships.
377-- **device_to_switch_links**: Describes NPU-to-switch connection relationships.
378- 
379-#### 4.3.2 Cluster Topology Configuration File Description
380- 
381-An Ascend cluster network is composed of one or more Server/Pod sub-topologies combined according to CLOS hierarchical network rules. Users can choose different Server/Pod topology types based on cluster size and requirements.
382- 
383-Users can define custom cluster topology configuration files according to the following format:
384- 
385-```yaml
386-name: "ascend950_cluster_32_server_normal"
387-description: "Ascend950 normal networking: 32 super nodes, 1 server per super node"
388- 
389-# Total number of super nodes
390-super_node_num: 4
391-# Total number of servers/pods
392-server_num: 32
393-server_list:
394- # 0-7 servers: all use ascend950_server_topo_normal topology type
395- - {super_pod_id: 0, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
396- - {super_pod_id: 1, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
397- - {super_pod_id: 2, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
398- - {super_pod_id: 3, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"}
399-```
400- 
401-The configuration file above describes a cluster topology with 4 super nodes, 32 Servers, and a total of 128 NPU cards. Each Server/Pod uses the `ascend950_server_topo_normal` topology type.
402- 
403-**Field Descriptions**:
404- 
405-- **super_node_num**: Total number of super nodes.
406-- **server_num**: Total number of servers/pods.
407-- **server_list**: Configuration information for each Server/Pod, including super node ID, device ID range, chip model, and Server/Pod topology configuration file path.
408- 
409-#### 4.3.3 Communication Domain Configuration File Description
410- 
411-In an Ascend cluster environment, users need to select different communication domain configuration files based on the communication domain required by the operator to be executed.
412- 
413-The tool provides the `hccl-vm mock-comm` command to read and configure operator communication domain configuration files. The communication domain configuration file format is yaml, located at `hccl_vm_install/config/topo_meta`. If the corresponding configuration file does not exist in the directory, the user needs to create one first.
414- 
415-The hccl-vm tool supports asymmetric topology communication domain configuration, as shown below:
416- 
417-```yaml
418-# 1. Global statistics: podNum, serNum, rankNum are all less than 1024
419-meta:
420- podNum: 1 # Total number of super nodes
421- serNum: 2 # Total number of servers
422- rankNum: 6 # Total number of ranks
423- 
424-# 2. Detailed topology structure
425-topology:
426- - podId: 0
427- servers:
428- - serId: 0
429- # Local IDs of ranks running on this server
430- ranks: [0, 2]
431- - serId: 1
432- # Local IDs of ranks running on this server
433- ranks: [1, 3, 5, 7]
434-```
435- 
436-**Notes**:
437- 
438-- When configuring the communication domain, the tool regenerates the `topo.json` and `ranktable.json` files based on the specified communication domain configuration ID.
439-- In the communication domain configuration yaml file above, the `ranks` field represents the list of local IDs (i.e., device physical IDs) of ranks actually running on each server.
440- 
441-#### 4.3.4 topo and ranktable.json File Description
442- 
443-The `topo.json` and `ranktable.json` files do not need to be created manually. The tool automatically generates them based on the following information:
444- 
445-- **Topology configuration ID**: The ID specified by the user at startup (e.g., 112, 113, etc.)
446-- **Chip type**: The chip type automatically identified based on the runtime environment.
447- 
448-Although the configuration files are automatically generated by the tool, understanding their structure helps in understanding topology configuration.
449- 
450-**topo.json Structure**:
451- 
452-`topo.json` describes the connection relationships among all devices within a server:
453- 
454-```json
455-{
456- "server": {
457- "device_count": 8,
458- "groups": [
459- {
460- "group_id": 0,
461- "device_start": 0,
462- "device_count": 8,
463- "topo_layout": "1D"
464- }
465- ]
466- },
467- "ports": [
468- {
469- "ccu": "die0",
470- "port_pattern": "0/{0-6}",
471- "protocol": "HCCS",
472- "func_id": 2,
473- "usage": "peer2peer",
474- "ip_binding": "independent"
475- },
476- {
477- "ccu": "die0",
478- "port_pattern": "0/7,0/8",
479- "protocol": "ROCE",
480- "func_id": 3,
481- "usage": "peer2net",
482- "ip_binding": "independent"
483- }
484- ],
485- "links": [
486- {
487- "net_layer": 0,
488- "link_type": "PEER2PEER",
489- "topo_type": "1DMESH",
490- "ccu": "die0",
491- "port_pattern": "0/{0-6}",
492- "connect_pattern": "full_mesh",
493- "group_id": 0
494- },
495- {
496- "net_layer": 1,
497- "link_type": "PEER2NET",
498- "topo_type": "CLOS",
499- "ccu": "die0",
500- "port_pattern": "0/7,0/8",
501- "connect_pattern": "all_to_net",
502- "group_id": 0
503- }
504- ]
505-}
506-```
507- 
508-**Field Descriptions**:
509- 
510-- `server.device_count`: Total number of devices.
511-- `server.groups`: Device grouping information.
512-- `ports`: Port configuration.
513- - `usage`: Port purpose (`peer2peer` for inter-device connections, `peer2net` for external connections)
514-- `links`: Link configuration.
515- - `link_type`: Link type (`PEER2PEER` or `PEER2NET`)
516- - `topo_type`: Topology type (`1DMESH`, `CLOS`, etc.)
517- 
518-**ranktable.json Structure**:
519- 
520-`ranktable.json` describes the device and IP mapping for the current run:
521- 
522-```json
523-{
524- "version": "1.0",
525- "server_count": 1,
526- "device_count": 8,
527- "server_list": [
528- {
529- "server_id": 0,
530- "device_id": 0,
531- "device_ip": "192.168.1.10",
532- "port": "2222"
533- }
534- ]
535-}
536-```
537- 
538-**Field Descriptions**:
539- 
540-- `server_count`: Number of servers.
541-- `device_count`: Total number of devices.
542-- `server_list`: Server and device list.
543- - `device_ip`: Device IP address.
544- - `port`: Device port number.
545- 
546-### 4.4 hccl_config.sh File Description
547- 
548-The `hccl_config.sh` file contains the environment variable configuration required for running HCCL_Test cases. The environment variables are consistent with those used for Hccl_Test cases on real hardware.
549-Users need to modify the `hccl_config.sh` script according to their own use cases and requirements to configure the HCCL test case runtime environment variables.
550- 
551-```bash
552-#!/bin/bash
553-# hccl_config.sh - HCCL environment variable configuration
554- 
555-remove_files_by_prefix() {
556- if [ "$#" -ne 1 ]; then
557- echo "Usage: remove_files_by_prefix <prefix>" >&2
558- return 2
559- fi
560- 
561- local prefix="$1"
562- if [ -z "$prefix" ]; then
563- return 0
564- fi
565- 
566- shopt -s nullglob
567- local any_deleted=0
568- for f in "${prefix}"*; do
569- if [ -f "$f" ]; then
570- rm -f -- "$f" && any_deleted=1
571- fi
572- done
573- shopt -u nullglob
574- 
575- # Return 0 regardless of whether files were deleted, to ensure script continues
576- return 0
577-}
578- 
579-# Clean up redundant files in the data/ directory (temporary files generated in CCU mode)
580-cd "${HCCL_VM_INSTALL_DIR}/data" 2>/dev/null && {
581- remove_files_by_prefix "sqe_info_rank_"
582- remove_files_by_prefix "mc_instr_info_rank_"
583- rm -f "all_rank_input_output.txt"
584- cd "${HCCL_VM_INSTALL_DIR}"
585-}
586- 
587-# Set CANN environment variables
588-source /home/workspace/Ascend/cann/set_env.sh
589- 
590-# Disable HCCL heartbeat function
591-export HCCL_DFS_CONFIG=cluster_heartbeat:off
592- 
593-# Set HCCL-VM installation path, inferred from the script itself (compatible with bin/ and script/ subdirectories)
594-_INSTALL_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
595-case "$(basename "${_INSTALL_SCRIPT_DIR}")" in
596- bin|script)
597- export HCCL_VM_INSTALL_DIR="$(dirname "${_INSTALL_SCRIPT_DIR}")"
598- ;;
599- *)
600- export HCCL_VM_INSTALL_DIR="${_INSTALL_SCRIPT_DIR}"
601- ;;
602-esac
603-unset _INSTALL_SCRIPT_DIR
604- 
605-# Configure LD_LIBRARY_PATH
606-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
607- 
608-# Set ranktable.json file path (consistent with mock-comm generation path)
609-export RANK_TABLE_FILE=${HCCL_VM_INSTALL_DIR}/data/ranktable.json
610- 
611-# Set log level
612-export ASCEND_GLOBAL_LOG_LEVEL=1
613- 
614-# Enable log output to stdout
615-export ASCEND_SLOG_PRINT_TO_STDOUT=1
616- 
617-# Set HCCL operation mode (CCU, AI_CPU, AIV, etc.)
618-export HCCL_OP_EXPANSION_MODE="CCU_SCHED"
619-# Or set HCCL runtime parameters (AI_CPU expansion mode) AI_CPU mode environment variables cannot be set simultaneously with other modes
620-# export HCCL_OP_EXPANSION_MODE="AI_CPU"
621-# Or set HCCL runtime parameters (AIV expansion mode) AIV mode environment variables cannot be set simultaneously with other modes
622-# export HCCL_OP_EXPANSION_MODE="AIV"
623- 
624-echo "HCCL-VM environment configured successfully!"
625-```
626- 
627-### 4.5 Tool Specification Constraints
628- 
629-**Supported Operator Types**:
630- 
631-The supported operator types include: allgather/allreduce/alltoall/reduce/reduce\_scatter/scatter/alltoallv.
632- 
633-**Supported Data Types**:
634- 
635-The HCCL-VM tool supports the following data types: int8/int16/int32/fp16/fp32/uint8/uint16/uint32/bfp16/hif8/fp8e4m3/fp8e5m2/fp8e8m0.
636- 
637-The HCCL-VM Runner plugin supports the following data types:
638- 
639-| ReduceOp | DataType |
640-|----------|------------------------------------|
641-| `ADD` | `int8/int16/int32/uint8` |
642-| `MIN` | `int8/int16/int32/uint8` |
643-| `MAX` | `int8/int16/int32/uint8` |
644- 
645-**Hardware Specifications**:
646- 
647-Currently, this tool only supports the Ascend950 chip. A single server supports a maximum of 8 cards. For more than 8 cards, cross-server execution is required.
648- 
649-### 4.6 HCCL-VM Plugin Features
650- 
651-#### 4.6.1 Runner Plugin
652- 
653-The Runner plugin simulates the execution of task sequences generated by HCCL operator orchestration and outputs data.
654-The simulation runner plugin is **disabled** by default during hccl_test case execution. After the hccl_test case calls the operator interface, it waits for the operator task to complete via the `aclrtSynchronizeStream` interface. The simulation runner tool waits until all ranks are in a waiting state, then starts simulating the execution of all rank tasks. After execution completes, it notifies each rank's test case to continue.
655- 
656-After test case execution, users can view the input buffer and output buffer data for each rank in the `all_rank_input_output.txt` file in the execution directory. This feature is disabled by default and can be enabled via the corresponding command before test execution.
657- 
658-**Installation and Uninstallation**:
659- 
660-The Runner plugin supports installation and uninstallation via the `hccl-vm plugin install/uninstall` command. The runner plugin must be installed after entering the hccl-vm tool command line and before executing the test case. Subsequent executions will then run the runner.
661- 
662-```bash
663-# Install runner plugin
664-(hvm)$> hccl-vm plugin install @runner
665- 
666-# Uninstall runner plugin
667-(hvm)$> hccl-vm plugin uninstall @runner
668-```
669- 
670-#### 4.6.2 Checker Plugin
671- 
672-The Checker plugin is an algorithm analyzer plugin. It forms a DAG from all tasks generated by HCCL and analyzes it to detect memory conflicts. By simulating execution of the DAG, it also detects semantic errors.
673- 
674-The Checker plugin is started manually by the user via a command.
675- 
676-The Checker plugin is currently in a transition period. Checker V3 is a refactored version of the original Checker, mainly improving validation performance. By default, the new Checker (Checker V3) is used. You can adjust this by modifying the configuration parameters in the Checker's `manifest.json` file.
677- 
678-```bash
679-# Configuration file located at /pathto/hccl_vm_install/plugin/checker/manifest.json
680- 
681-{
682- "name": "checker", // Checker plugin name
683- "version": "1.0.0", // Checker plugin version
684- "entry": "./checker", // Checker plugin startup command
685- "dependency": {
686- "min_core_version": "1.0.0"
687- },
688- "setting": { // Checker plugin configuration
689- "enable_new_checker": true, // Whether to enable the new Checker (Checker V3, enabled by default)
690- "enable_old_checker": false, // Whether to enable the old Checker (disabled by default)
691- "enable_insight_dump": false, // Whether to enable visualization data output (disabled by default, only supported by old Checker)
692- "enable_memory_snapshot_dump": false // Whether to enable visualization memory snapshot data output (disabled by default, only supported by old Checker, requires visualization data output "enable_insight_dump" to be enabled first)
693- }
694-}
695-```
696- 
697-### 4.7 hccl_rootinfo.json File
698- 
699-Currently, the tool uses the `ranktable.json` file for communication domain initialization. Therefore, the `hccl_rootinfo.json` file is only needed to provide the path to the `topo.json` file.
700-If the `hccl_rootinfo.json` file does not exist under the `/etc` path, users need to create it with the following content:
701- 
702-```json
703-{
704- "version": "2.0",
705- "topo_file_path": "/home/workspace/hcomm/test/hccl_vm/hccl_vm_install/data/topo.json"
706-}
707-```
708- 
709-### 4.8 Differences in Running Cases Between OpenMPI and MPICH Environments
710- 
711-Before running hccl_test cases, users can use the `which` command to determine which mpirun is being used in the current environment.
712- 
713-#### 4.8.1 Environment Variable Configuration Differences
714- 
715-OpenMPI is generally the default configuration in most environments. If using OpenMPI to run cases, no additional environment variable configuration is typically needed.
716-If using the MPICH environment to run cases, environment variables must be configured as follows:
717- 
718-```bash
719-# Configure mpich environment variables
720-export LD_LIBRARY_PATH=/usr/lib/mpich/lib/:${ASCEND_HOME_PATH}/lib64/:${ASCEND_HOME_PATH}/x86_64-linux/devlib:$LD_LIBRARY_PATH
721-export PATH=/usr/lib/mpich/bin:$PATH
722-```
723- 
724-#### 4.8.2 mpirun Command Parameter Differences
725- 
726-In the OpenMPI environment, run hccl_test cases as follows:
727- 
728-```bash
729-export HCCL_TEST_PATH=/home/workspace/Ascend/cann/tools/hccl_test
730-mpirun --allow-run-as-root --oversubscribe -np 2 ${HCCL_TEST_PATH}/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1
731-```
732- 
733-**Parameter Descriptions**:
734- 
735-- --allow-run-as-root: OpenMPI-specific parameter that allows MPI processes to run as the root user, for use in environments without root privileges.
736-- --oversubscribe: OpenMPI-specific parameter, removes CPU slot limits, allowing a single node to launch processes when the number of processes exceeds the number of logical CPU cores—i.e., running with oversubscription/excess allocation.
737-- -np 2: Specifies 2 processes, consistent with the number of nodes.
738- 
739-In the MPICH environment, run hccl_test cases as follows:
740- 
741-```bash
742-export HCCL_TEST_PATH=/home/workspace/Ascend/cann/tools/hccl_test
743-mpirun -np 2 ${HCCL_TEST_PATH}/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1
744-```
745- 
746-**Parameter Descriptions**:
747- 
748-- -np 2: Specifies 2 processes, consistent with the number of nodes.
749- 
750-### 4.9 Result Viewing
751- 
752-#### 4.9.1 Runner Plugin Results
753- 
754-If the runner plugin is installed via `hccl-vm plugin install @runner` in the hccl-vm terminal, the runner plugin is automatically triggered after operator execution completes. The final result depends on hccl_test verification. Users should check the redirected log file for `[error]` level logs and the final verification result:
755- 
756-```bash
757-data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result:
758-64 | 1000.00 | 0.00006 | success
759-```
760- 
761-#### 4.9.2 Checker Plugin Results
762- 
763-After executing `hccl-vm plugin run @checker` in the hccl-vm terminal, the Checker validation process and results are printed to the terminal. Users should check for `[error]` level logs and the final validation result:
764- 
765-```bash
766-[info][PID:144373][TID:144880][main.cc][RunChecker] [RunChecker] op[0] Checker Success.
767-```
768- 
769----
770- 
771-### 4.10 Large Memory Reuse (Check-Only Mode)
772- 
773-Check-only mode is used for scenarios where only Checker validation is needed in large-scale clusters. When enabled, a single large memory allocation of 200MB to 4GB reuses the same 4GB shared pool `HcclCommPool`, shared across all ranks and allowing mutual overwriting. This significantly reduces `/dev/shm` usage. In this mode, the content of large blocks is not guaranteed to be correct, making it suitable only for the Checker V3 validation pipeline that does not read buffer data. Do not enable this mode when numerically correct results are needed.
774- 
775-Check-only mode is a session-level switch. Append `--check-only` after the `start` subcommand to explicitly enable it. Without this flag, the default normal mode is used, where large blocks use real independent allocation with no correctness impact. Allocations smaller than 200MB always use real allocation. A single block exceeding 4GB in check-only mode results in a direct error. Check-only mode does not conflict with Runner, but if Runner is installed while check-only mode is enabled, large block reuse still takes effect and may overwrite Runner data. The tool prints a warning in this case.
776- 
777-```bash
778-# Start the tool with check-only mode enabled
779-./hccl-vm start ascend950_cluster_32_server_normal.yaml --check-only
780-```
781- 
782----
783- 
784-## 5 Appendix
785- 
786-### Open Source Third-Party Software Dependencies
787- 
788-When compiling this project, the following third-party open-source software is required. For offline compilation, download and rename the packages, then place them in the `third_party` directory under the project.
789- 
790-| Open Source Software | Version | Download URL |
791-|---------------------|---------|--------------|
792-| CLI11 | 2.2.0 | [cli11-2.2.0.tar.gz](https://raw.gitcode.com/src-openeuler/cli11/blobs/58c912141164a5c0f0139bfa91343fefe151d525/cli11-2.2.0.tar.gz) |
793-| json | 3.11.3 | [include.zip](https://gitcode.com/cann-src-third-party/json/releases/download/v3.11.3/include.zip) |
794-| spdlog | 1.11.0 | [spdlog-v1.11.0.tar.gz](https://raw.gitcode.com/src-openeuler/spdlog/blobs/c2dfb1aca26c607393665c836155613ff283de66/v1.11.0.tar.gz) |
795-| yaml-cpp | 0.8.0 | [yaml-cpp-0.8.0.tar.gz](https://raw.gitcode.com/src-openeuler/yaml-cpp/blobs/d1ead4fff417073b9cdbf98b8b55eb0efc00b0ba/yaml-cpp-0.8.0.tar.gz) |
796-| sqlite | 3.51.0 | [sqlite-amalgamation-3510300.zip](https://www.sqlite.org/2026/sqlite-amalgamation-3510300.zip) |
797-| googletest | 1.14.0 | [googletest-1.14.0.tar.gz](https://gitcode.com/cann-src-third-party/googletest/releases/download/v1.14.0/googletest-1.14.0.tar.gz) |
798- 
799-### Glossary
800- 
801-| Term | Description |
802-|------------|------------------------------------------------------------------|
803-| HCCL | Huawei Collective Communication Library |
804-| NPU | Neural Processing Unit |
805-| CANN | Compute Architecture for Neural Networks, Huawei Ascend AI processor software stack |
806-| MPI | Message Passing Interface |
807-| CCU | Collective Communication Unit |
808-| Topology | Device connection relationship |
809-| Rank | Process identifier in MPI |
810- 
811----
812- 
813-**Document Version**: v1.1.
814-**Last Updated**: 2026-06-30.
Mtest/hccl_vm/docs/checker_quick_intro.md+2-11
@@ -49,7 +49,6 @@ flowchart TD
49---49---
50 50 
51## 3 任务图51## 3 任务图
52- 
53任务图是 Checker 的核心数据结构，它用来表达一个算子执行所生成的任务节点及其依赖关系，任务图是在成图阶段基于 Checker 输入生成的52任务图是 Checker 的核心数据结构，它用来表达一个算子执行所生成的任务节点及其依赖关系，任务图是在成图阶段基于 Checker 输入生成的
54 53 
55### 3.1 节点与边54### 3.1 节点与边
@@ -100,7 +99,6 @@ flowchart LR
100### 3.2 任务图示例99### 3.2 任务图示例
101 100 
102#### 3.2.1 AICPU模式101#### 3.2.1 AICPU模式
103- 
104AICPU 模式下，任务图通常由 `RECORD`、`WAIT` 和 `TRANS_MEM` 这几类节点组合而成。下面给出一个典型的 2-rank、每个 rank 含两条 stream 的 `AllGather` 示例，按实际执行序列展示 stream 内顺序边，并用虚线表示 `RECORD` 到 `WAIT` 的同步依赖。102AICPU 模式下，任务图通常由 `RECORD`、`WAIT` 和 `TRANS_MEM` 这几类节点组合而成。下面给出一个典型的 2-rank、每个 rank 含两条 stream 的 `AllGather` 示例，按实际执行序列展示 stream 内顺序边，并用虚线表示 `RECORD` 到 `WAIT` 的同步依赖。
105 103 
106```mermaid104```mermaid
@@ -140,7 +138,6 @@ flowchart LR
140```138```
141 139 
142#### 3.2.2 CCU模式140#### 3.2.2 CCU模式
143- 
144在 CCU 模式下，Checker 会把 CCU 指令展开成 CCU 子图。下面沿用 2-rank `AllReduce` 的数据流，省略了 CCU 子图外部的同步操作，只保留 CCU 子图内部的任务序列。CCU 使用同步字段从 `notifyId` 变为 `cke` / `mask`，中间缓冲区使用的是 CCU 的 `MS` 类型。141在 CCU 模式下，Checker 会把 CCU 指令展开成 CCU 子图。下面沿用 2-rank `AllReduce` 的数据流，省略了 CCU 子图外部的同步操作，只保留 CCU 子图内部的任务序列。CCU 使用同步字段从 `notifyId` 变为 `cke` / `mask`，中间缓冲区使用的是 CCU 的 `MS` 类型。
145 142 
146```mermaid143```mermaid
@@ -168,7 +165,6 @@ flowchart LR
168```165```
169 166 
170#### 3.2.3 Graphviz 简易可视化167#### 3.2.3 Graphviz 简易可视化
171- 
172Checker 提供了任务图导出能力，用于把任务图输出成 Graphviz 的 `.dot` 文件。168Checker 提供了任务图导出能力，用于把任务图输出成 Graphviz 的 `.dot` 文件。
173 169 
174它导出的内容不只是“有哪些节点”，还会尽量把调试时常用的信息直接放进图里：170它导出的内容不只是“有哪些节点”，还会尽量把调试时常用的信息直接放进图里：
@@ -183,10 +179,10 @@ Checker 提供了任务图导出能力，用于把任务图输出成 Graphviz
183 179 
184> 拿到 `.dot` 文件后，可使用 `Microsoft VS Code` 相关的插件如 `Graphviz Interactive Preview` 来实现即时浏览。180> 拿到 `.dot` 文件后，可使用 `Microsoft VS Code` 相关的插件如 `Graphviz Interactive Preview` 来实现即时浏览。
185 181 
182+ 
186---183---
187 184 
188## 4. 单任务校验185## 4. 单任务校验
189- 
190此阶段主要检查单条任务的内存区间是否合法，并校验从流的头尾结构186此阶段主要检查单条任务的内存区间是否合法，并校验从流的头尾结构
191 187 
192### 4.1 内存区间 (MemSlice)188### 4.1 内存区间 (MemSlice)
@@ -210,10 +206,10 @@ MemSlice = { rankId, type, offset, len }
210 - `len` 表示本次内存访问的长度206 - `len` 表示本次内存访问的长度
211 - 访问区间采用半开表示，即 `[offset, offset + length)` 为本次访问的内存段207 - 访问区间采用半开表示，即 `[offset, offset + length)` 为本次访问的内存段
212 208 
209+ 
213单条任务内存区间的校验点如下：210单条任务内存区间的校验点如下：
214- `offset + length` 不能溢出 `uint64` 上界，否则会报错211- `offset + length` 不能溢出 `uint64` 上界，否则会报错
215- 同一任务内部的多个 MemSlice 在相同 `(rankId, memType)` 下不能有区间重叠，否则会报错212- 同一任务内部的多个 MemSlice 在相同 `(rankId, memType)` 下不能有区间重叠，否则会报错
216- 
217 ```mermaid213 ```mermaid
218 gantt214 gantt
219 title MemSlice 区间对比215 title MemSlice 区间对比
@@ -229,10 +225,8 @@ MemSlice = { rankId, type, offset, len }
229 Slice A 0x000-0x600 : crit, 0, 600225 Slice A 0x000-0x600 : crit, 0, 600
230 Slice B 0x400-0x800 : crit, 400, 800226 Slice B 0x400-0x800 : crit, 400, 800
231 ```227 ```
232- 
233- 不同 `type` 是相互独立的地址空间，同一 `offset` 在不同 `type` 下不视为重叠228- 不同 `type` 是相互独立的地址空间，同一 `offset` 在不同 `type` 下不视为重叠
234- `offset + length` 不能越过当前 `type` 地址空间的边界229- `offset + length` 不能越过当前 `type` 地址空间的边界
235- 
236 ```mermaid230 ```mermaid
237 gantt231 gantt
238 title MemSlice 边界检查232 title MemSlice 边界检查
@@ -254,7 +248,6 @@ MemSlice = { rankId, type, offset, len }
254从流用于执行算子的辅助任务，例如数据预搬运。在HCCL编程模型中主流通过同步任务 `RECORD -> WAIT` 触发从流，从流完成后，再通过另一组同步任务通知主流，所以从流必须满足固定的首尾结构：首任务为 `WAIT` && 末任务为 `RECORD`248从流用于执行算子的辅助任务，例如数据预搬运。在HCCL编程模型中主流通过同步任务 `RECORD -> WAIT` 触发从流，从流完成后，再通过另一组同步任务通知主流，所以从流必须满足固定的首尾结构：首任务为 `WAIT` && 末任务为 `RECORD`
255 249 
256下图为一个错误示例，标红的节点都是违规节点：250下图为一个错误示例，标红的节点都是违规节点：
257- 
258```mermaid251```mermaid
259flowchart LR252flowchart LR
260 START(["start"])253 START(["start"])
@@ -298,7 +291,6 @@ flowchart LR
298Checker会高效地校验每一对内存访问类任务节点，保证不发生漏报291Checker会高效地校验每一对内存访问类任务节点，保证不发生漏报
299 292 
300### 5.2 内存冲突示例293### 5.2 内存冲突示例
301- 
302下图为一个存在内存冲突的任务图示例：294下图为一个存在内存冲突的任务图示例：
303 295 
304```mermaid296```mermaid
@@ -359,7 +351,6 @@ flowchart LR
359| `node X, action=read/write` | 任务图中的节点 ID 与本次访问的读写类型。只要两条访问中至少一条是 `write`，就可能报冲突 |351| `node X, action=read/write` | 任务图中的节点 ID 与本次访问的读写类型。只要两条访问中至少一条是 `write`，就可能报冲突 |
360| `access range : [start,end)` | 这条访问自身覆盖的完整地址区间，不一定与 `Overlap range` 完全相同 |352| `access range : [start,end)` | 这条访问自身覆盖的完整地址区间，不一定与 `Overlap range` 完全相同 |
361| `task :` | 具体任务详情（任务类型、节点 ID、位置、src/dst 内存区间等） |353| `task :` | 具体任务详情（任务类型、节点 ID、位置、src/dst 内存区间等） |
362- 
363---354---
364 355 
365## 6. 语义校验356## 6. 语义校验
Dtest/hccl_vm/docs/checker_quick_intro_en.md+0-737
@@ -1,737 +0,0 @@
1-# Checker Quick Start Guide
2- 
3-This document introduces the basic concepts and processing flow of Checker. For error code specific scenarios, see [Error Code FAQ](faq/modules/checker_faq_en.md).
4- 
5-[toc]
6- 
7-## 1 What is Checker
8- 
9-Checker is a static verification tool. It does not intervene in operator execution. Instead, it reads the records left after operator execution, reconstructs the execution graph, and performs static analysis to determine whether the operator execution is logically correct.
10-- Checker input: operator information + task data of each rank + CCU instruction sequence
11-- Checker output: verification conclusion (success/failure) + error log
12- 
13----
14- 
15-## 2 Checker Overall Flow
16- 
17-### 2.1 Key Terms
18- 
19-| Term | Description |
20-|------|-------------|
21-| Communication domain | A group of communication members, describing the communication scope |
22-| Communication member | Usually referred to as a rank, the smallest logical entity participating in communication. Each rank is assigned a unique identifier called `rankId` |
23-| Communication operator | A collective communication operation, such as `AllReduce` or `AllGather`. Different communication algorithms may be used depending on network topology, data volume, hardware resources, etc. |
24-| Task | The core data structure of Checker, describing a record of an atomic operation for a rank, such as memory copy, Reduce, memory move, etc. |
25-| Task graph | The core data structure of Checker, used to express the task nodes generated by an operator execution and their dependency relationships |
26-| Node | Each node on the task graph is a task |
27-| Stream | A queue within a rank that executes tasks sequentially. Each task uses `streamId` to record which queue it belongs to |
28-| Task type | Different task types serve different purposes, such as memory copy, Reduce, or synchronization |
29- 
30-### 2.2 Checker Processing Flow
31- 
32-```mermaid
33-flowchart TD
34- INPUT["Input"]
35- GG["Graph Generation\nGenGraph"]
36- ST["Single Task Check\nSingleTaskCheck"]
37- MC["Memory Conflict Detection\nMemConflict"]
38- SC["Semantic Check\nSemanticCheck"]
39- INPUT --> GG --> ST --> MC --> SC
40-```
41- 
42-| Phase | Purpose |
43-|-------|---------|
44-| Graph Generation | Generate the task graph based on Checker input. In CCU mode, CCU instructions are converted into the task graph |
45-| Single Task Check | Check whether the memory range of a single task is valid, and verify the slave stream head/tail structure |
46-| Memory Conflict Detection | Check for unprotected memory overlaps between concurrent tasks |
47-| Semantic Check | Simulate the operator execution process and verify whether the final output meets operator expectations |
48- 
49----
50- 
51-## 3 Task Graph
52- 
53-The task graph is the core data structure of Checker, used to express the task nodes generated by an operator execution and their dependency relationships. It is generated during the graph generation phase based on Checker input.
54- 
55-### 3.1 Nodes and Edges
56- 
57-Each node on the task graph is a task with a specific task type indicating the operation it performs. Common types are:
58- 
59-| Task Type | Description | Core Fields |
60-|-----------|-------------|-------------|
61-| `TRANS_MEM` | Memory data copy | `srcRankId`, `srcOffset`<br>`dstRankId`, `dstOffset`<br>`len`, `type` |
62-| `BATCH_TRANS_MEM` | Batch memory data copy, containing multiple `(src -> dst)` relationships per node | `srcs[]`<br>`dsts[]` |
63-| `REDUCE` | Data reduce | `srcRankId`, `srcOffset`<br>`dstRankId`, `dstOffset`<br>`type`, `dataCount`, `dataType`, `reduceOp` |
64-| `BATCH_REDUCE` | Batch data reduce, containing multiple `(src -> dst)` reduce relationships per node | `srcs[][]`<br>`dsts[]`<br>`dataType`, `reduceOp` |
65-| `RECORD` / `WAIT` | Synchronization tasks, representing sending and waiting for synchronization signals respectively | `srcRankId` (sender)<br>`dstRankId` (waiter)<br>`notifyId` |
66- 
67-In addition to the real execution tasks above, the task graph also includes `START` / `END` virtual boundary nodes. They do not correspond to actual data copy or computation, but serve to mark the boundaries of the main graph, subgraphs, and Loop structures.
68- 
69-| Virtual Node Type | Supported `boundaryType` | Description |
70-|-------------------|--------------------------|-------------|
71-| `START` | `MAIN_GRAPH`, `CCU_SUB_GRAPH`, `AIV_SUB_GRAPH`, `LOOP` | Start boundary node. Marks the entry of the entire task graph, or the start of a CCU/AIV subgraph or Loop fragment |
72-| `END` | `CCU_SUB_GRAPH`, `AIV_SUB_GRAPH`, `LOOP` | End boundary node. Marks the end of a CCU/AIV subgraph or Loop fragment, and converges the tail nodes within the boundary |
73- 
74-Edges represent the execution order relationships between nodes. A directed edge means the tail node executes after the head node. Edges can be categorized as follows:
75- 
76-| Edge Type | Description |
77-|-----------|-------------|
78-| Sequential edge | Dependency edge connecting tasks in execution order within the same stream. Example: two sequential tasks on `rank0/stream0`, three sequential tasks on `rank1/stream0` |
79-| Synchronization edge | Dependency edge between synchronization task nodes. Example: a `WAIT` node needs to wait for a signal from a `RECORD` node before it can proceed. An edge from `RECORD` to `WAIT` represents this dependency |
80- 
81-```mermaid
82-flowchart LR
83- START(["start"])
84- 
85- subgraph SG0["rank0 / stream0"]
86- direction LR
87- R0T1["TRANS_MEM"] --> R0REC["RECORD\nnotifyId=42"]
88- end
89- 
90- subgraph SG1["rank1 / stream0"]
91- direction LR
92- R1T1["TRANS_MEM"] --> R1WAI["WAIT\nnotifyId=42"] --> R1T2["REDUCE"]
93- end
94- 
95- START --> R0T1
96- START --> R1T1
97- R0REC -.-> R1WAI
98-```
99- 
100-### 3.2 Task Graph Examples
101- 
102-#### 3.2.1 AICPU Mode
103- 
104-In AICPU mode, the task graph typically consists of `RECORD`, `WAIT`, and `TRANS_MEM` nodes. Below is a typical 2-rank `AllGather` example with each rank containing two streams, showing sequential edges in actual execution order and dashed lines for `RECORD` to `WAIT` synchronization dependencies.
105- 
106-```mermaid
107-flowchart LR
108- subgraph R0["rank0"]
109- direction TB
110- subgraph R0S0["stream0 main stream"]
111- direction LR
112- R0REC0["RECORD\nnotifyId=101"] --> R0WAI1S0["WAIT\nnotifyId=101"]
113- end
114- subgraph R0S1["stream1 slave stream"]
115- direction LR
116- R0WAI1S1["WAIT\nnotifyId=100"] --> R0TM0["TRANS_MEM\nsrc=Rank0-INPUT-[0x0,0x100)\ndst=Rank0-CCL-[0x0,0x100)"] --> R0REC2["RECORD\nnotifyId=102"] --> R0WAI3["WAIT\nnotifyId=103"] --> R0TM1["TRANS_MEM\nsrc=Rank0-CCL-[0x0,0x100)\ndst=Rank1-CCL-[0x0,0x100)"] --> R0REC4["RECORD\nnotifyId=104"] --> R0WAI5["WAIT\nnotifyId=105"] --> R0TM2["TRANS_MEM\nsrc=Rank0-CCL-[0x0,0x200)\ndst=Rank0-OUTPUT-[0x0,0x200)"] --> R0REC1["RECORD\nnotifyId=100"]
117- end
118- end
119- 
120- subgraph R1["rank1"]
121- direction TB
122- subgraph R1S0["stream0 main stream"]
123- direction LR
124- R1REC6["RECORD\nnotifyId=106"] --> R1WAI7["WAIT\nnotifyId=107"]
125- end
126- subgraph R1S1["stream1 slave stream"]
127- direction LR
128- R1WAI6["WAIT\nnotifyId=106"] --> R1TM0["TRANS_MEM\nsrc=Rank1-INPUT-[0x0,0x100)\ndst=Rank1-CCL-[0x100,0x200)"] --> R1REC3["RECORD\nnotifyId=103"] --> R1WAI2["WAIT\nnotifyId=102"] --> R1TM1["TRANS_MEM\nsrc=Rank1-CCL-[0x100,0x200)\ndst=Rank0-CCL-[0x100,0x200)"] --> R1REC5["RECORD\nnotifyId=105"] --> R1WAI4["WAIT\nnotifyId=104"] --> R1TM2["TRANS_MEM\nsrc=Rank1-CCL-[0x0,0x200)\ndst=Rank1-OUTPUT-[0x0,0x200)"] --> R1REC7["RECORD\nnotifyId=107"]
129- end
130- end
131- 
132- R0REC0 -.-> R0WAI1S1
133- R0REC1 -.-> R0WAI1S0
134- R1REC6 -.-> R1WAI6
135- R1REC7 -.-> R1WAI7
136- R0REC2 -.-> R1WAI2
137- R1REC3 -.-> R0WAI3
138- R0REC4 -.-> R1WAI4
139- R1REC5 -.-> R0WAI5
140-```
141- 
142-#### 3.2.2 CCU Mode
143- 
144-In CCU mode, Checker expands CCU instructions into CCU subgraphs. Using the 2-rank `AllReduce` data flow, the synchronization operations outside the CCU subgraph are omitted, retaining only the internal task sequence of the CCU subgraph. CCU uses `cke` / `mask` instead of `notifyId` for synchronization, and uses the CCU `MS` type for the intermediate buffer.
145- 
146-```mermaid
147-flowchart LR
148- subgraph R0["rank0"]
149- direction TB
150- subgraph R0Q0["Stream0 / CCU subgraph"]
151- direction LR
152- R0START["START\nboundaryType=CCU_SUB_GRAPH"] --> R0WAI0["WAIT\ncke=100\nmask=0x1"] --> R0TM0_CCU["TRANS_MEM\nsrc=Rank0-INPUT-[0x0,0x100)\ndst=Rank0-MS-[0x0,0x100)"] --> R0REC2_CCU["RECORD\ncke=102\nmask=0x1"] --> R0WAI3_CCU["WAIT\ncke=103\nmask=0x1"] --> R0TM1_CCU["TRANS_MEM\nsrc=Rank0-MS-[0x0,0x100)\ndst=Rank1-MS-[0x0,0x100)"] --> R0REC4_CCU["RECORD\ncke=104\nmask=0x1"] --> R0WAI5_CCU["WAIT\ncke=105\nmask=0x1"] --> R0TM2_CCU["TRANS_MEM\nsrc=Rank0-MS-[0x0,0x200)\ndst=Rank0-OUTPUT-[0x0,0x200)"] --> R0REC1_CCU["RECORD\ncke=100\nmask=0x1"] --> R0END["END\nboundaryType=CCU_SUB_GRAPH"]
153- end
154- end
155- 
156- subgraph R1["rank1"]
157- direction TB
158- subgraph R1Q0["Stream0 / CCU subgraph"]
159- direction LR
160- R1START["START\nboundaryType=CCU_SUB_GRAPH"] --> R1WAI6_CCU["WAIT\ncke=106\nmask=0x1"] --> R1TM0_CCU["TRANS_MEM\nsrc=Rank1-INPUT-[0x0,0x100)\ndst=Rank1-MS-[0x100,0x200)"] --> R1REC3_CCU["RECORD\ncke=103\nmask=0x1"] --> R1WAI2_CCU["WAIT\ncke=102\nmask=0x1"] --> R1TM1_CCU["TRANS_MEM\nsrc=Rank1-MS-[0x100,0x200)\ndst=Rank0-MS-[0x100,0x200)"] --> R1REC5_CCU["RECORD\ncke=105\nmask=0x1"] --> R1WAI4_CCU["WAIT\ncke=104\nmask=0x1"] --> R1TM2_CCU["TRANS_MEM\nsrc=Rank1-MS-[0x0,0x200)\ndst=Rank1-OUTPUT-[0x0,0x200)"] --> R1REC7_CCU["RECORD\ncke=107\nmask=0x1"] --> R1END["END\nboundaryType=CCU_SUB_GRAPH"]
161- end
162- end
163- 
164- R0REC2_CCU -.-> R1WAI2_CCU
165- R1REC3_CCU -.-> R0WAI3_CCU
166- R0REC4_CCU -.-> R1WAI4_CCU
167- R1REC5_CCU -.-> R0WAI5_CCU
168-```
169- 
170-#### 3.2.3 Graphviz Visualization
171- 
172-Checker provides task graph export capability for outputting the task graph as a Graphviz `.dot` file.
173- 
174-The exported content includes not only "which nodes exist" but also commonly used debugging information directly in the graph:
175-- Nodes arranged by `rank / stream` for easy observation of sequential relationships on the same execution queue
176-- Solid lines for normal dependency edges, dashed lines for `RECORD -> WAIT` synchronization dependencies
177-- Node labels include task type, `nodeId`, location information, key fields such as memory slices, `notifyId`, or `cke/mask`
178- 
179-Usage:
180-- Checker automatically attempts to export the `.dot` file after graph generation without requiring an additional switch
181-- After successful export, search the logs for `[GraphvizDot]` to find the output path, typically `hccl_vm_install/data/`
182-- Output file name format: `TaskGraph_YYYYMMDDHHMMSS.dot`
183- 
184-> After obtaining the `.dot` file, use `Microsoft VS Code` plugins such as `Graphviz Interactive Preview` for instant browsing.
185- 
186----
187- 
188-## 4. Single Task Check
189- 
190-This phase checks whether the memory range of a single task is valid and verifies the slave stream head/tail structure.
191- 
192-### 4.1 MemSlice
193- 
194-The most important information in memory copy and reduce tasks is the memory slice (MemSlice). A memory slice consists of the following:
195- 
196-```
197-MemSlice = { rankId, type, offset, len }
198-```
199- 
200-- `rankId` indicates which rank the memory belongs to
201-- `type` indicates the memory type
202- | Memory Type | Usage |
203- |-------------|-------|
204- | INPUT | Operator input buffer |
205- | OUTPUT | Operator output buffer |
206- | CCL | CCL buffer |
207- | MS_CCU | CCU MS |
208-- `offset` and `len` together define the memory access range
209- - `offset` is the starting address of this access on the memory slice
210- - `len` is the length of this memory access
211- - The access range uses half-open notation: `[offset, offset + length)`
212- 
213-The check points for a single task's memory slice are:
214-- `offset + length` must not overflow the `uint64` upper bound, otherwise an error is reported
215-- Multiple MemSlices within the same task with the same `(rankId, memType)` must not overlap, otherwise an error is reported
216- 
217- ```mermaid
218- gantt
219- title MemSlice Range Comparison
220- dateFormat x
221- axisFormat %L
222- tickInterval 100millisecond
223- 
224- section Valid (no overlap)
225- Slice A 0x000-0x400 : 0, 400
226- Slice B 0x400-0x800 : 400, 800
227- 
228- section Invalid (overlap)
229- Slice A 0x000-0x600 : crit, 0, 600
230- Slice B 0x400-0x800 : crit, 400, 800
231- ```
232- 
233-- Different `type` values represent independent address spaces. The same `offset` under different `type` values is not considered overlapping
234-- `offset + length` must not exceed the boundary of the current `type` address space
235- 
236- ```mermaid
237- gantt
238- title MemSlice Boundary Check
239- dateFormat x
240- axisFormat %L
241- tickInterval 100millisecond
242- 
243- section Valid (within bounds)
244- Type Space [0x000,0x800) : 0, 800
245- MemSlice [0x200,0x500) : 200, 500
246- 
247- section Invalid (out of bounds)
248- Type Space [0x000,0x800) : 0, 800
249- MemSlice [0x600,0x900) : crit, 600, 900
250- ```
251- 
252-### 4.2 Slave Stream Structure Check
253- 
254-The slave stream executes auxiliary tasks for the operator, such as data pre-copy. In the HCCL programming model, the main stream triggers the slave stream via a synchronization task `RECORD -> WAIT`. After the slave stream completes, it notifies the main stream through another set of synchronization tasks. Therefore, the slave stream must satisfy a fixed head/tail structure: first task is `WAIT` && last task is `RECORD`.
255- 
256-The following diagram shows an incorrect example, with the offending nodes highlighted in red:
257- 
258-```mermaid
259-flowchart LR
260- START(["start"])
261- 
262- subgraph MAIN["rank0 / stream0 (main stream)"]
263- M0["TRANS_MEM"]
264- M1["RECORD\nnotifyId=10\ntriggers slave stream"]
265- M2["WAIT\nnotifyId=11\nwaiting for slave stream"]
266- M3["REDUCE"]
267- M0 --> M1 --> M2 --> M3
268- end
269- 
270- subgraph SLAVE["rank0 / stream1 (slave stream, invalid example)"]
271- S0["TRANS_MEM\nerror: first task is not WAIT"]
272- S1["REDUCE"]
273- S2["WAIT\nerror: last task is not RECORD"]
274- S0 --> S1 --> S2
275- end
276- 
277- START --> M0
278- M1 -. sync edge .-> S0
279- S2 -. sync edge .-> M2
280- 
281- classDef invalid fill:#fde2e2,stroke:#c62828,stroke-width:2px,color:#7f1d1d;
282- class S0,S2 invalid;
283-```
284- 
285----
286- 
287-## 5. Memory Conflict Check
288- 
289-This phase checks whether there is a potential memory conflict in the task graph. A memory conflict occurs when multiple memory operations access the same memory segment at the same time, and at least one operation is a write. When a memory conflict occurs, the value of the conflicting memory segment is indeterminate, leading to accuracy issues in collective communication operators.
290- 
291-### 5.1 Memory Conflict Criteria
292- 
293-Two memory-accessing task nodes are judged as having a memory conflict when all three conditions below are met:
294-1. The two nodes may execute concurrently (no path exists between the two nodes on the task graph)
295-2. The accessed memory address ranges overlap
296-3. At least one is a write operation
297- 
298-Checker efficiently checks every pair of memory-accessing task nodes to ensure no false negatives.
299- 
300-### 5.2 Memory Conflict Example
301- 
302-The following diagram shows a task graph with a memory conflict:
303- 
304-```mermaid
305-flowchart LR
306- START(["start"])
307- 
308- subgraph R0["rank0 / stream0"]
309- R0A["R0A\n\nTRANS_MEM\nsrc={rankId=0, type=INPUT, offset=0x000, len=0x400}\ndst={rankId=0, type=CCL, offset=0x000, len=0x400}"]
310- R0REC["RECORD\nnotifyId=7"]
311- R0B["R0B\n\nTRANS_MEM\nsrc={rankId=0, type=CCL, offset=0x200, len=0x200}\ndst={rankId=1, type=OUTPUT, offset=0x000, len=0x200}"]
312- R0A --> R0REC --> R0B
313- end
314- 
315- subgraph R1["rank1 / stream0"]
316- R1WAIT["WAIT\nnotifyId=7"]
317- R1B["R1B\n\nTRANS_MEM\nsrc={rankId=0, type=CCL, offset=0x000, len=0x200}\ndst={rankId=1, type=OUTPUT, offset=0x000, len=0x200}"]
318- R1WAIT --> R1B
319- end
320- 
321- START --> R0A
322- START --> R1WAIT
323- R0REC -.-> R1WAIT
324- 
325- classDef conflict fill:#fde2e2,stroke:#c62828,stroke-width:2px,color:#7f1d1d;
326- class R0B,R1B conflict;
327-```
328- 
329-- `R0A` and `R0B` execute sequentially on the same stream, so concurrent execution is not possible and no memory conflict occurs
330-- `R0A` and `R1B` have their execution order constrained by synchronization nodes: `R0A -> R0RECORD -> R1WAIT -> R1B`, so no memory conflict occurs
331-- `R0B` and `R1B` can execute concurrently, and their write memory `dst` completely overlaps, so a memory conflict exists
332- 
333-### 5.3 Conflict Log Interpretation
334- 
335-The error log format for memory conflict is as follows:
336- 
337-```text
338-[ErrorCode: 302] Two tasks may access the same memory range in parallel, and at least one access is a write.
339- Conflict memory : rank 0 OUTPUT
340- Overlap range : [0x0,0xc80)
341- Conflict task 1:
342- node 17, action=write
343- access range : [0x0,0xc80)
344- task : [TaskTransMem] node=17, rank=1, stream=0, queue=0, protocol=SDMA, src=rank 1 CCL [0x0,0xc80), dst=rank 0 OUTPUT [0x0,0xc80)
345- Conflict task 2:
346- node 23, action=write
347- access range : [0x0,0xc80)
348- task : [TaskTransMem] node=23, rank=2, stream=0, queue=0, protocol=SDMA, src=rank 2 CCL [0x0,0xc80), dst=rank 0 OUTPUT [0x0,0xc80)
349-```
350- 
351-Log description:
352- 
353-| Field | Meaning |
354-|-------|---------|
355-| `[ErrorCode: 302]` | Memory conflict error code, corresponding to `MEMCONFLICT_DETECTED` |
356-| `Conflict memory : rank X TYPE` | Location of the conflicting memory |
357-| `Overlap range : [start,end)` | The actual overlapping address range of the two accesses |
358-| `Conflict task 1 / Conflict task 2` | The two accesses determined to be "concurrently executable with overlapping addresses" |
359-| `node X, action=read/write` | Node ID on the task graph and the read/write type of this access. If at least one is `write`, a conflict may be reported |
360-| `access range : [start,end)` | The complete address range covered by this access, may not be identical to the `Overlap range` |
361-| `task :` | Specific task details (task type, node ID, location, src/dst memory ranges, etc.) |
362- 
363----
364- 
365-## 6. Semantic Check
366- 
367-The semantic check phase traverses the task graph topologically, simulates the operator execution process, and verifies whether the final output meets operator expectations.
368- 
369-### 6.1 BufferSemantic
370- 
371-During the semantic check, Checker maintains data source records for each memory segment:
372- 
373-```
374-BufferSemantic = {
375- startAddr: Memory segment start address (offset)
376- size: Memory segment length (len)
377- srcBufs: Set of memory sources, each item is {rankId, bufferType, srcAddr}
378- isReduce: Whether it is a Reduce operation
379- reduceType: Reduce operation type SUM/MAX/MIN/...
380-}
381-```
382- 
383-`srcBufs` records the data source of the memory. `bufferType` indicates the buffer type the source belongs to, such as `INPUT`, `OUTPUT`, or `CCL`.
384- 
385-When each node executes, each `(src -> dst)` relationship is translated into one of the following two operations, then written back to the target address space. `(src -> dst)` represents a set of copy or reduce relationships from source address to destination address:
386- 
387-| TaskType | Operation | Behavior |
388-|----------|-----------|----------|
389-| `TRANS_MEM` / `BATCH_TRANS_MEM` | overwrite | First clears the existing semantics of the target range, then copies the source semantics over |
390-| `REDUCE` / `BATCH_REDUCE` | reduce | Requires that the target range is pre-filled with semantics, otherwise an error is reported. Then appends the new source to `srcBufs` and sets `isReduce=true` |
391- 
392-### 6.2 OUTPUT Expectations by Operator
393- 
394-The goal of semantic check is to determine whether each rank's `OUTPUT` meets the current operator's expectations. The expectations for different operators are:
395- 
396-| Operator | OUTPUT Semantic Expectation |
397-|----------|-----------------------------|
398-| AllReduce | Each rank's OUTPUT is the reduce result of all ranks' INPUT |
399-| AllGather | Each rank's OUTPUT is the concatenation of all ranks' INPUT in order |
400-| ReduceScatter | Each rank's OUTPUT is the fragment of the global reduce assigned to this rank |
401-| AllGatherV | Same as AllGather, but each rank contributes a different size |
402-| ReduceScatterV | Same as ReduceScatter, but each rank's fragment size differs |
403-| Send/Recv | The target rank's OUTPUT equals the source rank's INPUT, single source without reduce |
404-| BatchSendRecv | Multiple pairs of Send/Recv simultaneously |
405-| Broadcast | All ranks' OUTPUT equal the root rank's INPUT |
406-| Reduce | Only the root rank's OUTPUT is the reduce result of all ranks' INPUT |
407-| All2All | Each rank's `OUTPUT[i]` equals `rank i`'s `INPUT[this rank's offset]` |
408- 
409-The diagrams below show the OUTPUT expectations for each operator using a 2-rank collective communication operator example:
410- 
411-**AllReduce**
412- 
413-```mermaid
414-gantt
415- title AllReduce (2-rank)
416- dateFormat x
417- axisFormat %L
418- tickInterval 100millisecond
419- 
420- section rank0.INPUT
421- rank0.INPUT : 0, 100
422- 
423- section rank1.INPUT
424- rank1.INPUT : 0, 100
425- 
426- section rank0.OUTPUT
427- rank0.INPUT + rank1.INPUT : 0, 100
428- 
429- section rank1.OUTPUT
430- rank0.INPUT + rank1.INPUT : 0, 100
431-```
432- 
433-**AllGather / AllGatherV**
434- 
435-```mermaid
436-gantt
437- title AllGather (2-rank)
438- dateFormat x
439- axisFormat %L
440- tickInterval 100millisecond
441- 
442- section rank0.INPUT
443- rank0.INPUT : 0, 100
444- 
445- section rank1.INPUT
446- rank1.INPUT : 0, 100
447- 
448- section rank0.OUTPUT
449- rank0.INPUT : 0, 100
450- rank1.INPUT : 100, 200
451- 
452- section rank1.OUTPUT
453- rank0.INPUT : 0, 100
454- rank1.INPUT : 100, 200
455-```
456- 
457-`AllGatherV` semantics are the same as above, except that each rank's contribution length can differ.
458- 
459-**ReduceScatter / ReduceScatterV**
460- 
461-```mermaid
462-gantt
463- title ReduceScatter (2-rank, each slice Len=100)
464- dateFormat x
465- axisFormat %L
466- tickInterval 100millisecond
467- 
468- section rank0.INPUT
469- rank0.INPUT1 : 0, 50
470- rank0.INPUT2 : 50, 100
471- 
472- section rank1.INPUT
473- rank1.INPUT1 : 0, 50
474- rank1.INPUT2 : 50, 100
475- 
476- section rank0.OUTPUT
477- rank0.INPUT1 + rank1.INPUT1 : 0, 50
478- 
479- section rank1.OUTPUT
480- rank0.INPUT2 + rank1.INPUT2 : 50, 100
481-```
482- 
483-`ReduceScatterV` semantics are the same as above, except that each rank's output fragment size can differ.
484- 
485-**Send/Recv**
486- 
487-```mermaid
488-gantt
489- title Send/Recv (2-rank, src=0, dst=1)
490- dateFormat x
491- axisFormat %L
492- tickInterval 100millisecond
493- 
494- section rank0.INPUT
495- rank0.INPUT : 0, 100
496- 
497- section rank1.INPUT
498- None: 0,0
499- 
500- section rank0.OUTPUT
501- None: 0,0
502- 
503- section rank1.OUTPUT
504- rank0.INPUT : 0, 100
505-```
506- 
507-**BatchSendRecv**
508- 
509-```mermaid
510-gantt
511- title BatchSendRecv (2-rank)
512- dateFormat x
513- axisFormat %L
514- tickInterval 100millisecond
515- 
516- section rank0.INPUT
517- rank0.INPUT : 0, 100
518- 
519- section rank1.INPUT
520- rank1.INPUT : 0, 100
521- 
522- section rank0.OUTPUT
523- rank1.INPUT : 0, 100
524- 
525- section rank1.OUTPUT
526- rank0.INPUT : 0, 100
527-```
528- 
529-**Broadcast**
530- 
531-```mermaid
532-gantt
533- title Broadcast (2-rank, root=0)
534- dateFormat x
535- axisFormat %L
536- tickInterval 100millisecond
537- 
538- section rank0.INPUT
539- rank0.INPUT : 0, 100
540- 
541- section rank1.INPUT
542- None: 0,0
543- 
544- section rank0.OUTPUT
545- rank0.INPUT : 0, 100
546- 
547- section rank1.OUTPUT
548- rank0.INPUT : 0, 100
549-```
550- 
551-**Reduce**
552- 
553-```mermaid
554-gantt
555- title Reduce (2-rank, root=0, SUM)
556- dateFormat x
557- axisFormat %L
558- tickInterval 100millisecond
559- 
560- section rank0.INPUT
561- rank0.INPUT : 0, 100
562- 
563- section rank1.INPUT
564- rank1.INPUT : 0, 100
565- 
566- section rank0.OUTPUT
567- rank0.INPUT + rank1.INPUT : 0, 100
568- 
569- section rank1.OUTPUT
570- None: 0,0
571-```
572- 
573-**All2All**
574- 
575-```mermaid
576-gantt
577- title All2All (2-rank)
578- dateFormat x
579- axisFormat %L
580- tickInterval 100millisecond
581- 
582- section rank0.INPUT
583- rank0.INPUT1 : 0, 100
584- rank0.INPUT2 : 100, 200
585- 
586- section rank1.INPUT
587- rank1.INPUT1 : 0, 100
588- rank1.INPUT2 : 100, 200
589- 
590- section rank0.OUTPUT
591- rank0.INPUT1 : 0, 100
592- rank1.INPUT1 : 100, 200
593- 
594- section rank1.OUTPUT
595- rank0.INPUT2 : 0, 100
596- rank1.INPUT2 : 100, 200
597-```
598- 
599-### 6.3 Final Verification Flow
600- 
601-```mermaid
602-flowchart LR
603- SIM["Topologically traverse the task graph, simulate each task, record BufferSemantic for each memory segment"]
604- CHK["Check whether each rank's OUTPUT meets operator expectations"]
605- PASS["Pass"]
606- FAIL["Fail"]
607- 
608- SIM --> CHK --> PASS
609- CHK --> FAIL
610-```
611- 
612-Using a 4-rank `AllReduce` as an example:
613- 
614-```
615-rank0.OUTPUT[0,L) expected:
616- sources = { rank0.INPUT, rank1.INPUT, rank2.INPUT, rank3.INPUT }
617- reduceType = SUM
618- 
619-Assuming sources is missing rank3.INPUT:
620- actualSourceRankCount=3, expectedRankSize=4 -> check fails
621-```
622- 
623-### 6.4 Semantic Propagation Example
624- 
625-Using a 2-rank `AllGather` (each rank INPUT size 100 bytes) as an example to illustrate the semantic propagation process.
626- 
627-**Initial State**
628- 
629-Each rank's INPUT already has its own initial semantics (source pointing to itself):
630- 
631-```
632-rank0.INPUT[0, 100): srcBufs = { (rank0, INPUT, 0) }
633-rank1.INPUT[0, 100): srcBufs = { (rank1, INPUT, 0) }
634-rank0.OUTPUT: empty
635-rank1.OUTPUT: empty
636-```
637- 
638-**Propagation Process**
639- 
640-The diagram below shows the semantic filling process of rank0.OUTPUT. Each arrow represents an overwrite operation: reading the source buffer's semantics and writing them to the corresponding range of the target buffer.
641- 
642-```mermaid
643-flowchart LR
644- subgraph SRC["Initial Semantics"]
645- I0["rank0.INPUT[0,100)\nsrc = rank0"]
646- I1["rank1.INPUT[0,100)\nsrc = rank1"]
647- end
648- 
649- subgraph TASK["TRANS_MEM task (overwrite)"]
650- T1["rank0.INPUT[0,100) → rank0.OUTPUT[0,100)"]
651- T2["rank1.INPUT[0,100) → rank0.OUTPUT[100,200)"]
652- end
653- 
654- subgraph DST["Final Semantics (rank0.OUTPUT)"]
655- F0A["[0,100)\nsrc = rank0.INPUT"]
656- F0B["[100,200)\nsrc = rank1.INPUT"]
657- end
658- 
659- I0 --> T1 --> F0A
660- I1 --> T2 --> F0B
661-```
662- 
663-The propagation process for rank1.OUTPUT is similar. Finally, both ranks' OUTPUT are filled, sources are correct, and the check passes.
664- 
665-**Range Splitting**
666- 
667-If the write range does not align with the existing semantic boundary, Checker splits first, then overwrites. For example, rank0.OUTPUT[0,100) already has a full semantic segment, and then [35,65) is written:
668- 
669-```mermaid
670-gantt
671- title Range Splitting Illustration
672- dateFormat x
673- axisFormat %L
674- tickInterval 100millisecond
675- 
676- section Before Write
677- rank0.OUTPUT[0,100) src=rank0.INPUT : 0, 100
678- 
679- section After Write
680- [0,35) retain original semantics : 0, 35
681- [35,65) replace with new source :crit, 35, 65
682- [65,100) retain original semantics : 65, 100
683-```
684- 
685-Before the write, the original semantic block is split at offsets 35 and 65, [35,65) is replaced with the new source, and the remaining parts are kept unchanged.
686- 
687-### 6.5 Two Root Causes of Semantic Errors
688- 
689-Semantic check failures essentially have only two types of problems:
690- 
691-- **Missing data**: The OUTPUT range was not written to, or was written incompletely (missing head, fragmented, or missing tail).
692-- **Incorrect data source**: The OUTPUT is fully written, but the source rank, offset, or reduce type does not match expectations.
693- 
694-When debugging, first determine the problem type: if data is missing, focus on whether the task scheduling lacks a transmission; if the data source is incorrect, focus on whether the src/dst addresses and reduceOp are correct.
695- 
696-### 6.6 Error Log Interpretation
697- 
698-```text
699-[ErrorCode: 407] AllGather output range [0x1000,0x1400) for rank 3 should come from rank 4, but it actually comes from rank 5.
700-Current result range detail:
701- range=[0x1000,0x1400), size=0x400, sourceCount=1
702- sources:
703- - sourceRank=5, sourceBufferType=INPUT, sourceAddr=0x0
704-```
705- 
706-Log description:
707- 
708-| Line | Meaning |
709-|------|---------|
710-| Line 1 | Error code and main error message. `407` indicates an output source attribute error; `output range [0x1000,0x1400) for rank 3` specifies the failing output rank and range; `should come from rank 4, but it actually comes from rank 5` indicates the expected source rank differs from the actual source rank |
711-| `Current result range detail` | Full semantic expansion of the current output range for further debugging |
712-| `range / size / sourceCount` | Address range, length, and number of sources for the current output semantic block |
713-| `sources` | Source list for the current range, each item includes source rank, source buffer type, and source address |
714- 
715----
716- 
717-## 7. Quick Term Reference
718- 
719-| Term | Description |
720-|------|-------------|
721-| Communication domain | A group of communication members, describing the communication scope |
722-| Communication member | Usually referred to as a rank, the smallest logical entity participating in communication. Each rank is assigned a unique identifier called `rankId` |
723-| Communication operator | A collective communication operation, such as `AllReduce` or `AllGather` |
724-| Task | The core data structure of Checker, which describes a record of one atomic operation performed by a given rank, such as memory transfer, Reduce, memory copy, etc |
725-| Task graph | The core data structure of Checker, used to express task nodes and their dependency relationships |
726-| Node | Each node on the task graph is a task |
727-| Stream | A queue within a rank that executes tasks in sequential order, where each task uses a streamId to record which queue it belongs to |
728-| Task type | Different task types serve different purposes, such as memory copy, Reduce, or synchronization |
729-| Queue | CCU internal serial instruction queue |
730-| MemSlice | Memory access range `{rankId, type, offset, len}` |
731-| Slave stream | An auxiliary stream for executing secondary tasks, requiring first `WAIT`, last `RECORD` |
732-| `RECORD` / `WAIT` | A pair of synchronization tasks for sending and waiting for synchronization signals |
733-| Synchronization edge | A cross-stream or cross-rank execution dependency established by `RECORD -> WAIT` |
734-| Memory conflict | Multiple memory operations access the same memory segment at the same time with at least one write |
735-| BufferSemantic | An important data structure in the semantic check phase, recording where a memory segment's data comes from |
736-| `reduceType` | The reduce type in semantics, such as `SUM`, `MAX`, `MIN`, used to describe how multi-source data is merged |
737-| OUTPUT expectation | The correct semantic definition that the final output of a communication operator should satisfy, used for final comparison with actual results |
Dtest/hccl_vm/docs/faq/faq_en.md+0-1372
@@ -1,1372 +0,0 @@
1-# HCCL-VM FAQ Test Document
2- 
3-> This document is used to test the FAQ HTML generation framework.
4- 
5----
6- 
7-## Module: HCCL-VM
8- 
9-### Submodule: Command Line
10- 
11----
12- 
13-#### FAQ-E001
14- 
15-**Title:** Communication domain not configured
16- 
17-**Error code:**
18-```
19-NA (4)
20-```
21- 
22-**Error function:**
23-```
24-db_sim_runner_common.cc::GetDeviceByRankId()
25-```
26- 
27-**Key log:**
28-```
29-[error][PID:173579][TID:173579][db_sim_runner_common.cc][GetDeviceByRankId] cannot find rank by rank id 0
30-[error][PID:173579][TID:173579][aclrt_device_stub.cc][aclrtSetDevice] [DEVICE_STUB]device not found by rankId:0
31-acl interface return err ./common/src/hccl_test_common.cc:861, retcode: 100000.
32-This is an error in device_init.
33-```
34- 
35-**Symptoms:** When executing a business case, the device with rank id 0 cannot be found.
36- 
37-**Troubleshooting:**
38-```
39-[Possible Causes]
40-Before executing the business case, users need to determine the communication domain size used by the operator and configure the communication domain using the `hccl-vm mock-comm aa` command. The aa.yaml file is located at $HCCL_VM_INSTALL_DIR/config/topo_meta/aa.yaml.
41-```
42----
43- 
44-#### FAQ-E002
45- 
46-**Title:** RANK_TABLE_FILE not set
47- 
48-**Error code:**
49-```
50-HCCL_SIM_E_PARA (1)
51-```
52- 
53-**Error function:**
54-```
55-hccl_comm_stub.cc::HcclCommInitRootInfo()
56-```
57- 
58-**Key log:**
59-```
60-RANK_TABLE_FILE env not set, please check your config.
61-```
62- 
63-**Symptoms:** The rank table configuration file cannot be found during communication domain initialization.
64- 
65-**Troubleshooting:**
66-```
67-[Possible Causes]
68-1. Environment variable not set
69-2. Incorrect file path
70- 
71-[Solution]
72-export RANK_TABLE_FILE=/path/to/rank_table.json
73-```
74----
75- 
76-#### FAQ-E003
77- 
78-**Title:** HCCL_VM_INSTALL_DIR not set
79- 
80-**Error code:**
81-```
82-HCCL_SIM_E_INTERNAL (4)
83-```
84- 
85-**Error function:**
86-```
87-hccl_op_stub.cc::VirtualExecuteAivKernel()
88-```
89- 
90-**Key log:**
91-```
92-[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set, can not locate <path> for kernel <name>
93-```
94- 
95-**Symptoms:** AIV kernel virtual execution failed; the corresponding .so file cannot be found.
96- 
97-**Troubleshooting:**
98-```
99-[Solution]
100-export HCCL_VM_INSTALL_DIR=/path/to/hccl_vm/install/dir
101-```
102----
103- 
104-#### FAQ-E004
105- 
106-**Title:** Repeated execution of start command in a subshell
107- 
108-**Error code:**
109-```
110-NA (no error code, only WARNING)
111-```
112- 
113-**Error function:**
114-```
115-subcmd_start.cc::StartCommand::Execute()
116-```
117- 
118-**Key log:**
119-```
120-[warning][PID:<PID>][TID:<TID>][subcmd_start.cc][Execute] hccl-vm has already started. Please do not start it again in a sub-bash.
121-```
122- 
123-**Symptoms:** In the hvm subshell environment, executing the `hccl-vm start` command again causes the system to prompt that it has already started and ignore this operation.
124- 
125-**Troubleshooting:**
126-```
127-[Possible Causes]
128-`hccl-vm start` forks a sub-bash process. When the user enters `hccl-vm start` again inside that sub-bash (prompt `(hvm)$>`), the system refuses to start again.
129- 
130-[Solution]
131-Do not execute `hccl-vm start` repeatedly inside the subshell. To restart the simulation environment, first exit the current subshell (type `exit`), then re-execute `hccl-vm start`.
132-```
133----
134- 
135-#### FAQ-E005
136- 
137-**Title:** Fork subprocess failed
138- 
139-**Error code:**
140-```
141-HCCL_SIM_HOST_ERROR_CMD (no standard error code)
142-```
143- 
144-**Error function:**
145-```
146-cmd_base_utils.cc::StartHvmCmd()
147-```
148- 
149-**Key log:**
150-```
151-fork failed: Resource temporarily unavailable
152-```
153- 
154-**Symptoms:** After executing the `hccl-vm start` command, the system cannot create a subshell process, and the simulation environment fails to start.
155- 
156-**Troubleshooting:**
157-```
158-[Possible Causes]
159-1. The system user process limit has been reached (ulimit -u)
160-2. Insufficient system memory to allocate resources for the new process
161-3. PID resources exhausted (/proc/sys/kernel/pid_max)
162- 
163-[Steps]
164-ulimit -u
165-cat /proc/sys/kernel/pid_max
166-free -m
167-ps -eLf | wc -l
168- 
169-[Solution]
170-1. Increase the user process limit: `ulimit -u <larger value>`
171-2. Clean up zombie processes remaining in the system
172-3. Check if other programs are consuming excessive system resources
173-```
174----
175- 
176-#### FAQ-E006
177- 
178-**Title:** Plugin name format error
179- 
180-**Error code:**
181-```
182-NA (CLI parameter validation)
183-```
184- 
185-**Error function:**
186-```
187-subcmd_plugin.cc::PluginCommand::Setup()
188-```
189- 
190-**Key log:**
191-```
192-[HVM] [ERROR] Install plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin).
193-[HVM] [ERROR] Uninstall plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin).
194-[HVM] [ERROR] Run plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin).
195-```
196- 
197-**Symptoms:** When executing the `hccl-vm plugin install/run/uninstall` command, CLI parameter validation fails and the operation is rejected.
198- 
199-**Troubleshooting:**
200-```
201-[Possible Causes]
202-The plugin name does not start with the `@` symbol. For example, entering `hccl-vm plugin install runner` instead of `hccl-vm plugin install @runner`.
203- 
204-[Solution]
205-Ensure the plugin name starts with `@`, for example:
206-hccl-vm plugin install @runner
207-hccl-vm plugin install @checker
208-hccl-vm plugin uninstall @runner
209-```
210----
211- 
212-#### FAQ-E007
213- 
214-**Title:** Topology configuration file not found
215- 
216-**Error code:**
217-```
218-NA (CLI parameter validation)
219-```
220- 
221-**Error function:**
222-```
223-cmd_base_utils.cc::FileInModelDir()
224-```
225- 
226-**Key log:**
227-```
228-[HVM] model File not found: <install_path>/config/topo_meta/<name>.yaml
229-```
230- 
231-**Symptoms:** When executing the `hccl-vm mock-comm <name>` command, the specified topology yaml configuration file does not exist, and CLI parameter validation directly rejects the operation. The communication domain configuration file describes the scale of the operator's communication domain (e.g., how many super nodes, how many servers, and which cards are selected within each server; see the file description for details).
232- 
233-**Troubleshooting:**
234-```
235-[Possible Causes]
236-1. The specified topology name is misspelled
237-2. The corresponding yaml file is not placed in the `$HCCL_VM_INSTALL_DIR/config/topo_meta/` directory
238-3. Incorrect file extension (should be `.yaml`)
239- 
240-[Steps]
241-ls $HCCL_VM_INSTALL_DIR/config/topo_meta/
242- 
243-[Solution]
244-Ensure the topology yaml file is placed in the correct directory and the filename matches the command parameter. For example, executing `hccl-vm mock-comm 121` requires the `config/topo_meta/121.yaml` file to exist.
245-```
246----
247- 
248-#### FAQ-E008
249- 
250-**Title:** YAML topology file format parsing error
251- 
252-**Error code:**
253-```
254-NA (runtime parsing error)
255-```
256- 
257-**Error function:**
258-```
259-cmd_cluster_model_utils.cc::ParseYamlTopoImpl()
260-```
261- 
262-**Key log:**
263-```
264-[error][PID:<PID>][TID:<TID>][cmd_cluster_model_utils.cc][ParseYamlTopoImpl] Exception when parsing YAML: <detail>
265-```
266- 
267-**Symptoms:** When executing the `hccl-vm mock-comm <name>` command, the YAML topology configuration file parsing fails, and communication domain initialization is interrupted.
268- 
269-**Troubleshooting:**
270-```
271-[Possible Causes]
272-1. Syntax errors in the YAML file (e.g., incorrect indentation, missing space after colon, illegal characters)
273-2. Unsupported field types or formats in the YAML file
274-3. YAML file encoding is not UTF-8
275- 
276-[Steps]
277-# Use python to verify the yaml format
278-python3 -c "import yaml; yaml.safe_load(open('$HCCL_VM_INSTALL_DIR/config/topo_meta/<name>.yaml'))"
279- 
280-[Solution]
281-Fix the YAML file syntax errors based on the `<detail>` information in the log. Common issues include:
282-1. Indentation must use spaces, not Tab
283-2. A space is required after the colon in key-value pairs
284-3. Indentation of list items (`-`) must be consistent with their parent level
285-```
286----
287- 
288-### Submodule: Memory Management
289- 
290----
291- 
292-#### FAQ-M001
293- 
294-**Title:** Device memory allocation exceeded limit
295- 
296-**Error code:**
297-```
298-HCCL_SIM_E_MEMORY (3)
299-```
300- 
301-**Error function:**
302-```
303-store_sim_device_memory_manager.cc::AllocPhyMem()
304-```
305- 
306-**Key log:**
307-```
308-dev:<N> alloc phy mem:<ADDR> size:<SIZE> exceeds pool ceiling:<CEILING>, reject
309-```
310- 
311-**Symptoms:** The device memory allocation request exceeds the simulated memory pool ceiling.
312- 
313-**Diagram:**
314-```mermaid
315-graph LR
316- A[Memory allocation request] --> B{Check pool ceiling}
317- B -->|Within limit| C[Allocation successful]
318- B -->|Exceeded| D[Allocation rejected]
319- D --> E[Error: exceeds pool ceiling]
320-```
321----
322- 
323-#### FAQ-M002
324- 
325-**Title:** Shared memory creation failed
326- 
327-**Error code:**
328-```
329-HCCL_SIM_E_SYSCALL (8)
330-```
331- 
332-**Error function:**
333-```
334-store_sim_shm_ops.cc::ShmCreate()
335-```
336- 
337-**Key log:**
338-```
339-[SHM_OPS] create: shm_open failed, name: <name>
340-[SHM_OPS] create: ftruncate failed, name: <name>
341-[SHM_OPS] create: mmap failed, name: <name>
342-```
343- 
344-**Symptoms:** Unable to create a shared memory segment.
345- 
346-**Troubleshooting:**
347-```
348-[Possible Causes]
349-1. Insufficient `/dev/shm` space
350-2. Insufficient permissions
351-3. A shared memory with the same name already exists and conflicts
352- 
353-[Steps]
354-df -h /dev/shm
355-ls /dev/shm/ | grep hccl
356-```
357----
358- 
359-#### FAQ-M003
360- 
361-**Title:** Communication memory allocation failed
362- 
363-**Error code:**
364-```
365-HCCL_SIM_E_NOT_FOUND (6)
366-```
367- 
368-**Error function:**
369-```
370-store_sim_comm_memory_manager.cc
371-```
372- 
373-**Key log:**
374-```
375-[COMM_MEM] alloc failed, name: <name>
376-[COMM_MEM] acquire failed, name: <name>
377-[COMM_MEM] write size too large, size: <N>, max: <MAX>
378-```
379- 
380-**Symptoms:** Cross-process communication memory operation failed.
381- 
382----
383- 
384-### Submodule: Stub Proxy
385- 
386----
387- 
388-#### FAQ-PX001
389- 
390-**Title:** AIV Kernel virtual execution failed
391- 
392-**Error code:**
393-```
394-HCCL_SIM_E_INTERNAL (4)
395-```
396- 
397-**Error function:**
398-```
399-hccl_op_stub.cc::VirtualExecuteAivKernel()
400-```
401- 
402-**Key log:**
403-```
404-[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set
405-[virtual-aiv] missing aiv stub shared library, kernel=<name>
406-[virtual-aiv] dlopen <so> failed, err = <error>
407-[virtual-aiv] dlsym <symbol> from <so> failed, err = <error>
408-```
409- 
410-**Symptoms:** AIV kernel execution failed in the virtual environment.
411- 
412-**Troubleshooting:**
413-```
414-[Steps]
415-echo $HCCL_VM_INSTALL_DIR
416-ls -la $HCCL_VM_INSTALL_DIR/lib/aiv/
417-nm -D $HCCL_VM_INSTALL_DIR/lib/aiv/<kernel>.so | grep <symbol>
418-```
419----
420- 
421-#### FAQ-PX002
422- 
423-**Title:** Operator database recording failed
424- 
425-**Error code:**
426-```
427-HCCL_SIM_E_INTERNAL (4)
428-```
429- 
430-**Error function:**
431-```
432-hccl_op_stub.cc::RecordOpDbInfo()
433-```
434- 
435-**Key log:**
436-```
437-[RecordOpDbInfo] insert op detail+mem failed
438-[HcclAllReduce] record op db info failed
439-```
440- 
441-**Symptoms:** HCCL collective communication operator parameters cannot be written to the simulation database.
442- 
443-**Affected operators:** AlltoAll, AlltoAllV, AllGather, Broadcast, AllReduce, Scatter, Reduce, ReduceScatter
444- 
445----
446- 
447-#### FAQ-PX003
448- 
449-**Title:** QP not found or status error
450- 
451-**Error code:**
452-```
453-HCCL_SIM_E_NOT_FOUND (6)
454-```
455- 
456-**Error function:**
457-```
458-hccp_stub.cc::RaSendWr()
459-```
460- 
461-**Key log:**
462-```
463-[HCCP] RaSendWr: QP <N> not found
464-[HCCP] RaSendWr: QP <N> not in RTS state, current state:<N>
465-```
466- 
467-**Symptoms:** RDMA QP operation failed—QP does not exist or has not reached the RTS state.
468- 
469-**Diagram:**
470-```mermaid
471-stateDiagram-v2
472- [*] --> INIT
473- INIT --> RTR: RaQpConnect
474- RTR --> RTS: RaTypicalQpModify
475- RTS --> [*]: Ready to send data
476- RTS --> ERROR: Abnormal state
477- INIT --> ERROR: Not properly initialized
478-```
479----
480- 
481-#### FAQ-PX004
482- 
483-**Title:** EndPoint lookup failed
484- 
485-**Error code:**
486-```
487-HCCL_SIM_E_NOT_FOUND (6)
488-```
489- 
490-**Error function:**
491-```
492-hccp_stub.cc::RaCtxQpImport()
493-```
494- 
495-**Key log:**
496-```
497-[HCCP] cannot find endpoint addr:<IP>
498-Get remote endpoint failed. ip:<IP>, eid:<EID>
499-```
500- 
501-**Symptoms:** Network endpoint lookup failed.
502- 
503-**Troubleshooting:**
504-```
505-[Possible Causes]
506-The IP address is not in the endpoint list configured in the rank table.
507-```
508----
509- 
510-#### FAQ-PX005
511- 
512-**Title:** CCU microcode loading failed
513- 
514-**Error code:**
515-```
516-HCCL_SIM_E_INTERNAL (4)
517-```
518- 
519-**Error function:**
520-```
521-hccp_ccu_stub.cc::LoadMicrocodeInstruction()
522-```
523- 
524-**Key log:**
525-```
526-[LoadMicrocodeInstruction] get device by logic id <N> failed.
527-[LoadMicrocodeInstruction] get ccu from device by die id <N> failed.
528-[LoadMicrocodeInstruction] insert instr failed
529-```
530- 
531-**Symptoms:** CCU microcode instruction failed to load into the simulator.
532- 
533----
534- 
535-#### FAQ-PX006
536- 
537-**Title:** Unable to get current Context
538- 
539-**Error code:**
540-```
541-HCCL_SIM_E_NOT_FOUND (6)
542-```
543- 
544-**Error function:**
545-```
546-hccp_stub.cc::RaRdevInit()
547-```
548- 
549-**Key log:**
550-```
551-[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaRdevInit] can not get CurrContext: <N>
552-```
553- 
554-**Symptoms:** During RDMA device initialization, the active Context cannot be obtained from the current Runner, causing RDMA device creation to fail.
555- 
556-**Troubleshooting:**
557-```
558-[Possible Causes]
559-1. The application layer did not call `aclrtSetDevice`/`aclrtCreateContext` to initialize the device and context
560-2. The Context was destroyed prematurely
561-3. The current_ctx_id in the Runner's TLS (Thread Local Storage) is invalid
562-4. The application layer called other runtime interfaces to obtain a context before calling `aclrtSetDevice` to initialize the device context
563- 
564-[Steps]
565-# Check the Context table
566-hccl-vm table show Context
567-# Check the current_ctx_id in the Runner table
568-hccl-vm table show Runner
569- 
570-[Solution]
571-Ensure that the application layer has correctly called `aclrtSetDevice` and `aclrtCreateContext` before performing RDMA operations, and that the Context has not been destroyed prematurely.
572-```
573----
574- 
575-#### FAQ-PX007
576- 
577-**Title:** AICPU binary file not found
578- 
579-**Error code:**
580-```
581-ACL_ERROR_RT_FEATURE_NOT_SUPPORT
582-```
583- 
584-**Error function:**
585-```
586-aclrt_kernel_stub.cc::aclrtDestroyBinary()
587-```
588- 
589-**Key log:**
590-```
591-[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][aclrtDestroyBinary] can not find this binary
592-```
593- 
594-**Symptoms:** When destroying an AICPU binary object, the corresponding binary handle cannot be found in the global kernel binary registry.
595- 
596-**Troubleshooting:**
597-```
598-[Possible Causes]
599-1. The binary file was not properly loaded (`aclrtLoadBinary` was not executed or failed)
600-2. The binary handle was destroyed twice (double-free)
601-3. The binary object was accessed concurrently in a multi-threaded environment, causing state inconsistency
602- 
603-[Steps]
604-# Check for duplicate destroy calls
605-# Verify the return value of aclrtLoadBinary
606- 
607-[Solution]
608-Ensure that `aclrtLoadBinary` returns successfully before calling `aclrtDestroyBinary`, and do not destroy the same binary object twice.
609-```
610----
611- 
612-#### FAQ-PX008
613- 
614-**Title:** AICPU device process exited abnormally
615- 
616-**Error code:**
617-```
618-NA (process-level error)
619-```
620- 
621-**Error function:**
622-```
623-aclrt_kernel_stub.cc::WaitAicpuProcess()
624-```
625- 
626-**Key log:**
627-```
628-[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] exited with status <N>
629-[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] killed by signal <N>
630-```
631- 
632-**Symptoms:** The AICPU device subprocess exited abnormally or was killed by a signal, causing the main process to also exit (`exit(EXIT_FAILURE)`).
633- 
634-**Troubleshooting:**
635-```
636-[Possible Causes]
637-1. Uncaught exception or segmentation fault inside the AICPU process
638-2. Insufficient system resources (memory, file descriptors, etc.) causing the subprocess to be killed by OOM killer
639-3. Bugs in the AICPU binary file itself
640-4. Missing shared libraries required by the subprocess
641- 
642-[Steps]
643-# Check system logs for OOM records
644-dmesg | grep -i "oom\|killed"
645-# Verify the integrity of the AICPU binary file
646-ls -la $HCCL_VM_INSTALL_DIR/bin/
647-# Check system resources
648-ulimit -a
649-free -m
650- 
651-[Solution]
652-1. Check whether the AICPU binary file is correctly compiled and deployed
653-2. Ensure sufficient system resources (memory, file descriptor limits, etc.)
654-3. If killed by a signal, further locate the cause based on the signal number (e.g., 11=SIGSEGV, 9=SIGKILL)
655-```
656----
657- 
658-#### FAQ-PX009
659- 
660-**Title:** No ranks found when CCU loads microcode
661- 
662-**Error code:**
663-```
664-HCCL_SIM_E_NOT_FOUND (6)
665-```
666- 
667-**Error function:**
668-```
669-hccp_ccu_stub.cc::LoadMicrocodeInstruction()
670-```
671- 
672-**Key log:**
673-```
674-[error][PID:<PID>][TID:<TID>][hccp_ccu_stub.cc][LoadMicrocodeInstruction] can not find any rank
675-```
676- 
677-**Symptoms:** During CCU microcode instruction loading, no rank records can be found in the Rank table corresponding to the current device.
678- 
679-**Troubleshooting:**
680-```
681-[Possible Causes]
682-1. The communication domain has not been initialized via the `mock-comm` command, so the Rank table is empty
683-2. The current device ID does not exist in the communication domain configuration
684- 
685-[Steps]
686-# Check if the Rank table has data
687-hccl-vm table show Rank
688-# Check the Device table
689-hccl-vm table show Device
690- 
691-[Solution]
692-Ensure that before performing CCU-related operations, the communication domain has been correctly initialized via the `hccl-vm mock-comm` command, and the communication domain configuration covers the current device.
693-```
694----
695- 
696-#### FAQ-PX010
697- 
698-**Title:** Failed to find device by rankId
699- 
700-**Error code:**
701-```
702-HCCL_E_NOT_FOUND
703-```
704- 
705-**Error function:**
706-```
707-aclrt_device_stub.cc::hrtSetDevice()
708-```
709- 
710-**Key log:**
711-```
712-[error][PID:<PID>][TID:<TID>][aclrt_device_stub.cc][hrtSetDevice] device not found by rankId:<N>
713-```
714- 
715-**Symptoms:** When calling `aclrtSetDevice` to set the current device, looking up the device by rankId fails.
716- 
717-**Troubleshooting:**
718-```
719-[Possible Causes]
720-1. The rankId exceeds the actual rank range in the communication domain — e.g., the communication domain is configured with 4 NPUs, but mpirun starts 6 NPU processes, causing rankIds 4 and 5 to report device not found.
721-2. The communication domain has not been initialized (the `mock-comm` command was not executed) — [High probability] The tool initializes the Rank table only after the communication domain is initialized.
722-3. The ranktable configuration does not match the actual number of ranks used — possibly `RANK_TABLE_FILE` points to the wrong file path.
723- 
724-[Steps]
725-# Check if the rankId is within the valid range
726-hccl-vm table show Rank
727- 
728-[Solution]
729-Ensure that the rankId is within the legal range of the communication domain configuration (0 to rank_count-1), and that the `RANK_TABLE_FILE` environment variable points to the correct ranktable.json file.
730-```
731----
732- 
733-#### FAQ-PX011
734- 
735-**Title:** Stub interface not yet implemented
736- 
737-**Error code:**
738-```
739-HCCL_SIM_E_INTERNAL (4) or NA
740-```
741- 
742-**Error function:**
743-```
744-Multiple stub function files (hccp_stub.cc, ascend_hal_stub.cc, aclrt_kernel_stub.cc, etc.)
745-```
746- 
747-**Key log:**
748-```
749-[warning][PID:<PID>][TID:<TID>][ascend_hal_stub.cc][*] [STUB] is empty
750-[warning][PID:<PID>][TID:<TID>][hccp_stub.cc][*] [STUB] is empty
751-[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetAuxInfo] Not support yet
752-[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetCrErrInfoList] Not support yet
753-```
754- 
755-**Symptoms:** The application layer called a low-level driver or runtime interface that is not yet implemented by the simulator. The log shows `[STUB] is empty` or `Not support yet` warnings/errors. Such stub functions return default values directly (usually 0 or success) without performing any actual operations.
756- 
757-**Troubleshooting:**
758-```
759-[Possible Causes]
760-The current version of the simulator only implements the core interface subset required for HCCL collective communication. Some low-level driver interfaces (such as drvGetDeviceCapability, RaCtxGetAuxInfo, drvMemPrefetch, etc.) are not on the core path of HCCL communication, so the stub function body is empty or marked as unsupported.
761- Generally, flows supported by the HCCL-VM tool do not call these interfaces, so such warnings should not occur. If the user calls the wrong application layer interface or enters an incorrect HCCL business flow, such warnings may appear.
762- 
763-[Solution]
764-1. Such warnings usually do not affect the correctness simulation of HCCL operators and can be safely ignored.
765-2. If the warning is accompanied by functional anomalies, it means the application depends on an unimplemented interface. Please report it to the simulator development team.
766-3. If a stub implementation for a specific interface is needed, contact the development team for prioritized adaptation.
767-```
768- 
769-**Main interface types involved:**
770-1. **Driver layer interfaces** (`ascend_hal_stub.cc`): drvGetDeviceCapability, drvMemPrefetch, drvStreamQuery, etc., approximately 315 interfaces
771-2. **RDMA interfaces** (`hccp_stub.cc`): RaRestoreSnapshot, RaRdevInitWithBackup, RaCtxGetAuxInfo, etc., approximately 44 interfaces
772-3. **Runtime adaptation layer** (`adapter_rts_stub.cc`): some aclrt extension interfaces
773-4. **TSD client** (`tsd_client_stub.cc`): TSD-related interfaces
774- 
775----
776- 
777-### Submodule: Networking
778- 
779----
780- 
781-#### FAQ-N001
782- 
783-**Title:** Ranktable environment variable configuration error
784- 
785-**Error code:**
786-```
787-NA (1)
788-```
789- 
790-**Error function:**
791-```
792-param_check_v2.cc::RanktableRealPath
793-```
794- 
795-**Key log:**
796-```
797-[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172019]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path
798- 
799-[info][PID:172021][TID:172021][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172021][hrtGetDeviceRefresh]deviceLogicId[3]
800-[error][PID:172020][TID:172020][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172020]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path
801- 
802-[info][PID:172018][TID:172018][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172018][hrtGetDeviceRefresh]deviceLogicId[0]
803-[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base_v2.cc:294][172019][HcclCommInitClusterInfoV2]call trace: hcclRet -> 1
804- 
805-[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base.cc:811] [172019][operator()]call trace: hcclRet -> 1
806-```
807- 
808-**Symptoms:** Running a test case fails to initialize the communication domain.
809- 
810-**Troubleshooting:**
811-```
812-[Possible Causes]
813-The ranktable.json file path is configured incorrectly. Check the RANK_TABLE_FILE environment variable. The ranktable.json is generated by the tool, and its path is $HCCL_VM_INSTALL_DIR/data/ranktable.json.
814- 
815-[Steps]
816-echo $RANK_TABLE_FILE
817- 
818-[Solution]
819-Ensure the RANK_TABLE_FILE environment variable is correctly set to point to the ranktable.json file path.
820-```
821----
822- 
823-#### FAQ-N002
824- 
825-**Title:** topo.json path configuration error
826- 
827-**Error code:**
828-```
829-NA (1)
830-```
831- 
832-**Error function:**
833-```
834-communicator_impl.cc::GetTopoFilePath
835-```
836- 
837-**Key log:**
838-```
839-[error][PID:172635][TID:172635][log_stub.cc][DlogPrintStub] [HCCL_LOG][communicator_impl.cc:1339][172635][GetTopoFilePath] topo_file_path[/home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/topo.json] is not a valid real path
840-```
841- 
842-**Symptoms:** Running a test case fails to initialize the communication domain.
843- 
844-**Troubleshooting:**
845-```
846-[Possible Causes]
847-The topo.json file path is incorrectly configured in the /etc/hccl_rootinfo.json file. Check the topo_file_path field. The topo.json is generated by the tool, and its path is $HCCL_VM_INSTALL_DIR/data/topo.json.
848- 
849-[Steps]
850-echo $TOPO_FILE_PATH
851- 
852-[Solution]
853-Ensure the TOPO_FILE_PATH environment variable is correctly set to point to the topo.json file path.
854-```
855----
856- 
857-#### FAQ-N003
858- 
859-**Title:** mock-comm command error
860- 
861-**Error code:**
862-```
863-NA
864-```
865- 
866-**Error function:**
867-```
868-db_sim_runner_ops.cc::GetServerKeyById
869-```
870- 
871-**Key log:**
872-```
873-(hvm)$> hccl-vm mock-comm 144
874-[error][PID:172799][TID:172875][db_sim_runner_ops.cc][GetServerKeyById] can not find server by id: 0, 2
875-[error][PID:172799][TID:172875][topo_ascend_cluster_parser.cc][InitDynamicModelData] cannot find device by physical id 0
876-[error][PID:172799][TID:172875][cmd_base_utils.cc][InitHvmCommEnv] [HVM] InitHvmCommEnv failed
877-[error][PID:172799][TID:172875][subcmd_mock_comm.cc][Execute] [HVM] Failed to initialize mock communication environment. Cleaning up environment.
878-```
879- 
880-**Symptoms:** Before running a test case, configuring the communication domain via the mock-comm command fails.
881- 
882-**Troubleshooting:**
883-```
884-[Possible Causes]
885-The communication domain 144 configured by the mock-comm command exceeds the cluster configuration used when starting the tool. For example, the cluster started by the tool has only 2 servers per super node, but communication domain 144 indicates that the super node has 4 servers.
886- 
887-[Steps]
888-Check the cluster configuration file used when starting the tool and the communication domain configuration file for the mock-comm command.
889- 
890-[Solution]
891-Check the cluster configuration used when starting the tool to confirm the number of servers per super node. If communication domain 144 is indeed needed, ensure the tool is started with a larger cluster networking configuration.
892-Ensure that the communication domain configured by the mock-comm command does not exceed the cluster configuration used when starting the tool.
893-```
894----
895- 
896-#### FAQ-N004
897- 
898-**Title:** EndPoint IP lookup failed
899- 
900-**Error code:**
901-```
902-HCCL_SIM_E_NOT_FOUND (6)
903-```
904- 
905-**Error function:**
906-```
907-topo_ascend_cluster_parser.cc::AddLinkInfo()
908-```
909- 
910-**Key log:**
911-```
912-cannot find endPoint by ip <IP_ADDR>
913-```
914- 
915-**Symptoms:** The IP address referenced in the network link configuration does not exist in the topology.
916- 
917----
918- 
919-#### FAQ-N005
920- 
921-**Title:** Superpod index out of range
922- 
923-**Error code:**
924-```
925-HCCL_SIM_E_NOT_FOUND (6)
926-```
927- 
928-**Error function:**
929-```
930-topo_ascend_cluster_parser.cc::InitDynamicModelData()
931-```
932- 
933-**Key log:**
934-```
935-[InitDynamicModelData] superpod index <N> out of range
936-```
937- 
938-**Symptoms:** When parsing the ranktable to generate ranktable.json, the referenced superpod index exceeds the actual number of superpods in the cluster, causing initialization to fail.
939- 
940-**Troubleshooting:**
941-```
942-[Possible Causes]
943-The number of superpods to which the devices in the ranktable belong exceeds the cluster networking configuration used when starting the tool. For example, the cluster has only 1 superpod, but the ranktable references a second superpod.
944- 
945-[Steps]
946-1. Check the cluster networking configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of superpods.
947-2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to verify that the referenced superpod index is within range.
948- 
949-[Solution]
950-Ensure that the number of superpods referenced by the communication domain configured via mock-comm does not exceed the cluster networking configuration. If more superpods are needed, start the tool with a larger cluster networking configuration.
951-```
952----
953- 
954-#### FAQ-N006
955- 
956-**Title:** Server index out of range
957- 
958-**Error code:**
959-```
960-HCCL_SIM_E_NOT_FOUND (6)
961-```
962- 
963-**Error function:**
964-```
965-topo_ascend_cluster_parser.cc::InitDynamicModelData()
966-```
967- 
968-**Key log:**
969-```
970-[InitDynamicModelData] server index <N> out of range in superpod <M>
971-```
972- 
973-**Symptoms:** When parsing the ranktable to generate ranktable.json, the referenced server index exceeds the actual number of servers in the superpod, causing initialization to fail.
974- 
975-**Troubleshooting:**
976-```
977-[Possible Causes]
978-The number of servers under a certain superpod in the ranktable exceeds the number of servers in that superpod in the cluster networking configuration used when starting the tool. For example, the cluster networking has 2 servers per superpod, but the ranktable references a third server.
979- 
980-[Steps]
981-1. Check the cluster networking configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of servers per superpod.
982-2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to verify that the referenced server index is within range.
983- 
984-[Solution]
985-Ensure that the number of servers per superpod in the communication domain configured via mock-comm does not exceed the cluster networking configuration. If more servers are needed, start the tool with a larger cluster networking configuration.
986-```
987----
988- 
989-#### FAQ-N007
990- 
991-**Title:** Failed to find device by physical ID
992- 
993-**Error code:**
994-```
995-HCCL_SIM_E_NOT_FOUND (6)
996-```
997- 
998-**Error function:**
999-```
1000-topo_ascend_cluster_parser.cc::InitDynamicModelData()
1001-```
1002- 
1003-**Key log:**
1004-```
1005-[InitDynamicModelData] cannot find device by physical id <N>
1006-```
1007- 
1008-**Symptoms:** When parsing the ranktable, finding a device by its physical device ID fails, typically occurring when configuring the communication domain via mock-comm.
1009- 
1010-**Troubleshooting:**
1011-```
1012-[Possible Causes]
1013-The physical device ID referenced in the communication domain configured by the mock-comm command exceeds the actual device range in the cluster networking. For example, the cluster has only 2 devices (physical id 0 and 1), but the communication domain configuration references physical id 2.
1014- 
1015-[Steps]
1016-1. Check the cluster networking configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of devices per server.
1017-2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to verify that the referenced device_id is within range.
1018- 
1019-[Solution]
1020-Ensure that the physical device IDs referenced in the communication domain configured via mock-comm do not exceed the device range in the cluster networking configuration. If more devices are needed, start the tool with a larger cluster networking configuration.
1021-```
1022----
1023- 
1024-### Submodule: Database
1025- 
1026----
1027- 
1028-#### FAQ-DB001
1029- 
1030-**Title:** SQLite database connection failed
1031- 
1032-**Error code:**
1033-```
1034-HCCL_SIM_E_OPEN_FILE_FAILURE (10)
1035-```
1036- 
1037-**Error function:**
1038-```
1039-db_hccl_db_sqlite.cc::Connect()
1040-```
1041- 
1042-**Key log:**
1043-```
1044-[dbInit] Connect database failed
1045-Connect database:<path> failed
1046-```
1047- 
1048-**Symptoms:** Unable to connect to the SQLite database file.
1049- 
1050-**Troubleshooting:**
1051-```
1052-[Possible Causes]
1053-1. The database file does not exist
1054-2. Insufficient file permissions
1055-3. The file is locked by another process
1056-```
1057----
1058- 
1059-#### FAQ-DB002
1060- 
1061-**Title:** Database backup file not found
1062- 
1063-**Error code:**
1064-```
1065-HCCL_SIM_E_OPEN_FILE_FAILURE (10)
1066-```
1067- 
1068-**Error function:**
1069-```
1070-sim_loader.cc::BackupDatabase()
1071-```
1072- 
1073-**Key log:**
1074-```
1075-[Loader] Backup database file not found: <dbPath>
1076-```
1077- 
1078-**Symptoms:** The Loader cannot find the simulation database file.
1079- 
1080-**Troubleshooting:**
1081-```
1082-[Possible Causes]
1083-1. Incorrect simulation data file path configuration
1084-2. Simulation data has not been generated yet
1085-3. Insufficient file permissions
1086- 
1087-[Steps]
1088-ls -la <dbPath>
1089-```
1090----
1091- 
1092-#### FAQ-DB003
1093- 
1094-**Title:** SQLite query failed
1095- 
1096-**Error code:**
1097-```
1098-HCCL_SIM_E_INTERNAL (4)
1099-```
1100- 
1101-**Error function:**
1102-```
1103-db_hccl_db_sqlite.cc
1104-```
1105- 
1106-**Key log:**
1107-```
1108-Prepare failed: <error> sql:<SQL>
1109-Step failed: <error>, sql:<SQL>
1110-```
1111- 
1112-**Symptoms:** SQL query execution failed.
1113- 
1114-**Troubleshooting:**
1115-```
1116-[Possible Causes]
1117-1. Database table structure mismatch (version incompatibility)
1118-2. Database file corruption
1119-3. Insufficient disk space
1120-```
1121----
1122- 
1123-## Module: Plugin
1124- 
1125-### Submodule: checker
1126- 
1127----
1128- 
1129-##### HCCL_SIM_E_INTERNAL (4)
1130- 
1131----
1132- 
1133-#### FAQ-C001
1134- 
1135-**Title:** Memory slice overflow
1136- 
1137-**Error function:**
1138-```
1139-task_graph_single_task_check_v3.cc::CheckMemorySlice()
1140-```
1141- 
1142-**Key log:**
1143-```
1144-[TaskGraphSingleTaskCheckV3] Memory slice overflow while accumulating coverage, <detail>
1145-[TaskGraphSingleTaskCheckV3] Memory slice overflow, node=<node>, slice=<slice>
1146-```
1147- 
1148-**Symptoms:** The memory slice coverage of a single task exceeds the total size of the target buffer.
1149- 
1150-**Troubleshooting:**
1151-```
1152-[Possible Causes]
1153-1. Incorrect memory offset calculation in the HCCL algorithm layer (HCCL business issue)
1154-2. Mismatch between the memory layout information in the simulation data and the task parameters (tool data issue)
1155-```
1156- 
1157-**Diagram:**
1158-```mermaid
1159-graph LR
1160- A[Task memory slice] --> B[Target Buffer]
1161- A -.->|Out of bounds| C[Overflow error]
1162- style C fill:#f96,stroke:#333
1163-```
1164----
1165- 
1166-#### FAQ-C002
1167- 
1168-**Title:** Buffer semantic incomplete
1169- 
1170-**Error function:**
1171-```
1172-task_graph_semantic_check_v3.cc::CheckBufferContinuity()
1173-```
1174- 
1175-**Key log:**
1176-```
1177-[TaskGraphSemanticCheckV3] Head gap, expect start=0x<ADDR>, actual start=0x<ADDR>
1178-[TaskGraphSemanticCheckV3] Middle gap, prev end=0x<ADDR>, cur start=0x<ADDR>
1179-[TaskGraphSemanticCheckV3] Tail gap, expect end=0x<ADDR>, actual end=0x<ADDR>
1180-```
1181- 
1182-**Symptoms:** There are gaps in the data semantic coverage of the target buffer.
1183- 
1184-**Troubleshooting:**
1185-```
1186-[Possible Causes]
1187-1. The HCCL algorithm missed some data regions
1188-2. The Checker failed to correctly trace the transfer path
1189-```
1190- 
1191-**Diagram:**
1192-```mermaid
1193-graph TB
1194- subgraph Target Buffer
1195- A[Address 0x0] --> B[Address 0x100]
1196- B -.->|Gap| C[Address 0x200]
1197- C --> D[Address 0x300]
1198- end
1199- style B fill:#ff9,stroke:#333
1200- style C fill:#ff9,stroke:#333
1201-```
1202----
1203- 
1204-#### FAQ-C003
1205- 
1206-**Title:** Reduce semantic error
1207- 
1208-**Error function:**
1209-```
1210-task_graph_semantic_check_v3.cc::CheckReduceSemantics()
1211-```
1212- 
1213-**Key log:**
1214-```
1215-[TaskGraphSemanticCheckV3] Reduce type mismatch, pair=<pair>
1216-[TaskGraphSemanticCheckV3] Duplicate reduce source, pair=<pair>, srcOffset=0x<ADDR>
1217-[TaskGraphSemanticCheckV3] Destination reduce semantic incomplete, pair=<pair>
1218-```
1219- 
1220-**Symptoms:** Data semantic validation of the Reduce operation failed.
1221- 
1222-**Troubleshooting:**
1223-```
1224-[Possible Causes]
1225-1. Data type mismatch in the Reduce operation
1226-2. Duplicate reduce source exists
1227-3. The target buffer is not fully covered by all reduce sources
1228-```
1229----
1230- 
1231-##### HCCL_SIM_E_PARA (1)
1232- 
1233----
1234- 
1235-#### FAQ-C004
1236- 
1237-**Title:** rankSize is zero
1238- 
1239-**Error function:**
1240-```
1241-task_graph_semantic_check_v3.cc
1242-```
1243- 
1244-**Key log:**
1245-```
1246-[TaskGraphSemanticCheckV3] rankSize is zero
1247-```
1248- 
1249-**Symptoms:** The number of ranks participating in communication is 0 during semantic check.
1250- 
1251-**Troubleshooting:**
1252-```
1253-[Possible Causes]
1254-The communication domain has not been properly initialized, or the rank table parsing failed.
1255-```
1256----
1257- 
1258-#### FAQ-C005
1259- 
1260-**Title:** Batch Trans pair size mismatch
1261- 
1262-**Error function:**
1263-```
1264-task_graph_single_task_check_v3.cc::CheckBatchTrans()
1265-```
1266- 
1267-**Key log:**
1268-```
1269-[TaskGraphSingleTaskCheckV3] Batch trans slice length mismatch, node=<node>, label=<label>, index=<N>
1270-[TaskGraphSingleTaskCheckV3] Batch trans pair size mismatch, node=<node>, label=<label>
1271-```
1272- 
1273-**Symptoms:** Slice length or pair count mismatch in batch transfer operations.
1274- 
1275-**Troubleshooting:**
1276-```
1277-[Possible Causes]
1278-Uneven data distribution among ranks for operators such as AlltoAll.
1279-```
1280----
1281- 
1282-##### HCCL_SIM_E_NOT_SUPPORT (5)
1283- 
1284----
1285- 
1286-#### FAQ-C006
1287- 
1288-**Title:** Unsupported memory type
1289- 
1290-**Error function:**
1291-```
1292-task_graph_single_task_check_v3.cc
1293-```
1294- 
1295-**Key log:**
1296-```
1297-[TaskGraphSingleTaskCheckV3] Unsupported memory type, node=<node>, slice=<slice>
1298-[TaskGraphSingleTaskCheckV3] Invalid memory slice, node=<node>, slice=<slice>
1299-```
1300- 
1301-**Symptoms:** The memory slice type is not within the range supported by Checker.
1302- 
1303----
1304- 
1305-##### HCCL_SIM_E_OPEN_FILE_FAILURE (10)
1306- 
1307----
1308- 
1309-#### FAQ-C007
1310- 
1311-**Title:** Dump file write failed
1312- 
1313-**Error function:**
1314-```
1315-dump_manager.cc, dump_v3_manager.cc
1316-```
1317- 
1318-**Key log:**
1319-```
1320-[DumpManager::WriteMsgpackFile] failed to open file: <path>
1321-[DumpManager::WriteJsonFile] json serialize failed: <error>, file: <path>
1322-[DumpV3Manager::WriteMsgpack] failed to open file: <path>
1323-```
1324- 
1325-**Symptoms:** Failed to write Checker intermediate result dump file.
1326- 
1327-**Troubleshooting:**
1328-```
1329-[Possible Causes]
1330-1. Insufficient disk space
1331-2. Directory does not exist or lacks write permissions
1332-```
1333----
1334- 
1335-#### FAQ-C008
1336- 
1337-**Title:** Binary file magic number mismatch
1338- 
1339-**Error function:**
1340-```
1341-binary_data_operator.cc::FileHeaderRead()
1342-```
1343- 
1344-**Key log:**
1345-```
1346-[FileHeaderRead] Unmatched magic number:0x<N>≠0x<M>
1347-```
1348- 
1349-**Symptoms:** When reading the simulation data file, the magic number in the file header does not match.
1350- 
1351-**Troubleshooting:**
1352-```
1353-[Possible Causes]
1354-1. The data file version is incompatible with the tool version
1355-2. The file is corrupted
1356-```
1357----
1358- 
1359-## Appendix: Error Code Quick Reference
1360- 
1361-| Error Code | Enum Value | Description |
1362-|--------|--------|------|
1363-| 0 | HCCL_SIM_SUCCESS | Success |
1364-| 1 | HCCL_SIM_E_PARA | Parameter error |
1365-| 2 | HCCL_SIM_E_PTR | Null pointer |
1366-| 3 | HCCL_SIM_E_MEMORY | Memory error |
1367-| 4 | HCCL_SIM_E_INTERNAL | Internal error |
1368-| 5 | HCCL_SIM_E_NOT_SUPPORT | Unsupported feature |
1369-| 6 | HCCL_SIM_E_NOT_FOUND | Resource not found |
1370-| 8 | HCCL_SIM_E_SYSCALL | System call error |
1371-| 9 | HCCL_SIM_E_TIMEOUT | Timeout |
1372-| 10 | HCCL_SIM_E_OPEN_FILE_FAILURE | File open failed |