已合并
[modify docs]修正文档中英文不一致性问题 #4700
zangyan创建于 17 天前
[modify docs]修正文档中英文不一致性问题 #4700
已合并
zangyan创建于 17 天前
52 个文件变更+12134-572
@@ -1,6 +1,6 @@
1# HCOMM Agent Rules1# HCOMM Agent Rules
2 2 
3-This file serves as the main entry point for AI Agent governance of the HCOMM repository. It is intended for AI programming tools that support the AGENTS.md standard. If a subdirectory has its own `AGENTS.md`, that file supplements this one. In case of conflicts, follow the user's explicit requirements first, and then the rules in the nearest subdirectory.3+This file serves as the main entry point for AI Agent governance of the HCOMM repository. It is intended for AI programming tools that support the AGENTS.md standard. If a subdirectory has its own `AGENTS.md`, the nearer file supplements this one. In case of conflicts, follow the user's explicit requirements first, and then the rules in the nearest subdirectory.
4 4 
5> This file provides only the **essential hard constraints and entry points**. Detailed content is progressively disclosed through links. Before modifying code, read the architecture constraints in Section 3.5> This file provides only the **essential hard constraints and entry points**. Detailed content is progressively disclosed through links. Before modifying code, read the architecture constraints in Section 3.
6 6 
@@ -52,9 +52,9 @@ Dependency direction is top-down: `coll_comm_ops` (hccl) → `coll_communicator_
52| Layer | Header Files | Audience |52| Layer | Header Files | Audience |
53|------|------|------|53|------|------|------|
54| L2-comm | `include/hccl/hccl_comm.h` | AI framework layer (communication domain creation) |54| L2-comm | `include/hccl/hccl_comm.h` | AI framework layer (communication domain creation) |
55-| L2-res-rank_graph | `include/hccl/hccl_res.h`, `include/hccl/hccl_rank_graph.h` | Operator developers (topology query, resource acquisition) |55+| L2-res-rank_graph | `include/hccl/hccl_res.h`, `include/hccl/hccl_channel.h`, `include/hccl/hccl_rank_graph.h` | Operator developers (topology query, resource acquisition) |
56| L3-prim | `include/hcomm_primitives.h` | Operator or communication library developers (data transfer and synchronization) |56| L3-prim | `include/hcomm_primitives.h` | Operator or communication library developers (data transfer and synchronization) |
57-| L3-res | `include/hcomm_res.h`, `include/hcomm_res_defs.h` | Communication library developers (device, channel, and memory resources) |57+| L3-res | `include/hcomm_res.h`, `include/hcomm_res_defs.h`, `include/hcomm_channel.h` | Communication library developers (device, channel, and memory resources) |
58| CCU | `include/ccu/` (`ccu_primitives.hpp`, `ccu_res.h`, `ccu_launch.h`) | CCU operator developers |58| CCU | `include/ccu/` (`ccu_primitives.hpp`, `ccu_res.h`, `ccu_launch.h`) | CCU operator developers |
59 59 
60Changes to `include/` must be backward compatible. `pkg_inc/` is for inter-package use between HCOMM, HCCL, GE, and so on, and is not externally stable. For the complete API layer relationships, see [Section 3.3 of the Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md).60Changes to `include/` must be backward compatible. `pkg_inc/` is for inter-package use between HCOMM, HCCL, GE, and so on, and is not externally stable. For the complete API layer relationships, see [Section 3.3 of the Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md).
@@ -77,7 +77,7 @@ Output: `build_out/cann-hcomm_<version>_linux-<arch>.run`. For a complete list o
77- Naming: Classes and functions use PascalCase; member variables use `camelCase_` (lower camelCase with a trailing underscore); constants and macros use `UPPER_SNAKE_CASE`.77- Naming: Classes and functions use PascalCase; member variables use `camelCase_` (lower camelCase with a trailing underscore); constants and macros use `UPPER_SNAKE_CASE`.
78- Style: Follow the `.clang-format` in the root directory (120 columns, 4 spaces, pointer right-aligned, K&R braces). Use C++14.78- Style: Follow the `.clang-format` in the root directory (120 columns, 4 spaces, pointer right-aligned, K&R braces). Use C++14.
79- Static warnings: Code must pass CANN static check requirements (verified during the CI codecheck stage) and compile without warnings.79- Static warnings: Code must pass CANN static check requirements (verified during the CI codecheck stage) and compile without warnings.
80-- pre-commit: clang-format v16 + OAT compliance check. New source files must include the CANN-2.0 license header.80+- pre-commit: clang-format v18.1.8 + OAT compliance check. New source files must include the CANN-2.0 license header.
81 81 
82References: [CANN Coding Standards](https://gitcode.com/cann/community/tree/master/contributor/coding-standards), [CANN CI Guide](https://gitcode.com/cann/community/blob/master/contributor/repository/ci-guide.md), [pre-commit Guide](./docs/en/build/pre-commit-guide.md), `.clang-format`, `OAT.xml`.82References: [CANN Coding Standards](https://gitcode.com/cann/community/tree/master/contributor/coding-standards), [CANN CI Guide](https://gitcode.com/cann/community/blob/master/contributor/repository/ci-guide.md), [pre-commit Guide](./docs/en/build/pre-commit-guide.md), `.clang-format`, `OAT.xml`.
83 83 
@@ -2,11 +2,11 @@
2 2 
3## Recommended User Account3## Recommended User Account
4 4 
5-For security reasons, avoid using administrator accounts such as root to execute commands. Follow the principle of least privilege.5+For security reasons, avoid using administrator accounts such as root to execute any commands. Follow the principle of least privilege.
6 6 
7## File Permission Control7## File Permission Control
8 8 
9-- It is recommended to set the umask value to 0027 or higher on the host machine (including the host machine and containers) to ensure that the default maximum permissions for new folders are 750 and for new files are 640.9+- It is recommended to set the umask value to 0027 or higher on the host (including the host machine and containers) to ensure that the default maximum permissions for new folders are 750 and for new files are 640.
10- Apply security measures such as permission control to sensitive content, including personal private data, business assets, and source files. For example, control the permissions of the installation directory and input public data files. For recommended permission settings, refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios).10- Apply security measures such as permission control to sensitive content, including personal private data, business assets, and source files. For example, control the permissions of the installation directory and input public data files. For recommended permission settings, refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios).
11- Control permissions during installation and usage. Refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios) for file permission references.11- Control permissions during installation and usage. Refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios) for file permission references.
12 12 
@@ -52,7 +52,7 @@ The software dependencies required for compiling this project are listed below.
52 ```52 ```
53 53 
54 - \$\{cann\_version\}: Indicates the CANN software package version number.54 - \$\{cann\_version\}: Indicates the CANN software package version number.
55- - \$\{arch\}$: Indicates the CPU architecture, for example, aarch64 or x86_64.55+ - \$\{arch\}: Indicates the CPU architecture, for example, aarch64 or x86_64.
56 - \$\{soc\_name\}: Indicates the NPU model name.56 - \$\{soc\_name\}: Indicates the NPU model name.
57 - \$\{install\_path\}: Indicates the specified installation path. The CANN ops operator package must be installed in the same path as the CANN Toolkit development kit package. The default installation path for the root user is `/usr/local/Ascend`.57 - \$\{install\_path\}: Indicates the specified installation path. The CANN ops operator package must be installed in the same path as the CANN Toolkit development kit package. The default installation path for the root user is `/usr/local/Ascend`.
58 58 
@@ -158,7 +158,7 @@ bash build.sh --ut
158 158 
159### On-Board Testing159### On-Board Testing
160 160 
161-> **Note**161+> [!Note]NOTE
162> Before on-board testing, ensure that the driver, firmware, CANN Toolkit development kit package, and CANN ops operator package are installed.162> Before on-board testing, ensure that the driver, firmware, CANN Toolkit development kit package, and CANN ops operator package are installed.
163 163 
164Developers can use the HCCL Test tool for collective communication function and performance testing on the board. The workflow for using the HCCL Test tool is as follows:164Developers can use the HCCL Test tool for collective communication function and performance testing on the board. The workflow for using the HCCL Test tool is as follows:
@@ -13,7 +13,7 @@ pre-commit is a Git Hooks framework that automatically runs code checking and fo
13 13 
14- **Git**: 2.0+14- **Git**: 2.0+
15- **Python**: 3.8+15- **Python**: 3.8+
16-- **clang-format**: 14.0+ (code formatting tool)16+- **clang-format**: v18.1.8 (code formatting tool, must match the `rev` in [.pre-commit-config.yaml](../../../.pre-commit-config.yaml))
17- **Java**: 17+ (required for OAT tool, can be installed automatically)17- **Java**: 17+ (required for OAT tool, can be installed automatically)
18- **Maven**: 3.6+ (required for OAT tool, can be installed automatically)18- **Maven**: 3.6+ (required for OAT tool, can be installed automatically)
19 19 
@@ -1,24 +1,24 @@
1-# EI0002 Communication_Error_Timeout1+# EI0002 Communication_Error_Timeout
2 2 
3## Symptom3## Symptom
4 4 
5The following is error format. The meanings of the placeholders %s in sequence are: rank id, task information, communication operator information, communicator information.5The following is error format. The meanings of the placeholders %s in sequence are: rank id, task information, communication operator information, communicator information.
6 6 
7```text7```text
8-An timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s.8+A timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s.
9```9```
10 10 
11Error example:11Error example:
12 12 
13```text13```text
14-An timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none.14+A timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none.
15```15```
16 16 
17## Possible Cause17## Possible Cause
18 18 
191. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed.191. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed.
20 20 
21-2. The execution speed on some NPU in the cluster is too slow to complete a communication operation within the timeout interval. \(The default timeout interval is 1800s, You can set the interval by using HCCL_EXEC_TIMEOUT.\)21+2. The execution speed on some NPUs in the cluster is too slow to complete a communication operation within the timeout interval. \(The default timeout interval is 1800s. You can set the interval by using HCCL_EXEC_TIMEOUT.\)
22 22 
233. The number of training samples of each NPU is inconsistent.233. The number of training samples of each NPU is inconsistent.
24 24 
@@ -22,4 +22,4 @@ The Toolkit (cann-hcomm) and ops (cann-hccl) versions are inconsistent. The loca
22 22 
23## Solution23## Solution
24 24 
25-Install the same version. The local Toolkit \(cann-hcomm\) and ops \(cann-hccl\) versions must be consistent with the corresponding remote versions.25+Install the same version. The local Toolkit (cann-hcomm) and ops (cann-hccl) versions must be consistent with the corresponding remote versions.
@@ -16,4 +16,4 @@ P2P communication failed. Reason: Device ID 0 in module 0 and device ID 9 in mod
16 16 
17## Solution17## Solution
18 18 
19-Ensure that the NPU card is normal and entering environment variables 'export HCCL_INTRA_ROCE_ENABLE=1'.19+Ensure that the NPU card is normal and enter environment variable `export HCCL_INTRA_ROCE_ENABLE=1`.
@@ -11,7 +11,7 @@ Value %s for config %s is invalid. Expected value: %s.
11Error example:11Error example:
12 12 
13```text13```text
14-Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A." .14+Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A".
15```15```
16 16 
17## Solution17## Solution
@@ -28,4 +28,4 @@ An error CQE occurred during operator execution. Local information: server az0-r
28 28 
292. Check whether the peer process exits first. If yes, check the reason why the process exit.292. Check whether the peer process exits first. If yes, check the reason why the process exit.
30 30 
31-3. Use the RAS fault check mechanism to check whether the hardware of the HBM or UB chip processing module of either device is abnormal.If the hardware is abnormal, contact Huawei technical support.31+3. Use the RAS fault check mechanism to check whether the hardware of the HBM or UB chip processing module of either device is abnormal. If the hardware is abnormal, contact Huawei technical support.
@@ -16,6 +16,6 @@ Failed to enable listening for the host network adapter socket. Reason: The IP a
16 16 
17## Solution17## Solution
18 18 
19-1. Check whether this port has been occupied by another process. If yes, you can make adjustment using the environment variable HCCL_IF_BASE_PORT and use sysctl -w net.ipv4.ip_local_reserved_ports=\*\*\*\*-\*\*\*\* to adjust the scope of reserved ports.19+1. Check whether this port has been occupied by another process. If yes, you can make adjustment using the environment variable HCCL_IF_BASE_PORT and use `sysctl -w net.ipv4.ip_local_reserved_ports=****-****` to adjust the scope of reserved ports.
20 20 
212. Check whether the service process is started multiple times on a device during this service.212. Check whether the service process is started multiple times on a device during this service.
@@ -59,6 +59,7 @@ Which interfaces of hcomm / hccl / other CANN components this depends on.
59This section covers only module-internal implementation. Organize by module; for cross-layer features, split into separate module subsections per layer, do not mix them.59This section covers only module-internal implementation. Organize by module; for cross-layer features, split into separate module subsections per layer, do not mix them.
60 60 
61Each module subsection includes:61Each module subsection includes:
62+ 
62- Module responsibility (one sentence)63- Module responsibility (one sentence)
63- Core data structures (UML may be used)64- Core data structures (UML may be used)
64- Key logic and algorithms65- Key logic and algorithms
@@ -0,0 +1,690 @@
1+# RFC: Topology-Based Cluster Communication Rapid Sub-Health Monitoring
2+ 
3+- Start date: 2026-06-01
4+- Revision dates: 2026-07-13, 2026-08-10
5+- RFC PR number: 2491
6+- Related issues: 249, 261
7+- Status: accept
8+ 
9+## Summary
10+ 
11+This RFC proposes a "layered, domain-based minimum-cost network probing + link-level fault localization" scheme for RoCE lossless clusters. The system identifies relationships among devices, hosts, ToR/Leaf switches, and upper-level network domains through topology discovery. It uses solvable ring-path sets within each domain and constructs probing tasks between adjacent domains on demand. The system then combines HCCN PingPong path latency and pass rates with RNIC PFC and CNP counters to produce L1 link metrics, L2 path metrics, and abnormal link candidates.
12+ 
13+This document defines the target design and user contract for this feature. Items that have not yet been validated in production or aligned with the implementation are explicitly marked as "release gate" or "to be confirmed" and are not considered capabilities of the current version.
14+ 
15+## 1. Background and Motivation
16+ 
17+### 1.1 Problem
18+ 
19+A stable, high-speed network is the foundation of HCCL performance. Link jitter, congestion, intra-host bottlenecks, and configuration deviations cause long-tail latency in collective communication, training job slowdowns, and even interruptions. Existing monitoring approaches have the following limitations:
20+ 
21+1. Full PingMesh probing scales quadratically with cluster size, resulting in high overhead.
22+2. Only end-to-end latency or packet loss is observed, making it impossible to attribute anomalies to specific access links.
23+3. Static thresholds do not adapt to different scales, layers, and load phases, leading to high false-positive rates.
24+4. A single control node collects data serially, which easily creates a control-plane bottleneck in large-scale clusters.
25+ 
26+### 1.2 Four Problem Categories to Address
27+ 
28+| ID | Problem | Approach in This Scheme |
29+| --- | --- | --- |
30+| Q1 | Complex topology awareness | Discover observable layers and local meshes of Spine-Leaf, HammingMesh, and 3D-Torus topologies based on Tracert and LLDP |
31+| Q2 | Network metric collection | Generate a solvable minimum probing set; collect P90/P99/Mean latency, pass rates, and PFC/CNP counters |
32+| Q3 | Anomaly analysis and localization | Solve path-link equations for L1; output cross-domain path metrics after subtracting access link metrics for L2; apply time-series anomaly detection |
33+| Q4 | Large-scale adaptability | Use layered, domain-based design with parallel host execution and control-plane aggregation to avoid global full-mesh probing |
34+ 
35+### 1.3 Use Cases
36+ 
37+- During large-scale distributed training, assist in locating latency jitter, throughput drops, and communication long-tail issues in collective communication operations such as AllReduce and RingAllReduce.
38+- In RoCE networks, detect link flapping, congestion, switch or NPU packet loss, and hash collisions.
39+- Perform network baseline checks before training jobs start, or run continuous low-frequency inspections during job execution.
40+ 
41+### 1.4 Non-Goals
42+ 
43+- Do not replace switch telemetry, NMS, or alerting platforms.
44+- Do not directly modify the HCCL/HCOMM data plane, QP creation process, or communication topology of training jobs.
45+- The current version does not guarantee unique localization of all L2 anomalies to a single physical switch link. L2 output provides path-level or communication-pair-level candidates first.
46+ 
47+## 2. Glossary
48+ 
49+| Term | Definition |
50+| --- | --- |
51+| Device / NPU | An accelerator that participates in HCCL communication and has HCCN/RNIC ports |
52+| Host | A server node that manages one or more devices and runs `rpc_host` |
53+| Controller | A central control node that runs `probe_topo`, `probe_controller`, and deployment scripts |
54+| Leaf / ToR | The top-of-rack switch to which devices connect; this document treats both as the same access-layer concept |
55+| L1 link | The access link from a device/RNIC to its directly connected Leaf/ToR; the current implementation can resolve this to link level through path equations |
56+| L2 path | The network segment between two different Leaf/ToR domains. The current implementation computes "end-to-end P99 latency minus the resolved L1 values at both ends" and does not correspond to a single physical link |
57+| Mesh | A group of nodes or sub-domains that share a parent network structure at the same network layer and can be modeled with a local path set |
58+| Probe path | A Tracert or PingPong path from a source device to a destination device using a specified source port |
59+| PingList | The set of PingPong tasks that the controller distributes to each source device; each element contains a source IP, destination IP, and source port |
60+| Sub-health | A state in which the network remains connected but latency, pass rates, or congestion counters persistently deviate from the same-layer baseline and have affected or may affect service performance |
61+| Slow fault | A fault that does not manifest as a complete outage but instead shows persistent or intermittent high latency, low throughput, retransmissions, or congestion |
62+| Long tail | A condition in which high-percentile latency such as P99 is significantly higher than the mean or median, causing synchronous collective communication to be slowed by the slowest path |
63+| 3σ | An anomaly threshold constructed as `μ + 3σ` using the mean `μ` and standard deviation `σ` of a time-domain or spatial-domain baseline |
64+| Turn | One complete cycle of PingPong sampling, metric reduction, link solving, and anomaly detection |
65+ 
66+## 3. Overall Architecture and Data Flow
67+ 
68+### 3.1 Two-Layer Architecture
69+ 
70+![Module architecture](figures/0001-topology-based-ccl-monitor-architecture.svg)
71+ 
72+The system consists of two layers:
73+ 
74+- Controller layer: Parses configurations, discovers topology, generates the PingList, distributes tasks in parallel, aggregates results, solves link metrics, and produces artifacts.
75+- Host layer: Runs `rpc_host`, calls `hccn_tool` and `hccn_rping` to execute device-side probing, collects RNIC counters, and returns results through RPC.
76+ 
77+The module relationships are summarized as follows:
78+ 
79+```mermaid
80+flowchart LR
81+ User[Configuration and CLI] --> Controller[probe_topo / probe_controller]
82+ Controller --> Topology[Topology Modeling and Probe Plan]
83+ Controller <-->|RPC| Host[rpc_host]
84+ Host --> HccnTool[hccn_tool: IP / LLDP / Tracert / Stat]
85+ Host --> Rping[hccn_rping: AddTarget / BatchPing]
86+ Controller --> Solver[Metric Reduction and Link Solving]
87+ Solver --> Analyse[Anomaly Analysis]
88+ Analyse --> Output[JSON / TXT / Charts]
89+```
90+ 
91+### 3.2 Topology Discovery Flow
92+ 
93+```mermaid
94+flowchart LR
95+ Config[probe.scope / probe_scope] --> Controller[Controller]
96+ Controller -->|Device list| Host[rpc_host]
97+ Host -->|hccn_tool -ip/-lldp/-traceroute| Device[Device/RNIC]
98+ Device -->|IP, ToR management IP, multi-hop paths| Host
99+ Host -->|RPC results| Controller
100+ Controller -->|Layering, domain partitioning, deduplication| Topology[MeshTopo]
101+ Topology --> Files[probe_topo.json / probe_topo_lldp.json / allpath.json]
102+```
103+ 
104+### 3.3 Network Probing Flow
105+ 
106+```mermaid
107+flowchart LR
108+ Topology[MeshTopo] --> Plan[Generate L1 ring tasks and L2 adjacent-domain tasks]
109+ Plan -->|PingList| Controller[Controller]
110+ Controller -->|RPC: AddTarget / BatchPing| Hosts[Multiple rpc_hosts]
111+ Hosts -->|hccn_rping| Devices[Multiple devices]
112+ Devices -->|P90/P99/Mean/Pass| Hosts
113+ Hosts -->|Batch results| Controller
114+```
115+ 
116+### 3.4 Anomaly Localization Flow
117+ 
118+```mermaid
119+flowchart LR
120+ Raw[PingPong raw results] --> Reduce[Metric reduction]
121+ Reduce --> L1[Path-L1 link equation solving]
122+ Reduce --> L2[End-to-end metrics minus L1 at both ends]
123+ Counter[PFC/CNP counters] --> Analyse[Joint analysis]
124+ L1 --> Analyse
125+ L2 --> Analyse
126+ Analyse --> Artifacts[link_*.txt / l2_status / metrics / bad_link*.txt]
127+```
128+ 
129+The data direction is uniformly `Device → Host → Controller → Artifacts`; control commands and task distribution flow in the opposite direction.
130+ 
131+## 4. Interface Design (User-Facing)
132+ 
133+### 4.1 Build and Environment
134+ 
135+```bash
136+cd <repo_dir>
137+ 
138+export THIRDLIB_ROOT=/usr/local/third_lib
139+export ASCEND_HOME_PATH=/usr/local/Ascend
140+export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux
141+source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh"
142+ 
143+cmake -S . -B build \
144+ -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \
145+ -DCMAKE_BUILD_TYPE=Release
146+cmake --build build -j"$(nproc)"
147+```
148+ 
149+### 4.2 Configuration File
150+ 
151+The user uses a JSON file to describe the deployment topology, probe scope, and runtime parameters. `schema_version=2` is the current recommended format; the default `schema_version` or `schema_version=1` remains backward compatible with the legacy format.
152+ 
153+```json
154+{
155+ "schema_version": 2,
156+ "controller": "node-01",
157+ "deploy": {
158+ "default_ssh_port": 22,
159+ "default_timeout": 5,
160+ "to_path": "~/disp_probe",
161+ "from_path": "."
162+ },
163+ "hosts": [
164+ {
165+ "id": "node-01",
166+ "ip": "10.90.15.67",
167+ "user": "root",
168+ "ssh_key": "/root/.ssh/id_ed25519"
169+ },
170+ {
171+ "id": "node-02",
172+ "ip": "10.90.15.69",
173+ "user": "root",
174+ "password_env": "NODE_02_PASS"
175+ }
176+ ],
177+ "probe": {
178+ "scope": {
179+ "node-01": {"device_range": [0, 3]},
180+ "node-02": {"devices": [4, 5, 6, 7]}
181+ },
182+ "topology": {
183+ "sport_begin": 49152,
184+ "sport_count": 32,
185+ "tree_probe_sport_count": 1,
186+ "topology_optimized": true,
187+ "l2_path_aware": true,
188+ "output_subdir": "",
189+ "allpath_output": "allpath.json",
190+ "l2_path_output": "l2_fullmesh_path.json"
191+ },
192+ "pingpong": {
193+ "times": 50,
194+ "turns": 1000,
195+ "payload_len": 12,
196+ "interval_ms": 1
197+ }
198+ }
199+}
200+```
201+ 
202+| Field | Type/Default | Description and Constraints |
203+| --- | --- | --- |
204+| `schema_version` | int / `1` | Configuration format version; defaults to v1 if absent; `2` enables the current recommended format |
205+| `controller` | string / required (v2) | The controller node host ID; must reference `hosts[].id` |
206+| `hosts` | array / required (v2) | Host inventory; `id` is a stable identifier and `ip` is the connection address; both must be unique |
207+| `hosts[].user` | string / `root` | SSH login user |
208+| `hosts[].ssh_key` | string / nullable | SSH private key path; backward compatible with the legacy name `key_filename`. When configured together with `password_env`, the SSH connection receives the private key path and attempts key/agent/local key authentication first |
209+| `hosts[].password_env` | string / nullable | Reads the SSH password from an environment variable to avoid writing plaintext in the configuration. When configured together with `ssh_key`, this password is passed to SSH for password authentication or as an encrypted private key passphrase; it also serves as the `su` fallback password when `su_password` is not configured |
210+| `deploy.default_ssh_port` | int / `22` | SSH port |
211+| `deploy.default_timeout` | int / `5` | Dispatcher SSH connection timeout in seconds |
212+| `deploy.control_topo` | array/object / nullable | The host tree for distribution and remote execution; in v2, host IDs can be used and are resolved to IPs during parsing |
213+| `deploy.to_path` | string / required | Remote deployment root directory; `~` is expanded for the control user |
214+| `deploy.from_path` | string / `"."` | Local source path; used as the default local sync source directory for the Dispatcher |
215+| `probe.scope` | object / required (v2) | The unique probe scope; keys are host IDs; values can be `{"device_range":[begin,end]}` or `{"devices":[...]}` |
216+| `probe.topology.sport_begin` | positive int / `49152` | Optional; Tracert source port start value |
217+| `probe.topology.sport_count` | positive int / `1` | Optional; number of source ports per directed pair for cross-domain multi-path coverage |
218+| `probe.topology.tree_probe_sport_count` | positive int / `1` | Optional; number of source ports per pair during ring-topology skeleton discovery |
219+| `probe.topology.topology_optimized` | bool / `true` | Optional; `true` uses adjacent-domain same-slot ring coverage; `false` uses cross-domain full-mesh Tracert |
220+| `probe.topology.l2_path_aware` | bool / `true` | Optional; whether to generate additional L2 path discovery artifacts between adjacent L1 domains |
221+| `probe.topology.output_subdir` | string / `""` | Optional; a relative directory under `output/`; must not be an absolute path and must not contain `.`, `..`, or empty path segments |
222+| `probe.topology.allpath_output` | filename / `allpath.json` | Optional; cross-domain path JSON filename; must not contain parent directory references |
223+| `probe.topology.l2_path_output` | filename / `l2_fullmesh_path.json` | Optional; L2 path JSON filename; must not contain parent directory references |
224+| `probe.pingpong.times` | positive int / `50` | Optional; number of samples per PingPong task per turn |
225+| `probe.pingpong.turns` | positive int / `1000` | Optional; number of PingPong turns; the actual cycle also includes execution, RPC, and solving time |
226+| `probe.pingpong.payload_len` | int / `12` | Optional; HCCN Rping payload size in bytes, range `1-1500`. Each target independently generates random bytes of the specified length during AddTarget; the payload is not treated as a string |
227+| `probe.pingpong.interval_ms` | positive int / `1` | Optional; HCCN Rping probe interval in milliseconds |
228+ 
229+The v1-compatible format continues to support the legacy `deploy.host_to_user_pair`, `deploy.controller`, `probe_scope`, `probe_topo.tracert`, and `probe_controller.pingpong` fields. The compatibility mapping is as follows:
230+ 
231+| v1 Field | v2 Field | Description |
232+| --- | --- | --- |
233+| `deploy.host_to_user_pair` | `hosts[]` | Host IP, user, and password/key configuration migrated to the host inventory; v2 uses `hosts[].id` as a stable reference |
234+| `deploy.controller` | `controller` | v1 uses a controller node IP-to-user mapping; v2 uses the controller node host ID |
235+| `deploy.host_to_key_filename` | `hosts[].ssh_key` | v2 consolidates private key paths into the corresponding host entry |
236+| `probe_scope` | `probe.scope` | v1 keys are host management IPs; v2 keys are host IDs and support `device_range` or explicit `devices` |
237+| `probe_topo.tracert` | `probe.topology` | Tracert topology discovery parameters remain compatible; field name prefix adjusted |
238+| `probe_controller.pingpong` | `probe.pingpong` | PingPong probe parameters remain compatible; field name prefix adjusted |
239+ 
240+In v2, `probe.scope` is required; `probe.topology` and `probe.pingpong` are optional and use the default parameters from the table above when omitted. Run `./run.sh migrate-config old.json new.json` to generate a v2 configuration template.
241+ 
242+### 4.3 CLI and Execution Order
243+ 
244+#### 4.3.1 `rpc_host`
245+ 
246+| Parameter | Description |
247+| --- | --- |
248+| `-f, --file <PATH>` | Control JSON, default `./control_json/910b2_info.json` |
249+| `-d, --dev <IFACE>` | Host management network interface name |
250+| `-i, --ip <IP>` | Host management IP |
251+| `-p, --port <PORT>` | RPC listening port |
252+| `--pingpong-local-log` | Save local PingPong result logs on each host; disabled by default |
253+| `--pingpong-log-dir <PATH>` | Local log root directory, default `/root/output` |
254+ 
255+#### 4.3.2 `probe_topo`
256+ 
257+| Parameter | Description |
258+| --- | --- |
259+| `-f, --file <PATH>` | Control JSON; execute topology discovery and generate topology artifacts |
260+ 
261+#### 4.3.3 `probe_controller`
262+ 
263+| Parameter | Description |
264+| --- | --- |
265+| `-f, --file <PATH>` | Control JSON |
266+| `--print-pingpong-plan` | Print the task plan only; do not distribute the PingList or execute PingPong |
267+| `--l1-only` | Probe and solve L1 only; disable L2 FullMesh tasks and L2 output |
268+| `--no-metrics` | Disable PFC/CNP counter collection |
269+ 
270+Recommended execution order:
271+ 
272+```bash
273+# Terminal 0: Clean up residual processes, distribute binaries and configuration, start rpc_host
274+./run.sh deploy
275+ 
276+# Optional: Enable NPU-side PingPong local logging when starting rpc_host
277+./run.sh deploy --pingpong-log /root/output
278+ 
279+# Equivalent expanded steps:
280+python3 ./dispatcher/exec_realtime_cmd.py "ps aux | grep 'rpc_host' | grep -v grep | tr ' ' '\n' | grep -E '^[0-9]+$' | tr '\n' ' ' | xargs -r kill"
281+ 
282+python3 ./dispatcher/disp_file_scp.py './build/rpc_host' './bin/rpc_host'
283+python3 ./dispatcher/disp_file_scp.py \
284+ './control_json/910b2_info.json' './control_json/910b2_info.json'
285+ 
286+python3 ./dispatcher/exec_realtime_cmd.py -l \
287+ './bin/rpc_host -f ./control_json/910b2_info.json'
288+ 
289+# Terminal 1: Discover topology
290+./build/probe_topo -f ./control_json/910b2_info.json
291+ 
292+# Terminal 1: Continuous probing and analysis
293+./build/probe_controller -f ./control_json/910b2_info.json
294+```
295+ 
296+`disp_file_scp.py` recursively determines distribution targets based on `deploy.control_topo`; only when this field is empty does it use all hosts from `host_to_user_pair`.
297+ 
298+### 4.4 Artifact Contract
299+ 
300+#### 4.4.1 Topology Discovery Artifacts
301+ 
302+| Artifact | Format | Content |
303+| --- | --- | --- |
304+| `output/<subdir>/probe_topo.json` | JSON | Network layers and mesh skeleton discovered by ring Tracert |
305+| `output/<subdir>/probe_topo_lldp.json` | JSON | L1 domain topology aggregated by LLDP management IP; the default topology input for `probe_controller` |
306+| `output/<subdir>/allpath.json` | JSON | Cross-domain pairs, port samples, multi-hop IPs, empty paths, and unique path statistics |
307+| `output/<subdir>/l2_fullmesh_path.json` | JSON | L2 path discovery results between adjacent L1 domains; generated only when `l2_path_aware=true` |
308+ 
309+The topology JSON uses `status` to indicate device count completeness: `complete` when the number of discovered devices matches the probe scope configuration (v2 `probe.scope` or v1 `probe_scope`); `incomplete` when they do not match. The JSON also records `configured_device_count`, `discovered_device_count`, the configured device list, and the discovered device IP list. Completeness compares only device counts and does not change `status` based on LLDP domain count, empty path rate, or path coverage rate.
310+ 
311+#### 4.4.2 Network Status Artifacts
312+ 
313+```text
314+output/<time>/
315+├── link_lat.txt
316+├── link_pass_rate.txt
317+├── bad_link.txt
318+├── bad_link_candidate.txt
319+├── l2_status/
320+│ ├── l2_path_lat.txt
321+│ └── l2_path_passrate.txt
322+└── metrics/
323+ ├── mac_tx_pfc_pkt_num.txt
324+ ├── mac_rx_pfc_pkt_num.txt
325+ ├── roce_tx_cnp_pkt_num.txt
326+ └── roce_rx_cnp_pkt_num.txt
327+```
328+ 
329+| Artifact | Format Contract |
330+| --- | --- |
331+| `link_lat.txt` | The first row contains fixed-width L1 link column names `[device_ip-tor_ip]`; each subsequent row contains floating-point latency values for one turn |
332+| `link_pass_rate.txt` | Column order matches `link_lat.txt`; value range target is `[0,1]` |
333+| `bad_link.txt` | Text alerts; each row contains alert type, link endpoints, latency, and pass rate; multiple links can appear separately in the same turn |
334+| `bad_link_candidate.txt` | Latency candidate events; written when either the time-domain or spatial-domain 3σ condition exceeds the threshold for 3 consecutive valid turns; not equivalent to a persistent alert |
335+| `l2_path_lat.txt` | Tab-separated: `turn, task_index, tag, from_label, from_ip, to_label, to_ip, src_sport, l2_path_lat` |
336+| `l2_path_passrate.txt` | Tab-separated; the last column is `pass_rate`; remaining fields are the same as above |
337+| `metrics/*.txt` | CSV: `time,port,value,valid`; `port` is formatted as `<host>_dev<id>`. When all four counter fields are parsed successfully, `valid=true`; when a key is missing, a value is non-numeric, or a command fails, the value is set to `0` as a placeholder and `valid=false`; the sampling interval is approximately 1 second |
338+ 
339+Run `python3 ./plot/topo_plot.py` and `python3 ./plot/status_plot.py --input-dir "output/<time>"` to generate topology and status charts. Charts are derived artifacts; JSON/TXT/CSV are the stable data interfaces.
340+ 
341+## 5. External Dependencies
342+ 
343+### 5.1 Runtime Interfaces Within the Ecosystem
344+ 
345+| Dependency/Interface | Invocation | Purpose | Input Convention | Output and Parsing Convention | Stability Source | Known Limitations |
346+| --- | --- | --- | --- | --- | --- | --- |
347+| HCCN Device IP | `hccn_tool -i <dev> -ip -g` | Maps device ID to HCCN IP | `<dev>` is the local non-negative device ID | Parse tool output line by line and extract valid IPs; missing or format changes are treated as query failures for that device | Assumed based on current CANN tool implementation | Text format has no versioned schema |
348+| LLDP Neighbor | `hccn_tool -i <dev> -lldp -g` | Obtain the directly connected Leaf/ToR management IP | Same as above | Extract the switch management IP; null values are assigned to the `unknown:<device>` domain | Assumed based on current implementation | Domain partitioning is unreliable when LLDP is disabled or permissions are insufficient |
349+| Tracert | `hccn_tool -i <dev> -traceroute ... -sport <port>` | Obtain multi-hop paths | Source device, destination IP, source port | Return an ordered multi-hop IP list; an empty list indicates a failed sample; single-hop results are padded at the controller to preserve path shape | Assumed based on current implementation | TC/DSCP and return-direction port controllability are limited; ECMP coverage depends on source port hashing |
350+| RNIC Stat | `hccn_tool -i <dev> -stat -g` | Obtain cumulative PFC/CNP counters | Device ID | Parse `mac_tx_pfc_pkt_num`, `mac_rx_pfc_pkt_num`, `roce_tx_cnp_pkt_num`, and `roce_rx_cnp_pkt_num` as `key:value`; the RPC line carries a validity flag; the controller outputs `valid=true/false` | Assumed based on current implementation | When any target key is missing, the value is not an unsigned integer, or the command fails, the entire row has `valid=false`; the value `0` is a placeholder only and is excluded from joint analysis |
351+| HCCN Rping Initialization | `HccnRpingInit/Deinit` | Initialize the probe context for each device | Local device ID | The return code must indicate success before probing can proceed | CANN/HCCL headers and libraries | The runtime environment must provide a matching version of `libhccl*` |
352+| HCCN Rping Target Configuration | `HccnRpingAddTarget` | Distribute destination IP, source/return ports | Target IP, source port, receive port, and other parameters | The return code maps to RPC success/failure | Same as above | Target count and port resource limits must be verified per version |
353+| HCCN Rping Batch Probing | `HccnRpingBatchPingStart/Stop/GetResult/GetPayload` | Execute PingPong and obtain statistics | Device, task set, `times` | Normalized to a `uint64_t` array of `[P90Lat, P99Lat, Mean, Pass]` | Same as above | The raw latency unit must be confirmed by the interface version and documented in the release documentation; current logs display values in ms |
354+ 
355+These interfaces are all runtime dependencies on the host. The project does not modify the HCCL/HCOMM data plane and remains consistent with the HCCL-HCOMM decoupling principle; however, the C++ build still requires CANN/HCCL/ACL headers and link libraries and cannot be described as "completely free of compile-time dependencies."
356+ 
357+### 5.2 General Third-Party Libraries
358+ 
359+| Dependency | Version | Purpose | License | Dependency Stage |
360+| --- | --- | --- | --- | --- |
361+| CLI11 | 2.5.0 | C++ CLI parsing | BSD-3-Clause | Compile-time, header-only |
362+| nlohmann_json | 3.12.0 | JSON configuration and artifacts | MIT | Compile-time, header-only |
363+| Eigen | 3.4.0 | SVD link equation solving | MPL-2.0 | Compile-time, header-only |
364+| rpclib | 2.3.0 | Controller-Host RPC | MIT | Compile-time static linking + runtime communication |
365+| fmt | 12.0.0 | Optional formatting support | MIT | Optional, compile-time |
366+| spdlog | 1.15.3 | Optional logging support | MIT | Optional, compile-time |
367+| paramiko | 5.0.0 | Dispatcher SSH | LGPL-2.1-or-later | Python runtime |
368+| scp | 0.15.0 | File distribution | LGPL-2.1-or-later | Python runtime |
369+| matplotlib | 3.9.4 | Chart plotting | PSF-based | Python plotting runtime |
370+ 
371+Release packages must generate third-party notices from `third_party/manifest.json`, `third_party/python/requirements.txt`, and the actual installed package metadata. If versions or licenses are inconsistent, the release build audit result takes precedence.
372+ 
373+## 6. Key Algorithm Design
374+ 
375+### 6.1 Layered Domain Partitioning and L1/L2 Definitions
376+ 
377+1. Use device IPs and ring Tracert to identify the network layer skeleton.
378+2. Use LLDP management IPs to aggregate devices into their directly connected Leaf/ToR domains.
379+3. Define the edges from in-domain devices to the Leaf/ToR as L1 links.
380+4. Define the remaining network segments between different Leaf/ToR domains as L2 paths. The current design provides only path-level candidates for L2 and does not guarantee unique physical link localization.
381+ 
382+### 6.2 Minimum-Cost Topology Probing
383+ 
384+Let the cluster have `D` L1 domains in total, where domain `i` has `n_i` devices, the total device count is `N=Σn_i`, and each pair uses `s` source port samples.
385+ 
386+- Skeleton discovery: All devices form a directed ring; each device probes the next device. The number of paths is approximately `N × tree_probe_sport_count`, which is `O(N)`.
387+- Optimized cross-domain coverage: Each domain connects only to the same slot in the next domain. The number of pairs is approximately `Σ min(n_i,n_(i+1))`, and the number of paths is multiplied by `sport_count`. With uniform domain sizes, this is approximately `N × s`, which is `O(Ns)`.
388+- Non-optimized cross-domain coverage: All devices in different domains form a directed full mesh. The number of pairs is `Σ_{i≠j} n_i n_j = N²-Σn_i²`, which is worst case `O(N²s)`.
389+- L1 PingPong: Each domain generates tasks based on solvable rings. With uniform domain sizes, this is approximately `O(N)` pairs.
390+- Current L2 PingPong: Adjacent domains form a ring, and FullMesh is executed between adjacent domains. The number of tasks is `Σ n_i n_(i+1)`; with uniform domain size `k`, this is `Dk²=Nk`. This is not strictly `O(N)`, so the task volume must be reviewed with `--print-pingpong-plan` before deployment, and `--l1-only` should be used when necessary.
391+ 
392+"Minimum cost" refers to reducing the number of paths under the constraint that the target link equations remain identifiable. It does not claim that the entire process always has linear complexity.
393+ 
394+### 6.3 PingPong Metric Reduction
395+ 
396+The raw arrays returned by the host map to the controller metric enumerations as follows:
397+ 
398+| Raw `PingpongResult` | Index | Controller `PingpongMetric` | Processing |
399+| --- | --- | --- | --- |
400+| `P90Lat` | 0 | `P90Lat` | Convert to `float` |
401+| `P99Lat` | 1 | `P99Lat` | Convert to `float` |
402+| `Mean` | 2 | `MeanLat` | Convert to `float`; renamed to avoid confusion with statistical mean |
403+| `Pass` | 3 | `LogPassRate` | Compute `log2(pass/times)` |
404+| `Size` | 4 | None | Enumeration sentinel indicating the array field count; not an observed metric |
405+ 
406+The pass rate is solved in the log domain:
407+ 
408+```text
409+r_path = pass / times
410+b_path = log2(r_path)
411+b_path = Σ log2(r_link)
412+```
413+ 
414+Boundary conventions:
415+ 
416+- `times <= 0`: Configuration or invocation error; fail immediately.
417+- `pass > times`: Data anomaly; the sample is marked invalid and excluded from solving; validation must be implemented before release.
418+- `pass = 0`: Mathematically `-∞`. The current implementation uses `-1e10` as a finite sentinel for solving; the target implementation should also carry `all_lost=true`, output a final pass rate of `0`, and prevent the sentinel from being misinterpreted as a comparable normal value.
419+- Missing fields in the raw array: Latency returns `NaN`; the pass rate is currently handled as `pass=0`. The target implementation should uniformly mark such samples as invalid.
420+ 
421+### 6.4 L1 Link Solving
422+ 
423+Build linear equations for each path sample:
424+ 
425+```text
426+A × x = b
427+```
428+ 
429+- `A[m,n]` indicates whether path `m` traverses L1 link `n`.
430+- For latency, `b` is the path P99 latency and `x` is the link latency.
431+- For pass rate, `b` is the `log2` value of the path pass rate; after solving, `exp2` recovers the link pass rate.
432+- Use Eigen `BDCSVD` to compute the least-squares solution.
433+ 
434+The probe plan must ensure that the columns corresponding to target links are identifiable. If the matrix rank is insufficient, the condition number is too large, the input contains invalid samples, or the solution is clearly out of bounds, the link results for that turn must be marked as `NaN/invalid` and no definitive fault conclusion is produced. The current implementation performs SVD rank and condition number checks and returns `NaN` when these checks fail.
435+ 
436+### 6.5 L2 Path Metrics
437+ 
438+The current L2 latency calculation is:
439+ 
440+```text
441+l2_path_lat = end_to_end_p99 - src_l1_lat - dst_l1_lat
442+```
443+ 
444+When L1 at either end is unsolvable, the output is `NaN`. The L2 pass rate directly uses the end-to-end pair pass rate and does not yet subtract the L1 pass rates at both ends; therefore, the artifact name uses `l2_path_passrate` rather than `l2_link_passrate`.
445+ 
446+### 6.6 Anomaly Detection
447+ 
448+The target anomaly detection handles latency and pass rate separately. Latency anomalies use two baselines: the time-domain baseline maintains historical statistics per L1 link, and the spatial-domain baseline maintains statistics across all valid L1 links in the same topology layer for the current turn.
449+ 
450+1. Exclude `NaN`, all-lost sentinels, and samples in known maintenance windows.
451+2. Time-domain baseline: For each L1 link, compute the mean `μ_t` and standard deviation `σ_t` using historical valid turns. The current implementation uses cumulative historical statistics; a rolling window or median/MAD can be adopted later.
452+3. Spatial-domain baseline: Compute the mean `μ_s` and standard deviation `σ_s` across all valid L1 link latencies in the same topology layer for the current turn. Links are no longer filtered by IP string or specific network segment.
453+4. Latency alert condition: Both `latency > μ_t + 3σ_t` and `latency > μ_s + 3σ_s` must be satisfied simultaneously. After the condition is met for 3 consecutive valid turns, write to `bad_link.txt`.
454+5. Pass rate anomaly condition: `pass_rate < 0.99`; write an alert immediately when the condition is met. The threshold will be made configurable later.
455+6. Latency candidate events: When either the time-domain or spatial-domain 3σ condition exceeds the threshold for 3 consecutive valid turns, write to `bad_link_candidate.txt` without generating a persistent alert.
456+7. PFC/CNP counters are currently collected and stored only and do not participate in alert confidence determination. In the future, when counters increase simultaneously, they can raise the confidence of congestion-type anomalies, but counters are never used as the sole alert condition.
457+ 
458+The current code uses mean and standard deviation to maintain L1 link time-domain and spatial-domain baselines.
459+ 
460+## 7. Internal Module Implementation
461+ 
462+### 7.1 Directory and Responsibilities
463+ 
464+```text
465+disp_probe-main/
466+├── control_json/ # User configuration samples
467+├── dispatcher/ # SSH/SCP distribution and remote execution
468+├── docs/ # RFC, user, and environment documentation
469+├── plot/ # Topology and status visualization
470+├── scripts/ # System and third-party dependency installation
471+├── src/
472+│ ├── probe_topo.cpp # Topology discovery entry point
473+│ ├── probe_controller.cpp # Probing, solving, and analysis entry point
474+│ ├── rpc_host.cpp # Host RPC service
475+│ └── util/
476+│ ├── helper/ # Workflow coordination, PingList, equation solving
477+│ ├── topo/ # Configuration, control topology, MeshTopo
478+│ ├── tool/ # Tracert, PingPong, metric collection
479+│ ├── rpc_call/ # RPC proxy
480+│ └── file_path/ # Working directory management
481+└── third_party/ # Dependency manifest
482+```
483+ 
484+### 7.2 Cross-Module Contracts
485+ 
486+The RFC retains only signatures that cross module or process boundaries; purely internal getters and setters are not listed.
487+ 
488+```cpp
489+using PingpongRawResult =
490+ std::vector<std::vector<std::vector<uint64_t>>>; // device -> task -> metric
491+using PingpongMetricMatrix =
492+ std::vector<std::vector<float>>; // device -> task
493+using LinkMetricVector = std::vector<float>; // link_global_id -> value
494+ 
495+using HccnDeviceIpListParallel =
496+ std::function<std::vector<std::vector<std::string>>(
497+ const std::vector<std::string>& control_devices,
498+ const std::vector<int>& device_counts)>;
499+ 
500+MeshTopo& probe_topo_ring();
501+MeshTopo& probe_topo();
502+std::map<std::string, std::vector<std::tuple<std::string, std::string, int>>>&
503+get_pinglist();
504+void set_pinglist(const PingList& pinglist);
505+PingpongRawResult get_pingpong_res(int times);
506+PingpongMetricMatrix reduce_pingpong_res(
507+ const PingpongRawResult&, PingpongMetric, int times);
508+LinkMetricVector solve_pingpong_res(
509+ const PingpongMetricMatrix&, const PingList&);
510+```
511+ 
512+Differences between `probe_topo_ring()` and `probe_topo()`:
513+ 
514+| Interface | Behavior | Use Case |
515+| --- | --- | --- |
516+| `probe_topo_ring()` | Executes only `gen_topo()` to generate the layer/mesh skeleton from ring Tracert | New configuration path; `probe_topo.cpp` subsequently supplements artifacts with LLDP and explicit cross-domain probing |
517+| `probe_topo()` | Executes `gen_topo()`, in-domain `mesh_full_trace()`, `gen_pinglist()`, supplementary Tracert, and link ID assignment | Legacy configuration compatibility path or when a complete solving state must be constructed in one pass |
518+ 
519+Host RPC boundary:
520+ 
521+```cpp
522+std::vector<std::vector<std::vector<std::string>>>
523+tracert_ports_multi_by_src_ip_with_sport_begin(
524+ std::string src_dev_ip,
525+ std::vector<std::string> targets,
526+ std::vector<int> port_num,
527+ int sport_begin);
528+ 
529+int pinglist_insert_muti(
530+ std::string src_dev_ip,
531+ std::vector<std::string> target_dev_ips,
532+ std::vector<int> src_ports,
533+ std::vector<int> return_ports,
534+ int rx_num);
535+ 
536+std::vector<std::vector<uint64_t>>
537+ud_pingpong_tx_muti(std::string src_dev_ip, int times);
538+ 
539+std::vector<std::string> get_metrics_counter_name();
540+std::vector<std::vector<uint64_t>> get_metrics_counter_value();
541+```
542+ 
543+### 7.3 `metrics_collector` and Analysis Module Responsibilities
544+ 
545+- `metrics_collector`: Calls `hccn_tool -stat -g` every second and parses four types of cumulative PFC/CNP counters. The in-memory queue retains up to 100 rows and is cleared after the controller pulls the data. Read errors are currently swallowed and zero values are written; the target implementation should return a validity status and error reason.
546+- Analysis logic (in `probe_controller.cpp`): Maintains time-domain and spatial-domain baselines, detects latency/pass-rate anomalies, and outputs `bad_link.txt` and `bad_link_candidate.txt`. PFC/CNP counters are currently collected and stored only and do not participate in alert confidence determination.
547+ 
548+## 8. Impact Analysis
549+ 
550+### 8.1 Production Intrusion Impact
551+ 
552+| Resource | Impact Source | Control Measures |
553+| --- | --- | --- |
554+| NPU/RNIC | HCCN Rping initialization, AddTarget, probe packet transmission and reception | Limit per-device task count and `times`; run `--print-pingpong-plan` before deployment; support `--l1-only` |
555+| Network bandwidth | Tracert and PingPong packets | Topology discovery runs at low frequency; status probing is domain-partitioned; configure probe intervals; stop sending when budget is exceeded |
556+| Collective communication latency | Probe traffic competes with production traffic for queues, hash paths, and port resources | A/B validation; gradual rollout; avoid full topology discovery during peak business hours |
557+| Host CPU/memory | RPC, SSH, result aggregation, 1-second counter polling | Host-level parallelism; counter queue capped at 100; monitor controller/host CPU and RSS |
558+| File system | Long-turn text artifacts grow continuously | Rotate by time directory; apply retention policies and compression; configure disk quotas before release |
559+ 
560+### 8.2 Further Evaluation
561+ 
562+For each RNIC, assuming 500 B probe packets are sent every 1 ms per turn over a 100 Gbps network, the bandwidth utilization is less than one thousandth. The scheme is further validated through multi-host hccl_test to demonstrate that the impact of probe traffic on hccl task completion time is less than 1%. The default probe cycle sends probe packets every 1 ms for `turns` consecutive turns, with `times` samples per turn.
563+ 
564+### 8.3 Failure Modes and Degradation
565+ 
566+| Failure Mode | Detection | Handling and Degradation | Artifact Semantics |
567+| --- | --- | --- | --- |
568+| Tracert failure/empty path | Empty hop list, timeout, or non-zero status returned | Retry limited times for the pair/port; if still failing, mark as unmeasured rather than guessing the path; terminate topology release when the empty path rate exceeds the threshold | `allpath.json` records empty samples and statistics |
569+| RPC timeout | Client exception or no response | Isolate the single host; other hosts continue; exponential backoff retry; do not treat timeouts as packet loss | Corresponding samples are `invalid` |
570+| Partial host unreachability | SSH/RPC health check failure | Remove from the current turn scope and report coverage; if matrix identifiability is compromised, stop link solving | Output the missing host list |
571+| Incomplete topology discovery | Discovered device count does not match the probe scope configuration count | `probe_topo` writes diagnostic artifacts and stops subsequent LLDP/path probing; `probe_controller` refuses to execute PingPong and alerting | Topology marked `status=incomplete`; configured and discovered counts recorded; LLDP domain or path coverage completeness is no longer checked |
572+| Counter read error | Missing key, non-numeric value, or command failure | Sample marked `valid=false`; excluded from joint analysis; the main probe flow continues | `metrics/*.txt` writes `value=0,valid=false`, distinguishable from genuine `value=0,valid=true` |
573+| PingPong total packet loss | `pass=0` | Generate a connectivity candidate alert directly; use a sentinel in the log domain but do not compare it as a normal value | Final pass rate is 0 with `all_lost` flag |
574+| Insufficient equation rank / ill-conditioned | Rank and condition number checks | Do not output definitive link localization; add probe paths or degrade to path-level alerts | Link values are `NaN/invalid` |
575+| Output file not writable | File open/write failure | Terminate the current controller run to avoid probing without artifacts | Process exits with non-zero code |
576+| Controller exit | Process signal or exception | Stop distributing tasks; host services can remain for reconnection; cleanup policies are managed by operations | Turns already written to disk are retained |
577+ 
578+### 8.4 Overhead Model
579+ 
580+Let:
581+ 
582+- `P`: Number of directed PingPong tasks per turn;
583+- `times`: Number of packets per task per turn;
584+- `B_req`, `B_rsp`: Average request and response bytes on the wire, including protocol overhead;
585+- `T_cycle`: Turn cycle in seconds;
586+- `C_link`: Port line rate in bit/s.
587+ 
588+The cluster-wide average probe bandwidth is estimated as:
589+ 
590+```text
591+BW_cluster = P × times × (B_req + B_rsp) × 8 / T_cycle
592+```
593+ 
594+The average utilization for a single device `d` is:
595+ 
596+```text
597+Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link)
598+```
599+ 
600+### 8.5 Security and Operations Impact
601+ 
602+- The Dispatcher uses SSH/SCP, and the configuration may contain plaintext passwords. In production environments, prefer private keys and restrict configuration file permissions.
603+- `hccn_tool` and device operations may require elevated privileges. Use least-privilege accounts and command allowlists.
604+- The RPC interface does not define authentication or encryption in this document and must be deployed in a controlled management network. A separate security design is required before use across security domains.
605+ 
606+## 9. Compatibility, Feature Toggles, and Gradual Rollout
607+ 
608+### 9.1 Compatibility
609+ 
610+- The tool runs as an independent process and does not modify training processes or HCCL/HCOMM interfaces.
611+- The current recommended configuration uses `schema_version=2`, `hosts[]`, and `probe.scope` as required contracts; `probe.topology` and `probe.pingpong` are optional parameter groups. The default `schema_version` or `schema_version=1` remains backward compatible with the `probe_scope + probe_topo.tracert + probe_controller.pingpong` format. When an entire probe configuration group is missing, the legacy ranktable path is preserved, but no new capabilities are added.
612+- Once JSON/TXT artifacts are consumed by external systems, new fields must remain backward compatible. Breaking changes must upgrade the schema or filename.
613+ 
614+### 9.2 Feature Toggles
615+ 
616+| Toggle | Effect | Recommended Scenario |
617+| --- | --- | --- |
618+| Do not start `probe_controller` | Completely disable continuous probing | Default off / rollback |
619+| `--print-pingpong-plan` | Review the plan only; no network probing | Pre-deployment capacity check |
620+| `--l1-only` | Disable L2 FullMesh | Initial rollout, bandwidth-constrained environments, or excessive L2 tasks |
621+| `--no-metrics` | Disable PFC/CNP collection | `hccn_tool -stat` is unstable or permissions are insufficient |
622+| `topology_optimized=true` | Use low-overhead cross-domain coverage | Default production mode |
623+| `l2_path_aware=false` | Do not generate additional L2 path discovery tasks | When only L1 is of interest |
624+ 
625+A unified top-level `feature_enabled` flag and configurable turn interval should be added in the future. The current version controls behavior through process startup and the toggles listed above.
626+ 
627+### 9.3 Gradual Rollout Strategy
628+ 
629+Lab environment with 2 hosts and 16 devices: Validate interfaces, artifacts, and fault injection.
630+ 
631+## 10. Test Plan
632+ 
633+### 10.1 Objective-Use Case Mapping
634+ 
635+| Use Case | Problem/Criteria | Scenario and Checkpoints | Level |
636+| --- | --- | --- | --- |
637+| T1 Configuration Parsing | Q1, compatibility | Required fields, defaults, duplicate devices, range strings, illegal output paths | UT |
638+| T2 Topology Modeling | Q1, topology completeness | Control topology, multiple LLDP domains, unknown LLDP, JSON round-trip, and exception access | UT |
639+| T3 Probe Plan Scale | Q4, bandwidth threshold | Validate L1 ring, L2 adjacent-domain FullMesh task formulas, and task deduplication | UT |
640+| T4 Metric Reduction | Q2 | P90/P99/Mean mapping; `times<=0`, `pass=0`, `pass>times`, missing fields | UT |
641+| T5 Equation Solving | Q3, localization accuracy | Solvable, overdetermined, rank-deficient, and NaN inputs | UT |
642+| T6 Counter Anomaly | Q2/Q3 | Missing key, illegal value, device ID normalization, counter increment and reset/wraparound | UT |
643+ 
644+### 10.2 UT Integration
645+ 
646+The current repository uses CMake `BUILD_TESTS` and CTest. There is no available root-level `build.sh -u/-s` entry point. The RFC follows the repository as it stands:
647+ 
648+```bash
649+cmake -S . -B build-test \
650+ -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \
651+ -DBUILD_TESTS=ON \
652+ -DCMAKE_BUILD_TYPE=Debug
653+cmake --build build-test -j"$(nproc)"
654+ctest --test-dir build-test --output-on-failure -L UT
655+```
656+ 
657+T1, T2, T3, T4, T5, and T6 are implemented under `tests/` and registered by `tests/CMakeLists.txt` with the CTest `UT` label. The tests do not depend on real NPUs, RPC hosts, or switches and can run in any development environment that satisfies the compilation dependencies. If the project later restores a unified `build.sh -u/-s`, it should wrap the CMake/CTest flow above rather than creating a separate test system.
658+ 
659+## 11. Risks and Mitigations
660+ 
661+| Risk | Impact | Mitigation | Status |
662+| --- | --- | --- | --- |
663+| `traffic_tc/dscp` takes effect only on the source side; return-direction TC/DSCP and source port are uncontrollable | Probe paths diverge from production paths, potentially causing false packet-loss or latency measurements | Drive interface capabilities through Issue 261; mark confidence levels and use multi-source-port and production A/B calibration until resolved | Known, being tracked |
664+| L2 is currently path-level rather than physical-link-level | Cannot uniquely locate a specific edge between Spine and Leaf | Name artifacts as "path"; output Top-10 pairs and paths; combine with switch telemetry for secondary localization | Mitigated but capability is limited |
665+| Simple Z-Score-based anomaly analysis is sensitive to non-stationary, heavy-tailed distributions | False positives or false negatives | Latency requires 3 consecutive turns and layered baselines; pass rate below threshold triggers immediate alert; evaluate MAD/quantile approaches | Currently using mean/standard deviation + 3σ for latency detection, validated on 16 devices; larger topologies require further validation |
666+ 
667+## 12. Alternatives
668+ 
669+| Approach | Probe Overhead | Localization Accuracy | Implementation/Deployment Complexity | RoCE Compatibility | Conclusion |
670+| --- | --- | --- | --- | --- | --- |
671+| R-pingmesh | Moderate; depends on Traceroute with simulated service five-tuples | Can observe paths closer to production traffic | High; depends on eBPF and requires five-tuple extraction at RDMA QP creation | If RoCE five-tuple generation moves to user space, kernel-side extraction becomes difficult | Not adopted for now |
672+| NetBouncer | Depends on probe frequency | Path/device-level | Requires special packet processing | Depends on IP-in-IP; not suitable for the current RoCE environment | Not adopted |
673+| RDMA Pingmesh | `O(N²)` for full mesh | End-to-end anomalies visible; link localization limited | Moderate | Native RDMA/RoCE, but large-scale probe cycles are long | Baseline comparison only |
674+| This scheme | L1 near `O(N)`; L2 currently adjacent-domain FullMesh `Σn_i n_(i+1)` | L1 link-level; L2 path/pair-level | Medium-high; requires topology discovery, RPC, solving, and analysis | Uses HCCN/Rping, designed for the current Ascend RoCE environment | Recommended for continued development |
675+ 
676+## 13. Open Questions
677+ 
678+Items that are already known and have mitigations are listed in Section 11 and are not repeated here.
679+ 
680+| ID | Question | Decision Needed |
681+| --- | --- | --- |
682+| O1 | Whether L2 must be further localized from pair-level to physical switch link | If needed, introduce switch telemetry or other information for further identification |
683+| O2 | Whether anomaly baselines should use mean/3σ or median/MAD | Decide the specific metric to adopt based on actual deployment conditions |
684+| O3 | RPC authentication, encryption, and version negotiation | Confirm that the system is limited to controlled management networks, or add an independent security design |
685+| O4 | `hccn_rping` raw latency unit, AddTarget limit, and CANN version compatibility matrix | Fix through upstream interface documentation or a versioned adaptation layer; confirm that hccn_tool-related UT test cases pass before use |
686+ 
687+## Review Record
688+ 
689+- PR 2491 first-round feedback: Direction acknowledged; requested chapter restructuring, impact analysis supplement, user/external dependency contract supplement, algorithm boundary clarification, and test mapping.
690+- This revision: Addressed each first-round comment item by item.
@@ -0,0 +1,1239 @@
1+# RFC: Host NIC Plugin Mechanism
2+ 
3+- Start Date: 2026-08-14
4+- RFC PR Number: 4610
5+- Status: accept
6+ 
7+---
8+ 
9+## 1. Summary
10+ 
11+Add a Host NIC (Network Interface Card) plugin mechanism to HCOMM that allows developers to replace built-in protocol implementations or extend new communication protocols as standalone `.so` files without modifying the original HCOMM source code.
12+ 
13+This design embeds a C function pointer table (ops table) into the `Channel` / `Endpoint` base classes. The corresponding ops table and ctx context are populated at creation time, and calls are dispatched directly through C function pointers at runtime.
14+ 
15+---
16+ 
17+## 2. Background and Motivation
18+ 
19+The communication protocol types currently supported by HCOMM (HCCS, RoCE, PCIe, SIO, UB series, and so on) are all implemented as built-in modules. Adding a new NIC protocol requires modifying the protocol enumeration and internal implementation, which leads to the following issues:
20+ 
21+- External developers cannot contribute new NIC support without modifying the HCOMM source code
22+- Built-in code is coupled with third-party code, which hinders community collaboration
23+- The barrier to experimenting with new protocols is high because a deep understanding of the internal HCOMM architecture is required
24+ 
25+### 2.1 Target Scenarios
26+ 
27+| Scenario | Description | Typical Use Case |
28+|----------|-------------|-----------------|
29+| **Protocol Replacement** | Replace the built-in implementation of a specific protocol in HCOMM | `COMM_PROTOCOL_ROCE` → custom optimized version |
30+| **Protocol Extension** | A hardware vendor registers a brand-new protocol number for a new RNIC | `COMM_PROTOCOL_CUSTOM_BASE + 0` → new NIC driver |
31+| **Performance Experimentation** | Rapidly validate a new communication backend in the `experimental/` directory | Custom URMA variant without touching the main branch code |
32+ 
33+---
34+ 
35+## 3. Design Goals and Non-Goals
36+ 
37+### 3.1 Design Goals
38+ 
39+| # | Goal | Description |
40+|---|------|-------------|
41+| G1 | Protocol Replacement | The plugin takes over an existing built-in protocol number; the built-in implementation no longer participates |
42+| G2 | Protocol Extension | The plugin registers a new protocol number (≥ `COMM_PROTOCOL_CUSTOM_BASE`) to implement a brand-new communication backend |
43+| G3 | Zero-Branch Dispatch for Interfaces | Except for some data-plane interfaces that are not exclusive to the 950 series, all other interfaces use zero-branch dispatch |
44+| G4 | Bind Ops Table at Creation Time | The Endpoint / Channel is populated with a C ops table at creation time, and runtime calls go directly through C function pointers |
45+| G5 | Independent Build | Gated by `ENABLE_EXPERIMENTAL`; the plugin .so is compiled and distributed independently without linking `libhcomm.so` |
46+| G6 | Optimal Performance | For data-plane Channel dispatch, Type 1 interfaces share the same overhead path for both built-in and plugin; Type 2 built-in interfaces use the legacy path |
47+| G7 | One Plugin per Protocol | Only one plugin binary is supported per communication protocol; a later-loaded plugin overwrites an earlier one |
48+ 
49+### 3.2 Non-Goals
50+ 
51+| # | Description |
52+|---|-------------|
53+| N1 | A single plugin taking over multiple different protocols is not supported |
54+| N2 | No modifications to any public header files under the `include/` directory |
55+| N3 | No modifications to the implementation code of any built-in Channel/Endpoint subclasses |
56+| N4 | Runtime unloading or hot-updating of plugins is not supported |
57+ 
58+---
59+ 
60+## 4. Overall Architecture
61+ 
62+### 4.1 Architecture Overview
63+ 
64+The core idea is to embed two fields in the `Channel` / `Endpoint` base classes: `nicOps_` (ops table pointer) and `nicCtx_` (ctx parameter). These fields are populated at creation time, and all runtime calls are unified through C function pointer invocations. The built-in implementation is integrated into the same mechanism through the `g_BuiltinChannelOps` / `g_BuiltinEndpointOps` wrapper function tables. The plugin implementation is output when `CreateEndpoint` and `CreateChannel` are called through `dlopen`-loaded symbols. The calling layer does not need to distinguish between built-in and plugin for most interfaces.
65+ 
66+### 4.2 Protocol Replacement vs. Protocol Extension
67+ 
68+Both modes share the same mechanism. The only difference is whether the protocol number registered by the plugin conflicts with a built-in protocol:
69+ 
70+| Mode | Protocol Number | Behavior |
71+|------|----------------|----------|
72+| **Replacement** | All non-RESERVED built-in enumeration values < `COMM_PROTOCOL_CUSTOM_BASE` (1000) | The plugin mapping table is hit during Endpoint creation, and the plugin path is used; the built-in Endpoint factory is not called |
73+| **Extension** | Custom protocol numbers ≥ `COMM_PROTOCOL_CUSTOM_BASE` (1000) | The framework has no built-in implementation; only the plugin path is used during Endpoint creation, and the built-in path returns an error because the protocol number is unrecognized |
74+ 
75+- The plugin declares its supported protocol number set (up to 4) in the `protocols[]` array returned by `HcommNicPluginGetInfo()`.
76+- COMM_PROTOCOL_CUSTOM_BASE=1000 is defined in nic_plugin_manager.h and serves as the boundary constant for plugin protocol numbers. The CommProtocol field carries custom values ≥1000 with int32_t semantics.
77+ 
78+### 4.3 Layered Architecture
79+ 
80+```text
81+┌──────────────────────────────────────────────────────────────────┐
82+│ include/hcomm_res.h / include/hcomm_primitives.h (Public API) │
83+├──────────────────────────────────────────────────────────────────┤
84+│ Calling Layer │
85+│ HcommWriteNbi(ch, ...) { │
86+│ auto* ch = CHANNEL_FROM_HANDLE(channel); │
87+│ return ch->GetNicOps()->writeNbi(ch->GetNicCtx(), ...); │
88+│ } │
89+│ HcommMemReg(ep, ...) { │
90+│ auto endpoint = GetEndpointMap().GetEndpoint(ep); │
91+│ return endpoint->GetNicOps()->registerMemory( │
92+│ endpoint->GetNicCtx(), ...); │
93+│ } │
94+├──────────────────────────────────────────────────────────────────┤
95+│ Channel / Endpoint Base Classes (2 new public fields each) │
96+│ class Channel { │
97+│ HcommNicChannelOps *nicOps_{nullptr}; │
98+│ void *nicCtx_{nullptr}; │
99+│ }; │
100+├──────────────────────────────────────────────────────────────────┤
101+│ Injection at Creation Time │
102+│ Built-in: ch->nicOps_ = &g_BuiltinChannelOps; ch->nicCtx_ = ch; │
103+│ Plugin: ch->nicOps_ = pluginOps; ch->nicCtx_ = pluginCtx │
104+├────────────────────────────────────────── ───────────────────────┤
105+│ g_BuiltinChannelOps (Global Ops Table) │
106+│ .writeNbi = ctx → ((Channel*)ctx)->Write(...) │
107+└──────────────────────────────────────────────────────────────────┘
108+```
109+ 
110+### 4.4 Logical View
111+ 
112+```mermaid
113+graph TB
114+ subgraph HCOMM["libhcomm.so"]
115+ API["Public C API<br/>HcommWriteNbi / HcommReadNbi / ..."]
116+ ChBase["class Channel<br/>+ nicOps_: HcommNicChannelOps*<br/>+ nicCtx_: void*<br/>12 pure virtual methods (unchanged)"]
117+ EpBase["class Endpoint<br/>+ nicOps_: HcommNicEndpointOps*<br/>+ nicCtx_: void*<br/>6+ pure virtual methods (unchanged)"]
118+ 
119+ subgraph BUILTIN["Built-in Path"]
120+ kB["g_BuiltinChannelOps<br/>g_BuiltinEndpointOps<br/>Global Ops Table<br/>ctx=this → virtual methods"]
121+ BuiltinCh["Built-in Channel Subclasses<br/>HostCpuRoceChannel, etc.<br/>(10+ subclasses, zero modifications)"]
122+ BuiltinEp["Built-in Endpoint Subclasses<br/>CpuRoceEndpoint, etc.<br/>(zero modifications)"]
123+ end
124+ 
125+ Loader["NicPluginLoader<br/>FindHostNicPlugin(protocol)<br/>map(Protocol → Entry*)"]
126+ Holder["PluginChannelHolder<br/>PluginEndpointHolder<br/>(placeholder subclasses, only carry ops table)"]
127+ end
128+ 
129+ subgraph SO["Plugin .so (Independently Compiled)"]
130+ Exports["3 Exported Symbols<br/>GetInfo() / CreateEndpoint() / CreateChannel()"]
131+ PluginOps["HcommNicChannelOps kMyOps<br/>HcommNicEndpointOps kMyOps<br/>(C function pointer tables implemented by the plugin)"]
132+ Impl["Data Plane I/O<br/>ibv_post_send / urma ops / ..."]
133+ end
134+ 
135+ API -->|"ch->nicOps_->writeNbi(ch->nicCtx_, ...)"| ChBase
136+ API -->|"ep->nicOps_->registerMemory(ep->nicCtx_, ...)"| EpBase
137+ 
138+ ChBase -->|"Built-in: SetNicChannelCtx(&g_BuiltinChannelOps, this)"| kB
139+ ChBase -.->|"Plugin: SetNicChannelCtx(pluginOps, pluginCtx)"| PluginOps
140+ EpBase -->|"Built-in: SetNicEndpointCtx(&g_BuiltinEndpointOps, this)"| kB
141+ EpBase -.->|"Plugin: SetNicEndpointCtx(pluginOps, pluginCtx)"| PluginOps
142+ 
143+ kB -.->|"ctx=this → Write() and other virtual methods"| BuiltinCh
144+ kB -.->|"ctx=this → virtual methods"| BuiltinEp
145+ 
146+ PluginOps -->|"Function pointer binding"| Impl
147+ 
148+ Loader -->|"dlopen + dlsym (runtime)"| Exports
149+ Exports -->|"Return"| PluginOps
150+ ChBase --- Holder
151+ EpBase --- Holder
152+ 
153+ HCOMM -.-|"Compile-time: include hcomm_nic_plugin.h<br/>(struct definitions)"| SO
154+ 
155+ style HCOMM fill:none,stroke:#01579b,stroke-width:2px
156+ style SO fill:none,stroke:#bf360c,stroke-width:2px,stroke-dasharray: 8 4
157+ style API fill:#e1f5fe,stroke:#01579b
158+ style BUILTIN fill:#c8e6c9,stroke:#2e7d32
159+ style Loader fill:#b2dfdb,stroke:#00695c
160+ style Exports fill:#ffe0b2,stroke:#e65100
161+ style PluginOps fill:#ffccbc,stroke:#bf360c
162+ style Impl fill:#fff3e0,stroke:#ef6c00
163+```
164+ 
165+**Connection Description**:
166+ 
167+| Line Type | Meaning |
168+|-----------|---------|
169+| Solid arrow `-->` | Data dispatch flow / strong dependency |
170+| Dashed arrow `-.->` | Injection at creation time / internal forwarding |
171+| Dotted line `-.-` | Compile-time dependency (plugin .so includes `hcomm_nic_plugin.h`) |
172+ 
173+**Core Relationships**:
174+ 
175+| Relationship | Direction | Description |
176+|---------------|-----------|-------------|
177+| API → Base Class | Dispatch | All `HcommXxx` functions uniformly go through `ch->nicOps_->xxx(ch->nicCtx_, ...) or ep->nicOps_->xxx(ep->nicCtx_, ...)` |
178+| Base Class → Ops Table | Hold | `nicOps_` points to `g_BuiltinXxxOps` (built-in) or `pluginOps` (plugin), mutually exclusive |
179+| Built-in Ops → Subclass | Forward | Built-in wrapper function, `ctx = Channel*``Write()` virtual method |
180+| Plugin Ops → Implementation | Bind | Function pointers are bound to plugin data-plane I/O logic |
181+| Loader → SO Symbols | Load | `dlopen` + `dlsym` to obtain 3 exported function pointers |
182+| SO Symbols → Ops Table | Return | `CreateEndpoint/CreateChannel` return the ops table populated by the plugin |
183+ 
184+#### 4.4.1 Module Responsibilities and Relationships
185+ 
186+| Side | Module | Responsibility | Relationship |
187+|------|--------|---------------|--------------|
188+| **libhcomm.so** | Public C API Layer | `include/` external interface declarations, unchanged | Caller entry point, directly converted to ops table calls |
189+| **libhcomm.so** | C Ops Table Definition | `hcomm_nic_plugin.h`, defines `HcommNicChannelOps` / `HcommNicEndpointOps` structs | Serves as both the built-in ops table type for libhcomm.so and the only header file dependency for plugin .so compilation |
190+| **libhcomm.so** | Channel / Endpoint Base Classes | `nicOps_` + `nicCtx_` pointer fields + `SetNic***Ctx()` setter | The ops table pointer is populated at creation time; `nicOps_` is immutable once set. The plugin-side ctx is opaque |
191+| **libhcomm.so** | Built-in Ops Table | `g_BuiltinChannelOps` / `g_BuiltinEndpointOps` global tables | Shared by all built-in subclasses; `ctx = this`; wrapper functions forward to virtual methods |
192+| **libhcomm.so** | Channel / Endpoint Subclasses | 10+ built-in subclasses require zero modifications; new `PluginXxxHolder` placeholder subclasses are added | Placeholder subclasses only carry the ops table and are stored in the global Map; data-plane logic is fully delegated to the ops table |
193+| **libhcomm.so** | Plugin Framework Layer | `NicPluginLoader`: loads .so, validates, and maps protocols | Serves only the creation path; no coupling with the Map |
194+| **Plugin .so** | Exported Symbols | `GetInfo()` / `CreateEndpoint()` / `CreateChannel()` — 3 C symbols | Discovered and invoked by libhcomm.so through `dlopen`/`dlsym` at runtime |
195+| **Plugin .so** | Plugin Ops Table | `HcommNicChannelOps` / `HcommNicEndpointOps` populated by the plugin | During Endpoint/Channel creation, the framework fills the plugin's ops table pointer into the base class `nicOps_` field; subsequent dispatch calls go directly through this pointer |
196+| **Plugin .so** | Plugin Data-Plane Implementation | Actual I/O logic for RDMA write/read/notify and so on | Ops table function pointers are bound to specific implementations (such as `ibv_post_send`), running in the caller's thread context |
197+ 
198+#### 4.4.2 Boundaries and Dependencies
199+ 
200+```text
201+libhcomm.so Plugin .so
202+┌──────────────────────┐ ┌──────────────────────────┐
203+│ │ dlopen/dlsym (runtime) │
204+│ NicPluginLoader │────────────────▶│ 3 Exported Symbols │
205+│ │ │ │
206+│ g_EndpointMap │ Holds ops │ │
207+│ g_ChannelMap ─────│──▶ PluginHolder──│▶ Ops Table (func ptrs) │
208+│ │ (placeholder │ ├─ registerMemory │
209+│ │ subclass) │ ├─ writeNbi │
210+│ │ │ ├─ readNbi │
211+│ │ │ └─ ... │
212+│ │ │ │ │
213+│ g_BuiltinXxxOps │ Statically │ ▼ │
214+│ (also implements │ populated at │ Data Plane I/O │
215+│ the same ops table) │ compile time │ ibv_post_send, etc. │
216+│ │ │ │
217+│ ← Compile-time dep. ─│── hcomm_nic_plugin.h ──│▶ Compile-time incl.│
218+│ (shared struct defs) │ │ │
219+└──────────────────────┘ └──────────────────────────┘
220+ 
221+Key Boundaries:
222+ ▸ The plugin .so depends only on hcomm_nic_plugin.h (pure C struct definitions) at compile time and does not link libhcomm.so
223+ ▸ libhcomm.so loads the plugin .so at runtime through dlopen and obtains function pointers through dlsym
224+ ▸ Both sides interoperate through the same HcommNicChannelOps / HcommNicEndpointOps table
225+ ▸ The plugin ctx is completely opaque to libhcomm.so — libhcomm.so only passes the nicCtx_ pointer without parsing its contents
226+ ▸ g_Builtin***Ops in libhcomm.so and the plugin ops table are mutually exclusive: a Channel/Endpoint object holds only one of them
227+```
228+ 
229+### 4.5 Overall Module Sequence
230+ 
231+```mermaid
232+sequenceDiagram
233+ participant App as Upper-Layer Application
234+ participant API as HCOMM C API
235+ participant Base as Channel/Endpoint Base Class
236+ participant Ops as C Ops Table
237+ participant Builtin as Built-in Implementation
238+ participant Mgr as NicPluginManager
239+ participant SO as Plugin .so
240+
241+ Note over App,SO: ═══ Creation Phase: Endpoint / Channel ═══
242+ 
243+ App->>API: HcommEndpointCreate(&desc, &handle)
244+ API->>Mgr: std::call_once triggers LoadAllNicPlugins()
245+ Mgr->>SO: dlopen + dlsym (GetInfo / CreateEndpoint / CreateChannel)
246+ SO-->>Mgr: Plugin metadata + creation function pointers
247+ Mgr->>Mgr: Validate + register into ProtocolPlugins map
248+ 
249+ API->>Mgr: FindHostNicPlugin(desc.protocol)
250+ 
251+ alt Plugin Hit
252+ Mgr-->>API: entry
253+ API->>SO: createEndpoint(&desc, &ctx, &ops)
254+ SO-->>API: pluginCtx + EndpointOps
255+ API->>Base: ep->SetNicEndpointCtx(pluginOps, pluginCtx)
256+ else Not Hit
257+ Mgr-->>API: null
258+ API->>Builtin: Endpoint::CreateEndpoint(desc)
259+ API->>Base: ep->SetNicEndpointCtx(&g_BuiltinEndpointOps, ep)
260+ end
261+ 
262+ App->>API: HcommChannelCreate(epHandle, ...)
263+ API->>Base: Is ep->nicOps_ a plugin ops?
264+ 
265+ alt Plugin Endpoint
266+ API->>SO: createChannel(epCtx, &desc, &chCtx, &chOps)
267+ SO-->>API: chCtx + ChannelOps
268+ API->>Base: ch->SetNicChannelCtx(chOps, chCtx)
269+ else Built-in Endpoint
270+ API->>Builtin: Channel::CreateChannel factory
271+ API->>Base: ch->SetNicChannelCtx(&g_BuiltinChannelOps, ch)
272+ end
273+ 
274+ Note over App,SO: ═══ Invocation Phase: Zero-Branch Data-Plane Dispatch ═══
275+ 
276+ App->>API: HcommWriteNbi(ch, dst, src, len)
277+ API->>Base: ch = (Channel*)ch
278+ API->>Ops: ch->nicOps_->writeNbi(ch->nicCtx_, dst, src, len)
279+ 
280+ alt Built-in Channel
281+ Ops->>Builtin: BuiltinWriteNbi(ctx, ...) → Channel::Write(...)
282+ Builtin-->>Ops: ret
283+ else Plugin Channel
284+ Ops->>SO: pluginWriteNbi(pluginCtx, ...)
285+ SO-->>Ops: ret
286+ end
287+ 
288+ Ops-->>API: ret
289+ API-->>App: ret
290+ 
291+ Note over API,SO: Zero-branch data-plane dispatch applies only to 950-series exclusive interfaces; non-950-series exclusive interfaces use the legacy path
292+```
293+ 
294+## 5. Detailed Design
295+ 
296+### 5.1 Core Data Structures
297+ 
298+#### 5.1.1 C ABI Interface Specification
299+ 
300+##### Version Number and Metadata
301+ 
302+```c
303+#define HCOMM_NIC_PLUGIN_INFO_VERSION 1U
304+#define HCOMM_NIC_CHANNEL_OPS_VERSION 1U
305+#define HCOMM_NIC_ENDPOINT_OPS_VERSION 1U
306+ 
307+typedef struct {
308+ CommAbiHeader header; // version / magic / size / reserved
309+ const char *name; // plugin name
310+ uint32_t protocolCount; // number of protocols
311+ CommProtocol protocols[HCOMM_NIC_PLUGIN_MAX_PROTOCOLS]; // up to 4
312+ uint64_t reserved[8]; // reserved for extension
313+} HcommNicPluginInfo;
314+```
315+ 
316+**Validation Rules**:
317+ 
318+- `header.magicWord` must equal `HCOMM_NIC_PLUGIN_INFO_MAGIC_WORD`
319+- `header.size` is rejected when it is less than `offsetof(HcommNicPluginInfo, protocols) + sizeof(protocols)`
320+- Each value in `protocols[]` must be either a non-RESERVED built-in enumeration value < `COMM_PROTOCOL_CUSTOM_BASE`, or a value ≥ `COMM_PROTOCOL_CUSTOM_BASE`
321+- COMM_PROTOCOL_CUSTOM_BASE=1000 is defined in nic_plugin_manager.h and serves as the boundary constant for plugin protocol numbers. The CommProtocol field carries custom values ≥1000 with int32_t semantics.
322+ 
323+##### Channel Ops Table
324+ 
325+```c
326+typedef struct {
327+ CommAbiHeader header;
328+ int32_t (*init)(void* ctx);
329+ int32_t (*destroy)(void* ctx);
330+ 
331+ int32_t (*getStatus)(void* ctx, int32_t* status);
332+ 
333+ int32_t (*writeNbi)(void* ctx, void* dst, const void* src, uint64_t len);
334+ int32_t (*writeNbiOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len);
335+ int32_t (*writeOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len);
336+ int32_t (*writeWithNotifyNbi)(void* ctx, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx);
337+ int32_t (*writeWithNotifyNbiOnThread)(
338+ void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx);
339+ int32_t (*writeWithNotifyOnThread)(
340+ void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx);
341+ int32_t (*writeReduceOnThread)(
342+ void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t count, HcommDataType dataType,
343+ HcommReduceOp reduceOp);
344+ int32_t (*writeReduceWithNotifyOnThread)(
345+ void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t count, HcommDataType dataType,
346+ HcommReduceOp reduceOp, uint32_t remoteNotifyIdx);
347+ 
348+ int32_t (*readNbi)(void* ctx, void* dst, const void* src, uint64_t len);
349+ int32_t (*readNbiOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len);
350+ int32_t (*readOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len);
351+ int32_t (*readReduceOnThread)(
352+ void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t count, HcommDataType dataType,
353+ HcommReduceOp reduceOp);
354+ 
355+ int32_t (*notifyRecord)(void* ctx, uint32_t remoteNotifyIdx);
356+ int32_t (*notifyRecordOnThread)(void* ctx, ThreadHandle thread, uint32_t remoteNotifyIdx);
357+ int32_t (*notifyWait)(void* ctx, uint32_t localNotifyIdx, uint32_t timeOut);
358+ int32_t (*notifyWaitOnThread)(void* ctx, ThreadHandle thread, uint32_t localNotifyIdx, uint32_t timeOut);
359+ int32_t (*notifyWaitOnThreadWithDefaultTimeout)(void* ctx, ThreadHandle thread, uint32_t localNotifyIdx);
360+ 
361+ int32_t (*batchTransferOnThread)(
362+ void* ctx, ThreadHandle thread, const HcommBatchTransferDesc* transferDescs, uint32_t transferDescNum);
363+ 
364+ int32_t (*fence)(void* ctx);
365+ int32_t (*fenceOnThread)(void* ctx, ThreadHandle thread);
366+ int32_t (*drainOnThread)(void* ctx, ThreadHandle thread);
367+} HcommNicChannelOps;
368+```
369+ 
370+**Required Interface for Plugins**: The plugin must implement the destroy interface to release the channel object created by the plugin.
371+ 
372+**API Mapping**:
373+ 
374+| Ops Entry | Corresponding Public API | Description |
375+| ---------------------------------------- | ------------------------------------------------ | ----------- |
376+| `destroy` | `HcommChannelDestroy` | The public API is a batch interface; the framework calls the corresponding ops entry for each channel |
377+| `getStatus` | `HcommChannelGetStatus` | The public API is a batch interface; the framework calls the corresponding ops entry for each channel |
378+| `writeNbi` | `HcommWriteNbi` | |
379+| `writeNbiOnThread` | `HcommWriteNbiOnThread` | |
380+| `writeOnThread` | `HcommWriteOnThread` | |
381+| `writeWithNotifyNbi` | `HcommWriteWithNotifyNbi` | |
382+| `writeWithNotifyNbiOnThread` | `HcommWriteWithNotifyNbiOnThread` | |
383+| `writeWithNotifyOnThread` | `HcommWriteWithNotifyOnThread` | |
384+| `writeReduceOnThread` | `HcommWriteReduceOnThread` | |
385+| `writeReduceWithNotifyOnThread` | `HcommWriteReduceWithNotifyOnThread` | |
386+| `readNbi` | `HcommReadNbi` | |
387+| `readNbiOnThread` | `HcommReadNbiOnThread` | |
388+| `readOnThread` | `HcommReadOnThread` | |
389+| `readReduceOnThread` | `HcommReadReduceOnThread` | |
390+| `notifyRecord` | `HcommNotifyRecord` | |
391+| `notifyRecordOnThread` | `HcommNotifyRecordOnThread` | |
392+| `notifyWait` | `HcommNotifyWait` | |
393+| `notifyWaitOnThread` | `HcommNotifyWaitOnThread` | |
394+| `notifyWaitOnThreadWithDefaultTimeout` | `HcommNotifyWaitOnThreadWithDefaultTimeout` | |
395+| `batchTransferOnThread` | `HcommBatchTransferOnThread` | |
396+| `fence` | `HcommFence` | |
397+| `fenceOnThread` | `HcommFenceOnThread` | |
398+| `drainOnThread` | `HcommChannelDrainOnThread` | |
399+ 
400+##### Endpoint Ops Table
401+ 
402+```c
403+typedef struct {
404+ CommAbiHeader header;
405+ int32_t (*init)(void* ctx);
406+ int32_t (*destroy)(void* ctx);
407+ 
408+ int32_t (*registerMemory)(void* ctx, const CommMem* mem, const char* tag, void** handle);
409+ int32_t (*unregisterMemory)(void* ctx, void* handle);
410+ int32_t (*memoryExport)(void* ctx, void* handle, void** desc, uint32_t* descLen);
411+ int32_t (*memoryImport)(void* ctx, const void* desc, uint32_t descLen, CommMem* outMem);
412+ int32_t (*memoryUnimport)(void* ctx, const void* desc, uint32_t descLen);
413+ int32_t (*getListenPort)(void* ctx, uint32_t* port);
414+} HcommNicEndpointOps;
415+```
416+ 
417+**API Mapping**:
418+ 
419+| Ops Entry | Corresponding Public API |
420+|-----------|-------------------------|
421+| `destroy` | `HcommEndpointDestroy` |
422+| `registerMemory` | `HcommMemReg` |
423+| `unregisterMemory` | `HcommMemUnreg` |
424+| `memoryExport` | `HcommMemExport` |
425+| `memoryImport` | `HcommMemImport` |
426+| `memoryUnimport` | `HcommMemUnimport` |
427+| `getListenPort` | `HcommEndpointGetListenPort` |
428+ 
429+**Required Interface for Plugins**: The plugin must implement the destroy interface to release the endpoint object created by the plugin.
430+ 
431+##### Exported Symbols
432+ 
433+```c
434+typedef const HcommNicPluginInfo *(*HcommNicPluginGetInfoFunc)(void);
435+ 
436+typedef int32_t (*HcommNicPluginCreateEndpointFunc)(
437+ const EndpointDesc *endpointDesc,
438+ void **outCtx, HcommNicEndpointOps **outOps);
439+ 
440+typedef int32_t (*HcommNicPluginCreateChannelFunc)(
441+ void *epCtx, const HcommChannelDesc *channelDesc,
442+ void **outCtx, HcommNicChannelOps **outOps);
443+ 
444+// The 3 C symbols that the plugin must export
445+const HcommNicPluginInfo *HcommNicPluginGetInfo(void);
446+int32_t HcommNicPluginCreateEndpoint(const EndpointDesc *desc,
447+ void **outCtx, HcommNicEndpointOps **outOps);
448+int32_t HcommNicPluginCreateChannel(void *epCtx, const HcommChannelDesc *desc,
449+ void **outCtx, HcommNicChannelOps **outOps);
450+```
451+ 
452+#### 5.1.2 New Fields in the Channel Base Class
453+ 
454+**File**: `src/base_comm/resources/endpoint_pairs/channels/channel.h`
455+ 
456+```cpp
457+class Channel {
458+protected:
459+ // ==== New nicOps_ and nicCtx_ ====
460+ HcommNicChannelOps* nicOps_{nullptr};
461+ void* nicCtx_{nullptr};
462+public:
463+ // ==== New set/get methods ====
464+ void SetNicChannelCtx(HcommNicChannelOps* nicOps, void* nicCtx)
465+ {
466+ nicOps_ = nicOps;
467+ nicCtx_ = nicCtx;
468+ }
469+ HcommNicChannelOps* GetNicOps() const { return nicOps_; }
470+ void* GetNicCtx() const { return nicCtx_; }
471+};
472+```
473+ 
474+**Boundary Constraints**:
475+ 
476+- `nicOps_` is set once at creation time and is read-only at runtime; it is immutable
477+- `nicCtx_` is `this` for built-in, and a private opaque context allocated by the plugin for plugin
478+- The new fields are located at the end of the class with a default value of `nullptr`; the memory layout of existing subclasses remains unchanged
479+- The new interfaces are non-virtual and do not affect the vtable
480+ 
481+#### 5.1.3 New Fields in the Endpoint Base Class
482+ 
483+**File**: `src/base_comm/resources/endpoints/endpoint.h` (same pattern as Channel)
484+ 
485+```cpp
486+class Endpoint {
487+protected:
488+ // ==== New nicOps_ and nicCtx_ ====
489+ HcommNicEndpointOps* nicOps_{nullptr};
490+ void* nicCtx_{nullptr};
491+public:
492+ // ==== New set/get methods ====
493+ void SetNicEndpointCtx(HcommNicEndpointOps* nicOps, void* nicCtx)
494+ {
495+ nicOps_ = nicOps;
496+ nicCtx_ = nicCtx;
497+ }
498+ HcommNicEndpointOps* GetNicOps() const { return nicOps_; }
499+ void* GetNicCtx() const { return nicCtx_; }
500+};
501+```
502+ 
503+#### 5.1.4 Plugin Registry and Plugin Subclasses
504+ 
505+```cpp
506+struct NicPluginEntry {
507+ void *soHandle; // dlopen handle
508+ const HcommNicPluginInfo *info; // plugin metadata
509+ HcommNicPluginCreateEndpointFunc createEndpoint; // V1 creation function
510+ HcommNicPluginCreateChannelFunc createChannel; // V1 creation function
511+};
512+ 
513+// Global singleton, protocol number → plugin entry
514+std::unordered_map<CommProtocol, const NicPluginEntry *> &ProtocolPlugins();
515+ 
516+// Placeholder subclasses (new): pure virtual method implementations return NOT_SUPPORT (data plane does not go through vtable; the dispatch layer uses the ops table); used only to carry object lifetime in the Map
517+class PluginEndpointHolder : public Endpoint {
518+ explicit PluginEndpointHolder(const EndpointDesc& endpointDesc, const NicPluginEntry* pluginEntry)
519+ : Endpoint(endpointDesc),
520+ pluginEntry_(pluginEntry)
521+ {}
522+ ~PluginEndpointHolder() override { DestroyNicPluginOpsAndCtx(nicOps_, nicCtx_); }
523+ 
524+ const NicPluginEntry* GetPluginEntry() const { return pluginEntry_; }
525+ // Pure virtual method implementation returns NOT_SUPPORT
526+ HcclResult RegisterMemory(HcommMem mem, const char* memTag, void** memHandle) override
527+ {
528+ (void)mem;
529+ (void)memTag;
530+ (void)memHandle;
531+ return HCCL_E_NOT_SUPPORT;
532+ }
533+};
534+class PluginChannelHolder : public Channel {
535+ explicit PluginChannelHolder(const NicPluginEntry* pluginEntry) : pluginEntry_(pluginEntry) {}
536+ ~PluginChannelHolder() override { DestroyNicPluginOpsAndCtx(nicOps_, nicCtx_); }
537+ 
538+ const NicPluginEntry* GetPluginEntry() const { return pluginEntry_; }
539+ // Pure virtual method implementation returns NOT_SUPPORT
540+ HcclResult Write(void* dst, const void* src, uint64_t len) override
541+ {
542+ (void)dst;
543+ (void)src;
544+ (void)len;
545+ return HCCL_E_NOT_SUPPORT;
546+ }
547+};
548+ 
549+// When the plugin placeholder subclass PluginEndpointHolder/PluginChannelHolder is released, call the plugin destroy function to destroy the plugin nicCtx object
550+template <typename Ops>
551+void DestroyNicPluginOpsAndCtx(Ops*& nicOps, void* nicCtx)
552+{
553+ if (nicOps != nullptr) {
554+ if (nicOps->destroy != nullptr) {
555+ int32_t ret = nicOps->destroy(nicCtx);
556+ if (ret != HCCL_SUCCESS) {
557+ HCCL_WARNING("[%s] plugin destroy failed, ret[%d].", __func__, ret);
558+ }
559+ }
560+ delete nicOps;
561+ nicOps = nullptr;
562+ }
563+}
564+```
565+ 
566+### 5.2 Built-in Ops Table Implementation
567+ 
568+#### 5.2.1 g_BuiltinChannelOps
569+ 
570+**File**: `src/base_comm/resources/endpoint_pairs/channels/builtin_channel_ops.h`
571+ 
572+```c
573+// Built-in implementation wrapper function
574+inline int32_t BuiltinWriteWithNotifyNbiOnThread(
575+ void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx)
576+{
577+ HCCL_INFO(
578+ "[%s] START. thread[0x%llx], channel[0x%llx], dst[0x%llx], src[0x%llx], len[%llu], remoteNotifyIdx[%u].",
579+ __func__, thread, ctx, dst, src, len, remoteNotifyIdx);
580+ 
581+ (void)thread;
582+ CHK_PTR_NULL(src);
583+ CHK_PTR_NULL(dst);
584+ HcclResult ret = HCCL_SUCCESS;
585+ DevType devType;
586+ CHK_RET(hrtGetDeviceType(devType));
587+ if (devType == DevType::DEV_TYPE_950 || devType == DevType::DEV_TYPE_960 || thread == 0) {
588+ auto* const channelPtr = reinterpret_cast<hcomm::Channel*>(ctx);
589+ CHK_PTR_NULL(channelPtr);
590+ ret = channelPtr->WriteWithNotify(dst, src, len, remoteNotifyIdx);
591+ } else {
592+ ret = HCCL_E_NOT_SUPPORT;
593+ }
594+ CHK_PRT_RET(
595+ ret != HCCL_SUCCESS,
596+ HCCL_ERROR(
597+ "[%s] FAIL. thread[0x%llx], channel[0x%llx], dst[0x%llx], src[0x%llx], len[%llu], remoteNotifyIdx[%u].",
598+ __func__, thread, ctx, dst, src, len, remoteNotifyIdx),
599+ ret);
600+ HCCL_INFO("[%s] SUCCESS.", __func__);
601+ return HCCL_SUCCESS;
602+}
603+// ... Other interfaces for built-in implementation follow the same pattern
604+ 
605+inline HcommNicChannelOps g_BuiltinChannelOps = {
606+ {HCOMM_NIC_CHANNEL_OPS_VERSION, HCOMM_NIC_CHANNEL_OPS_MAGIC_WORD, sizeof(HcommNicChannelOps), 0},
607+ BuiltinChannelInit, // init
608+ BuiltinChannelDestroy, // destroy
609+ BuiltinGetStatus, // getStatus
610+ BuiltinWriteNbi, // writeNbi
611+ BuiltinWriteNbiOnThread, // writeNbiOnThread
612+ hcomm::DefaultChannelWriteOnThread, // writeOnThread
613+ BuiltinWriteWithNotifyNbi, // writeWithNotifyNbi
614+ BuiltinWriteWithNotifyNbiOnThread, // writeWithNotifyNbiOnThread
615+ hcomm::DefaultChannelWriteWithNotifyOnThread, // writeWithNotifyOnThread
616+ hcomm::DefaultChannelWriteReduceOnThread, // writeReduceOnThread
617+ hcomm::DefaultChannelWriteReduceWithNotifyOnThread, // writeReduceWithNotifyOnThread
618+ BuiltinReadNbi, // readNbi
619+ BuiltinReadNbiOnThread, // readNbiOnThread
620+ hcomm::DefaultChannelReadOnThread, // readOnThread
621+ hcomm::DefaultChannelReadReduceOnThread, // readReduceOnThread
622+ BuiltinNotifyRecord, // notifyRecord
623+ BuiltinNotifyRecordOnThread, // notifyRecordOnThread
624+ BuiltinNotifyWait, // notifyWait
625+ BuiltinNotifyWaitOnThread, // notifyWaitOnThread
626+ hcomm::DefaultChannelNotifyWaitOnThreadWithDefaultTimeout, // notifyWaitOnThreadWithDefaultTimeout
627+ hcomm::DefaultChannelBatchTransferOnThread, // batchTransferOnThread
628+ BuiltinFence, // fence
629+ BuiltinFenceOnThread, // fenceOnThread
630+ hcomm::DefaultChannelDrainOnThread, // drainOnThread
631+};
632+```
633+ 
634+**Design Points**:
635+ 
636+- 1. The public APIs corresponding to Channel ops entries differ in product support. They are divided into two categories: APIs that support only the 950 series and APIs that also support non-950 series products. For APIs that support only the 950 series, wrapper functions must be implemented. For APIs that also support non-950 series products, unified dispatch is not possible due to flow differences, so wrapper functions are not implemented. For such interfaces, the dispatch point must distinguish between plugin and non-plugin scenarios.
637+- 2. For interfaces without wrapper functions, default interfaces are populated to avoid null pointer call exceptions at the dispatch point.
638+ 
639+#### 5.2.2 g_BuiltinEndpointOps
640+ 
641+**File**: `src/base_comm/resources/endpoints/builtin_endpoint_ops.h`
642+ 
643+```c
644+// Built-in implementation wrapper function
645+inline int32_t BuiltinRegisterMemory(void* ctx, const CommMem* mem, const char* tag, void** handle)
646+{
647+ CHK_PTR_NULL(mem);
648+ CHK_PTR_NULL(handle);
649+ EXCEPTION_HANDLE_BEGIN(void) HcommResMgrInit();
650+ EndpointHandle epHandle = reinterpret_cast<EndpointHandle>(ctx);
651+ HCCL_INFO("[%s] START. endpointHandle[0x%llx].", __func__, epHandle);
652+ auto endpoint = GetEndpointMap().GetEndpoint(epHandle);
653+ CHK_PRT_RET(
654+ endpoint == nullptr, HCCL_ERROR("[%s] endpoint not found, endpointHandle[0x%llx]", __func__, epHandle),
655+ HCCL_E_NOT_FOUND);
656+ CHK_RET(RefreshEndpointContext(endpoint->GetEndpointDesc()));
657+ CHK_RET(endpoint->RegisterMemory(*mem, tag, handle));
658+ 
659+ EXCEPTION_HANDLE_END
660+ return HCCL_SUCCESS;
661+}
662+// ... Other interfaces for built-in implementation follow the same pattern
663+ 
664+inline HcommNicEndpointOps g_BuiltinEndpointOps = {
665+ {HCOMM_NIC_ENDPOINT_OPS_VERSION, HCOMM_NIC_ENDPOINT_OPS_MAGIC_WORD, sizeof(HcommNicEndpointOps), 0},
666+ BuiltinEndpointInit, // init
667+ BuiltinEndpointDestroy, // destroy
668+ BuiltinRegisterMemory, // registerMemory
669+ BuiltinUnregisterMemory, // unregisterMemory
670+ BuiltinMemoryExport, // memoryExport
671+ BuiltinMemoryImport, // memoryImport
672+ BuiltinMemoryUnimport, // memoryUnimport
673+ BuiltinGetListenPort, // getListenPort
674+};
675+```
676+ 
677+### 5.3 Creation Path
678+ 
679+#### 5.3.1 Endpoint Creation
680+ 
681+```c
682+HcommResult HcommEndpointCreate(const EndpointDesc *endpoint, EndpointHandle *handle) {
683+// Shows the distinction between plugin and built-in; context is omitted
684+ ...
685+ if (endpoint->loc.locType == ENDPOINT_LOC_TYPE_HOST) {
686+ const NicPluginEntry* pluginEntry = FindHostNicPlugin(endpoint->protocol);
687+ if (pluginEntry != nullptr) {
688+ return CreatePluginEndpointHolder(endpoint, pluginEntry, endpointHandle);
689+ }
690+ }
691+ ...
692+ CHK_RET(CreateBuiltinEndpoint(endpoint, endpointHandle));
693+ ...
694+}
695+```
696+ 
697+**Ops Population**:
698+ 
699+- Built-in path: `Endpoint::CreateEndpoint(*endpoint, ep)``ep->SetNicEndpointCtx(&g_BuiltinEndpointOps, ep.get())` → store into `g_EndpointMap`
700+- Plugin path: `pluginEntry->createEndpoint(endpoint, &pluginCtx, &pluginOps)` → validate pluginOps → create `pluginHolderOps`, copy pluginOps and populate unimplemented interfaces → create `PluginEndpointHolder``holder->SetNicEndpointCtx(pluginHolderOps, pluginCtx)` → store into `g_EndpointMap`
701+ 
702+#### 5.3.2 Channel Creation
703+ 
704+```c
705+HcommResult HcommChannelCreate(
706+ EndpointHandle endpointHandle, CommEngine engine, HcommChannelDesc* channelDescs, uint32_t channelNum,
707+ ChannelHandle* channels)
708+{
709+ // Shows the distinction between plugin and built-in; context is omitted
710+ ...
711+ auto endpoint = GetEndpointMap().GetEndpoint(endpointHandle);
712+ if (endpoint != nullptr && endpoint->GetNicOps() != nullptr && endpoint->GetNicOps() != &g_BuiltinEndpointOps) {
713+ CHK_RET(
714+ static_cast<HcclResult>(CreatePluginChannels(endpoint, channelDescFinals.data(), channelNum, channels)));
715+ return HCCL_SUCCESS;
716+ }
717+ ...
718+ CHK_RET(ChannelProcess::CreateChannelsLoop(
719+ endpointHandle, engine, channelDescFinals.data(), channelNum, targetChannels));
720+ CHK_RET(
721+ ChannelProcess::PrepareUserChannels(targetChannels, channels, channelDescFinals.data(), channelNum, engine));
722+ ...
723+}
724+```
725+ 
726+**Ops Population**:
727+ 
728+- Built-in path: `Channel::CreateChannel` factory → `ch->SetNicChannelCtx(&g_BuiltinChannelOps, ch.get())` → store into `g_ChannelMap`
729+- Plugin path: `pluginEntry->createChannel(epCtx, channelDesc, &pluginCtx, &pluginOps)` → validate pluginOps → create `pluginHolderOps`, copy pluginOps and populate unimplemented interfaces → create `PluginChannelHolder``holder->SetNicChannelCtx(pluginHolderOps, pluginCtx)` → store into `g_ChannelMap`
730+ 
731+#### 5.3.3 Decision Points in the Creation Path
732+ 
733+| Decision Point | Location | Frequency | Method |
734+|---------------|----------|-----------|--------|
735+| Whether to enable plugin | `HcommEndpointCreate` | Once per Endpoint creation | `NicPluginEntry* pluginEntry = FindHostNicPlugin(endpoint->protocol)`, whether the plugin supports the corresponding protocol |
736+| Whether it is a plugin endpoint | `HcommChannelCreate` | Once per Channel creation | `endpoint->GetNicOps() != &g_BuiltinEndpointOps`, whether the endpoint was created by a plugin |
737+ 
738+The creation path is not a hot path, so the branching overhead is negligible.
739+ 
740+### 5.4 Dispatch Path
741+ 
742+> **Core Principle**: Among the public API interfaces corresponding to channel ops entries, interfaces that support non-950 series products require an additional plugin check. Other dispatch functions do not contain any `if/else`, `#ifdef`, or tag-bit checks.
743+ 
744+#### 5.4.1 Channel Dispatch Type 1
745+ 
746+```c
747+int32_t HcommWriteNbi(ChannelHandle channel, void* dst, const void* src, uint64_t len)
748+{
749+ auto* ch = CHANNEL_FROM_HANDLE(channel);
750+ CHK_PTR_NULL(ch);
751+ // No distinction between plugin and built-in
752+ return ch->GetNicOps()->writeNbi(ch->GetNicCtx(), dst, src, len);
753+}
754+```
755+ 
756+#### 5.4.2 Channel Dispatch Type 2
757+ 
758+```c
759+int32_t HcommWriteOnThread(ThreadHandle thread, ChannelHandle channel, void* dst, const void* src, uint64_t len)
760+{
761+ if (IS_PLUGIN_HANDLE(channel)) {
762+ // Plugin scenario
763+ auto* ch = CHANNEL_FROM_HANDLE(channel);
764+ CHK_PTR_NULL(ch);
765+ return ch->GetNicOps()->writeOnThread(ch->GetNicCtx(), thread, dst, src, len);
766+ }
767+ // Built-in scenario
768+ ...
769+ HcclResult ret = HcclRemoteWrite(stream, reinterpret_cast<void*>(channel), &rmtBuf, &locBuf);
770+ ...
771+}
772+```
773+ 
774+#### 5.4.3 Endpoint Dispatch
775+ 
776+```c
777+HcommResult
778+HcommMemReg(EndpointHandle endpointHandle, const char* memTag, const CommMem* mem, HcommMemHandle* memHandle)
779+{
780+ auto endpoint = GetEndpointMap().GetEndpoint(endpointHandle);
781+ CHK_PRT_RET(
782+ endpoint == nullptr, HCCL_ERROR("[%s] endpoint not found, endpointHandle[%p]", __func__, endpointHandle),
783+ HCCL_E_NOT_FOUND);
784+ // No distinction between plugin and built-in
785+ return static_cast<HcclResult>(endpoint->GetNicOps()->registerMemory(
786+ endpoint->GetNicCtx(), mem, memTag, reinterpret_cast<void**>(memHandle)));
787+}
788+```
789+ 
790+### 5.5 Plugin Discovery and Loading
791+ 
792+- **Trigger Timing**: `std::call_once` is executed when the first endpoint is created
793+- **Default Path**: `opendir($ASCEND_HOME_PATH/hcomm_plugin/)` → each `*.so`
794+ 
795+Loading flow for a single .so:
796+ 
797+```text
798+dlopen(path, RTLD_NOW)
799+ ├─ Failure → WARNING log, skip
800+ └─ Success →
801+ dlsym("HcommNicPluginGetInfo")
802+ dlsym("HcommNicPluginCreateEndpoint")
803+ dlsym("HcommNicPluginCreateChannel")
804+ getInfo() → validate magic / version / size / protocols[]
805+ ├─ Pass → protocolPlugins[p] = entry (later load overwrites + WARNING)
806+ └─ Failure → dlclose, WARNING log
807+```
808+ 
809+**Lifecycle**: `dlclose` is not called after `dlopen`; the plugin has process-level singleton lifecycle. Runtime unloading is not supported. When the same protocol conflicts, the later-loaded plugin overwrites the earlier one, and a WARNING log is recorded.
810+ 
811+---
812+ 
813+## 6. Key Interaction Flows
814+ 
815+### 6.1 Endpoint Creation
816+ 
817+```mermaid
818+sequenceDiagram
819+ participant Caller as Caller
820+ participant EC as HcommEndpointCreate
821+ participant Mgr as NicPluginManager
822+ participant Map as g_EndpointMap
823+ participant SO as Plugin .so
824+ 
825+ Caller->>EC: Create(&desc, &handle)
826+ 
827+ EC->>Mgr: FindHostNicPlugin(desc.protocol)
828+ Mgr-->>EC: entry / null
829+ 
830+ alt entry != null (Plugin Endpoint)
831+ EC->>SO: createEndpoint(&desc, &pluginCtx, &pluginOps)
832+ SO-->>EC: pluginCtx + pluginOps
833+ EC->>EC: Validate pluginOps, validate ABI header and required interfaces
834+ EC->>EC: Create pluginHolderOps, copy pluginOps and populate unimplemented interfaces
835+ EC->>EC: Initialize plugin object context pluginHolderOps->init(pluginCtx)
836+ EC->>EC: Create PluginEndpointHolder(desc, pluginEntry)
837+ EC->>EC: holder->SetNicEndpointCtx(pluginHolderOps, pluginCtx)
838+ EC->>Map: AddEndpoint(handle, ep)
839+ Map-->>EC: ok
840+ else entry == null (Built-in Endpoint)
841+ EC->>EC: ep = Endpoint::CreateEndpoint(*endpoint, ep)
842+ EC->>EC: ep->SetNicEndpointCtx(&g_BuiltinEndpointOps, ep.get())
843+ EC->>Map: AddEndpoint(handle, ep)
844+ Map-->>EC: ok
845+ end
846+ 
847+ EC-->>Caller: handle
848+```
849+ 
850+### 6.2 Channel Creation
851+ 
852+```mermaid
853+sequenceDiagram
854+ participant Caller as Caller
855+ participant CC as HcommChannelCreate
856+ participant CP as ChannelProcess
857+ participant SO as Plugin .so
858+ 
859+ Caller->>CC: Create(epHandle, engine, descs, num, &channels)
860+ 
861+ alt ep->nicOps_ != &g_BuiltinEndpointOps (Plugin Endpoint)
862+ CC->>CC: entry = epHolder->GetPluginEntry()
863+ loop i in [0, num)
864+ CC->>SO: entry->createChannel(epCtx, channelDesc, &pluginCtx, &pluginOps)
865+ SO-->>CC: pluginCtx + pluginOps
866+ CC->>CC: Validate pluginOps, validate ABI header and required interfaces
867+ CC->>CC: Create pluginHolderOps, copy pluginOps and populate unimplemented interfaces
868+ CC->>CC: Initialize plugin object context pluginHolderOps->init(pluginCtx)
869+ CC->>CC: Create PluginChannelHolder(pluginEntry)
870+ CC->>CC: holder->SetNicChannelCtx(pluginHolderOps, pluginCtx)
871+ CC->>CC: ChannelHandle handle = (ChannelHandle)holder.get()
872+ CC->>Map: InsertPluginChannelToMap(handle, std::move(holder))
873+ Map-->>CC: ok
874+ CC->>CC: Set high bit to mark plugin creation: channels[i] = MAKE_PLUGIN_CH_HANDLE(handle);
875+ end
876+ else ep->nicOps_ == &g_BuiltinEndpointOps (Built-in Endpoint)
877+ CC->>CP: CreateChannelsLoop(ep, eng, descs, num, channels)
878+ CP->>CP: Channel::CreateChannel factory → ch->SetNicChannelCtx(&g_BuiltinChannelOps, ch.get())
879+ CP-->>CC: ok
880+ end
881+ 
882+ CC-->>Caller: channels
883+```
884+ 
885+### 6.3 Data-Plane Invocation (Zero-Branch)
886+ 
887+```mermaid
888+sequenceDiagram
889+ participant Caller as Upper-Layer Operator
890+ participant WN as HcommWriteNbi
891+ participant Ops as HcommNicChannelOps
892+ participant Impl as Implementation
893+ 
894+ Caller->>WN: WriteNbi(ch, dst, src, len)
895+ 
896+ WN->>WN: Obtain channel with restored bit: ch = CHANNEL_FROM_HANDLE(channel)
897+ WN->>Ops: ch->nicOps_->writeNbi(ch->nicCtx_, dst, src, len)
898+ 
899+ alt Built-in (nicOps_ = &g_BuiltinChannelOps)
900+ Ops->>Impl: BuiltinWriteNbi(ctx, ...)<br/>→ ((Channel*)ctx)->Write(...)
901+ Impl->>Impl: HostCpuRoceChannel::Write (ibv_post_send)
902+ Impl-->>Ops: ret
903+ else Plugin (nicOps_ = pluginOps)
904+ Ops->>Impl: pluginWriteNbi(pluginCtx, ...)
905+ Impl->>Impl: Plugin data plane (custom RDMA/URMA)
906+ Impl-->>Ops: ret
907+ end
908+ 
909+ Ops-->>WN: ret
910+ WN-->>Caller: ret
911+ 
912+ Note over WN,Impl: Built-in and plugin follow the exact same path: handle decode + 2 loads + 1 indirect call
913+```
914+ 
915+### 6.4 Data-Plane Invocation (Branch Decision)
916+ 
917+```mermaid
918+sequenceDiagram
919+ participant Caller as Upper-Layer Operator
920+ participant WO as HcommChannelNotifyRecordOnThread
921+ participant Ops as HcommNicChannelOps
922+ participant Ch as Channel
923+ participant HR as HcclRemoteWrite
924+ participant Impl as Plugin Implementation
925+ 
926+ Caller->>WO: notifyRecordOnThread(thread, ch, remoteNotifyIdx)
927+ alt Plugin (IS_PLUGIN_HANDLE, ch high bit set)
928+ WO->>WO: Obtain channel with restored bit: ch = CHANNEL_FROM_HANDLE(channel)
929+ WO->>Ops: ch->GetNicOps()->notifyRecordOnThread(ch->GetNicCtx(), thread, remoteNotifyIdx)
930+ Ops->>Impl: pluginNotifyRecordOnThread(pluginCtx, ...)
931+ Impl->>Impl: Plugin data plane (custom RDMA/URMA)
932+ Impl-->>Ops: ret
933+ Ops-->>WO: ret
934+ else
935+ WO->>WO: hrtGetDeviceType(dt)
936+ alt DEV_TYPE_950
937+ WO->>Ch: ((Channel*)ch)->Write(...)
938+ Ch-->>WO: ret
939+ else Non-950
940+ WO->>HR: HcclRemoteNotifyRecord(stream, channel, remoteNotifyIdx)
941+ HR-->>WO: ret
942+ end
943+ end
944+ WO-->>Caller: ret
945+```
946+ 
947+### 6.5 Plugin Loading
948+ 
949+```mermaid
950+sequenceDiagram
951+ participant Lib as libhcomm.so
952+ participant Loader as NicPluginLoader
953+ participant FS as File System
954+ participant DL as dl Runtime
955+ 
956+ Lib->>Loader: First FindHostNicPlugin() triggers std::call_once
957+ 
958+ Loader->>FS: opendir($ASCEND_HOME_PATH/hcomm_plugin/)
959+ FS-->>Loader: *.so list
960+ 
961+ loop for each .so
962+ Loader->>DL: dlopen(path, RTLD_NOW)
963+ DL-->>Loader: soHandle
964+ 
965+ alt dlopen Succeeded
966+ Loader->>DL: dlsym("HcommNicPluginGetInfo")
967+ Loader->>DL: dlsym("HcommNicPluginCreateEndpoint")
968+ Loader->>DL: dlsym("HcommNicPluginCreateChannel")
969+ 
970+ alt Any Symbol Missing
971+ Loader->>DL: dlclose(soHandle)
972+ Note over Loader: WARNING log
973+ else All Symbols Available
974+ Loader->>SO: getInfo()
975+ SO-->>Loader: info{name, protocols[], magic, version}
976+ 
977+ Loader->>Loader: Validate: magic/version/size/protocols
978+ 
979+ alt Validation Passed
980+ Loader->>Loader: protocolPlugins[protocol] = entry
981+ Note over Loader: Same-protocol conflict: later load overwrites + WARNING
982+ else Validation Failed
983+ Loader->>DL: dlclose(soHandle)
984+ Note over Loader: WARNING log
985+ end
986+ end
987+ else dlopen Failed
988+ Note over Loader: WARNING log, skip
989+ end
990+ end
991+```
992+ 
993+---
994+ 
995+## 7. Plugin Development Guide
996+ 
997+### 7.1 Directory Structure
998+ 
999+```text
1000+experimental/base_comm/nic_plugin/<my_plugin>/
1001+├── CMakeLists.txt # Independent build; does not link libhcomm.so
1002+└── src/
1003+ └── my_plugin.c # 3 exported functions + ops table implementation
1004+```
1005+ 
1006+### 7.2 Three Required Exported Symbols
1007+ 
1008+```c
1009+// 1. Return plugin metadata
1010+const HcommNicPluginInfo *HcommNicPluginGetInfo(void) {
1011+ static const HcommNicPluginInfo info = {
1012+ .header = {
1013+ .magicWord = HCOMM_NIC_PLUGIN_INFO_MAGIC_WORD,
1014+ .version = HCOMM_NIC_PLUGIN_INFO_VERSION,
1015+ .size = sizeof(HcommNicPluginInfo),
1016+ .reserved = 0,
1017+ },
1018+ .name = "my_plugin",
1019+ .protocolCount = 1,
1020+ .protocols = {COMM_PROTOCOL_ROCE}, // Replacement mode
1021+ // or .protocols = {COMM_PROTOCOL_CUSTOM_BASE + 0}, // Extension mode
1022+ };
1023+ return &info;
1024+}
1025+ 
1026+// 2. Create Endpoint
1027+int32_t HcommNicPluginCreateEndpoint(const EndpointDesc *desc,
1028+ void **outCtx, HcommNicEndpointOps **outOps) {
1029+ MyEndpointCtx *ctx = malloc(sizeof(MyEndpointCtx));
1030+ // Initialize ctx...
1031+ *outCtx = ctx;
1032+ *outOps = &kMyEndpointOps; // HcommNicEndpointOps table implemented by the plugin
1033+ return 0;
1034+}
1035+ 
1036+// 3. Create Channel
1037+int32_t HcommNicPluginCreateChannel(void *epCtx, const HcommChannelDesc *desc,
1038+ void **outCtx, HcommNicChannelOps **outOps) {
1039+ MyChannelCtx *ctx = malloc(sizeof(MyChannelCtx));
1040+ // Initialize ctx (establish connections, exchange memory information, etc.)...
1041+ *outCtx = ctx;
1042+ *outOps = &kMyChannelOps; // HcommNicChannelOps table implemented by the plugin
1043+ return 0;
1044+}
1045+```
1046+ 
1047+### 7.3 Implement the Ops Table
1048+ 
1049+```c
1050+// ---- HcommNicEndpointOps ----
1051+static int32_t RegisterMemory(void *ctx, const CommMem *mem,
1052+ const char *tag, void **handle) {
1053+ // Implement memory registration (e.g., ibv_reg_mr / urma_reg_mr)
1054+ return 0;
1055+}
1056+static int32_t UnregisterMemory(void *ctx, void *handle) {
1057+ // Implement memory unregistration
1058+ return 0;
1059+}
1060+// ... memoryExport, memoryImport, memoryUnimport, init, destroy follow the same pattern
1061+ 
1062+static HcommNicEndpointOps kMyEndpointOps = {
1063+ {HCOMM_NIC_ENDPOINT_OPS_VERSION, HCOMM_NIC_ENDPOINT_OPS_MAGIC_WORD, sizeof(HcommNicEndpointOps), 0},
1064+ InitEndpoint, // init
1065+ DestroyEndpoint, // destroy
1066+ RegisterMemory, // registerMemory
1067+ UnregisterMemory, // unregisterMemory
1068+ MemoryExport, // memoryExport
1069+ MemoryImport, // memoryImport
1070+ MemoryUnimport, // memoryUnimport
1071+ GetListenPort, // getListenPort
1072+};
1073+ 
1074+// ---- HcommNicChannelOps ----
1075+// Implement data-plane entries as needed; destroy must be implemented
1076+static int32_t WriteNbi(void *ctx, void *dst, const void *src, uint64_t len) {
1077+ MyChannelCtx *ch = (MyChannelCtx*)ctx;
1078+ // Implement data write operations (e.g., ibv_post_send)
1079+ return 0;
1080+}
1081+ 
1082+static HcommNicChannelOps kMyChannelOps = {
1083+ {HCOMM_NIC_CHANNEL_OPS_VERSION, HCOMM_NIC_CHANNEL_OPS_MAGIC_WORD, sizeof(HcommNicChannelOps), 0},
1084+ InitChannel, // init
1085+ DestroyChannel, // destroy
1086+ GetStatus, // getStatus
1087+ WriteNbi, // writeNbi
1088+ WriteNbiOnThread, // writeNbiOnThread
1089+ nullptr, // writeOnThread
1090+ WriteWithNotifyNbi, // writeWithNotifyNbi
1091+ WriteWithNotifyNbiOnThread, // writeWithNotifyNbiOnThread
1092+ nullptr, // writeWithNotifyOnThread
1093+ nullptr, // writeReduceOnThread
1094+ nullptr, // writeReduceWithNotifyOnThread
1095+ ReadNbi, // readNbi
1096+ ReadNbiOnThread, // readNbiOnThread
1097+ nullptr, // readOnThread
1098+ nullptr, // readReduceOnThread
1099+ NotifyRecord, // notifyRecord
1100+ NotifyRecordOnThread, // notifyRecordOnThread
1101+ NotifyWait, // notifyWait
1102+ NotifyWaitOnThread, // notifyWaitOnThread
1103+ nullptr, // notifyWaitOnThreadWithDefaultTimeout
1104+ nullptr, // batchTransferOnThread
1105+ Fence, // fence
1106+ FenceOnThread, // fenceOnThread
1107+ nullptr, // drainOnThread
1108+};
1109+```
1110+ 
1111+### 7.4 Build and Deployment
1112+ 
1113+```bash
1114+# 1. Build
1115+cd experimental/base_comm/nic_plugin/<my_plugin>
1116+mkdir build && cd build
1117+cmake .. && make -j
1118+ 
1119+# CMakeLists.txt key points:
1120+# - Do not link libhcomm.so
1121+# - Align compilation options with the hcomm_nic_plugin.h header file path
1122+# - Generate a .so file (e.g., libmy_plugin.so)
1123+ 
1124+# 2. Deploy
1125+cp build/libmy_plugin.so ${ASCEND_HOME_PATH}/hcomm_plugin/
1126+ 
1127+# 3. Verify
1128+# Restart the process and check for the "[NicPlugin] protocol[X] is handled by plugin[my_plugin]" message in the log
1129+ 
1130+# 4. Debug (alternative path)
1131+export HCOMM_NIC_PLUGIN_SO=/path/to/build/libmy_plugin.so
1132+```
1133+ 
1134+### 7.5 Protocol Number Selection
1135+ 
1136+| Category | Protocol Number Range | Description |
1137+|----------|----------------------|-------------|
1138+| Built-in Protocol | All non-RESERVED built-in enumeration values < `COMM_PROTOCOL_CUSTOM_BASE` (1000) | Replacement mode: overwrite the built-in implementation |
1139+| Custom Protocol | ≥ `COMM_PROTOCOL_CUSTOM_BASE` (1000) | Extension mode: add a new protocol |
1140+ 
1141+The `protocols[]` array returned by `HcommNicPluginGetInfo()` can contain both built-in protocol numbers (replacement) and custom protocol numbers (extension), with a maximum of 4 entries.
1142+ 
1143+---
1144+ 
1145+## 8. Performance Analysis
1146+ 
1147+### 8.1 Call Chain Overhead
1148+ 
1149+```text
1150+Built-in and plugin share the same call path:
1151+ and rsi, ~HCOMM_PLUGIN_HANDLE_FLAG ; handle decode (1 cycle)
1152+ test rsi, rsi ; null check (parallel, 0 extra)
1153+ mov rax, [rsi+ nicOps_offs] ; load nicOps_ (4 cycles)
1154+ mov rdi, [rsi + nicCtx_offs] ; load nicCtx_ (parallel, 0 extra)
1155+ call [rax + writeNbi_offs] ; load writeNbi (4 cycles) + indirect call (1-2 cycles)
1156+ ; callee: BuiltinWriteNbi
1157+ ; Total: ~10-11 cycles (all loads hit L1, indirect call hits BTB)
1158+```
1159+ 
1160+---
1161+ 
1162+## 9. Edge Cases
1163+ 
1164+| Scenario | Handling Strategy |
1165+|----------|------------------|
1166+| Some ops table entries are NULL | The interfaces to be implemented are divided into two categories: (1) Interfaces that the plugin must implement (destroy) — the framework performs strict validation. (2) Interfaces that the plugin is not required to implement — the plugin can fill NULL for such interfaces, and the framework populates default implementations. Default implementation return values have two types: a. For the init interface, return SUCCESS. b. For other interfaces, return NOT_SUPPORT. |
1167+| Multiple plugins conflict on the same protocol | Load in filename lexicographic order; later load overwrites earlier load with a WARNING log |
1168+| Plugin .so fails to load | `dlclose` releases the handle, WARNING log is recorded, and processing continues with the next .so |
1169+ 
1170+---
1171+ 
1172+## 10. Constraints and Limitations
1173+ 
1174+| # | Constraint | Type | Description |
1175+|---|-----------|------|-------------|
1176+| C1 | `nicOps_` is immutable | Design Constraint | Set once at creation time; read-only at runtime |
1177+| C2 | Plugin only for HOST endpoints | Runtime Constraint | `ENDPOINT_LOC_TYPE_HOST` |
1178+| C3 | Process-level lifecycle | Runtime Constraint | No `dlclose`; the OS reclaims resources on process exit |
1179+| C4 | One plugin per protocol | Design Constraint | `ProtocolPlugins` is a 1:1 mapping |
1180+| C5 | No modifications to built-in subclasses | Design Constraint | 10+ subclasses require zero modifications |
1181+| C6 | Public API unchanged | Compatibility Constraint | Zero changes to `include/` |
1182+| C7 | Op thread safety guaranteed by the plugin | Design Constraint | The plugin must ensure that ops can be called concurrently by multiple threads; the framework does not perform serialization |
1183+ 
1184+---
1185+ 
1186+## 11. Compatibility
1187+ 
1188+- **API**: `include/hcomm_primitives.h`, `include/hcomm_res.h`, `include/hcomm_res_defs.h` — zero changes
1189+- **ABI**: `src/base_comm/primitives/api_c_adpt/nic_plugin/hcomm_nic_plugin.h` — this path is stable as part of the SDK; plugin builds must configure it as an include path
1190+- **Build**: Behavior is equivalent to the original version when `ENABLE_EXPERIMENTAL=OFF`
1191+ 
1192+---
1193+ 
1194+## 12. Test Plan
1195+ 
1196+| Level | Content | Method |
1197+|-------|---------|--------|
1198+| Unit | ABI signature stability | Verify that exported symbol types remain unchanged |
1199+| Unit | `NicPluginLoader` loading/validation/mapping/conflict | Full coverage with mock .so |
1200+| Unit | `g_BuiltinChannelOps` / `g_BuiltinEndpointOps` wrapper functions | Verify forwarding correctness |
1201+| Integration | Plugin Endpoint → Channel → Write/Read/Notify/Fence | Experimental RoCE/UB plugin + UT |
1202+| Integration | Extended protocol (custom protocol number) end-to-end | Mock new protocol plugin |
1203+| Integration | Built-in path regression | Built-in channels function normally with `ENABLE_EXPERIMENTAL=ON` |
1204+ 
1205+---
1206+ 
1207+## 13. Risk Assessment
1208+ 
1209+| Risk | Level | Mitigation |
1210+|------|-------|-----------|
1211+| New fields in the base class affect existing subclass ABI | Low | Non-virtual fields at the end of the protected section, default `nullptr`; UT covers all subclasses |
1212+| `g_Builtin**Ops` duplicated across .o files | Low | Inline definition in header file |
1213+| Plugin .so load failure affects built-in channels | Very Low | Early return within `std::call_once` |
1214+| `nicOps_` is NULL when dispatched | Very Low | The creation flow ensures that every object is populated with an ops table; UT provides coverage |
1215+ 
1216+---
1217+ 
1218+## 14. Alternatives
1219+ 
1220+| Approach | Advantages | Disadvantages | Conclusion |
1221+|----------|-----------|---------------|-----------|
1222+| Entry-level tag-bit branch dispatch | Minimal changes | 30+ hot-path branches | Not adopted |
1223+| C++ virtual function adapter subclass | No base class changes | Virtual function overhead | Not adopted |
1224+| C ops table embedded in base class | Zero branching | Requires base class modification (non-invasive) | Adopted |
1225+ 
1226+---
1227+ 
1228+## 15. Open Questions
1229+ 
1230+| Number | Question | Description |
1231+|--------|----------|-------------|
1232+| O1 | Whether runtime unloading or updating of plugins is needed | Not currently supported |
1233+| O2 | Conflict arbitration strategy for one plugin per protocol | Currently, the later-loaded plugin overwrites the earlier one |
1234+ 
1235+---
1236+ 
1237+## 16. Review Records
1238+ 
1239+The review process takes place in the PR comment section. For detailed review comments, refer to the corresponding PR comments.
@@ -23,6 +23,7 @@ This file registers all assigned RFC numbers. Before adding a new RFC, claim the
23 23 
24| Number | Title | Author | Status | PR |24| Number | Title | Author | Status | PR |
25|------|------|------|------|-----|25|------|------|------|------|-----|
26+|0001|RFC: Topology-Based Cluster Communication Rapid Sub-Health Monitoring | Wang Yunhao (Xidian University)|`accepted`|#2491|
26 27 
27## Numbering Rules28## Numbering Rules
28 29 
@@ -0,0 +1 @@
1+<svg width="3222" height="1801" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" overflow="hidden"><g transform="translate(-664 -325)"><g><rect x="835.5" y="1371.5" width="2104" height="715" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none"/><rect x="1318.5" y="1505.5" width="1588" height="545" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#E8E8E8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 868.401 1455)">Agent</text><rect x="706.5" y="455.5" width="1009" height="427" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><rect x="1768.5" y="455.5" width="1234" height="422" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><rect x="2435.5" y="1740.5" width="428" height="157" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#A6CAEC" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2520.48 1840)">Time Analysis</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 729.986 532)">Controller</text><rect x="765.5" y="580.5" width="432" height="228" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 788.838 716)">Topo Partition</text><rect x="1341" y="1629" width="857" height="198" fill="#4E95D9" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1521.93 1697)">Counter Collection</text><path d="M2412.34 1820.11 2187.12 1794.92C2185.23 1794.71 2183.87 1793 2184.08 1791.12 2184.3 1789.23 2186 1787.87 2187.88 1788.08L2413.1 1813.28C2414.99 1813.49 2416.35 1815.19 2416.14 1817.07 2415.93 1818.96 2414.23 1820.32 2412.34 1820.11ZM2409.69 1802.52 2435.5 1819.24 2406.64 1829.85ZM2185.97 1805.16C2178.42 1804.32 2172.99 1797.52 2173.84 1789.97 2174.68 1782.42 2181.48 1776.99 2189.03 1777.84 2196.58 1778.68 2202.01 1785.48 2201.17 1793.03 2200.32 1800.58 2193.52 1806.01 2185.97 1805.16Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2264.44 1939)">Latency,</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2264.44 2005)">Pass Rate</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2258.42 1679)">PFC, </text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2258.42 1745)">CNP,…</text><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="83" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 3505.66 1617)"></text><rect x="869.5" y="1508.5" width="382" height="545" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 931.448 1725)">Topo Probing</text><rect x="2362.5" y="672.5" width="616" height="99" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#96DCF8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2477.85 732)">Link Solving</text><rect x="1808.5" y="643.5" width="507" height="165" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#96DCF8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1883.53 703)">Z-Score-Based</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1868.63 780)">Link Assessment</text><rect x="1341" y="1838" width="856" height="198" fill="#4E95D9" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1603.68 1906)">Active Probing</text><rect x="1032.5" y="1267.5" width="297" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1086.81 1352)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1242.07 1352)">0</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="83" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2107.46 1345)"></text><rect x="1640.5" y="1277.5" width="296" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1694.49 1362)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1849.75 1362)">1</text><rect x="2356.5" y="1279.5" width="314" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2407.21 1364)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2562.47 1364)">M</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1800.96 543)">Analyzer</text><path d="M0 0 276.615 110.458" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1 0 0 -1 1180.5 1266.96)"/><path d="M0 0 19.2323 72.3369" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1 0 0 -1 1784.5 1279.84)"/><path d="M2392.16 1279.28 2117.5 1121.5" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><path d="M1474.9 1025.98C1465.57 994.312 1513.62 965.151 1582.23 960.844 1610.03 959.098 1638.3 961.707 1662.5 968.253 1688.15 945.937 1748.09 937.455 1796.38 949.309 1804.83 951.382 1812.58 954.01 1819.39 957.109 1839.37 938.61 1888.03 931.095 1928.08 940.323 1939.16 942.877 1948.83 946.57 1956.34 951.12 1988.54 933.626 2045.44 931.465 2083.43 946.294 2099.4 952.526 2110.16 961.122 2113.87 970.608 2166.63 977.253 2197.75 1002.42 2183.39 1026.82 2182.18 1028.87 2180.66 1030.88 2178.85 1032.84 2221.16 1058.27 2210.8 1094.71 2155.71 1114.24 2138.57 1120.32 2118.31 1124.25 2096.9 1125.67 2096.43 1153.02 2047.96 1175.02 1988.65 1174.8 1968.83 1174.73 1949.45 1172.13 1932.64 1167.29 1912.58 1197.96 1842.6 1215.29 1776.34 1206 1748.57 1202.11 1724.58 1193.88 1708.46 1182.72 1640.62 1201.61 1552.56 1191.45 1511.78 1160.03 1511.27 1159.63 1510.76 1159.23 1510.27 1158.83 1465.87 1161.23 1425.65 1146.6 1420.44 1126.17 1417.66 1115.27 1425.45 1104.37 1441.74 1096.36 1403.29 1085.91 1390.38 1062.98 1412.89 1045.14 1425.88 1034.85 1448.66 1028.05 1474.23 1026.82Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-rule="evenodd" fill-opacity="1"/><path d="M1489.62 1100.32C1473.21 1100.91 1456.8 1099.16 1442.6 1095.3M1531.12 1155.23C1524.52 1156.45 1517.6 1157.25 1510.54 1157.64M1708.41 1181.62C1703.45 1178.19 1699.29 1174.51 1696.01 1170.67M1937.67 1154.3C1936.94 1158.38 1935.28 1162.41 1932.72 1166.32M2036.1 1080.03C2073.31 1088.39 2096.79 1105.86 2096.46 1124.95M2178.47 1032.17C2172.45 1038.67 2163.25 1044.44 2151.59 1049.02M2113.98 969.664C2115 972.289 2115.48 974.953 2115.4 977.62M1942.33 960.383C1945.83 956.712 1950.47 953.294 1956.1 950.236M1813.54 965.218C1814.97 962.185 1817.21 959.245 1820.21 956.467M1662.41 968.19C1671.17 970.559 1679.27 973.41 1686.54 976.68M1479.12 1034.91C1477.21 1031.99 1475.8 1029 1474.91 1025.98" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="73" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1583.12 1090)">RoCE</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="73" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1781.08 1090)">Fabric</text><path d="M0 3.4375-27.5 3.4375-27.5-3.4375 0-3.4375ZM-48.125 3.4375-75.625 3.4375-75.625-3.4375-48.125-3.4375ZM-79.0617 13.7508-79.0617 41.2508-85.9367 41.2508-85.9367 13.7508ZM-79.0617 61.8758-79.0617 89.3757-85.9367 89.3757-85.9367 61.8758ZM-79.0617 110.001-79.0617 137.501-85.9367 137.501-85.9367 110.001ZM-79.0617 158.126-79.0617 185.626-85.9367 185.626-85.9367 158.126ZM-79.0617 206.251-79.0617 233.751-85.9367 233.751-85.9367 206.251ZM-79.0617 254.376-79.0617 281.876-85.9367 281.876-85.9367 254.376ZM-79.0617 302.501-79.0617 330.001-85.9367 330.001-85.9367 302.501ZM-79.0617 350.626-79.0617 378.126-85.9367 378.126-85.9367 350.626ZM-79.0617 398.751-79.0617 426.251-85.9367 426.251-85.9367 398.751ZM-79.0617 446.876-79.0617 474.376-85.9367 474.376-85.9367 446.876ZM-79.0617 495.001-79.0617 522.501-85.9367 522.501-85.9367 495.001ZM-79.0617 543.126-79.0617 570.626-85.9367 570.626-85.9367 543.126ZM-79.0617 591.251-79.0617 618.751-85.9367 618.751-85.9367 591.251ZM-79.0617 639.376-79.0617 666.876-85.9367 666.876-85.9367 639.376ZM-79.0617 687.501-79.0617 715.001-85.9367 715.001-85.9367 687.501ZM-79.0617 735.626-79.0617 763.126-85.9367 763.126-85.9367 735.626ZM-79.0617 783.751-79.0617 811.251-85.9367 811.251-85.9367 783.751ZM-79.0617 831.876-79.0617 842.294-82.4992 838.857-65.418 838.857-65.418 845.732-85.9367 845.732-85.9367 831.876ZM-44.793 838.857-17.293 838.857-17.293 845.732-44.793 845.732ZM3.33198 838.857 30.832 838.857 30.832 845.732 3.33198 845.732ZM51.457 838.857 78.957 838.857 78.957 845.732 51.457 845.732ZM99.582 838.857 115.806 838.857 115.806 857.008 108.931 857.008 108.931 842.294 112.368 845.732 99.582 845.732ZM115.806 877.633 115.806 905.133 108.931 905.133 108.931 877.633ZM115.806 925.758 115.806 949.418 108.931 949.418 108.931 925.758ZM126.118 944.835 112.368 972.335 98.6182 944.835Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 -1.22465e-16 -1.22465e-16 -1 869.5 1780.83)"/><path d="M0 3.4375-27.5 3.4375-27.5-3.4375 0-3.4375ZM-48.125 3.4375-75.625 3.4375-75.625-3.4375-48.125-3.4375ZM-96.25 3.4375-123.75 3.4375-123.75-3.4375-96.25-3.4375ZM-144.375 3.4375-150.117 3.4375-150.117-25.1958-143.242-25.1958-143.242 0-146.679-3.4375-144.375-3.4375ZM-150.117-45.8209-150.117-73.3208-143.242-73.3208-143.242-45.8209ZM-150.117-93.9458-150.117-121.446-143.242-121.446-143.242-93.9458ZM-150.117-142.071-150.117-169.571-143.242-169.571-143.242-142.071ZM-150.117-190.196-150.117-217.696-143.242-217.696-143.242-190.196ZM-150.117-238.321-150.117-265.821-143.242-265.821-143.242-238.321ZM-150.117-286.446-150.117-313.946-143.242-313.946-143.242-286.446ZM-150.117-334.571-150.117-362.071-143.242-362.071-143.242-334.571ZM-150.117-382.696-150.117-410.196-143.242-410.196-143.242-382.696ZM-150.117-430.821-150.117-458.321-143.242-458.321-143.242-430.821ZM-150.117-478.946-150.117-506.446-143.242-506.446-143.242-478.946ZM-150.117-527.071-150.117-554.571-143.242-554.571-143.242-527.071ZM-150.117-575.196-150.117-602.696-143.242-602.696-143.242-575.196ZM-150.117-623.321-150.117-650.821-143.242-650.821-143.242-623.321ZM-150.117-671.446-150.117-698.946-143.242-698.946-143.242-671.446ZM-150.117-719.571-150.117-747.071-143.242-747.071-143.242-719.571ZM-150.117-767.696-150.117-787.395-135.44-787.395-135.44-780.52-146.679-780.52-143.242-783.957-143.242-767.696ZM-114.815-787.395-87.3153-787.395-87.3153-780.52-114.815-780.52ZM-66.6903-787.395-39.1903-787.395-39.1903-780.52-66.6903-780.52ZM-18.5653-787.395 8.93467-787.395 8.93467-780.52-18.5653-780.52ZM29.5597-787.395 57.0597-787.395 57.0597-780.52 29.5597-780.52ZM77.6847-787.395 105.185-787.395 105.185-780.52 77.6847-780.52ZM125.81-787.395 153.31-787.395 153.31-780.52 125.81-780.52ZM173.935-787.395 201.435-787.395 201.435-780.52 173.935-780.52ZM222.06-787.395 249.56-787.395 249.56-780.52 222.06-780.52ZM270.185-787.395 297.685-787.395 297.685-780.52 270.185-780.52ZM318.31-787.395 345.81-787.395 345.81-780.52 318.31-780.52ZM366.435-787.395 393.935-787.395 393.935-780.52 366.435-780.52ZM414.56-787.395 442.06-787.395 442.06-780.52 414.56-780.52ZM462.685-787.395 490.185-787.395 490.185-780.52 462.685-780.52ZM510.81-787.395 538.31-787.395 538.31-780.52 510.81-780.52ZM558.935-787.395 586.435-787.395 586.435-780.52 558.935-780.52ZM607.06-787.395 634.56-787.395 634.56-780.52 607.06-780.52ZM655.185-787.395 682.685-787.395 682.685-780.52 655.185-780.52ZM703.31-787.395 730.81-787.395 730.81-780.52 703.31-780.52ZM751.435-787.395 778.935-787.395 778.935-780.52 751.435-780.52ZM799.56-787.395 827.06-787.395 827.06-780.52 799.56-780.52ZM847.685-787.395 875.185-787.395 875.185-780.52 847.685-780.52ZM895.81-787.395 923.31-787.395 923.31-780.52 895.81-780.52ZM943.935-787.395 971.435-787.395 971.435-780.52 943.935-780.52ZM992.06-787.395 1019.56-787.395 1019.56-780.52 992.06-780.52ZM1040.18-787.395 1067.68-787.395 1067.68-780.52 1040.18-780.52ZM1088.31-787.395 1115.81-787.395 1115.81-780.52 1088.31-780.52ZM1136.43-787.395 1163.93-787.395 1163.93-780.52 1136.43-780.52ZM1184.56-787.395 1212.06-787.395 1212.06-780.52 1184.56-780.52ZM1232.68-787.395 1260.18-787.395 1260.18-780.52 1232.68-780.52ZM1280.81-787.395 1308.31-787.395 1308.31-780.52 1280.81-780.52ZM1328.93-787.395 1356.43-787.395 1356.43-780.52 1328.93-780.52ZM1377.06-787.395 1404.56-787.395 1404.56-780.52 1377.06-780.52ZM1425.18-787.395 1452.68-787.395 1452.68-780.52 1425.18-780.52ZM1473.31-787.395 1500.81-787.395 1500.81-780.52 1473.31-780.52ZM1521.43-787.395 1542.1-787.395 1542.1-773.682 1535.22-773.682 1535.22-783.957 1538.66-780.52 1521.43-780.52ZM1542.1-753.057 1542.1-725.557 1535.22-725.557 1535.22-753.057ZM1542.1-704.932 1542.1-677.432 1535.22-677.432 1535.22-704.932ZM1542.1-656.807 1542.1-629.307 1535.22-629.307 1535.22-656.807ZM1542.1-608.682 1542.1-581.182 1535.22-581.182 1535.22-608.682ZM1542.1-560.557 1542.1-533.057 1535.22-533.057 1535.22-560.557ZM1542.1-512.432 1542.1-484.932 1535.22-484.932 1535.22-512.432ZM1542.1-464.307 1542.1-436.807 1535.22-436.807 1535.22-464.307ZM1542.1-416.182 1542.1-388.682 1535.22-388.682 1535.22-416.182ZM1542.1-368.057 1542.1-340.557 1535.22-340.557 1535.22-368.057ZM1542.1-319.932 1542.1-292.432 1535.22-292.432 1535.22-319.932ZM1542.1-271.807 1542.1-244.307 1535.22-244.307 1535.22-271.807ZM1542.1-223.682 1542.1-196.182 1535.22-196.182 1535.22-223.682ZM1542.1-175.557 1542.1-148.057 1535.22-148.057 1535.22-175.557ZM1542.1-127.432 1542.1-99.9321 1535.22-99.9321 1535.22-127.432ZM1542.1-79.3071 1542.1-51.8071 1535.22-51.8071 1535.22-79.3071ZM1542.1-31.1821 1542.1-3.68209 1535.22-3.68209 1535.22-31.1821ZM1542.1 16.9429 1542.1 44.4429 1535.22 44.4429 1535.22 16.9429ZM1542.1 65.0679 1542.1 92.5679 1535.22 92.5679 1535.22 65.0679ZM1542.1 113.193 1542.1 140.693 1535.22 140.693 1535.22 113.193ZM1542.1 161.318 1542.1 188.818 1535.22 188.818 1535.22 161.318ZM1542.1 209.443 1542.1 236.943 1535.22 236.943 1535.22 209.443ZM1542.1 257.568 1542.1 285.068 1535.22 285.068 1535.22 257.568ZM1537.99 308.458 1510.49 308.458 1510.49 301.583 1537.99 301.583ZM1489.86 308.458 1479.07 308.458 1479.07 301.583 1489.86 301.583ZM1483.65 318.77 1456.15 305.02 1483.65 291.27Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1.83697e-16 1 1 -1.83697e-16 1464.5 580.5)"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1361.81 1597)">Executor</text><path d="M0-3.4375 27.5-3.4375 27.5 3.4375 0 3.4375ZM48.125-3.4375 75.625-3.4375 75.625 3.4375 48.125 3.4375ZM96.25-3.4375 123.75-3.4375 123.75 3.4375 96.25 3.4375ZM144.375-3.4375 171.875-3.4375 171.875 3.4375 144.375 3.4375ZM192.5-3.4375 201.903-3.4375 201.903 21.5343 195.028 21.5343 195.028 0 198.466 3.4375 192.5 3.4375ZM201.903 42.1593 201.903 69.6593 195.028 69.6593 195.028 42.1593ZM201.903 90.2843 201.903 117.784 195.028 117.784 195.028 90.2843ZM201.903 138.409 201.903 165.909 195.028 165.909 195.028 138.409ZM201.903 186.534 201.903 214.034 195.028 214.034 195.028 186.534ZM201.903 234.659 201.903 262.159 195.028 262.159 195.028 234.659ZM201.903 282.784 201.903 310.284 195.028 310.284 195.028 282.784ZM201.903 330.909 201.903 358.409 195.028 358.409 195.028 330.909ZM201.903 379.034 201.903 406.534 195.028 406.534 195.028 379.034ZM201.903 427.159 201.903 454.659 195.028 454.659 195.028 427.159ZM201.903 475.284 201.903 502.784 195.028 502.784 195.028 475.284ZM201.903 523.409 201.903 550.909 195.028 550.909 195.028 523.409ZM201.903 571.534 201.903 599.034 195.028 599.034 195.028 571.534ZM201.903 619.659 201.903 647.159 195.028 647.159 195.028 619.659ZM201.903 667.784 201.903 695.284 195.028 695.284 195.028 667.784ZM201.903 715.909 201.903 743.409 195.028 743.409 195.028 715.909ZM201.903 764.034 201.903 791.534 195.028 791.534 195.028 764.034ZM201.903 812.159 201.903 839.659 195.028 839.659 195.028 812.159ZM201.903 860.284 201.903 887.784 195.028 887.784 195.028 860.284ZM201.903 908.409 201.903 935.909 195.028 935.909 195.028 908.409ZM201.903 956.534 201.903 984.034 195.028 984.034 195.028 956.534ZM201.903 1004.66 201.903 1032.16 195.028 1032.16 195.028 1004.66ZM201.903 1052.78 201.903 1080.28 195.028 1080.28 195.028 1052.78ZM194.789 1100.67 167.289 1100.67 167.289 1093.8 194.789 1093.8ZM146.664 1100.67 138.882 1100.67 138.882 1093.8 146.664 1093.8ZM143.466 1110.98 115.966 1097.23 143.466 1083.48Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 0 0 -1 2863.5 1818.73)"/><path d="M23.1157-1.6294 47.9136 0.32168C49.8062 0.470591 51.2197 2.12559 51.0708 4.01822 50.9219 5.91085 49.2669 7.32441 47.3743 7.1755L22.5764 5.22442C20.6838 5.07551 19.2702 3.42051 19.4192 1.52788 19.5681-0.364746 21.2231-1.77831 23.1157-1.6294ZM26.3368 15.8647 0 0 28.4938-11.5506ZM48.7224-9.95905C56.293-9.36341 61.9472-2.74342 61.3516 4.8271 60.7559 12.3976 54.1359 18.0519 46.5654 17.4562 38.9949 16.8606 33.3406 10.2406 33.9363 2.67008 34.5319-4.90044 41.1519-10.5547 48.7224-9.95905Z" fill="#000000" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 0 0 -1 2315.5 725.249)"/><path d="M2240.44 1135.69 2158.17 810.343C2157.7 808.502 2158.82 806.633 2160.66 806.167 2162.5 805.702 2164.37 806.817 2164.83 808.657L2247.1 1134C2247.57 1135.84 2246.46 1137.71 2244.61 1138.18 2242.77 1138.64 2240.91 1137.53 2240.44 1135.69ZM2255.98 1127.03 2249.39 1157.06 2229.32 1133.77ZM2148.17 812.871C2146.31 805.509 2150.77 798.031 2158.13 796.17 2165.49 794.308 2172.97 798.767 2174.83 806.129 2176.69 813.491 2172.23 820.969 2164.87 822.83 2157.51 824.692 2150.03 820.233 2148.17 812.871Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><path d="M2219 1185.5C2219 1169.21 2232.21 1156 2248.5 1156 2264.79 1156 2278 1169.21 2278 1185.5 2278 1201.79 2264.79 1215 2248.5 1215 2232.21 1215 2219 1201.79 2219 1185.5Z" stroke="#042433" stroke-width="13.75" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><path d="M2269 1207 2294 1235.94" stroke="#0E2841" stroke-width="13.75" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><rect x="2297.5" y="1009.5" width="622" height="169" stroke="#000000" stroke-width="4.58333" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#F6C6AD" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2443.01 1111)">Anomaly Output</text><path d="M1118 1501.87 1158.63 1461.16 1146.99 1449.54 1193.53 1449.5 1193.57 1496.04 1181.92 1484.41 1141.29 1525.12Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEBF8" fill-rule="evenodd" fill-opacity="1"/><path d="M1819.73 1529.21 1829.18 1472.48 1812.95 1469.77 1850.82 1442.72 1877.87 1480.59 1861.64 1477.89 1852.18 1534.62Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#D1D1D1" fill-rule="evenodd" fill-opacity="1"/><path d="M0 0 53.6921 0 53.6921 1332.72 12.4421 1332.72" stroke="#0E2841" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1.83697e-16 1 1 -1.83697e-16 1180.5 1392.5)"/><path d="M0 0 0.000360892 45.4724" stroke="#0E2841" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(-1 0 0 1 1783.5 1402.5)"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(6.12323e-17 1 -1 6.12323e-17 721.193 1156)">Topo Probing Results</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 839.244 409)">Probe Path List</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(6.12323e-17 1 -1 6.12323e-17 2990.24 985)">Network Status Results</text><rect x="1248.5" y="580.5" width="432" height="232" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1271.16 718)">Probe Plan</text><path d="M1242.76 697.124 1200.48 696.937C1198.59 696.929 1197.05 695.383 1197.06 693.485 1197.07 691.586 1198.62 690.054 1200.52 690.062L1242.79 690.249 1242.79 690.249C1244.69 690.258 1246.22 691.803 1246.21 693.702 1246.2 695.6 1244.66 697.132 1242.76 697.124ZM1238.25 679.917 1265.69 693.788 1238.13 707.416ZM1200.44 707.25C1192.85 707.216 1186.72 701.033 1186.75 693.439 1186.78 685.845 1192.97 679.716 1200.56 679.75 1208.15 679.784 1214.28 685.967 1214.25 693.561 1214.22 701.154 1208.03 707.283 1200.44 707.25Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><path d="M22.9023 3.53092 252.373 74.9825C254.186 75.5469 255.198 77.4739 254.633 79.2865 254.069 81.0992 252.142 82.1111 250.329 81.5466L20.8584 10.0951C19.0457 9.53065 18.0338 7.60367 18.5982 5.79103 19.1627 3.97839 21.0896 2.96651 22.9023 3.53092ZM22.1688 21.304 0 0 30.3444-4.95264ZM255.439 65.1363C262.689 67.3939 266.737 75.1018 264.479 82.3524 262.222 89.603 254.514 93.6505 247.263 91.3929 240.013 89.1352 235.965 81.4273 238.223 74.1767 240.48 66.9262 248.188 62.8786 255.439 65.1363Z" fill="#000000" fill-rule="nonzero" fill-opacity="1" transform="matrix(-1 0 0 1 2435.85 1819.5)"/><rect x="902.5" y="1791.5" width="315" height="172" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 932.581 1853)">hccn_tool</text><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 930.862 1930)">traceroute</text><rect x="1394.5" y="1919.5" width="723" height="89" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1609.33 1981)">HccnRping</text><rect x="1411.5" y="1718.5" width="723" height="89" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1593.27 1780)">hccn_tool</text><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1862.54 1780)">stat</text></g></g></svg>
@@ -52,7 +52,7 @@ CCU_IF(condExpr) {
52## 约束说明52## 约束说明
53 53 
54- `CCU_IF(condExpr)`后必须紧跟{}包裹的then代码块54- `CCU_IF(condExpr)`后必须紧跟{}包裹的then代码块
55-- A5代际的`CCU_IF`当前的`condExpr`只支持==和!=两种判断模式55+- 针对Ascend 950PR/Ascend 950DT,`CCU_IF`当前的`condExpr`只支持==和!=两种判断模式
56- `CCU_IF`可以单独使用(无`CCU_ELSE`),也可以配合`CCU_ELSE`使用,两种写法均合法。56- `CCU_IF`可以单独使用(无`CCU_ELSE`),也可以配合`CCU_ELSE`使用,两种写法均合法。
57- 支持嵌套:`CCU_IF`内部可以再嵌套`CCU_IF`57- 支持嵌套:`CCU_IF`内部可以再嵌套`CCU_IF`
58- `CCU_IF`不建议在硬件Loop(`ccu::Loop`)的body lambda内部使用——硬件Loop body内不支持软件分支;框架不强制校验,但行为未定义。58- `CCU_IF`不建议在硬件Loop(`ccu::Loop`)的body lambda内部使用——硬件Loop body内不支持软件分支;框架不强制校验,但行为未定义。
@@ -1,17 +1,17 @@
1-# EI0002 Communication_Error_Timeout1+# EI0002 Communication_Error_Timeout
2 2 
3## 错误信息3## 错误信息
4 4 
5报错格式如下,占位符%s的含义依次为对端Rank ID、任务信息、通信算子信息、通信域信息:5报错格式如下,占位符%s的含义依次为对端Rank ID、任务信息、通信算子信息、通信域信息:
6 6 
7```text7```text
8-An timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s.8+A timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s.
9```9```
10 10 
11报错示例如下:11报错示例如下:
12 12 
13```text13```text
14-An timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none.14+A timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none.
15```15```
16 16 
17## 可能原因17## 可能原因
@@ -22,4 +22,4 @@ The Toolkit (cann-hcomm) and ops (cann-hccl) versions are inconsistent. The loca
22 22 
23## 解决方法23## 解决方法
24 24 
25-请安装相同的版本,本端的toolkit\(cann-hcomm\)版本、ops\(cann-hccl\)版本必须和远端对应的版本一致。25+请安装相同的版本,本端的toolkit (cann-hcomm)版本、ops (cann-hccl)版本必须和远端对应的版本一致。
@@ -11,7 +11,7 @@ Value %s for config %s is invalid. Expected value: %s.
11报错示例如下:11报错示例如下:
12 12 
13```text13```text
14-Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A." .14+Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A".
15```15```
16 16 
17## 解决方法17## 解决方法
@@ -16,6 +16,6 @@ Failed to enable listening for the host network adapter socket. Reason: The IP a
16 16 
17## 解决方法17## 解决方法
18 18 
19-1. 请确认是否已经有其他进程占用此端口,若已被占用可以通过环境变量HCCL_IF_BASE_PORT进行调整,并通过**sysctl -w net.ipv4.ip_local_reserved_ports=\*\*\*\*-\*\*\*\***调整预留端口范围。19+1. 请确认是否已经有其他进程占用此端口,若已被占用可以通过环境变量HCCL_IF_BASE_PORT进行调整,并通过`sysctl -w net.ipv4.ip_local_reserved_ports=****-****`调整预留端口范围。
20 20 
212. 请确认是否可能存在本次业务中一个device被多次拉起业务进程的情况。212. 请确认是否可能存在本次业务中一个device被多次拉起业务进程的情况。
@@ -59,6 +59,7 @@
59本节只写模块内部实现。按模块分节,跨层功能按层拆分多个模块子节,不混写。59本节只写模块内部实现。按模块分节,跨层功能按层拆分多个模块子节,不混写。
60 60 
61每个模块子节包含:61每个模块子节包含:
62+ 
62- 模块职责(一句话)63- 模块职责(一句话)
63- 核心数据结构,可以使用UML展示64- 核心数据结构,可以使用UML展示
64- 关键逻辑与算法65- 关键逻辑与算法
@@ -148,8 +148,7 @@ cmake --build build -j"$(nproc)"
148 148 
149### 4.2 配置文件149### 4.2 配置文件
150 150 
151-用户使用 JSON 文件同时描述部署拓扑、探测范围和运行参数。`schema_version=2` 为当前推荐格式;缺省151+用户使用 JSON 文件同时描述部署拓扑、探测范围和运行参数。`schema_version=2` 为当前推荐格式;缺省`schema_version` 或 `schema_version=1` 继续兼容旧格式。
152-`schema_version``schema_version=1` 继续兼容旧格式。
153 152 
154```json153```json
155{154{
@@ -227,8 +226,7 @@ cmake --build build -j"$(nproc)"
227| `probe.pingpong.payload_len` | int / `12` | 可选;HCCN Rping Payload 字节数,范围 `1~1500`。每个目标在 AddTarget 时独立生成指定长度的随机字节,不按字符串处理 |226| `probe.pingpong.payload_len` | int / `12` | 可选;HCCN Rping Payload 字节数,范围 `1~1500`。每个目标在 AddTarget 时独立生成指定长度的随机字节,不按字符串处理 |
228| `probe.pingpong.interval_ms` | positive int / `1` | 可选;HCCN Rping 的探测周期,单位毫秒 |227| `probe.pingpong.interval_ms` | positive int / `1` | 可选;HCCN Rping 的探测周期,单位毫秒 |
229 228 
230-v1 兼容格式继续支持旧的 `deploy.host_to_user_pair`、`deploy.controller`、`probe_scope`、`probe_topo.tracert`、229+v1 兼容格式继续支持旧的 `deploy.host_to_user_pair`、`deploy.controller`、`probe_scope`、`probe_topo.tracert`、`probe_controller.pingpong` 字段。兼容关系如下:
231-`probe_controller.pingpong` 字段。兼容关系如下:
232 230 
233| v1 字段 | v2 字段 | 说明 |231| v1 字段 | v2 字段 | 说明 |
234| --- | --- | --- |232| --- | --- | --- |
@@ -239,8 +237,7 @@ v1 兼容格式继续支持旧的 `deploy.host_to_user_pair`、`deploy.controlle
239| `probe_topo.tracert` | `probe.topology` | Tracert 拓扑发现参数保持兼容,字段名前缀调整 |237| `probe_topo.tracert` | `probe.topology` | Tracert 拓扑发现参数保持兼容,字段名前缀调整 |
240| `probe_controller.pingpong` | `probe.pingpong` | PingPong 探测参数保持兼容,字段名前缀调整 |238| `probe_controller.pingpong` | `probe.pingpong` | PingPong 探测参数保持兼容,字段名前缀调整 |
241 239 
242-v2 中 `probe.scope` 为必填;`probe.topology` 与 `probe.pingpong` 可缺省,缺省时使用上表默认参数。可用240+v2 中 `probe.scope` 为必填;`probe.topology` 与 `probe.pingpong` 可缺省,缺省时使用上表默认参数。可用`./run.sh migrate-config old.json new.json` 生成 v2 配置模板。
243-`./run.sh migrate-config old.json new.json` 生成 v2 配置模板。
244 241 
245### 4.3 CLI 与执行顺序242### 4.3 CLI 与执行顺序
246 243 
@@ -600,7 +597,6 @@ BW_cluster = P × times × (B_req + B_rsp) × 8 / T_cycle
600Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link)597Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link)
601```598```
602 599 
603- 
604### 8.5 安全与运维影响600### 8.5 安全与运维影响
605 601 
606- Dispatcher 使用 SSH/SCP,配置中可能包含明文密码;生产环境优先使用私钥,并限制配置文件权限。602- Dispatcher 使用 SSH/SCP,配置中可能包含明文密码;生产环境优先使用私钥,并限制配置文件权限。
@@ -612,8 +608,7 @@ Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link)
612### 9.1 兼容性608### 9.1 兼容性
613 609 
614- 工具以独立进程运行,不修改训练进程或 HCCL/HCOMM 接口。610- 工具以独立进程运行,不修改训练进程或 HCCL/HCOMM 接口。
615-- 当前推荐配置以 `schema_version=2`、`hosts[]`、`probe.scope` 为必需契约,`probe.topology`、`probe.pingpong` 为可选参数组;缺省611+- 当前推荐配置以 `schema_version=2`、`hosts[]`、`probe.scope` 为必需契约,`probe.topology`、`probe.pingpong` 为可选参数组;缺省 `schema_version` 或 `schema_version=1` 继续兼容 `probe_scope + probe_topo.tracert + probe_controller.pingpong` 格式,缺少整组探测配置时保留旧 ranktable 路径,但不新增能力。
616- `schema_version``schema_version=1` 继续兼容 `probe_scope + probe_topo.tracert + probe_controller.pingpong` 格式,缺少整组探测配置时保留旧 ranktable 路径,但不新增能力。
617- JSON/TXT 产物一旦被外部系统消费,新增字段应保持向后兼容;破坏性修改必须升级 Schema 或文件名。612- JSON/TXT 产物一旦被外部系统消费,新增字段应保持向后兼容;破坏性修改必须升级 Schema 或文件名。
618 613 
619### 9.2 特性开关614### 9.2 特性开关
@@ -633,7 +628,6 @@ Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link)
633 628 
634实验室 2 Host/16 Device:验证接口、产物和故障注入。629实验室 2 Host/16 Device:验证接口、产物和故障注入。
635 630 
636- 
637## 10. 测试方案631## 10. 测试方案
638 632 
639### 10.1 目标—用例映射633### 10.1 目标—用例映射
@@ -647,7 +641,6 @@ Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link)
647| T5 方程求解 | Q3、定位准确率 | 可解、超定、秩不足和 NaN 输入 | UT |641| T5 方程求解 | Q3、定位准确率 | 可解、超定、秩不足和 NaN 输入 | UT |
648| T6 计数器异常 | Q2/Q3 | Key 缺失、非法值、Device ID 归一化、计数器增长与重置/回绕 | UT |642| T6 计数器异常 | Q2/Q3 | Key 缺失、非法值、Device ID 归一化、计数器增长与重置/回绕 | UT |
649 643 
650- 
651### 10.2 UT 接入644### 10.2 UT 接入
652 645 
653当前仓库使用 CMake `BUILD_TESTS` 和 CTest,没有可用的根目录 `build.sh -u/-s` 入口。RFC 以仓库现状为准:646当前仓库使用 CMake `BUILD_TESTS` 和 CTest,没有可用的根目录 `build.sh -u/-s` 入口。RFC 以仓库现状为准:
@@ -36,4 +36,4 @@
36## 相关36## 相关
37 37 
38- [RFC 模板](./0000-template.md)38- [RFC 模板](./0000-template.md)
39-- [RFC 流程说明](./README.md)39+- [RFC 流程说明](./README.md)
@@ -512,7 +512,7 @@ classDiagram
5125. **Null pointer protection**: All reporting interfaces check for non-null pointers before calling the Reporter to avoid null pointer dereferences.5125. **Null pointer protection**: All reporting interfaces check for non-null pointers before calling the Reporter to avoid null pointer dereferences.
5136. **EXCEPTION_CATCH macro**: `ReportKernel` uses the `EXCEPTION_CATCH` macro to catch exceptions during `ProfilingHandler` reporting, returning `HCCL_E_PTR` on failure.5136. **EXCEPTION_CATCH macro**: `ReportKernel` uses the `EXCEPTION_CATCH` macro to catch exceptions during `ProfilingHandler` reporting, returning `HCCL_E_PTR` on failure.
5147. **MC2 Stream group reporting**: `ReportMc2CommInfo` groups every 8 streamIds into a group and reports them through `ProfilingDeviceCommResInfo`. The `commStreamIds` array size is fixed at 8.5147. **MC2 Stream group reporting**: `ReportMc2CommInfo` groups every 8 streamIds into a group and reports them through `ProfilingDeviceCommResInfo`. The `commStreamIds` array size is fixed at 8.
515-8. **Device-side device type restriction**: `HcommProfilingReportDeviceOp`, `HcommProfilingReportKernelStartTask`, and `HcommProfilingReportKernelEndTask` only execute on `DEV_TYPE_950` devices. For other device types, they directly return success.515+8. **Device-side device type restriction**: `HcommProfilingReportDeviceOp`, `HcommProfilingReportKernelStartTask`, and `HcommProfilingReportKernelEndTask` only execute on `DEV_TYPE_950` and `DEV_TYPE_960` devices. For other device types, they directly return success.
516 516 
517### Known Limitations517### Known Limitations
518 518 
@@ -1,6 +1,6 @@
1# topo addr info1# topo addr info
2 2 
3-# 简介3+## 简介
4 4 
5昇腾芯片在950代际中, 超平面使用Unified Bus总线组网,在不同产品形态中使用多种不同的拓扑组网方式, 本模块用于在发现在不同拓扑下每条边的端点地址5昇腾芯片在950代际中, 超平面使用Unified Bus总线组网,在不同产品形态中使用多种不同的拓扑组网方式, 本模块用于在发现在不同拓扑下每条边的端点地址
6 6 
@@ -26,10 +26,9 @@ chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run
26./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend26./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend
27```27```
28 28 
29- 
30### 2.2 hccl_test编译29### 2.2 hccl_test编译
31 30 
32-hccl_test是昇腾官方提供的HCCL性能测试工具,详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html),HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。31+hccl_test是昇腾官方提供的HCCL性能测试工具,详见[HCCL性能测试工具](https://www.hiascend.com/document/redirect/CANNCommunityToolHcclTest),HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。
33 32 
34备注:可选,未来支持Pytorch用例。33备注:可选,未来支持Pytorch用例。
35 34 
@@ -56,6 +55,7 @@ curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/master/test/hccl_vm/hccl_vm_in
56**hccl_test**:默认一并编译 OpenMPI 与 hccl_test 性能测试工具,`--skip-hccl-test` 可关闭。55**hccl_test**:默认一并编译 OpenMPI 与 hccl_test 性能测试工具,`--skip-hccl-test` 可关闭。
57 56 
58**常用参数**57**常用参数**
58+ 
59- `--profile <名>`:配套方案(默认 `main``--list-profiles` 列全部)59- `--profile <名>`:配套方案(默认 `main``--list-profiles` 列全部)
60- `--workspace <路径>`:工作目录,源码/编译/产物所在(默认当前目录)60- `--workspace <路径>`:工作目录,源码/编译/产物所在(默认当前目录)
61- `--ascend-path <路径>`:指定 CANN 目录,有则复用、无则装到此处61- `--ascend-path <路径>`:指定 CANN 目录,有则复用、无则装到此处
@@ -102,44 +102,44 @@ bash build_pkg.sh
102 102 
103#### 3.3.2 CCU模式103#### 3.3.2 CCU模式
104 104 
105-1. 环境变量配置105+1. 配置环境变量。
106 106 
107-```bash107+ ```bash
108-# 进入工具安装目录108+ # 进入工具安装目录
109-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install109+ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install
110-source /home/workspace/Ascend/cann/set_env.sh110+ source /home/workspace/Ascend/cann/set_env.sh
111-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH111+ export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
112-export RANK_TABLE_FILE=$(pwd)/data/ranktable.json112+ export RANK_TABLE_FILE=$(pwd)/data/ranktable.json
113-export HCCL_OP_EXPANSION_MODE="CCU_SCHED"113+ export HCCL_OP_EXPANSION_MODE="CCU_SCHED"
114-```114+ ```
115 115 
116-2. 执行116+2. 执行
117 117 
118+ ```bash
119+ # 需要进入到新的bin文件目录下执行hccl-vm
120+ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
121+
122+ # 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行
123+ ./hccl-vm start ascend950_cluster_32_server_normal.yaml
124+
125+ # 如需启用runner插件(可选)
126+ (hvm)$> hccl-vm plugin install @runner
127+
128+ # 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例)
129+ (hvm)$> hccl-vm mock-comm 112
130+ (hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
131+
132+ # 执行checker校验
133+ (hvm)$> hccl-vm plugin run @checker
134+
135+ # 退出工具终端
136+ (hvm)$> exit
137+ ```
118 138 
119-```bash139+3. 验证hccl_test用例运行结果。
120-# 需要进入到新的bin文件目录下执行hccl-vm
121-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
122 140 
123-# 选择昇腾集群拓扑配置文,启动工具,初始化集群环境,进入工具命令行141+ [Runner结果查看](#491-runner插结果)
124-./hccl-vm start ascend950_cluster_32_server_normal.yaml142+ [Checker结果查看](#492-checker插件结果)
125- 
126-# 如需启用runner插件(可选)
127-(hvm)$> hccl-vm plugin install @runner
128- 
129-# 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例)
130-(hvm)$> hccl-vm mock-comm 112
131-(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
132- 
133-# 执行checker校验
134-(hvm)$> hccl-vm plugin run @checker
135- 
136-# 退出工具终端
137-(hvm)$> exit
138-```
139- 
140-3. 验证hccl_test用例运行结果
141-[Runner结果查看](#491-runner插件结果)
142-[Checker结果查看](#492-checker插件结果)
143 143 
144#### 3.3.3 AICPU模式144#### 3.3.3 AICPU模式
145 145 
@@ -147,37 +147,37 @@ AICPU展开模式需要将算法展开步骤放到设备侧执行,因此hccl-v
147 147 
1481. 环境变量配置。1481. 环境变量配置。
149 149 
150-```bash150+ ```bash
151-# 进入工具安装目录151+ # 进入工具安装目录
152-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install152+ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install
153-source /home/workspace/Ascend/cann/set_env.sh153+ source /home/workspace/Ascend/cann/set_env.sh
154-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH154+ export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
155-export RANK_TABLE_FILE=$(pwd)/data/ranktable.json155+ export RANK_TABLE_FILE=$(pwd)/data/ranktable.json
156-export HCCL_OP_EXPANSION_MODE="AI_CPU"156+ export HCCL_OP_EXPANSION_MODE="AI_CPU"
157-```157+ ```
158 158 
1592. 执行1592. 执行
160 160 
161-```bash161+ ```bash
162-# 需要进入到新的bin文件目录下执行hccl-vm162+ # 需要进入到新的bin文件目录下执行hccl-vm
163-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin163+ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
164- 164+
165-# 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行165+ # 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行
166-./hccl-vm start ascend950_cluster_32_server_normal.yaml166+ ./hccl-vm start ascend950_cluster_32_server_normal.yaml
167- 167+
168-# 如需启用runner插件(可选)168+ # 如需启用runner插件(可选)
169-(hvm)$> hccl-vm plugin install @runner169+ (hvm)$> hccl-vm plugin install @runner
170- 170+
171-# 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例)171+ # 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例)
172-(hvm)$> hccl-vm mock-comm 112172+ (hvm)$> hccl-vm mock-comm 112
173-(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt173+ (hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
174- 174+
175-# 执行checker校验175+ # 执行checker校验
176-(hvm)$> hccl-vm plugin run @checker176+ (hvm)$> hccl-vm plugin run @checker
177- 177+
178-# 退出工具终端178+ # 退出工具终端
179-(hvm)$> exit179+ (hvm)$> exit
180-```180+ ```
181 181 
1823. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)1823. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)
183 183 
@@ -185,37 +185,37 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
185 185 
1861. 环境变量配置。1861. 环境变量配置。
187 187 
188-```bash188+ ```bash
189-# 进入工具安装目录189+ # 进入工具安装目录
190-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install190+ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install
191-source /home/workspace/Ascend/cann/set_env.sh191+ source /home/workspace/Ascend/cann/set_env.sh
192-export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH192+ export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH
193-export RANK_TABLE_FILE=$(pwd)/data/ranktable.json193+ export RANK_TABLE_FILE=$(pwd)/data/ranktable.json
194-export HCCL_OP_EXPANSION_MODE="AIV"194+ export HCCL_OP_EXPANSION_MODE="AIV"
195-```195+ ```
196 196 
1972. 执行1972. 执行
198 198 
199-```bash199+ ```bash
200-# 需要进入到新的bin文件目录下执行hccl-vm200+ # 需要进入到新的bin文件目录下执行hccl-vm
201-cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin201+ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin
202- 202+
203-# 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行203+ # 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行
204-./hccl-vm start ascend950_cluster_32_server_normal.yaml204+ ./hccl-vm start ascend950_cluster_32_server_normal.yaml
205- 205+
206-# 如需启用runner插件(可选)206+ # 如需启用runner插件(可选)
207-(hvm)$> hccl-vm plugin install @runner207+ (hvm)$> hccl-vm plugin install @runner
208- 208+
209-# 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例)209+ # 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例)
210-(hvm)$> hccl-vm mock-comm 112210+ (hvm)$> hccl-vm mock-comm 112
211-(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt211+ (hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt
212- 212+
213-# 执行checker校验213+ # 执行checker校验
214-(hvm)$> hccl-vm plugin run @checker214+ (hvm)$> hccl-vm plugin run @checker
215- 215+
216-# 退出工具终端216+ # 退出工具终端
217-(hvm)$> exit217+ (hvm)$> exit
218-```218+ ```
219 219 
2203. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)2203. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果)
221 221 
@@ -263,31 +263,31 @@ hccl_test用例源码在CANN包安装目录下,支持OpenMPI和MPICH两种环
263 263 
2641. 安装OpenMPI2641. 安装OpenMPI
265 265 
266-```bash266+ ```bash
267-sudo apt-get update267+ sudo apt-get update
268-sudo apt install openmpi-bin libopenmpi-dev268+ sudo apt install openmpi-bin libopenmpi-dev
269-```269+ ```
270 270 
2712. 编译hccl_test2712. 编译hccl_test
272 272 
273-```bash273+ ```bash
274-# 修改CANN安装目录权限274+ # 修改CANN安装目录权限
275-chmod -R 755 /home/workspace/Ascend275+ chmod -R 755 /home/workspace/Ascend
276- 276+
277-# 进入hccl_test用例源码目录277+ # 进入hccl_test用例源码目录
278-cd /home/workspace/Ascend/cann/tools/hccl_test278+ cd /home/workspace/Ascend/cann/tools/hccl_test
279- 279+
280-# 设置CANN环境变量280+ # 设置CANN环境变量
281-source /home/workspace/Ascend/cann/set_env.sh281+ source /home/workspace/Ascend/cann/set_env.sh
282- 282+
283-# 临时修改Makefile脚本283+ # 临时修改Makefile脚本
284-if ! grep -q '\-lmpi_cxx' Makefile; then284+ if ! grep -q '\-lmpi_cxx' Makefile; then
285- sed -i 's/-lmpi/-lmpi -lmpi_cxx/g' Makefile285+ sed -i 's/-lmpi/-lmpi -lmpi_cxx/g' Makefile
286-fi286+ fi
287- 287+
288-# 编译hccl_test用例288+ # 编译hccl_test用例
289-MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH}289+ MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH}
290-```290+ ```
291 291 
292#### 4.2.2 MPICH环境编译292#### 4.2.2 MPICH环境编译
293 293 
@@ -369,8 +369,8 @@ links:
369 - **port_group**: 描述哪些port合并为一个portGroup,相同portGroup的port对应IP地址相同。没有配置的port,则默认每个port为一个portGroup。369 - **port_group**: 描述哪些port合并为一个portGroup,相同portGroup的port对应IP地址相同。没有配置的port,则默认每个port为一个portGroup。
370 - **links**: 链路配置表,描述Server/Pod内的所有NPU卡之间,以及NPU与交换机之间的连接关系。370 - **links**: 链路配置表,描述Server/Pod内的所有NPU卡之间,以及NPU与交换机之间的连接关系。
371 - **NPU直连关系**: 工具提供了两种方式配置NPU直连关系:371 - **NPU直连关系**: 工具提供了两种方式配置NPU直连关系:
372- - **link_mode == "fullmesh"**: 表示所有Device基于一个Die的Port进行全连接。后续有新增典型的连接方式,可以新增link_mode类型,如"ring"。372+ - **link_mode == "fullmesh"**: 表示所有Device基于一个Die的Port进行全连接。后续有新增典型的连接方式,可以新增link_mode类型,如"ring"。
373- - **link_mode == "enum"**: 枚举法。当Server/Pod内的NPU连接方式比较复杂时,可以通过枚举所有的链路关系来描述。373+ - **link_mode == "enum"**: 枚举法。当Server/Pod内的NPU连接方式比较复杂时,可以通过枚举所有的链路关系来描述。
374 - **NPU与交换机连接关系**: 用户可以通过枚举法配置NPU与交换机之间的连接关系。374 - **NPU与交换机连接关系**: 用户可以通过枚举法配置NPU与交换机之间的连接关系。
375 - **device_to_device_links**: 描述NPU与NPU之间的连接关系。375 - **device_to_device_links**: 描述NPU与NPU之间的连接关系。
376 - **device_to_switch_links**: 描述NPU与交换机之间的连接关系。376 - **device_to_switch_links**: 描述NPU与交换机之间的连接关系。
@@ -770,6 +770,7 @@ data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result:
770```770```
771 771 
772---772---
773+ 
773### 4.10 大块内存复用(仅校验模式)774### 4.10 大块内存复用(仅校验模式)
774 775 
775仅校验模式用于大规模集群仅运行 Checker 校验的场景。开启后,单块 200MB 到 4GB 的大内存申请复用同一块 4GB 共享区 `HcclCommPool`,各 rank 共享、允许互相覆盖,以此大幅降低 `/dev/shm` 占用。此时大块内容不保证正确,仅适用于不读取缓冲区数据的 Checker V3 校验链路,需要数值正确的结果时请勿开启。776仅校验模式用于大规模集群仅运行 Checker 校验的场景。开启后,单块 200MB 到 4GB 的大内存申请复用同一块 4GB 共享区 `HcclCommPool`,各 rank 共享、允许互相覆盖,以此大幅降低 `/dev/shm` 占用。此时大块内容不保证正确,仅适用于不读取缓冲区数据的 Checker V3 校验链路,需要数值正确的结果时请勿开启。
@@ -782,7 +783,7 @@ data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result:
782./hccl-vm start ascend950_cluster_32_server_normal.yaml --check-only783./hccl-vm start ascend950_cluster_32_server_normal.yaml --check-only
783```784```
784 785 
785-***786+---
786 787 
787## 5 附录788## 5 附录
788 789