已合并
[modify docs]修正文档中英文不一致性问题 #4700
zangyan创建于 17 天前
[modify docs]修正文档中英文不一致性问题 #4700
已合并
共 52 个文件变更+12134-572
| @@ -1,6 +1,6 @@ | |||
| 1 | # HCOMM Agent Rules | 1 | # HCOMM Agent Rules |
| 2 | 2 | ||
| 3 | -This file serves as the main entry point for AI Agent governance of the HCOMM repository. It is intended for AI programming tools that support the AGENTS.md standard. If a subdirectory has its own `AGENTS.md`, that file supplements this one. In case of conflicts, follow the user's explicit requirements first, and then the rules in the nearest subdirectory. | 3 | +This file serves as the main entry point for AI Agent governance of the HCOMM repository. It is intended for AI programming tools that support the AGENTS.md standard. If a subdirectory has its own `AGENTS.md`, the nearer file supplements this one. In case of conflicts, follow the user's explicit requirements first, and then the rules in the nearest subdirectory. |
| 4 | 4 | ||
| 5 | > This file provides only the **essential hard constraints and entry points**. Detailed content is progressively disclosed through links. Before modifying code, read the architecture constraints in Section 3. | 5 | > This file provides only the **essential hard constraints and entry points**. Detailed content is progressively disclosed through links. Before modifying code, read the architecture constraints in Section 3. |
| 6 | 6 | ||
| @@ -52,9 +52,9 @@ Dependency direction is top-down: `coll_comm_ops` (hccl) → `coll_communicator_ | |||
| 52 | | Layer | Header Files | Audience | | 52 | | Layer | Header Files | Audience | |
| 53 | |------|------|------| | 53 | |------|------|------| |
| 54 | | L2-comm | `include/hccl/hccl_comm.h` | AI framework layer (communication domain creation) | | 54 | | L2-comm | `include/hccl/hccl_comm.h` | AI framework layer (communication domain creation) | |
| 55 | -| L2-res-rank_graph | `include/hccl/hccl_res.h`, `include/hccl/hccl_rank_graph.h` | Operator developers (topology query, resource acquisition) | | 55 | +| L2-res-rank_graph | `include/hccl/hccl_res.h`, `include/hccl/hccl_channel.h`, `include/hccl/hccl_rank_graph.h` | Operator developers (topology query, resource acquisition) | |
| 56 | | L3-prim | `include/hcomm_primitives.h` | Operator or communication library developers (data transfer and synchronization) | | 56 | | L3-prim | `include/hcomm_primitives.h` | Operator or communication library developers (data transfer and synchronization) | |
| 57 | -| L3-res | `include/hcomm_res.h`, `include/hcomm_res_defs.h` | Communication library developers (device, channel, and memory resources) | | 57 | +| L3-res | `include/hcomm_res.h`, `include/hcomm_res_defs.h`, `include/hcomm_channel.h` | Communication library developers (device, channel, and memory resources) | |
| 58 | | CCU | `include/ccu/` (`ccu_primitives.hpp`, `ccu_res.h`, `ccu_launch.h`) | CCU operator developers | | 58 | | CCU | `include/ccu/` (`ccu_primitives.hpp`, `ccu_res.h`, `ccu_launch.h`) | CCU operator developers | |
| 59 | 59 | ||
| 60 | Changes to `include/` must be backward compatible. `pkg_inc/` is for inter-package use between HCOMM, HCCL, GE, and so on, and is not externally stable. For the complete API layer relationships, see [Section 3.3 of the Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md). | 60 | Changes to `include/` must be backward compatible. `pkg_inc/` is for inter-package use between HCOMM, HCCL, GE, and so on, and is not externally stable. For the complete API layer relationships, see [Section 3.3 of the Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md). |
| @@ -77,7 +77,7 @@ Output: `build_out/cann-hcomm_<version>_linux-<arch>.run`. For a complete list o | |||
| 77 | - Naming: Classes and functions use PascalCase; member variables use `camelCase_` (lower camelCase with a trailing underscore); constants and macros use `UPPER_SNAKE_CASE`. | 77 | - Naming: Classes and functions use PascalCase; member variables use `camelCase_` (lower camelCase with a trailing underscore); constants and macros use `UPPER_SNAKE_CASE`. |
| 78 | - Style: Follow the `.clang-format` in the root directory (120 columns, 4 spaces, pointer right-aligned, K&R braces). Use C++14. | 78 | - Style: Follow the `.clang-format` in the root directory (120 columns, 4 spaces, pointer right-aligned, K&R braces). Use C++14. |
| 79 | - Static warnings: Code must pass CANN static check requirements (verified during the CI codecheck stage) and compile without warnings. | 79 | - Static warnings: Code must pass CANN static check requirements (verified during the CI codecheck stage) and compile without warnings. |
| 80 | -- pre-commit: clang-format v16 + OAT compliance check. New source files must include the CANN-2.0 license header. | 80 | +- pre-commit: clang-format v18.1.8 + OAT compliance check. New source files must include the CANN-2.0 license header. |
| 81 | 81 | ||
| 82 | References: [CANN Coding Standards](https://gitcode.com/cann/community/tree/master/contributor/coding-standards), [CANN CI Guide](https://gitcode.com/cann/community/blob/master/contributor/repository/ci-guide.md), [pre-commit Guide](./docs/en/build/pre-commit-guide.md), `.clang-format`, `OAT.xml`. | 82 | References: [CANN Coding Standards](https://gitcode.com/cann/community/tree/master/contributor/coding-standards), [CANN CI Guide](https://gitcode.com/cann/community/blob/master/contributor/repository/ci-guide.md), [pre-commit Guide](./docs/en/build/pre-commit-guide.md), `.clang-format`, `OAT.xml`. |
| 83 | 83 | ||
| @@ -2,11 +2,11 @@ | |||
| 2 | 2 | ||
| 3 | ## Recommended User Account | 3 | ## Recommended User Account |
| 4 | 4 | ||
| 5 | -For security reasons, avoid using administrator accounts such as root to execute commands. Follow the principle of least privilege. | 5 | +For security reasons, avoid using administrator accounts such as root to execute any commands. Follow the principle of least privilege. |
| 6 | 6 | ||
| 7 | ## File Permission Control | 7 | ## File Permission Control |
| 8 | 8 | ||
| 9 | -- It is recommended to set the umask value to 0027 or higher on the host machine (including the host machine and containers) to ensure that the default maximum permissions for new folders are 750 and for new files are 640. | 9 | +- It is recommended to set the umask value to 0027 or higher on the host (including the host machine and containers) to ensure that the default maximum permissions for new folders are 750 and for new files are 640. |
| 10 | - Apply security measures such as permission control to sensitive content, including personal private data, business assets, and source files. For example, control the permissions of the installation directory and input public data files. For recommended permission settings, refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios). | 10 | - Apply security measures such as permission control to sensitive content, including personal private data, business assets, and source files. For example, control the permissions of the installation directory and input public data files. For recommended permission settings, refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios). |
| 11 | - Control permissions during installation and usage. Refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios) for file permission references. | 11 | - Control permissions during installation and usage. Refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios) for file permission references. |
| 12 | 12 | ||
| @@ -52,7 +52,7 @@ The software dependencies required for compiling this project are listed below. | |||
| 52 | ``` | 52 | ``` |
| 53 | 53 | ||
| 54 | - \$\{cann\_version\}: Indicates the CANN software package version number. | 54 | - \$\{cann\_version\}: Indicates the CANN software package version number. |
| 55 | - - \$\{arch\}$: Indicates the CPU architecture, for example, aarch64 or x86_64. | 55 | + - \$\{arch\}: Indicates the CPU architecture, for example, aarch64 or x86_64. |
| 56 | - \$\{soc\_name\}: Indicates the NPU model name. | 56 | - \$\{soc\_name\}: Indicates the NPU model name. |
| 57 | - \$\{install\_path\}: Indicates the specified installation path. The CANN ops operator package must be installed in the same path as the CANN Toolkit development kit package. The default installation path for the root user is `/usr/local/Ascend`. | 57 | - \$\{install\_path\}: Indicates the specified installation path. The CANN ops operator package must be installed in the same path as the CANN Toolkit development kit package. The default installation path for the root user is `/usr/local/Ascend`. |
| 58 | 58 | ||
| @@ -158,7 +158,7 @@ bash build.sh --ut | |||
| 158 | 158 | ||
| 159 | ### On-Board Testing | 159 | ### On-Board Testing |
| 160 | 160 | ||
| 161 | -> **Note** | 161 | +> [!Note]NOTE |
| 162 | > Before on-board testing, ensure that the driver, firmware, CANN Toolkit development kit package, and CANN ops operator package are installed. | 162 | > Before on-board testing, ensure that the driver, firmware, CANN Toolkit development kit package, and CANN ops operator package are installed. |
| 163 | 163 | ||
| 164 | Developers can use the HCCL Test tool for collective communication function and performance testing on the board. The workflow for using the HCCL Test tool is as follows: | 164 | Developers can use the HCCL Test tool for collective communication function and performance testing on the board. The workflow for using the HCCL Test tool is as follows: |
| @@ -13,7 +13,7 @@ pre-commit is a Git Hooks framework that automatically runs code checking and fo | |||
| 13 | 13 | ||
| 14 | - **Git**: 2.0+ | 14 | - **Git**: 2.0+ |
| 15 | - **Python**: 3.8+ | 15 | - **Python**: 3.8+ |
| 16 | -- **clang-format**: 14.0+ (code formatting tool) | 16 | +- **clang-format**: v18.1.8 (code formatting tool, must match the `rev` in [.pre-commit-config.yaml](../../../.pre-commit-config.yaml)) |
| 17 | - **Java**: 17+ (required for OAT tool, can be installed automatically) | 17 | - **Java**: 17+ (required for OAT tool, can be installed automatically) |
| 18 | - **Maven**: 3.6+ (required for OAT tool, can be installed automatically) | 18 | - **Maven**: 3.6+ (required for OAT tool, can be installed automatically) |
| 19 | 19 | ||
| @@ -1,24 +1,24 @@ | |||
| 1 | -# EI0002 Communication_Error_Timeout | 1 | +# EI0002 Communication_Error_Timeout |
| 2 | 2 | ||
| 3 | ## Symptom | 3 | ## Symptom |
| 4 | 4 | ||
| 5 | The following is error format. The meanings of the placeholders %s in sequence are: rank id, task information, communication operator information, communicator information. | 5 | The following is error format. The meanings of the placeholders %s in sequence are: rank id, task information, communication operator information, communicator information. |
| 6 | 6 | ||
| 7 | ```text | 7 | ```text |
| 8 | -An timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s. | 8 | +A timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s. |
| 9 | ``` | 9 | ``` |
| 10 | 10 | ||
| 11 | Error example: | 11 | Error example: |
| 12 | 12 | ||
| 13 | ```text | 13 | ```text |
| 14 | -An timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none. | 14 | +A timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none. |
| 15 | ``` | 15 | ``` |
| 16 | 16 | ||
| 17 | ## Possible Cause | 17 | ## Possible Cause |
| 18 | 18 | ||
| 19 | 1. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed. | 19 | 1. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed. |
| 20 | 20 | ||
| 21 | -2. The execution speed on some NPU in the cluster is too slow to complete a communication operation within the timeout interval. \(The default timeout interval is 1800s, You can set the interval by using HCCL_EXEC_TIMEOUT.\) | 21 | +2. The execution speed on some NPUs in the cluster is too slow to complete a communication operation within the timeout interval. \(The default timeout interval is 1800s. You can set the interval by using HCCL_EXEC_TIMEOUT.\) |
| 22 | 22 | ||
| 23 | 3. The number of training samples of each NPU is inconsistent. | 23 | 3. The number of training samples of each NPU is inconsistent. |
| 24 | 24 | ||
| @@ -22,4 +22,4 @@ The Toolkit (cann-hcomm) and ops (cann-hccl) versions are inconsistent. The loca | |||
| 22 | 22 | ||
| 23 | ## Solution | 23 | ## Solution |
| 24 | 24 | ||
| 25 | -Install the same version. The local Toolkit \(cann-hcomm\) and ops \(cann-hccl\) versions must be consistent with the corresponding remote versions. | 25 | +Install the same version. The local Toolkit (cann-hcomm) and ops (cann-hccl) versions must be consistent with the corresponding remote versions. |
| @@ -16,4 +16,4 @@ P2P communication failed. Reason: Device ID 0 in module 0 and device ID 9 in mod | |||
| 16 | 16 | ||
| 17 | ## Solution | 17 | ## Solution |
| 18 | 18 | ||
| 19 | -Ensure that the NPU card is normal and entering environment variables 'export HCCL_INTRA_ROCE_ENABLE=1'. | 19 | +Ensure that the NPU card is normal and enter environment variable `export HCCL_INTRA_ROCE_ENABLE=1`. |
| @@ -11,7 +11,7 @@ Value %s for config %s is invalid. Expected value: %s. | |||
| 11 | Error example: | 11 | Error example: |
| 12 | 12 | ||
| 13 | ```text | 13 | ```text |
| 14 | -Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A." . | 14 | +Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A". |
| 15 | ``` | 15 | ``` |
| 16 | 16 | ||
| 17 | ## Solution | 17 | ## Solution |
| @@ -28,4 +28,4 @@ An error CQE occurred during operator execution. Local information: server az0-r | |||
| 28 | 28 | ||
| 29 | 2. Check whether the peer process exits first. If yes, check the reason why the process exit. | 29 | 2. Check whether the peer process exits first. If yes, check the reason why the process exit. |
| 30 | 30 | ||
| 31 | -3. Use the RAS fault check mechanism to check whether the hardware of the HBM or UB chip processing module of either device is abnormal.If the hardware is abnormal, contact Huawei technical support. | 31 | +3. Use the RAS fault check mechanism to check whether the hardware of the HBM or UB chip processing module of either device is abnormal. If the hardware is abnormal, contact Huawei technical support. |
| @@ -16,6 +16,6 @@ Failed to enable listening for the host network adapter socket. Reason: The IP a | |||
| 16 | 16 | ||
| 17 | ## Solution | 17 | ## Solution |
| 18 | 18 | ||
| 19 | -1. Check whether this port has been occupied by another process. If yes, you can make adjustment using the environment variable HCCL_IF_BASE_PORT and use sysctl -w net.ipv4.ip_local_reserved_ports=\*\*\*\*-\*\*\*\* to adjust the scope of reserved ports. | 19 | +1. Check whether this port has been occupied by another process. If yes, you can make adjustment using the environment variable HCCL_IF_BASE_PORT and use `sysctl -w net.ipv4.ip_local_reserved_ports=****-****` to adjust the scope of reserved ports. |
| 20 | 20 | ||
| 21 | 2. Check whether the service process is started multiple times on a device during this service. | 21 | 2. Check whether the service process is started multiple times on a device during this service. |
| @@ -59,6 +59,7 @@ Which interfaces of hcomm / hccl / other CANN components this depends on. | |||
| 59 | This section covers only module-internal implementation. Organize by module; for cross-layer features, split into separate module subsections per layer, do not mix them. | 59 | This section covers only module-internal implementation. Organize by module; for cross-layer features, split into separate module subsections per layer, do not mix them. |
| 60 | 60 | ||
| 61 | Each module subsection includes: | 61 | Each module subsection includes: |
| 62 | + | ||
| 62 | - Module responsibility (one sentence) | 63 | - Module responsibility (one sentence) |
| 63 | - Core data structures (UML may be used) | 64 | - Core data structures (UML may be used) |
| 64 | - Key logic and algorithms | 65 | - Key logic and algorithms |
| @@ -0,0 +1,690 @@ | |||
| 1 | +# RFC: Topology-Based Cluster Communication Rapid Sub-Health Monitoring | ||
| 2 | + | ||
| 3 | +- Start date: 2026-06-01 | ||
| 4 | +- Revision dates: 2026-07-13, 2026-08-10 | ||
| 5 | +- RFC PR number: 2491 | ||
| 6 | +- Related issues: 249, 261 | ||
| 7 | +- Status: accept | ||
| 8 | + | ||
| 9 | +## Summary | ||
| 10 | + | ||
| 11 | +This RFC proposes a "layered, domain-based minimum-cost network probing + link-level fault localization" scheme for RoCE lossless clusters. The system identifies relationships among devices, hosts, ToR/Leaf switches, and upper-level network domains through topology discovery. It uses solvable ring-path sets within each domain and constructs probing tasks between adjacent domains on demand. The system then combines HCCN PingPong path latency and pass rates with RNIC PFC and CNP counters to produce L1 link metrics, L2 path metrics, and abnormal link candidates. | ||
| 12 | + | ||
| 13 | +This document defines the target design and user contract for this feature. Items that have not yet been validated in production or aligned with the implementation are explicitly marked as "release gate" or "to be confirmed" and are not considered capabilities of the current version. | ||
| 14 | + | ||
| 15 | +## 1. Background and Motivation | ||
| 16 | + | ||
| 17 | +### 1.1 Problem | ||
| 18 | + | ||
| 19 | +A stable, high-speed network is the foundation of HCCL performance. Link jitter, congestion, intra-host bottlenecks, and configuration deviations cause long-tail latency in collective communication, training job slowdowns, and even interruptions. Existing monitoring approaches have the following limitations: | ||
| 20 | + | ||
| 21 | +1. Full PingMesh probing scales quadratically with cluster size, resulting in high overhead. | ||
| 22 | +2. Only end-to-end latency or packet loss is observed, making it impossible to attribute anomalies to specific access links. | ||
| 23 | +3. Static thresholds do not adapt to different scales, layers, and load phases, leading to high false-positive rates. | ||
| 24 | +4. A single control node collects data serially, which easily creates a control-plane bottleneck in large-scale clusters. | ||
| 25 | + | ||
| 26 | +### 1.2 Four Problem Categories to Address | ||
| 27 | + | ||
| 28 | +| ID | Problem | Approach in This Scheme | | ||
| 29 | +| --- | --- | --- | | ||
| 30 | +| Q1 | Complex topology awareness | Discover observable layers and local meshes of Spine-Leaf, HammingMesh, and 3D-Torus topologies based on Tracert and LLDP | | ||
| 31 | +| Q2 | Network metric collection | Generate a solvable minimum probing set; collect P90/P99/Mean latency, pass rates, and PFC/CNP counters | | ||
| 32 | +| Q3 | Anomaly analysis and localization | Solve path-link equations for L1; output cross-domain path metrics after subtracting access link metrics for L2; apply time-series anomaly detection | | ||
| 33 | +| Q4 | Large-scale adaptability | Use layered, domain-based design with parallel host execution and control-plane aggregation to avoid global full-mesh probing | | ||
| 34 | + | ||
| 35 | +### 1.3 Use Cases | ||
| 36 | + | ||
| 37 | +- During large-scale distributed training, assist in locating latency jitter, throughput drops, and communication long-tail issues in collective communication operations such as AllReduce and RingAllReduce. | ||
| 38 | +- In RoCE networks, detect link flapping, congestion, switch or NPU packet loss, and hash collisions. | ||
| 39 | +- Perform network baseline checks before training jobs start, or run continuous low-frequency inspections during job execution. | ||
| 40 | + | ||
| 41 | +### 1.4 Non-Goals | ||
| 42 | + | ||
| 43 | +- Do not replace switch telemetry, NMS, or alerting platforms. | ||
| 44 | +- Do not directly modify the HCCL/HCOMM data plane, QP creation process, or communication topology of training jobs. | ||
| 45 | +- The current version does not guarantee unique localization of all L2 anomalies to a single physical switch link. L2 output provides path-level or communication-pair-level candidates first. | ||
| 46 | + | ||
| 47 | +## 2. Glossary | ||
| 48 | + | ||
| 49 | +| Term | Definition | | ||
| 50 | +| --- | --- | | ||
| 51 | +| Device / NPU | An accelerator that participates in HCCL communication and has HCCN/RNIC ports | | ||
| 52 | +| Host | A server node that manages one or more devices and runs `rpc_host` | | ||
| 53 | +| Controller | A central control node that runs `probe_topo`, `probe_controller`, and deployment scripts | | ||
| 54 | +| Leaf / ToR | The top-of-rack switch to which devices connect; this document treats both as the same access-layer concept | | ||
| 55 | +| L1 link | The access link from a device/RNIC to its directly connected Leaf/ToR; the current implementation can resolve this to link level through path equations | | ||
| 56 | +| L2 path | The network segment between two different Leaf/ToR domains. The current implementation computes "end-to-end P99 latency minus the resolved L1 values at both ends" and does not correspond to a single physical link | | ||
| 57 | +| Mesh | A group of nodes or sub-domains that share a parent network structure at the same network layer and can be modeled with a local path set | | ||
| 58 | +| Probe path | A Tracert or PingPong path from a source device to a destination device using a specified source port | | ||
| 59 | +| PingList | The set of PingPong tasks that the controller distributes to each source device; each element contains a source IP, destination IP, and source port | | ||
| 60 | +| Sub-health | A state in which the network remains connected but latency, pass rates, or congestion counters persistently deviate from the same-layer baseline and have affected or may affect service performance | | ||
| 61 | +| Slow fault | A fault that does not manifest as a complete outage but instead shows persistent or intermittent high latency, low throughput, retransmissions, or congestion | | ||
| 62 | +| Long tail | A condition in which high-percentile latency such as P99 is significantly higher than the mean or median, causing synchronous collective communication to be slowed by the slowest path | | ||
| 63 | +| 3σ | An anomaly threshold constructed as `μ + 3σ` using the mean `μ` and standard deviation `σ` of a time-domain or spatial-domain baseline | | ||
| 64 | +| Turn | One complete cycle of PingPong sampling, metric reduction, link solving, and anomaly detection | | ||
| 65 | + | ||
| 66 | +## 3. Overall Architecture and Data Flow | ||
| 67 | + | ||
| 68 | +### 3.1 Two-Layer Architecture | ||
| 69 | + | ||
| 70 | + | ||
| 71 | + | ||
| 72 | +The system consists of two layers: | ||
| 73 | + | ||
| 74 | +- Controller layer: Parses configurations, discovers topology, generates the PingList, distributes tasks in parallel, aggregates results, solves link metrics, and produces artifacts. | ||
| 75 | +- Host layer: Runs `rpc_host`, calls `hccn_tool` and `hccn_rping` to execute device-side probing, collects RNIC counters, and returns results through RPC. | ||
| 76 | + | ||
| 77 | +The module relationships are summarized as follows: | ||
| 78 | + | ||
| 79 | +```mermaid | ||
| 80 | +flowchart LR | ||
| 81 | + User[Configuration and CLI] --> Controller[probe_topo / probe_controller] | ||
| 82 | + Controller --> Topology[Topology Modeling and Probe Plan] | ||
| 83 | + Controller <-->|RPC| Host[rpc_host] | ||
| 84 | + Host --> HccnTool[hccn_tool: IP / LLDP / Tracert / Stat] | ||
| 85 | + Host --> Rping[hccn_rping: AddTarget / BatchPing] | ||
| 86 | + Controller --> Solver[Metric Reduction and Link Solving] | ||
| 87 | + Solver --> Analyse[Anomaly Analysis] | ||
| 88 | + Analyse --> Output[JSON / TXT / Charts] | ||
| 89 | +``` | ||
| 90 | + | ||
| 91 | +### 3.2 Topology Discovery Flow | ||
| 92 | + | ||
| 93 | +```mermaid | ||
| 94 | +flowchart LR | ||
| 95 | + Config[probe.scope / probe_scope] --> Controller[Controller] | ||
| 96 | + Controller -->|Device list| Host[rpc_host] | ||
| 97 | + Host -->|hccn_tool -ip/-lldp/-traceroute| Device[Device/RNIC] | ||
| 98 | + Device -->|IP, ToR management IP, multi-hop paths| Host | ||
| 99 | + Host -->|RPC results| Controller | ||
| 100 | + Controller -->|Layering, domain partitioning, deduplication| Topology[MeshTopo] | ||
| 101 | + Topology --> Files[probe_topo.json / probe_topo_lldp.json / allpath.json] | ||
| 102 | +``` | ||
| 103 | + | ||
| 104 | +### 3.3 Network Probing Flow | ||
| 105 | + | ||
| 106 | +```mermaid | ||
| 107 | +flowchart LR | ||
| 108 | + Topology[MeshTopo] --> Plan[Generate L1 ring tasks and L2 adjacent-domain tasks] | ||
| 109 | + Plan -->|PingList| Controller[Controller] | ||
| 110 | + Controller -->|RPC: AddTarget / BatchPing| Hosts[Multiple rpc_hosts] | ||
| 111 | + Hosts -->|hccn_rping| Devices[Multiple devices] | ||
| 112 | + Devices -->|P90/P99/Mean/Pass| Hosts | ||
| 113 | + Hosts -->|Batch results| Controller | ||
| 114 | +``` | ||
| 115 | + | ||
| 116 | +### 3.4 Anomaly Localization Flow | ||
| 117 | + | ||
| 118 | +```mermaid | ||
| 119 | +flowchart LR | ||
| 120 | + Raw[PingPong raw results] --> Reduce[Metric reduction] | ||
| 121 | + Reduce --> L1[Path-L1 link equation solving] | ||
| 122 | + Reduce --> L2[End-to-end metrics minus L1 at both ends] | ||
| 123 | + Counter[PFC/CNP counters] --> Analyse[Joint analysis] | ||
| 124 | + L1 --> Analyse | ||
| 125 | + L2 --> Analyse | ||
| 126 | + Analyse --> Artifacts[link_*.txt / l2_status / metrics / bad_link*.txt] | ||
| 127 | +``` | ||
| 128 | + | ||
| 129 | +The data direction is uniformly `Device → Host → Controller → Artifacts`; control commands and task distribution flow in the opposite direction. | ||
| 130 | + | ||
| 131 | +## 4. Interface Design (User-Facing) | ||
| 132 | + | ||
| 133 | +### 4.1 Build and Environment | ||
| 134 | + | ||
| 135 | +```bash | ||
| 136 | +cd <repo_dir> | ||
| 137 | + | ||
| 138 | +export THIRDLIB_ROOT=/usr/local/third_lib | ||
| 139 | +export ASCEND_HOME_PATH=/usr/local/Ascend | ||
| 140 | +export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux | ||
| 141 | +source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" | ||
| 142 | + | ||
| 143 | +cmake -S . -B build \ | ||
| 144 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | ||
| 145 | + -DCMAKE_BUILD_TYPE=Release | ||
| 146 | +cmake --build build -j"$(nproc)" | ||
| 147 | +``` | ||
| 148 | + | ||
| 149 | +### 4.2 Configuration File | ||
| 150 | + | ||
| 151 | +The user uses a JSON file to describe the deployment topology, probe scope, and runtime parameters. `schema_version=2` is the current recommended format; the default `schema_version` or `schema_version=1` remains backward compatible with the legacy format. | ||
| 152 | + | ||
| 153 | +```json | ||
| 154 | +{ | ||
| 155 | + "schema_version": 2, | ||
| 156 | + "controller": "node-01", | ||
| 157 | + "deploy": { | ||
| 158 | + "default_ssh_port": 22, | ||
| 159 | + "default_timeout": 5, | ||
| 160 | + "to_path": "~/disp_probe", | ||
| 161 | + "from_path": "." | ||
| 162 | + }, | ||
| 163 | + "hosts": [ | ||
| 164 | + { | ||
| 165 | + "id": "node-01", | ||
| 166 | + "ip": "10.90.15.67", | ||
| 167 | + "user": "root", | ||
| 168 | + "ssh_key": "/root/.ssh/id_ed25519" | ||
| 169 | + }, | ||
| 170 | + { | ||
| 171 | + "id": "node-02", | ||
| 172 | + "ip": "10.90.15.69", | ||
| 173 | + "user": "root", | ||
| 174 | + "password_env": "NODE_02_PASS" | ||
| 175 | + } | ||
| 176 | + ], | ||
| 177 | + "probe": { | ||
| 178 | + "scope": { | ||
| 179 | + "node-01": {"device_range": [0, 3]}, | ||
| 180 | + "node-02": {"devices": [4, 5, 6, 7]} | ||
| 181 | + }, | ||
| 182 | + "topology": { | ||
| 183 | + "sport_begin": 49152, | ||
| 184 | + "sport_count": 32, | ||
| 185 | + "tree_probe_sport_count": 1, | ||
| 186 | + "topology_optimized": true, | ||
| 187 | + "l2_path_aware": true, | ||
| 188 | + "output_subdir": "", | ||
| 189 | + "allpath_output": "allpath.json", | ||
| 190 | + "l2_path_output": "l2_fullmesh_path.json" | ||
| 191 | + }, | ||
| 192 | + "pingpong": { | ||
| 193 | + "times": 50, | ||
| 194 | + "turns": 1000, | ||
| 195 | + "payload_len": 12, | ||
| 196 | + "interval_ms": 1 | ||
| 197 | + } | ||
| 198 | + } | ||
| 199 | +} | ||
| 200 | +``` | ||
| 201 | + | ||
| 202 | +| Field | Type/Default | Description and Constraints | | ||
| 203 | +| --- | --- | --- | | ||
| 204 | +| `schema_version` | int / `1` | Configuration format version; defaults to v1 if absent; `2` enables the current recommended format | | ||
| 205 | +| `controller` | string / required (v2) | The controller node host ID; must reference `hosts[].id` | | ||
| 206 | +| `hosts` | array / required (v2) | Host inventory; `id` is a stable identifier and `ip` is the connection address; both must be unique | | ||
| 207 | +| `hosts[].user` | string / `root` | SSH login user | | ||
| 208 | +| `hosts[].ssh_key` | string / nullable | SSH private key path; backward compatible with the legacy name `key_filename`. When configured together with `password_env`, the SSH connection receives the private key path and attempts key/agent/local key authentication first | | ||
| 209 | +| `hosts[].password_env` | string / nullable | Reads the SSH password from an environment variable to avoid writing plaintext in the configuration. When configured together with `ssh_key`, this password is passed to SSH for password authentication or as an encrypted private key passphrase; it also serves as the `su` fallback password when `su_password` is not configured | | ||
| 210 | +| `deploy.default_ssh_port` | int / `22` | SSH port | | ||
| 211 | +| `deploy.default_timeout` | int / `5` | Dispatcher SSH connection timeout in seconds | | ||
| 212 | +| `deploy.control_topo` | array/object / nullable | The host tree for distribution and remote execution; in v2, host IDs can be used and are resolved to IPs during parsing | | ||
| 213 | +| `deploy.to_path` | string / required | Remote deployment root directory; `~` is expanded for the control user | | ||
| 214 | +| `deploy.from_path` | string / `"."` | Local source path; used as the default local sync source directory for the Dispatcher | | ||
| 215 | +| `probe.scope` | object / required (v2) | The unique probe scope; keys are host IDs; values can be `{"device_range":[begin,end]}` or `{"devices":[...]}` | | ||
| 216 | +| `probe.topology.sport_begin` | positive int / `49152` | Optional; Tracert source port start value | | ||
| 217 | +| `probe.topology.sport_count` | positive int / `1` | Optional; number of source ports per directed pair for cross-domain multi-path coverage | | ||
| 218 | +| `probe.topology.tree_probe_sport_count` | positive int / `1` | Optional; number of source ports per pair during ring-topology skeleton discovery | | ||
| 219 | +| `probe.topology.topology_optimized` | bool / `true` | Optional; `true` uses adjacent-domain same-slot ring coverage; `false` uses cross-domain full-mesh Tracert | | ||
| 220 | +| `probe.topology.l2_path_aware` | bool / `true` | Optional; whether to generate additional L2 path discovery artifacts between adjacent L1 domains | | ||
| 221 | +| `probe.topology.output_subdir` | string / `""` | Optional; a relative directory under `output/`; must not be an absolute path and must not contain `.`, `..`, or empty path segments | | ||
| 222 | +| `probe.topology.allpath_output` | filename / `allpath.json` | Optional; cross-domain path JSON filename; must not contain parent directory references | | ||
| 223 | +| `probe.topology.l2_path_output` | filename / `l2_fullmesh_path.json` | Optional; L2 path JSON filename; must not contain parent directory references | | ||
| 224 | +| `probe.pingpong.times` | positive int / `50` | Optional; number of samples per PingPong task per turn | | ||
| 225 | +| `probe.pingpong.turns` | positive int / `1000` | Optional; number of PingPong turns; the actual cycle also includes execution, RPC, and solving time | | ||
| 226 | +| `probe.pingpong.payload_len` | int / `12` | Optional; HCCN Rping payload size in bytes, range `1-1500`. Each target independently generates random bytes of the specified length during AddTarget; the payload is not treated as a string | | ||
| 227 | +| `probe.pingpong.interval_ms` | positive int / `1` | Optional; HCCN Rping probe interval in milliseconds | | ||
| 228 | + | ||
| 229 | +The v1-compatible format continues to support the legacy `deploy.host_to_user_pair`, `deploy.controller`, `probe_scope`, `probe_topo.tracert`, and `probe_controller.pingpong` fields. The compatibility mapping is as follows: | ||
| 230 | + | ||
| 231 | +| v1 Field | v2 Field | Description | | ||
| 232 | +| --- | --- | --- | | ||
| 233 | +| `deploy.host_to_user_pair` | `hosts[]` | Host IP, user, and password/key configuration migrated to the host inventory; v2 uses `hosts[].id` as a stable reference | | ||
| 234 | +| `deploy.controller` | `controller` | v1 uses a controller node IP-to-user mapping; v2 uses the controller node host ID | | ||
| 235 | +| `deploy.host_to_key_filename` | `hosts[].ssh_key` | v2 consolidates private key paths into the corresponding host entry | | ||
| 236 | +| `probe_scope` | `probe.scope` | v1 keys are host management IPs; v2 keys are host IDs and support `device_range` or explicit `devices` | | ||
| 237 | +| `probe_topo.tracert` | `probe.topology` | Tracert topology discovery parameters remain compatible; field name prefix adjusted | | ||
| 238 | +| `probe_controller.pingpong` | `probe.pingpong` | PingPong probe parameters remain compatible; field name prefix adjusted | | ||
| 239 | + | ||
| 240 | +In v2, `probe.scope` is required; `probe.topology` and `probe.pingpong` are optional and use the default parameters from the table above when omitted. Run `./run.sh migrate-config old.json new.json` to generate a v2 configuration template. | ||
| 241 | + | ||
| 242 | +### 4.3 CLI and Execution Order | ||
| 243 | + | ||
| 244 | +#### 4.3.1 `rpc_host` | ||
| 245 | + | ||
| 246 | +| Parameter | Description | | ||
| 247 | +| --- | --- | | ||
| 248 | +| `-f, --file <PATH>` | Control JSON, default `./control_json/910b2_info.json` | | ||
| 249 | +| `-d, --dev <IFACE>` | Host management network interface name | | ||
| 250 | +| `-i, --ip <IP>` | Host management IP | | ||
| 251 | +| `-p, --port <PORT>` | RPC listening port | | ||
| 252 | +| `--pingpong-local-log` | Save local PingPong result logs on each host; disabled by default | | ||
| 253 | +| `--pingpong-log-dir <PATH>` | Local log root directory, default `/root/output` | | ||
| 254 | + | ||
| 255 | +#### 4.3.2 `probe_topo` | ||
| 256 | + | ||
| 257 | +| Parameter | Description | | ||
| 258 | +| --- | --- | | ||
| 259 | +| `-f, --file <PATH>` | Control JSON; execute topology discovery and generate topology artifacts | | ||
| 260 | + | ||
| 261 | +#### 4.3.3 `probe_controller` | ||
| 262 | + | ||
| 263 | +| Parameter | Description | | ||
| 264 | +| --- | --- | | ||
| 265 | +| `-f, --file <PATH>` | Control JSON | | ||
| 266 | +| `--print-pingpong-plan` | Print the task plan only; do not distribute the PingList or execute PingPong | | ||
| 267 | +| `--l1-only` | Probe and solve L1 only; disable L2 FullMesh tasks and L2 output | | ||
| 268 | +| `--no-metrics` | Disable PFC/CNP counter collection | | ||
| 269 | + | ||
| 270 | +Recommended execution order: | ||
| 271 | + | ||
| 272 | +```bash | ||
| 273 | +# Terminal 0: Clean up residual processes, distribute binaries and configuration, start rpc_host | ||
| 274 | +./run.sh deploy | ||
| 275 | + | ||
| 276 | +# Optional: Enable NPU-side PingPong local logging when starting rpc_host | ||
| 277 | +./run.sh deploy --pingpong-log /root/output | ||
| 278 | + | ||
| 279 | +# Equivalent expanded steps: | ||
| 280 | +python3 ./dispatcher/exec_realtime_cmd.py "ps aux | grep 'rpc_host' | grep -v grep | tr ' ' '\n' | grep -E '^[0-9]+$' | tr '\n' ' ' | xargs -r kill" | ||
| 281 | + | ||
| 282 | +python3 ./dispatcher/disp_file_scp.py './build/rpc_host' './bin/rpc_host' | ||
| 283 | +python3 ./dispatcher/disp_file_scp.py \ | ||
| 284 | + './control_json/910b2_info.json' './control_json/910b2_info.json' | ||
| 285 | + | ||
| 286 | +python3 ./dispatcher/exec_realtime_cmd.py -l \ | ||
| 287 | + './bin/rpc_host -f ./control_json/910b2_info.json' | ||
| 288 | + | ||
| 289 | +# Terminal 1: Discover topology | ||
| 290 | +./build/probe_topo -f ./control_json/910b2_info.json | ||
| 291 | + | ||
| 292 | +# Terminal 1: Continuous probing and analysis | ||
| 293 | +./build/probe_controller -f ./control_json/910b2_info.json | ||
| 294 | +``` | ||
| 295 | + | ||
| 296 | +`disp_file_scp.py` recursively determines distribution targets based on `deploy.control_topo`; only when this field is empty does it use all hosts from `host_to_user_pair`. | ||
| 297 | + | ||
| 298 | +### 4.4 Artifact Contract | ||
| 299 | + | ||
| 300 | +#### 4.4.1 Topology Discovery Artifacts | ||
| 301 | + | ||
| 302 | +| Artifact | Format | Content | | ||
| 303 | +| --- | --- | --- | | ||
| 304 | +| `output/<subdir>/probe_topo.json` | JSON | Network layers and mesh skeleton discovered by ring Tracert | | ||
| 305 | +| `output/<subdir>/probe_topo_lldp.json` | JSON | L1 domain topology aggregated by LLDP management IP; the default topology input for `probe_controller` | | ||
| 306 | +| `output/<subdir>/allpath.json` | JSON | Cross-domain pairs, port samples, multi-hop IPs, empty paths, and unique path statistics | | ||
| 307 | +| `output/<subdir>/l2_fullmesh_path.json` | JSON | L2 path discovery results between adjacent L1 domains; generated only when `l2_path_aware=true` | | ||
| 308 | + | ||
| 309 | +The topology JSON uses `status` to indicate device count completeness: `complete` when the number of discovered devices matches the probe scope configuration (v2 `probe.scope` or v1 `probe_scope`); `incomplete` when they do not match. The JSON also records `configured_device_count`, `discovered_device_count`, the configured device list, and the discovered device IP list. Completeness compares only device counts and does not change `status` based on LLDP domain count, empty path rate, or path coverage rate. | ||
| 310 | + | ||
| 311 | +#### 4.4.2 Network Status Artifacts | ||
| 312 | + | ||
| 313 | +```text | ||
| 314 | +output/<time>/ | ||
| 315 | +├── link_lat.txt | ||
| 316 | +├── link_pass_rate.txt | ||
| 317 | +├── bad_link.txt | ||
| 318 | +├── bad_link_candidate.txt | ||
| 319 | +├── l2_status/ | ||
| 320 | +│ ├── l2_path_lat.txt | ||
| 321 | +│ └── l2_path_passrate.txt | ||
| 322 | +└── metrics/ | ||
| 323 | + ├── mac_tx_pfc_pkt_num.txt | ||
| 324 | + ├── mac_rx_pfc_pkt_num.txt | ||
| 325 | + ├── roce_tx_cnp_pkt_num.txt | ||
| 326 | + └── roce_rx_cnp_pkt_num.txt | ||
| 327 | +``` | ||
| 328 | + | ||
| 329 | +| Artifact | Format Contract | | ||
| 330 | +| --- | --- | | ||
| 331 | +| `link_lat.txt` | The first row contains fixed-width L1 link column names `[device_ip-tor_ip]`; each subsequent row contains floating-point latency values for one turn | | ||
| 332 | +| `link_pass_rate.txt` | Column order matches `link_lat.txt`; value range target is `[0,1]` | | ||
| 333 | +| `bad_link.txt` | Text alerts; each row contains alert type, link endpoints, latency, and pass rate; multiple links can appear separately in the same turn | | ||
| 334 | +| `bad_link_candidate.txt` | Latency candidate events; written when either the time-domain or spatial-domain 3σ condition exceeds the threshold for 3 consecutive valid turns; not equivalent to a persistent alert | | ||
| 335 | +| `l2_path_lat.txt` | Tab-separated: `turn, task_index, tag, from_label, from_ip, to_label, to_ip, src_sport, l2_path_lat` | | ||
| 336 | +| `l2_path_passrate.txt` | Tab-separated; the last column is `pass_rate`; remaining fields are the same as above | | ||
| 337 | +| `metrics/*.txt` | CSV: `time,port,value,valid`; `port` is formatted as `<host>_dev<id>`. When all four counter fields are parsed successfully, `valid=true`; when a key is missing, a value is non-numeric, or a command fails, the value is set to `0` as a placeholder and `valid=false`; the sampling interval is approximately 1 second | | ||
| 338 | + | ||
| 339 | +Run `python3 ./plot/topo_plot.py` and `python3 ./plot/status_plot.py --input-dir "output/<time>"` to generate topology and status charts. Charts are derived artifacts; JSON/TXT/CSV are the stable data interfaces. | ||
| 340 | + | ||
| 341 | +## 5. External Dependencies | ||
| 342 | + | ||
| 343 | +### 5.1 Runtime Interfaces Within the Ecosystem | ||
| 344 | + | ||
| 345 | +| Dependency/Interface | Invocation | Purpose | Input Convention | Output and Parsing Convention | Stability Source | Known Limitations | | ||
| 346 | +| --- | --- | --- | --- | --- | --- | --- | | ||
| 347 | +| HCCN Device IP | `hccn_tool -i <dev> -ip -g` | Maps device ID to HCCN IP | `<dev>` is the local non-negative device ID | Parse tool output line by line and extract valid IPs; missing or format changes are treated as query failures for that device | Assumed based on current CANN tool implementation | Text format has no versioned schema | | ||
| 348 | +| LLDP Neighbor | `hccn_tool -i <dev> -lldp -g` | Obtain the directly connected Leaf/ToR management IP | Same as above | Extract the switch management IP; null values are assigned to the `unknown:<device>` domain | Assumed based on current implementation | Domain partitioning is unreliable when LLDP is disabled or permissions are insufficient | | ||
| 349 | +| Tracert | `hccn_tool -i <dev> -traceroute ... -sport <port>` | Obtain multi-hop paths | Source device, destination IP, source port | Return an ordered multi-hop IP list; an empty list indicates a failed sample; single-hop results are padded at the controller to preserve path shape | Assumed based on current implementation | TC/DSCP and return-direction port controllability are limited; ECMP coverage depends on source port hashing | | ||
| 350 | +| RNIC Stat | `hccn_tool -i <dev> -stat -g` | Obtain cumulative PFC/CNP counters | Device ID | Parse `mac_tx_pfc_pkt_num`, `mac_rx_pfc_pkt_num`, `roce_tx_cnp_pkt_num`, and `roce_rx_cnp_pkt_num` as `key:value`; the RPC line carries a validity flag; the controller outputs `valid=true/false` | Assumed based on current implementation | When any target key is missing, the value is not an unsigned integer, or the command fails, the entire row has `valid=false`; the value `0` is a placeholder only and is excluded from joint analysis | | ||
| 351 | +| HCCN Rping Initialization | `HccnRpingInit/Deinit` | Initialize the probe context for each device | Local device ID | The return code must indicate success before probing can proceed | CANN/HCCL headers and libraries | The runtime environment must provide a matching version of `libhccl*` | | ||
| 352 | +| HCCN Rping Target Configuration | `HccnRpingAddTarget` | Distribute destination IP, source/return ports | Target IP, source port, receive port, and other parameters | The return code maps to RPC success/failure | Same as above | Target count and port resource limits must be verified per version | | ||
| 353 | +| HCCN Rping Batch Probing | `HccnRpingBatchPingStart/Stop/GetResult/GetPayload` | Execute PingPong and obtain statistics | Device, task set, `times` | Normalized to a `uint64_t` array of `[P90Lat, P99Lat, Mean, Pass]` | Same as above | The raw latency unit must be confirmed by the interface version and documented in the release documentation; current logs display values in ms | | ||
| 354 | + | ||
| 355 | +These interfaces are all runtime dependencies on the host. The project does not modify the HCCL/HCOMM data plane and remains consistent with the HCCL-HCOMM decoupling principle; however, the C++ build still requires CANN/HCCL/ACL headers and link libraries and cannot be described as "completely free of compile-time dependencies." | ||
| 356 | + | ||
| 357 | +### 5.2 General Third-Party Libraries | ||
| 358 | + | ||
| 359 | +| Dependency | Version | Purpose | License | Dependency Stage | | ||
| 360 | +| --- | --- | --- | --- | --- | | ||
| 361 | +| CLI11 | 2.5.0 | C++ CLI parsing | BSD-3-Clause | Compile-time, header-only | | ||
| 362 | +| nlohmann_json | 3.12.0 | JSON configuration and artifacts | MIT | Compile-time, header-only | | ||
| 363 | +| Eigen | 3.4.0 | SVD link equation solving | MPL-2.0 | Compile-time, header-only | | ||
| 364 | +| rpclib | 2.3.0 | Controller-Host RPC | MIT | Compile-time static linking + runtime communication | | ||
| 365 | +| fmt | 12.0.0 | Optional formatting support | MIT | Optional, compile-time | | ||
| 366 | +| spdlog | 1.15.3 | Optional logging support | MIT | Optional, compile-time | | ||
| 367 | +| paramiko | 5.0.0 | Dispatcher SSH | LGPL-2.1-or-later | Python runtime | | ||
| 368 | +| scp | 0.15.0 | File distribution | LGPL-2.1-or-later | Python runtime | | ||
| 369 | +| matplotlib | 3.9.4 | Chart plotting | PSF-based | Python plotting runtime | | ||
| 370 | + | ||
| 371 | +Release packages must generate third-party notices from `third_party/manifest.json`, `third_party/python/requirements.txt`, and the actual installed package metadata. If versions or licenses are inconsistent, the release build audit result takes precedence. | ||
| 372 | + | ||
| 373 | +## 6. Key Algorithm Design | ||
| 374 | + | ||
| 375 | +### 6.1 Layered Domain Partitioning and L1/L2 Definitions | ||
| 376 | + | ||
| 377 | +1. Use device IPs and ring Tracert to identify the network layer skeleton. | ||
| 378 | +2. Use LLDP management IPs to aggregate devices into their directly connected Leaf/ToR domains. | ||
| 379 | +3. Define the edges from in-domain devices to the Leaf/ToR as L1 links. | ||
| 380 | +4. Define the remaining network segments between different Leaf/ToR domains as L2 paths. The current design provides only path-level candidates for L2 and does not guarantee unique physical link localization. | ||
| 381 | + | ||
| 382 | +### 6.2 Minimum-Cost Topology Probing | ||
| 383 | + | ||
| 384 | +Let the cluster have `D` L1 domains in total, where domain `i` has `n_i` devices, the total device count is `N=Σn_i`, and each pair uses `s` source port samples. | ||
| 385 | + | ||
| 386 | +- Skeleton discovery: All devices form a directed ring; each device probes the next device. The number of paths is approximately `N × tree_probe_sport_count`, which is `O(N)`. | ||
| 387 | +- Optimized cross-domain coverage: Each domain connects only to the same slot in the next domain. The number of pairs is approximately `Σ min(n_i,n_(i+1))`, and the number of paths is multiplied by `sport_count`. With uniform domain sizes, this is approximately `N × s`, which is `O(Ns)`. | ||
| 388 | +- Non-optimized cross-domain coverage: All devices in different domains form a directed full mesh. The number of pairs is `Σ_{i≠j} n_i n_j = N²-Σn_i²`, which is worst case `O(N²s)`. | ||
| 389 | +- L1 PingPong: Each domain generates tasks based on solvable rings. With uniform domain sizes, this is approximately `O(N)` pairs. | ||
| 390 | +- Current L2 PingPong: Adjacent domains form a ring, and FullMesh is executed between adjacent domains. The number of tasks is `Σ n_i n_(i+1)`; with uniform domain size `k`, this is `Dk²=Nk`. This is not strictly `O(N)`, so the task volume must be reviewed with `--print-pingpong-plan` before deployment, and `--l1-only` should be used when necessary. | ||
| 391 | + | ||
| 392 | +"Minimum cost" refers to reducing the number of paths under the constraint that the target link equations remain identifiable. It does not claim that the entire process always has linear complexity. | ||
| 393 | + | ||
| 394 | +### 6.3 PingPong Metric Reduction | ||
| 395 | + | ||
| 396 | +The raw arrays returned by the host map to the controller metric enumerations as follows: | ||
| 397 | + | ||
| 398 | +| Raw `PingpongResult` | Index | Controller `PingpongMetric` | Processing | | ||
| 399 | +| --- | --- | --- | --- | | ||
| 400 | +| `P90Lat` | 0 | `P90Lat` | Convert to `float` | | ||
| 401 | +| `P99Lat` | 1 | `P99Lat` | Convert to `float` | | ||
| 402 | +| `Mean` | 2 | `MeanLat` | Convert to `float`; renamed to avoid confusion with statistical mean | | ||
| 403 | +| `Pass` | 3 | `LogPassRate` | Compute `log2(pass/times)` | | ||
| 404 | +| `Size` | 4 | None | Enumeration sentinel indicating the array field count; not an observed metric | | ||
| 405 | + | ||
| 406 | +The pass rate is solved in the log domain: | ||
| 407 | + | ||
| 408 | +```text | ||
| 409 | +r_path = pass / times | ||
| 410 | +b_path = log2(r_path) | ||
| 411 | +b_path = Σ log2(r_link) | ||
| 412 | +``` | ||
| 413 | + | ||
| 414 | +Boundary conventions: | ||
| 415 | + | ||
| 416 | +- `times <= 0`: Configuration or invocation error; fail immediately. | ||
| 417 | +- `pass > times`: Data anomaly; the sample is marked invalid and excluded from solving; validation must be implemented before release. | ||
| 418 | +- `pass = 0`: Mathematically `-∞`. The current implementation uses `-1e10` as a finite sentinel for solving; the target implementation should also carry `all_lost=true`, output a final pass rate of `0`, and prevent the sentinel from being misinterpreted as a comparable normal value. | ||
| 419 | +- Missing fields in the raw array: Latency returns `NaN`; the pass rate is currently handled as `pass=0`. The target implementation should uniformly mark such samples as invalid. | ||
| 420 | + | ||
| 421 | +### 6.4 L1 Link Solving | ||
| 422 | + | ||
| 423 | +Build linear equations for each path sample: | ||
| 424 | + | ||
| 425 | +```text | ||
| 426 | +A × x = b | ||
| 427 | +``` | ||
| 428 | + | ||
| 429 | +- `A[m,n]` indicates whether path `m` traverses L1 link `n`. | ||
| 430 | +- For latency, `b` is the path P99 latency and `x` is the link latency. | ||
| 431 | +- For pass rate, `b` is the `log2` value of the path pass rate; after solving, `exp2` recovers the link pass rate. | ||
| 432 | +- Use Eigen `BDCSVD` to compute the least-squares solution. | ||
| 433 | + | ||
| 434 | +The probe plan must ensure that the columns corresponding to target links are identifiable. If the matrix rank is insufficient, the condition number is too large, the input contains invalid samples, or the solution is clearly out of bounds, the link results for that turn must be marked as `NaN/invalid` and no definitive fault conclusion is produced. The current implementation performs SVD rank and condition number checks and returns `NaN` when these checks fail. | ||
| 435 | + | ||
| 436 | +### 6.5 L2 Path Metrics | ||
| 437 | + | ||
| 438 | +The current L2 latency calculation is: | ||
| 439 | + | ||
| 440 | +```text | ||
| 441 | +l2_path_lat = end_to_end_p99 - src_l1_lat - dst_l1_lat | ||
| 442 | +``` | ||
| 443 | + | ||
| 444 | +When L1 at either end is unsolvable, the output is `NaN`. The L2 pass rate directly uses the end-to-end pair pass rate and does not yet subtract the L1 pass rates at both ends; therefore, the artifact name uses `l2_path_passrate` rather than `l2_link_passrate`. | ||
| 445 | + | ||
| 446 | +### 6.6 Anomaly Detection | ||
| 447 | + | ||
| 448 | +The target anomaly detection handles latency and pass rate separately. Latency anomalies use two baselines: the time-domain baseline maintains historical statistics per L1 link, and the spatial-domain baseline maintains statistics across all valid L1 links in the same topology layer for the current turn. | ||
| 449 | + | ||
| 450 | +1. Exclude `NaN`, all-lost sentinels, and samples in known maintenance windows. | ||
| 451 | +2. Time-domain baseline: For each L1 link, compute the mean `μ_t` and standard deviation `σ_t` using historical valid turns. The current implementation uses cumulative historical statistics; a rolling window or median/MAD can be adopted later. | ||
| 452 | +3. Spatial-domain baseline: Compute the mean `μ_s` and standard deviation `σ_s` across all valid L1 link latencies in the same topology layer for the current turn. Links are no longer filtered by IP string or specific network segment. | ||
| 453 | +4. Latency alert condition: Both `latency > μ_t + 3σ_t` and `latency > μ_s + 3σ_s` must be satisfied simultaneously. After the condition is met for 3 consecutive valid turns, write to `bad_link.txt`. | ||
| 454 | +5. Pass rate anomaly condition: `pass_rate < 0.99`; write an alert immediately when the condition is met. The threshold will be made configurable later. | ||
| 455 | +6. Latency candidate events: When either the time-domain or spatial-domain 3σ condition exceeds the threshold for 3 consecutive valid turns, write to `bad_link_candidate.txt` without generating a persistent alert. | ||
| 456 | +7. PFC/CNP counters are currently collected and stored only and do not participate in alert confidence determination. In the future, when counters increase simultaneously, they can raise the confidence of congestion-type anomalies, but counters are never used as the sole alert condition. | ||
| 457 | + | ||
| 458 | +The current code uses mean and standard deviation to maintain L1 link time-domain and spatial-domain baselines. | ||
| 459 | + | ||
| 460 | +## 7. Internal Module Implementation | ||
| 461 | + | ||
| 462 | +### 7.1 Directory and Responsibilities | ||
| 463 | + | ||
| 464 | +```text | ||
| 465 | +disp_probe-main/ | ||
| 466 | +├── control_json/ # User configuration samples | ||
| 467 | +├── dispatcher/ # SSH/SCP distribution and remote execution | ||
| 468 | +├── docs/ # RFC, user, and environment documentation | ||
| 469 | +├── plot/ # Topology and status visualization | ||
| 470 | +├── scripts/ # System and third-party dependency installation | ||
| 471 | +├── src/ | ||
| 472 | +│ ├── probe_topo.cpp # Topology discovery entry point | ||
| 473 | +│ ├── probe_controller.cpp # Probing, solving, and analysis entry point | ||
| 474 | +│ ├── rpc_host.cpp # Host RPC service | ||
| 475 | +│ └── util/ | ||
| 476 | +│ ├── helper/ # Workflow coordination, PingList, equation solving | ||
| 477 | +│ ├── topo/ # Configuration, control topology, MeshTopo | ||
| 478 | +│ ├── tool/ # Tracert, PingPong, metric collection | ||
| 479 | +│ ├── rpc_call/ # RPC proxy | ||
| 480 | +│ └── file_path/ # Working directory management | ||
| 481 | +└── third_party/ # Dependency manifest | ||
| 482 | +``` | ||
| 483 | + | ||
| 484 | +### 7.2 Cross-Module Contracts | ||
| 485 | + | ||
| 486 | +The RFC retains only signatures that cross module or process boundaries; purely internal getters and setters are not listed. | ||
| 487 | + | ||
| 488 | +```cpp | ||
| 489 | +using PingpongRawResult = | ||
| 490 | + std::vector<std::vector<std::vector<uint64_t>>>; // device -> task -> metric | ||
| 491 | +using PingpongMetricMatrix = | ||
| 492 | + std::vector<std::vector<float>>; // device -> task | ||
| 493 | +using LinkMetricVector = std::vector<float>; // link_global_id -> value | ||
| 494 | + | ||
| 495 | +using HccnDeviceIpListParallel = | ||
| 496 | + std::function<std::vector<std::vector<std::string>>( | ||
| 497 | + const std::vector<std::string>& control_devices, | ||
| 498 | + const std::vector<int>& device_counts)>; | ||
| 499 | + | ||
| 500 | +MeshTopo& probe_topo_ring(); | ||
| 501 | +MeshTopo& probe_topo(); | ||
| 502 | +std::map<std::string, std::vector<std::tuple<std::string, std::string, int>>>& | ||
| 503 | +get_pinglist(); | ||
| 504 | +void set_pinglist(const PingList& pinglist); | ||
| 505 | +PingpongRawResult get_pingpong_res(int times); | ||
| 506 | +PingpongMetricMatrix reduce_pingpong_res( | ||
| 507 | + const PingpongRawResult&, PingpongMetric, int times); | ||
| 508 | +LinkMetricVector solve_pingpong_res( | ||
| 509 | + const PingpongMetricMatrix&, const PingList&); | ||
| 510 | +``` | ||
| 511 | + | ||
| 512 | +Differences between `probe_topo_ring()` and `probe_topo()`: | ||
| 513 | + | ||
| 514 | +| Interface | Behavior | Use Case | | ||
| 515 | +| --- | --- | --- | | ||
| 516 | +| `probe_topo_ring()` | Executes only `gen_topo()` to generate the layer/mesh skeleton from ring Tracert | New configuration path; `probe_topo.cpp` subsequently supplements artifacts with LLDP and explicit cross-domain probing | | ||
| 517 | +| `probe_topo()` | Executes `gen_topo()`, in-domain `mesh_full_trace()`, `gen_pinglist()`, supplementary Tracert, and link ID assignment | Legacy configuration compatibility path or when a complete solving state must be constructed in one pass | | ||
| 518 | + | ||
| 519 | +Host RPC boundary: | ||
| 520 | + | ||
| 521 | +```cpp | ||
| 522 | +std::vector<std::vector<std::vector<std::string>>> | ||
| 523 | +tracert_ports_multi_by_src_ip_with_sport_begin( | ||
| 524 | + std::string src_dev_ip, | ||
| 525 | + std::vector<std::string> targets, | ||
| 526 | + std::vector<int> port_num, | ||
| 527 | + int sport_begin); | ||
| 528 | + | ||
| 529 | +int pinglist_insert_muti( | ||
| 530 | + std::string src_dev_ip, | ||
| 531 | + std::vector<std::string> target_dev_ips, | ||
| 532 | + std::vector<int> src_ports, | ||
| 533 | + std::vector<int> return_ports, | ||
| 534 | + int rx_num); | ||
| 535 | + | ||
| 536 | +std::vector<std::vector<uint64_t>> | ||
| 537 | +ud_pingpong_tx_muti(std::string src_dev_ip, int times); | ||
| 538 | + | ||
| 539 | +std::vector<std::string> get_metrics_counter_name(); | ||
| 540 | +std::vector<std::vector<uint64_t>> get_metrics_counter_value(); | ||
| 541 | +``` | ||
| 542 | + | ||
| 543 | +### 7.3 `metrics_collector` and Analysis Module Responsibilities | ||
| 544 | + | ||
| 545 | +- `metrics_collector`: Calls `hccn_tool -stat -g` every second and parses four types of cumulative PFC/CNP counters. The in-memory queue retains up to 100 rows and is cleared after the controller pulls the data. Read errors are currently swallowed and zero values are written; the target implementation should return a validity status and error reason. | ||
| 546 | +- Analysis logic (in `probe_controller.cpp`): Maintains time-domain and spatial-domain baselines, detects latency/pass-rate anomalies, and outputs `bad_link.txt` and `bad_link_candidate.txt`. PFC/CNP counters are currently collected and stored only and do not participate in alert confidence determination. | ||
| 547 | + | ||
| 548 | +## 8. Impact Analysis | ||
| 549 | + | ||
| 550 | +### 8.1 Production Intrusion Impact | ||
| 551 | + | ||
| 552 | +| Resource | Impact Source | Control Measures | | ||
| 553 | +| --- | --- | --- | | ||
| 554 | +| NPU/RNIC | HCCN Rping initialization, AddTarget, probe packet transmission and reception | Limit per-device task count and `times`; run `--print-pingpong-plan` before deployment; support `--l1-only` | | ||
| 555 | +| Network bandwidth | Tracert and PingPong packets | Topology discovery runs at low frequency; status probing is domain-partitioned; configure probe intervals; stop sending when budget is exceeded | | ||
| 556 | +| Collective communication latency | Probe traffic competes with production traffic for queues, hash paths, and port resources | A/B validation; gradual rollout; avoid full topology discovery during peak business hours | | ||
| 557 | +| Host CPU/memory | RPC, SSH, result aggregation, 1-second counter polling | Host-level parallelism; counter queue capped at 100; monitor controller/host CPU and RSS | | ||
| 558 | +| File system | Long-turn text artifacts grow continuously | Rotate by time directory; apply retention policies and compression; configure disk quotas before release | | ||
| 559 | + | ||
| 560 | +### 8.2 Further Evaluation | ||
| 561 | + | ||
| 562 | +For each RNIC, assuming 500 B probe packets are sent every 1 ms per turn over a 100 Gbps network, the bandwidth utilization is less than one thousandth. The scheme is further validated through multi-host hccl_test to demonstrate that the impact of probe traffic on hccl task completion time is less than 1%. The default probe cycle sends probe packets every 1 ms for `turns` consecutive turns, with `times` samples per turn. | ||
| 563 | + | ||
| 564 | +### 8.3 Failure Modes and Degradation | ||
| 565 | + | ||
| 566 | +| Failure Mode | Detection | Handling and Degradation | Artifact Semantics | | ||
| 567 | +| --- | --- | --- | --- | | ||
| 568 | +| Tracert failure/empty path | Empty hop list, timeout, or non-zero status returned | Retry limited times for the pair/port; if still failing, mark as unmeasured rather than guessing the path; terminate topology release when the empty path rate exceeds the threshold | `allpath.json` records empty samples and statistics | | ||
| 569 | +| RPC timeout | Client exception or no response | Isolate the single host; other hosts continue; exponential backoff retry; do not treat timeouts as packet loss | Corresponding samples are `invalid` | | ||
| 570 | +| Partial host unreachability | SSH/RPC health check failure | Remove from the current turn scope and report coverage; if matrix identifiability is compromised, stop link solving | Output the missing host list | | ||
| 571 | +| Incomplete topology discovery | Discovered device count does not match the probe scope configuration count | `probe_topo` writes diagnostic artifacts and stops subsequent LLDP/path probing; `probe_controller` refuses to execute PingPong and alerting | Topology marked `status=incomplete`; configured and discovered counts recorded; LLDP domain or path coverage completeness is no longer checked | | ||
| 572 | +| Counter read error | Missing key, non-numeric value, or command failure | Sample marked `valid=false`; excluded from joint analysis; the main probe flow continues | `metrics/*.txt` writes `value=0,valid=false`, distinguishable from genuine `value=0,valid=true` | | ||
| 573 | +| PingPong total packet loss | `pass=0` | Generate a connectivity candidate alert directly; use a sentinel in the log domain but do not compare it as a normal value | Final pass rate is 0 with `all_lost` flag | | ||
| 574 | +| Insufficient equation rank / ill-conditioned | Rank and condition number checks | Do not output definitive link localization; add probe paths or degrade to path-level alerts | Link values are `NaN/invalid` | | ||
| 575 | +| Output file not writable | File open/write failure | Terminate the current controller run to avoid probing without artifacts | Process exits with non-zero code | | ||
| 576 | +| Controller exit | Process signal or exception | Stop distributing tasks; host services can remain for reconnection; cleanup policies are managed by operations | Turns already written to disk are retained | | ||
| 577 | + | ||
| 578 | +### 8.4 Overhead Model | ||
| 579 | + | ||
| 580 | +Let: | ||
| 581 | + | ||
| 582 | +- `P`: Number of directed PingPong tasks per turn; | ||
| 583 | +- `times`: Number of packets per task per turn; | ||
| 584 | +- `B_req`, `B_rsp`: Average request and response bytes on the wire, including protocol overhead; | ||
| 585 | +- `T_cycle`: Turn cycle in seconds; | ||
| 586 | +- `C_link`: Port line rate in bit/s. | ||
| 587 | + | ||
| 588 | +The cluster-wide average probe bandwidth is estimated as: | ||
| 589 | + | ||
| 590 | +```text | ||
| 591 | +BW_cluster = P × times × (B_req + B_rsp) × 8 / T_cycle | ||
| 592 | +``` | ||
| 593 | + | ||
| 594 | +The average utilization for a single device `d` is: | ||
| 595 | + | ||
| 596 | +```text | ||
| 597 | +Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link) | ||
| 598 | +``` | ||
| 599 | + | ||
| 600 | +### 8.5 Security and Operations Impact | ||
| 601 | + | ||
| 602 | +- The Dispatcher uses SSH/SCP, and the configuration may contain plaintext passwords. In production environments, prefer private keys and restrict configuration file permissions. | ||
| 603 | +- `hccn_tool` and device operations may require elevated privileges. Use least-privilege accounts and command allowlists. | ||
| 604 | +- The RPC interface does not define authentication or encryption in this document and must be deployed in a controlled management network. A separate security design is required before use across security domains. | ||
| 605 | + | ||
| 606 | +## 9. Compatibility, Feature Toggles, and Gradual Rollout | ||
| 607 | + | ||
| 608 | +### 9.1 Compatibility | ||
| 609 | + | ||
| 610 | +- The tool runs as an independent process and does not modify training processes or HCCL/HCOMM interfaces. | ||
| 611 | +- The current recommended configuration uses `schema_version=2`, `hosts[]`, and `probe.scope` as required contracts; `probe.topology` and `probe.pingpong` are optional parameter groups. The default `schema_version` or `schema_version=1` remains backward compatible with the `probe_scope + probe_topo.tracert + probe_controller.pingpong` format. When an entire probe configuration group is missing, the legacy ranktable path is preserved, but no new capabilities are added. | ||
| 612 | +- Once JSON/TXT artifacts are consumed by external systems, new fields must remain backward compatible. Breaking changes must upgrade the schema or filename. | ||
| 613 | + | ||
| 614 | +### 9.2 Feature Toggles | ||
| 615 | + | ||
| 616 | +| Toggle | Effect | Recommended Scenario | | ||
| 617 | +| --- | --- | --- | | ||
| 618 | +| Do not start `probe_controller` | Completely disable continuous probing | Default off / rollback | | ||
| 619 | +| `--print-pingpong-plan` | Review the plan only; no network probing | Pre-deployment capacity check | | ||
| 620 | +| `--l1-only` | Disable L2 FullMesh | Initial rollout, bandwidth-constrained environments, or excessive L2 tasks | | ||
| 621 | +| `--no-metrics` | Disable PFC/CNP collection | `hccn_tool -stat` is unstable or permissions are insufficient | | ||
| 622 | +| `topology_optimized=true` | Use low-overhead cross-domain coverage | Default production mode | | ||
| 623 | +| `l2_path_aware=false` | Do not generate additional L2 path discovery tasks | When only L1 is of interest | | ||
| 624 | + | ||
| 625 | +A unified top-level `feature_enabled` flag and configurable turn interval should be added in the future. The current version controls behavior through process startup and the toggles listed above. | ||
| 626 | + | ||
| 627 | +### 9.3 Gradual Rollout Strategy | ||
| 628 | + | ||
| 629 | +Lab environment with 2 hosts and 16 devices: Validate interfaces, artifacts, and fault injection. | ||
| 630 | + | ||
| 631 | +## 10. Test Plan | ||
| 632 | + | ||
| 633 | +### 10.1 Objective-Use Case Mapping | ||
| 634 | + | ||
| 635 | +| Use Case | Problem/Criteria | Scenario and Checkpoints | Level | | ||
| 636 | +| --- | --- | --- | --- | | ||
| 637 | +| T1 Configuration Parsing | Q1, compatibility | Required fields, defaults, duplicate devices, range strings, illegal output paths | UT | | ||
| 638 | +| T2 Topology Modeling | Q1, topology completeness | Control topology, multiple LLDP domains, unknown LLDP, JSON round-trip, and exception access | UT | | ||
| 639 | +| T3 Probe Plan Scale | Q4, bandwidth threshold | Validate L1 ring, L2 adjacent-domain FullMesh task formulas, and task deduplication | UT | | ||
| 640 | +| T4 Metric Reduction | Q2 | P90/P99/Mean mapping; `times<=0`, `pass=0`, `pass>times`, missing fields | UT | | ||
| 641 | +| T5 Equation Solving | Q3, localization accuracy | Solvable, overdetermined, rank-deficient, and NaN inputs | UT | | ||
| 642 | +| T6 Counter Anomaly | Q2/Q3 | Missing key, illegal value, device ID normalization, counter increment and reset/wraparound | UT | | ||
| 643 | + | ||
| 644 | +### 10.2 UT Integration | ||
| 645 | + | ||
| 646 | +The current repository uses CMake `BUILD_TESTS` and CTest. There is no available root-level `build.sh -u/-s` entry point. The RFC follows the repository as it stands: | ||
| 647 | + | ||
| 648 | +```bash | ||
| 649 | +cmake -S . -B build-test \ | ||
| 650 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | ||
| 651 | + -DBUILD_TESTS=ON \ | ||
| 652 | + -DCMAKE_BUILD_TYPE=Debug | ||
| 653 | +cmake --build build-test -j"$(nproc)" | ||
| 654 | +ctest --test-dir build-test --output-on-failure -L UT | ||
| 655 | +``` | ||
| 656 | + | ||
| 657 | +T1, T2, T3, T4, T5, and T6 are implemented under `tests/` and registered by `tests/CMakeLists.txt` with the CTest `UT` label. The tests do not depend on real NPUs, RPC hosts, or switches and can run in any development environment that satisfies the compilation dependencies. If the project later restores a unified `build.sh -u/-s`, it should wrap the CMake/CTest flow above rather than creating a separate test system. | ||
| 658 | + | ||
| 659 | +## 11. Risks and Mitigations | ||
| 660 | + | ||
| 661 | +| Risk | Impact | Mitigation | Status | | ||
| 662 | +| --- | --- | --- | --- | | ||
| 663 | +| `traffic_tc/dscp` takes effect only on the source side; return-direction TC/DSCP and source port are uncontrollable | Probe paths diverge from production paths, potentially causing false packet-loss or latency measurements | Drive interface capabilities through Issue 261; mark confidence levels and use multi-source-port and production A/B calibration until resolved | Known, being tracked | | ||
| 664 | +| L2 is currently path-level rather than physical-link-level | Cannot uniquely locate a specific edge between Spine and Leaf | Name artifacts as "path"; output Top-10 pairs and paths; combine with switch telemetry for secondary localization | Mitigated but capability is limited | | ||
| 665 | +| Simple Z-Score-based anomaly analysis is sensitive to non-stationary, heavy-tailed distributions | False positives or false negatives | Latency requires 3 consecutive turns and layered baselines; pass rate below threshold triggers immediate alert; evaluate MAD/quantile approaches | Currently using mean/standard deviation + 3σ for latency detection, validated on 16 devices; larger topologies require further validation | | ||
| 666 | + | ||
| 667 | +## 12. Alternatives | ||
| 668 | + | ||
| 669 | +| Approach | Probe Overhead | Localization Accuracy | Implementation/Deployment Complexity | RoCE Compatibility | Conclusion | | ||
| 670 | +| --- | --- | --- | --- | --- | --- | | ||
| 671 | +| R-pingmesh | Moderate; depends on Traceroute with simulated service five-tuples | Can observe paths closer to production traffic | High; depends on eBPF and requires five-tuple extraction at RDMA QP creation | If RoCE five-tuple generation moves to user space, kernel-side extraction becomes difficult | Not adopted for now | | ||
| 672 | +| NetBouncer | Depends on probe frequency | Path/device-level | Requires special packet processing | Depends on IP-in-IP; not suitable for the current RoCE environment | Not adopted | | ||
| 673 | +| RDMA Pingmesh | `O(N²)` for full mesh | End-to-end anomalies visible; link localization limited | Moderate | Native RDMA/RoCE, but large-scale probe cycles are long | Baseline comparison only | | ||
| 674 | +| This scheme | L1 near `O(N)`; L2 currently adjacent-domain FullMesh `Σn_i n_(i+1)` | L1 link-level; L2 path/pair-level | Medium-high; requires topology discovery, RPC, solving, and analysis | Uses HCCN/Rping, designed for the current Ascend RoCE environment | Recommended for continued development | | ||
| 675 | + | ||
| 676 | +## 13. Open Questions | ||
| 677 | + | ||
| 678 | +Items that are already known and have mitigations are listed in Section 11 and are not repeated here. | ||
| 679 | + | ||
| 680 | +| ID | Question | Decision Needed | | ||
| 681 | +| --- | --- | --- | | ||
| 682 | +| O1 | Whether L2 must be further localized from pair-level to physical switch link | If needed, introduce switch telemetry or other information for further identification | | ||
| 683 | +| O2 | Whether anomaly baselines should use mean/3σ or median/MAD | Decide the specific metric to adopt based on actual deployment conditions | | ||
| 684 | +| O3 | RPC authentication, encryption, and version negotiation | Confirm that the system is limited to controlled management networks, or add an independent security design | | ||
| 685 | +| O4 | `hccn_rping` raw latency unit, AddTarget limit, and CANN version compatibility matrix | Fix through upstream interface documentation or a versioned adaptation layer; confirm that hccn_tool-related UT test cases pass before use | | ||
| 686 | + | ||
| 687 | +## Review Record | ||
| 688 | + | ||
| 689 | +- PR 2491 first-round feedback: Direction acknowledged; requested chapter restructuring, impact analysis supplement, user/external dependency contract supplement, algorithm boundary clarification, and test mapping. | ||
| 690 | +- This revision: Addressed each first-round comment item by item. | ||
| @@ -0,0 +1,1239 @@ | |||
| 1 | +# RFC: Host NIC Plugin Mechanism | ||
| 2 | + | ||
| 3 | +- Start Date: 2026-08-14 | ||
| 4 | +- RFC PR Number: 4610 | ||
| 5 | +- Status: accept | ||
| 6 | + | ||
| 7 | +--- | ||
| 8 | + | ||
| 9 | +## 1. Summary | ||
| 10 | + | ||
| 11 | +Add a Host NIC (Network Interface Card) plugin mechanism to HCOMM that allows developers to replace built-in protocol implementations or extend new communication protocols as standalone `.so` files without modifying the original HCOMM source code. | ||
| 12 | + | ||
| 13 | +This design embeds a C function pointer table (ops table) into the `Channel` / `Endpoint` base classes. The corresponding ops table and ctx context are populated at creation time, and calls are dispatched directly through C function pointers at runtime. | ||
| 14 | + | ||
| 15 | +--- | ||
| 16 | + | ||
| 17 | +## 2. Background and Motivation | ||
| 18 | + | ||
| 19 | +The communication protocol types currently supported by HCOMM (HCCS, RoCE, PCIe, SIO, UB series, and so on) are all implemented as built-in modules. Adding a new NIC protocol requires modifying the protocol enumeration and internal implementation, which leads to the following issues: | ||
| 20 | + | ||
| 21 | +- External developers cannot contribute new NIC support without modifying the HCOMM source code | ||
| 22 | +- Built-in code is coupled with third-party code, which hinders community collaboration | ||
| 23 | +- The barrier to experimenting with new protocols is high because a deep understanding of the internal HCOMM architecture is required | ||
| 24 | + | ||
| 25 | +### 2.1 Target Scenarios | ||
| 26 | + | ||
| 27 | +| Scenario | Description | Typical Use Case | | ||
| 28 | +|----------|-------------|-----------------| | ||
| 29 | +| **Protocol Replacement** | Replace the built-in implementation of a specific protocol in HCOMM | `COMM_PROTOCOL_ROCE` → custom optimized version | | ||
| 30 | +| **Protocol Extension** | A hardware vendor registers a brand-new protocol number for a new RNIC | `COMM_PROTOCOL_CUSTOM_BASE + 0` → new NIC driver | | ||
| 31 | +| **Performance Experimentation** | Rapidly validate a new communication backend in the `experimental/` directory | Custom URMA variant without touching the main branch code | | ||
| 32 | + | ||
| 33 | +--- | ||
| 34 | + | ||
| 35 | +## 3. Design Goals and Non-Goals | ||
| 36 | + | ||
| 37 | +### 3.1 Design Goals | ||
| 38 | + | ||
| 39 | +| # | Goal | Description | | ||
| 40 | +|---|------|-------------| | ||
| 41 | +| G1 | Protocol Replacement | The plugin takes over an existing built-in protocol number; the built-in implementation no longer participates | | ||
| 42 | +| G2 | Protocol Extension | The plugin registers a new protocol number (≥ `COMM_PROTOCOL_CUSTOM_BASE`) to implement a brand-new communication backend | | ||
| 43 | +| G3 | Zero-Branch Dispatch for Interfaces | Except for some data-plane interfaces that are not exclusive to the 950 series, all other interfaces use zero-branch dispatch | | ||
| 44 | +| G4 | Bind Ops Table at Creation Time | The Endpoint / Channel is populated with a C ops table at creation time, and runtime calls go directly through C function pointers | | ||
| 45 | +| G5 | Independent Build | Gated by `ENABLE_EXPERIMENTAL`; the plugin .so is compiled and distributed independently without linking `libhcomm.so` | | ||
| 46 | +| G6 | Optimal Performance | For data-plane Channel dispatch, Type 1 interfaces share the same overhead path for both built-in and plugin; Type 2 built-in interfaces use the legacy path | | ||
| 47 | +| G7 | One Plugin per Protocol | Only one plugin binary is supported per communication protocol; a later-loaded plugin overwrites an earlier one | | ||
| 48 | + | ||
| 49 | +### 3.2 Non-Goals | ||
| 50 | + | ||
| 51 | +| # | Description | | ||
| 52 | +|---|-------------| | ||
| 53 | +| N1 | A single plugin taking over multiple different protocols is not supported | | ||
| 54 | +| N2 | No modifications to any public header files under the `include/` directory | | ||
| 55 | +| N3 | No modifications to the implementation code of any built-in Channel/Endpoint subclasses | | ||
| 56 | +| N4 | Runtime unloading or hot-updating of plugins is not supported | | ||
| 57 | + | ||
| 58 | +--- | ||
| 59 | + | ||
| 60 | +## 4. Overall Architecture | ||
| 61 | + | ||
| 62 | +### 4.1 Architecture Overview | ||
| 63 | + | ||
| 64 | +The core idea is to embed two fields in the `Channel` / `Endpoint` base classes: `nicOps_` (ops table pointer) and `nicCtx_` (ctx parameter). These fields are populated at creation time, and all runtime calls are unified through C function pointer invocations. The built-in implementation is integrated into the same mechanism through the `g_BuiltinChannelOps` / `g_BuiltinEndpointOps` wrapper function tables. The plugin implementation is output when `CreateEndpoint` and `CreateChannel` are called through `dlopen`-loaded symbols. The calling layer does not need to distinguish between built-in and plugin for most interfaces. | ||
| 65 | + | ||
| 66 | +### 4.2 Protocol Replacement vs. Protocol Extension | ||
| 67 | + | ||
| 68 | +Both modes share the same mechanism. The only difference is whether the protocol number registered by the plugin conflicts with a built-in protocol: | ||
| 69 | + | ||
| 70 | +| Mode | Protocol Number | Behavior | | ||
| 71 | +|------|----------------|----------| | ||
| 72 | +| **Replacement** | All non-RESERVED built-in enumeration values < `COMM_PROTOCOL_CUSTOM_BASE` (1000) | The plugin mapping table is hit during Endpoint creation, and the plugin path is used; the built-in Endpoint factory is not called | | ||
| 73 | +| **Extension** | Custom protocol numbers ≥ `COMM_PROTOCOL_CUSTOM_BASE` (1000) | The framework has no built-in implementation; only the plugin path is used during Endpoint creation, and the built-in path returns an error because the protocol number is unrecognized | | ||
| 74 | + | ||
| 75 | +- The plugin declares its supported protocol number set (up to 4) in the `protocols[]` array returned by `HcommNicPluginGetInfo()`. | ||
| 76 | +- COMM_PROTOCOL_CUSTOM_BASE=1000 is defined in nic_plugin_manager.h and serves as the boundary constant for plugin protocol numbers. The CommProtocol field carries custom values ≥1000 with int32_t semantics. | ||
| 77 | + | ||
| 78 | +### 4.3 Layered Architecture | ||
| 79 | + | ||
| 80 | +```text | ||
| 81 | +┌──────────────────────────────────────────────────────────────────┐ | ||
| 82 | +│ include/hcomm_res.h / include/hcomm_primitives.h (Public API) │ | ||
| 83 | +├──────────────────────────────────────────────────────────────────┤ | ||
| 84 | +│ Calling Layer │ | ||
| 85 | +│ HcommWriteNbi(ch, ...) { │ | ||
| 86 | +│ auto* ch = CHANNEL_FROM_HANDLE(channel); │ | ||
| 87 | +│ return ch->GetNicOps()->writeNbi(ch->GetNicCtx(), ...); │ | ||
| 88 | +│ } │ | ||
| 89 | +│ HcommMemReg(ep, ...) { │ | ||
| 90 | +│ auto endpoint = GetEndpointMap().GetEndpoint(ep); │ | ||
| 91 | +│ return endpoint->GetNicOps()->registerMemory( │ | ||
| 92 | +│ endpoint->GetNicCtx(), ...); │ | ||
| 93 | +│ } │ | ||
| 94 | +├──────────────────────────────────────────────────────────────────┤ | ||
| 95 | +│ Channel / Endpoint Base Classes (2 new public fields each) │ | ||
| 96 | +│ class Channel { │ | ||
| 97 | +│ HcommNicChannelOps *nicOps_{nullptr}; │ | ||
| 98 | +│ void *nicCtx_{nullptr}; │ | ||
| 99 | +│ }; │ | ||
| 100 | +├──────────────────────────────────────────────────────────────────┤ | ||
| 101 | +│ Injection at Creation Time │ | ||
| 102 | +│ Built-in: ch->nicOps_ = &g_BuiltinChannelOps; ch->nicCtx_ = ch; │ | ||
| 103 | +│ Plugin: ch->nicOps_ = pluginOps; ch->nicCtx_ = pluginCtx │ | ||
| 104 | +├────────────────────────────────────────── ───────────────────────┤ | ||
| 105 | +│ g_BuiltinChannelOps (Global Ops Table) │ | ||
| 106 | +│ .writeNbi = ctx → ((Channel*)ctx)->Write(...) │ | ||
| 107 | +└──────────────────────────────────────────────────────────────────┘ | ||
| 108 | +``` | ||
| 109 | + | ||
| 110 | +### 4.4 Logical View | ||
| 111 | + | ||
| 112 | +```mermaid | ||
| 113 | +graph TB | ||
| 114 | + subgraph HCOMM["libhcomm.so"] | ||
| 115 | + API["Public C API<br/>HcommWriteNbi / HcommReadNbi / ..."] | ||
| 116 | + ChBase["class Channel<br/>+ nicOps_: HcommNicChannelOps*<br/>+ nicCtx_: void*<br/>12 pure virtual methods (unchanged)"] | ||
| 117 | + EpBase["class Endpoint<br/>+ nicOps_: HcommNicEndpointOps*<br/>+ nicCtx_: void*<br/>6+ pure virtual methods (unchanged)"] | ||
| 118 | + | ||
| 119 | + subgraph BUILTIN["Built-in Path"] | ||
| 120 | + kB["g_BuiltinChannelOps<br/>g_BuiltinEndpointOps<br/>Global Ops Table<br/>ctx=this → virtual methods"] | ||
| 121 | + BuiltinCh["Built-in Channel Subclasses<br/>HostCpuRoceChannel, etc.<br/>(10+ subclasses, zero modifications)"] | ||
| 122 | + BuiltinEp["Built-in Endpoint Subclasses<br/>CpuRoceEndpoint, etc.<br/>(zero modifications)"] | ||
| 123 | + end | ||
| 124 | + | ||
| 125 | + Loader["NicPluginLoader<br/>FindHostNicPlugin(protocol)<br/>map(Protocol → Entry*)"] | ||
| 126 | + Holder["PluginChannelHolder<br/>PluginEndpointHolder<br/>(placeholder subclasses, only carry ops table)"] | ||
| 127 | + end | ||
| 128 | + | ||
| 129 | + subgraph SO["Plugin .so (Independently Compiled)"] | ||
| 130 | + Exports["3 Exported Symbols<br/>GetInfo() / CreateEndpoint() / CreateChannel()"] | ||
| 131 | + PluginOps["HcommNicChannelOps kMyOps<br/>HcommNicEndpointOps kMyOps<br/>(C function pointer tables implemented by the plugin)"] | ||
| 132 | + Impl["Data Plane I/O<br/>ibv_post_send / urma ops / ..."] | ||
| 133 | + end | ||
| 134 | + | ||
| 135 | + API -->|"ch->nicOps_->writeNbi(ch->nicCtx_, ...)"| ChBase | ||
| 136 | + API -->|"ep->nicOps_->registerMemory(ep->nicCtx_, ...)"| EpBase | ||
| 137 | + | ||
| 138 | + ChBase -->|"Built-in: SetNicChannelCtx(&g_BuiltinChannelOps, this)"| kB | ||
| 139 | + ChBase -.->|"Plugin: SetNicChannelCtx(pluginOps, pluginCtx)"| PluginOps | ||
| 140 | + EpBase -->|"Built-in: SetNicEndpointCtx(&g_BuiltinEndpointOps, this)"| kB | ||
| 141 | + EpBase -.->|"Plugin: SetNicEndpointCtx(pluginOps, pluginCtx)"| PluginOps | ||
| 142 | + | ||
| 143 | + kB -.->|"ctx=this → Write() and other virtual methods"| BuiltinCh | ||
| 144 | + kB -.->|"ctx=this → virtual methods"| BuiltinEp | ||
| 145 | + | ||
| 146 | + PluginOps -->|"Function pointer binding"| Impl | ||
| 147 | + | ||
| 148 | + Loader -->|"dlopen + dlsym (runtime)"| Exports | ||
| 149 | + Exports -->|"Return"| PluginOps | ||
| 150 | + ChBase --- Holder | ||
| 151 | + EpBase --- Holder | ||
| 152 | + | ||
| 153 | + HCOMM -.-|"Compile-time: include hcomm_nic_plugin.h<br/>(struct definitions)"| SO | ||
| 154 | + | ||
| 155 | + style HCOMM fill:none,stroke:#01579b,stroke-width:2px | ||
| 156 | + style SO fill:none,stroke:#bf360c,stroke-width:2px,stroke-dasharray: 8 4 | ||
| 157 | + style API fill:#e1f5fe,stroke:#01579b | ||
| 158 | + style BUILTIN fill:#c8e6c9,stroke:#2e7d32 | ||
| 159 | + style Loader fill:#b2dfdb,stroke:#00695c | ||
| 160 | + style Exports fill:#ffe0b2,stroke:#e65100 | ||
| 161 | + style PluginOps fill:#ffccbc,stroke:#bf360c | ||
| 162 | + style Impl fill:#fff3e0,stroke:#ef6c00 | ||
| 163 | +``` | ||
| 164 | + | ||
| 165 | +**Connection Description**: | ||
| 166 | + | ||
| 167 | +| Line Type | Meaning | | ||
| 168 | +|-----------|---------| | ||
| 169 | +| Solid arrow `-->` | Data dispatch flow / strong dependency | | ||
| 170 | +| Dashed arrow `-.->` | Injection at creation time / internal forwarding | | ||
| 171 | +| Dotted line `-.-` | Compile-time dependency (plugin .so includes `hcomm_nic_plugin.h`) | | ||
| 172 | + | ||
| 173 | +**Core Relationships**: | ||
| 174 | + | ||
| 175 | +| Relationship | Direction | Description | | ||
| 176 | +|---------------|-----------|-------------| | ||
| 177 | +| API → Base Class | Dispatch | All `HcommXxx` functions uniformly go through `ch->nicOps_->xxx(ch->nicCtx_, ...) or ep->nicOps_->xxx(ep->nicCtx_, ...)` | | ||
| 178 | +| Base Class → Ops Table | Hold | `nicOps_` points to `g_BuiltinXxxOps` (built-in) or `pluginOps` (plugin), mutually exclusive | | ||
| 179 | +| Built-in Ops → Subclass | Forward | Built-in wrapper function, `ctx = Channel*` → `Write()` virtual method | | ||
| 180 | +| Plugin Ops → Implementation | Bind | Function pointers are bound to plugin data-plane I/O logic | | ||
| 181 | +| Loader → SO Symbols | Load | `dlopen` + `dlsym` to obtain 3 exported function pointers | | ||
| 182 | +| SO Symbols → Ops Table | Return | `CreateEndpoint/CreateChannel` return the ops table populated by the plugin | | ||
| 183 | + | ||
| 184 | +#### 4.4.1 Module Responsibilities and Relationships | ||
| 185 | + | ||
| 186 | +| Side | Module | Responsibility | Relationship | | ||
| 187 | +|------|--------|---------------|--------------| | ||
| 188 | +| **libhcomm.so** | Public C API Layer | `include/` external interface declarations, unchanged | Caller entry point, directly converted to ops table calls | | ||
| 189 | +| **libhcomm.so** | C Ops Table Definition | `hcomm_nic_plugin.h`, defines `HcommNicChannelOps` / `HcommNicEndpointOps` structs | Serves as both the built-in ops table type for libhcomm.so and the only header file dependency for plugin .so compilation | | ||
| 190 | +| **libhcomm.so** | Channel / Endpoint Base Classes | `nicOps_` + `nicCtx_` pointer fields + `SetNic***Ctx()` setter | The ops table pointer is populated at creation time; `nicOps_` is immutable once set. The plugin-side ctx is opaque | | ||
| 191 | +| **libhcomm.so** | Built-in Ops Table | `g_BuiltinChannelOps` / `g_BuiltinEndpointOps` global tables | Shared by all built-in subclasses; `ctx = this`; wrapper functions forward to virtual methods | | ||
| 192 | +| **libhcomm.so** | Channel / Endpoint Subclasses | 10+ built-in subclasses require zero modifications; new `PluginXxxHolder` placeholder subclasses are added | Placeholder subclasses only carry the ops table and are stored in the global Map; data-plane logic is fully delegated to the ops table | | ||
| 193 | +| **libhcomm.so** | Plugin Framework Layer | `NicPluginLoader`: loads .so, validates, and maps protocols | Serves only the creation path; no coupling with the Map | | ||
| 194 | +| **Plugin .so** | Exported Symbols | `GetInfo()` / `CreateEndpoint()` / `CreateChannel()` — 3 C symbols | Discovered and invoked by libhcomm.so through `dlopen`/`dlsym` at runtime | | ||
| 195 | +| **Plugin .so** | Plugin Ops Table | `HcommNicChannelOps` / `HcommNicEndpointOps` populated by the plugin | During Endpoint/Channel creation, the framework fills the plugin's ops table pointer into the base class `nicOps_` field; subsequent dispatch calls go directly through this pointer | | ||
| 196 | +| **Plugin .so** | Plugin Data-Plane Implementation | Actual I/O logic for RDMA write/read/notify and so on | Ops table function pointers are bound to specific implementations (such as `ibv_post_send`), running in the caller's thread context | | ||
| 197 | + | ||
| 198 | +#### 4.4.2 Boundaries and Dependencies | ||
| 199 | + | ||
| 200 | +```text | ||
| 201 | +libhcomm.so Plugin .so | ||
| 202 | +┌──────────────────────┐ ┌──────────────────────────┐ | ||
| 203 | +│ │ dlopen/dlsym (runtime) │ | ||
| 204 | +│ NicPluginLoader │────────────────▶│ 3 Exported Symbols │ | ||
| 205 | +│ │ │ │ | ||
| 206 | +│ g_EndpointMap │ Holds ops │ │ | ||
| 207 | +│ g_ChannelMap ─────│──▶ PluginHolder──│▶ Ops Table (func ptrs) │ | ||
| 208 | +│ │ (placeholder │ ├─ registerMemory │ | ||
| 209 | +│ │ subclass) │ ├─ writeNbi │ | ||
| 210 | +│ │ │ ├─ readNbi │ | ||
| 211 | +│ │ │ └─ ... │ | ||
| 212 | +│ │ │ │ │ | ||
| 213 | +│ g_BuiltinXxxOps │ Statically │ ▼ │ | ||
| 214 | +│ (also implements │ populated at │ Data Plane I/O │ | ||
| 215 | +│ the same ops table) │ compile time │ ibv_post_send, etc. │ | ||
| 216 | +│ │ │ │ | ||
| 217 | +│ ← Compile-time dep. ─│── hcomm_nic_plugin.h ──│▶ Compile-time incl.│ | ||
| 218 | +│ (shared struct defs) │ │ │ | ||
| 219 | +└──────────────────────┘ └──────────────────────────┘ | ||
| 220 | + | ||
| 221 | +Key Boundaries: | ||
| 222 | + ▸ The plugin .so depends only on hcomm_nic_plugin.h (pure C struct definitions) at compile time and does not link libhcomm.so | ||
| 223 | + ▸ libhcomm.so loads the plugin .so at runtime through dlopen and obtains function pointers through dlsym | ||
| 224 | + ▸ Both sides interoperate through the same HcommNicChannelOps / HcommNicEndpointOps table | ||
| 225 | + ▸ The plugin ctx is completely opaque to libhcomm.so — libhcomm.so only passes the nicCtx_ pointer without parsing its contents | ||
| 226 | + ▸ g_Builtin***Ops in libhcomm.so and the plugin ops table are mutually exclusive: a Channel/Endpoint object holds only one of them | ||
| 227 | +``` | ||
| 228 | + | ||
| 229 | +### 4.5 Overall Module Sequence | ||
| 230 | + | ||
| 231 | +```mermaid | ||
| 232 | +sequenceDiagram | ||
| 233 | + participant App as Upper-Layer Application | ||
| 234 | + participant API as HCOMM C API | ||
| 235 | + participant Base as Channel/Endpoint Base Class | ||
| 236 | + participant Ops as C Ops Table | ||
| 237 | + participant Builtin as Built-in Implementation | ||
| 238 | + participant Mgr as NicPluginManager | ||
| 239 | + participant SO as Plugin .so | ||
| 240 | + | ||
| 241 | + Note over App,SO: ═══ Creation Phase: Endpoint / Channel ═══ | ||
| 242 | + | ||
| 243 | + App->>API: HcommEndpointCreate(&desc, &handle) | ||
| 244 | + API->>Mgr: std::call_once triggers LoadAllNicPlugins() | ||
| 245 | + Mgr->>SO: dlopen + dlsym (GetInfo / CreateEndpoint / CreateChannel) | ||
| 246 | + SO-->>Mgr: Plugin metadata + creation function pointers | ||
| 247 | + Mgr->>Mgr: Validate + register into ProtocolPlugins map | ||
| 248 | + | ||
| 249 | + API->>Mgr: FindHostNicPlugin(desc.protocol) | ||
| 250 | + | ||
| 251 | + alt Plugin Hit | ||
| 252 | + Mgr-->>API: entry | ||
| 253 | + API->>SO: createEndpoint(&desc, &ctx, &ops) | ||
| 254 | + SO-->>API: pluginCtx + EndpointOps | ||
| 255 | + API->>Base: ep->SetNicEndpointCtx(pluginOps, pluginCtx) | ||
| 256 | + else Not Hit | ||
| 257 | + Mgr-->>API: null | ||
| 258 | + API->>Builtin: Endpoint::CreateEndpoint(desc) | ||
| 259 | + API->>Base: ep->SetNicEndpointCtx(&g_BuiltinEndpointOps, ep) | ||
| 260 | + end | ||
| 261 | + | ||
| 262 | + App->>API: HcommChannelCreate(epHandle, ...) | ||
| 263 | + API->>Base: Is ep->nicOps_ a plugin ops? | ||
| 264 | + | ||
| 265 | + alt Plugin Endpoint | ||
| 266 | + API->>SO: createChannel(epCtx, &desc, &chCtx, &chOps) | ||
| 267 | + SO-->>API: chCtx + ChannelOps | ||
| 268 | + API->>Base: ch->SetNicChannelCtx(chOps, chCtx) | ||
| 269 | + else Built-in Endpoint | ||
| 270 | + API->>Builtin: Channel::CreateChannel factory | ||
| 271 | + API->>Base: ch->SetNicChannelCtx(&g_BuiltinChannelOps, ch) | ||
| 272 | + end | ||
| 273 | + | ||
| 274 | + Note over App,SO: ═══ Invocation Phase: Zero-Branch Data-Plane Dispatch ═══ | ||
| 275 | + | ||
| 276 | + App->>API: HcommWriteNbi(ch, dst, src, len) | ||
| 277 | + API->>Base: ch = (Channel*)ch | ||
| 278 | + API->>Ops: ch->nicOps_->writeNbi(ch->nicCtx_, dst, src, len) | ||
| 279 | + | ||
| 280 | + alt Built-in Channel | ||
| 281 | + Ops->>Builtin: BuiltinWriteNbi(ctx, ...) → Channel::Write(...) | ||
| 282 | + Builtin-->>Ops: ret | ||
| 283 | + else Plugin Channel | ||
| 284 | + Ops->>SO: pluginWriteNbi(pluginCtx, ...) | ||
| 285 | + SO-->>Ops: ret | ||
| 286 | + end | ||
| 287 | + | ||
| 288 | + Ops-->>API: ret | ||
| 289 | + API-->>App: ret | ||
| 290 | + | ||
| 291 | + Note over API,SO: Zero-branch data-plane dispatch applies only to 950-series exclusive interfaces; non-950-series exclusive interfaces use the legacy path | ||
| 292 | +``` | ||
| 293 | + | ||
| 294 | +## 5. Detailed Design | ||
| 295 | + | ||
| 296 | +### 5.1 Core Data Structures | ||
| 297 | + | ||
| 298 | +#### 5.1.1 C ABI Interface Specification | ||
| 299 | + | ||
| 300 | +##### Version Number and Metadata | ||
| 301 | + | ||
| 302 | +```c | ||
| 303 | +#define HCOMM_NIC_PLUGIN_INFO_VERSION 1U | ||
| 304 | +#define HCOMM_NIC_CHANNEL_OPS_VERSION 1U | ||
| 305 | +#define HCOMM_NIC_ENDPOINT_OPS_VERSION 1U | ||
| 306 | + | ||
| 307 | +typedef struct { | ||
| 308 | + CommAbiHeader header; // version / magic / size / reserved | ||
| 309 | + const char *name; // plugin name | ||
| 310 | + uint32_t protocolCount; // number of protocols | ||
| 311 | + CommProtocol protocols[HCOMM_NIC_PLUGIN_MAX_PROTOCOLS]; // up to 4 | ||
| 312 | + uint64_t reserved[8]; // reserved for extension | ||
| 313 | +} HcommNicPluginInfo; | ||
| 314 | +``` | ||
| 315 | + | ||
| 316 | +**Validation Rules**: | ||
| 317 | + | ||
| 318 | +- `header.magicWord` must equal `HCOMM_NIC_PLUGIN_INFO_MAGIC_WORD` | ||
| 319 | +- `header.size` is rejected when it is less than `offsetof(HcommNicPluginInfo, protocols) + sizeof(protocols)` | ||
| 320 | +- Each value in `protocols[]` must be either a non-RESERVED built-in enumeration value < `COMM_PROTOCOL_CUSTOM_BASE`, or a value ≥ `COMM_PROTOCOL_CUSTOM_BASE` | ||
| 321 | +- COMM_PROTOCOL_CUSTOM_BASE=1000 is defined in nic_plugin_manager.h and serves as the boundary constant for plugin protocol numbers. The CommProtocol field carries custom values ≥1000 with int32_t semantics. | ||
| 322 | + | ||
| 323 | +##### Channel Ops Table | ||
| 324 | + | ||
| 325 | +```c | ||
| 326 | +typedef struct { | ||
| 327 | + CommAbiHeader header; | ||
| 328 | + int32_t (*init)(void* ctx); | ||
| 329 | + int32_t (*destroy)(void* ctx); | ||
| 330 | + | ||
| 331 | + int32_t (*getStatus)(void* ctx, int32_t* status); | ||
| 332 | + | ||
| 333 | + int32_t (*writeNbi)(void* ctx, void* dst, const void* src, uint64_t len); | ||
| 334 | + int32_t (*writeNbiOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len); | ||
| 335 | + int32_t (*writeOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len); | ||
| 336 | + int32_t (*writeWithNotifyNbi)(void* ctx, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx); | ||
| 337 | + int32_t (*writeWithNotifyNbiOnThread)( | ||
| 338 | + void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx); | ||
| 339 | + int32_t (*writeWithNotifyOnThread)( | ||
| 340 | + void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx); | ||
| 341 | + int32_t (*writeReduceOnThread)( | ||
| 342 | + void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t count, HcommDataType dataType, | ||
| 343 | + HcommReduceOp reduceOp); | ||
| 344 | + int32_t (*writeReduceWithNotifyOnThread)( | ||
| 345 | + void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t count, HcommDataType dataType, | ||
| 346 | + HcommReduceOp reduceOp, uint32_t remoteNotifyIdx); | ||
| 347 | + | ||
| 348 | + int32_t (*readNbi)(void* ctx, void* dst, const void* src, uint64_t len); | ||
| 349 | + int32_t (*readNbiOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len); | ||
| 350 | + int32_t (*readOnThread)(void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len); | ||
| 351 | + int32_t (*readReduceOnThread)( | ||
| 352 | + void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t count, HcommDataType dataType, | ||
| 353 | + HcommReduceOp reduceOp); | ||
| 354 | + | ||
| 355 | + int32_t (*notifyRecord)(void* ctx, uint32_t remoteNotifyIdx); | ||
| 356 | + int32_t (*notifyRecordOnThread)(void* ctx, ThreadHandle thread, uint32_t remoteNotifyIdx); | ||
| 357 | + int32_t (*notifyWait)(void* ctx, uint32_t localNotifyIdx, uint32_t timeOut); | ||
| 358 | + int32_t (*notifyWaitOnThread)(void* ctx, ThreadHandle thread, uint32_t localNotifyIdx, uint32_t timeOut); | ||
| 359 | + int32_t (*notifyWaitOnThreadWithDefaultTimeout)(void* ctx, ThreadHandle thread, uint32_t localNotifyIdx); | ||
| 360 | + | ||
| 361 | + int32_t (*batchTransferOnThread)( | ||
| 362 | + void* ctx, ThreadHandle thread, const HcommBatchTransferDesc* transferDescs, uint32_t transferDescNum); | ||
| 363 | + | ||
| 364 | + int32_t (*fence)(void* ctx); | ||
| 365 | + int32_t (*fenceOnThread)(void* ctx, ThreadHandle thread); | ||
| 366 | + int32_t (*drainOnThread)(void* ctx, ThreadHandle thread); | ||
| 367 | +} HcommNicChannelOps; | ||
| 368 | +``` | ||
| 369 | + | ||
| 370 | +**Required Interface for Plugins**: The plugin must implement the destroy interface to release the channel object created by the plugin. | ||
| 371 | + | ||
| 372 | +**API Mapping**: | ||
| 373 | + | ||
| 374 | +| Ops Entry | Corresponding Public API | Description | | ||
| 375 | +| ---------------------------------------- | ------------------------------------------------ | ----------- | | ||
| 376 | +| `destroy` | `HcommChannelDestroy` | The public API is a batch interface; the framework calls the corresponding ops entry for each channel | | ||
| 377 | +| `getStatus` | `HcommChannelGetStatus` | The public API is a batch interface; the framework calls the corresponding ops entry for each channel | | ||
| 378 | +| `writeNbi` | `HcommWriteNbi` | | | ||
| 379 | +| `writeNbiOnThread` | `HcommWriteNbiOnThread` | | | ||
| 380 | +| `writeOnThread` | `HcommWriteOnThread` | | | ||
| 381 | +| `writeWithNotifyNbi` | `HcommWriteWithNotifyNbi` | | | ||
| 382 | +| `writeWithNotifyNbiOnThread` | `HcommWriteWithNotifyNbiOnThread` | | | ||
| 383 | +| `writeWithNotifyOnThread` | `HcommWriteWithNotifyOnThread` | | | ||
| 384 | +| `writeReduceOnThread` | `HcommWriteReduceOnThread` | | | ||
| 385 | +| `writeReduceWithNotifyOnThread` | `HcommWriteReduceWithNotifyOnThread` | | | ||
| 386 | +| `readNbi` | `HcommReadNbi` | | | ||
| 387 | +| `readNbiOnThread` | `HcommReadNbiOnThread` | | | ||
| 388 | +| `readOnThread` | `HcommReadOnThread` | | | ||
| 389 | +| `readReduceOnThread` | `HcommReadReduceOnThread` | | | ||
| 390 | +| `notifyRecord` | `HcommNotifyRecord` | | | ||
| 391 | +| `notifyRecordOnThread` | `HcommNotifyRecordOnThread` | | | ||
| 392 | +| `notifyWait` | `HcommNotifyWait` | | | ||
| 393 | +| `notifyWaitOnThread` | `HcommNotifyWaitOnThread` | | | ||
| 394 | +| `notifyWaitOnThreadWithDefaultTimeout` | `HcommNotifyWaitOnThreadWithDefaultTimeout` | | | ||
| 395 | +| `batchTransferOnThread` | `HcommBatchTransferOnThread` | | | ||
| 396 | +| `fence` | `HcommFence` | | | ||
| 397 | +| `fenceOnThread` | `HcommFenceOnThread` | | | ||
| 398 | +| `drainOnThread` | `HcommChannelDrainOnThread` | | | ||
| 399 | + | ||
| 400 | +##### Endpoint Ops Table | ||
| 401 | + | ||
| 402 | +```c | ||
| 403 | +typedef struct { | ||
| 404 | + CommAbiHeader header; | ||
| 405 | + int32_t (*init)(void* ctx); | ||
| 406 | + int32_t (*destroy)(void* ctx); | ||
| 407 | + | ||
| 408 | + int32_t (*registerMemory)(void* ctx, const CommMem* mem, const char* tag, void** handle); | ||
| 409 | + int32_t (*unregisterMemory)(void* ctx, void* handle); | ||
| 410 | + int32_t (*memoryExport)(void* ctx, void* handle, void** desc, uint32_t* descLen); | ||
| 411 | + int32_t (*memoryImport)(void* ctx, const void* desc, uint32_t descLen, CommMem* outMem); | ||
| 412 | + int32_t (*memoryUnimport)(void* ctx, const void* desc, uint32_t descLen); | ||
| 413 | + int32_t (*getListenPort)(void* ctx, uint32_t* port); | ||
| 414 | +} HcommNicEndpointOps; | ||
| 415 | +``` | ||
| 416 | + | ||
| 417 | +**API Mapping**: | ||
| 418 | + | ||
| 419 | +| Ops Entry | Corresponding Public API | | ||
| 420 | +|-----------|-------------------------| | ||
| 421 | +| `destroy` | `HcommEndpointDestroy` | | ||
| 422 | +| `registerMemory` | `HcommMemReg` | | ||
| 423 | +| `unregisterMemory` | `HcommMemUnreg` | | ||
| 424 | +| `memoryExport` | `HcommMemExport` | | ||
| 425 | +| `memoryImport` | `HcommMemImport` | | ||
| 426 | +| `memoryUnimport` | `HcommMemUnimport` | | ||
| 427 | +| `getListenPort` | `HcommEndpointGetListenPort` | | ||
| 428 | + | ||
| 429 | +**Required Interface for Plugins**: The plugin must implement the destroy interface to release the endpoint object created by the plugin. | ||
| 430 | + | ||
| 431 | +##### Exported Symbols | ||
| 432 | + | ||
| 433 | +```c | ||
| 434 | +typedef const HcommNicPluginInfo *(*HcommNicPluginGetInfoFunc)(void); | ||
| 435 | + | ||
| 436 | +typedef int32_t (*HcommNicPluginCreateEndpointFunc)( | ||
| 437 | + const EndpointDesc *endpointDesc, | ||
| 438 | + void **outCtx, HcommNicEndpointOps **outOps); | ||
| 439 | + | ||
| 440 | +typedef int32_t (*HcommNicPluginCreateChannelFunc)( | ||
| 441 | + void *epCtx, const HcommChannelDesc *channelDesc, | ||
| 442 | + void **outCtx, HcommNicChannelOps **outOps); | ||
| 443 | + | ||
| 444 | +// The 3 C symbols that the plugin must export | ||
| 445 | +const HcommNicPluginInfo *HcommNicPluginGetInfo(void); | ||
| 446 | +int32_t HcommNicPluginCreateEndpoint(const EndpointDesc *desc, | ||
| 447 | + void **outCtx, HcommNicEndpointOps **outOps); | ||
| 448 | +int32_t HcommNicPluginCreateChannel(void *epCtx, const HcommChannelDesc *desc, | ||
| 449 | + void **outCtx, HcommNicChannelOps **outOps); | ||
| 450 | +``` | ||
| 451 | + | ||
| 452 | +#### 5.1.2 New Fields in the Channel Base Class | ||
| 453 | + | ||
| 454 | +**File**: `src/base_comm/resources/endpoint_pairs/channels/channel.h` | ||
| 455 | + | ||
| 456 | +```cpp | ||
| 457 | +class Channel { | ||
| 458 | +protected: | ||
| 459 | + // ==== New nicOps_ and nicCtx_ ==== | ||
| 460 | + HcommNicChannelOps* nicOps_{nullptr}; | ||
| 461 | + void* nicCtx_{nullptr}; | ||
| 462 | +public: | ||
| 463 | + // ==== New set/get methods ==== | ||
| 464 | + void SetNicChannelCtx(HcommNicChannelOps* nicOps, void* nicCtx) | ||
| 465 | + { | ||
| 466 | + nicOps_ = nicOps; | ||
| 467 | + nicCtx_ = nicCtx; | ||
| 468 | + } | ||
| 469 | + HcommNicChannelOps* GetNicOps() const { return nicOps_; } | ||
| 470 | + void* GetNicCtx() const { return nicCtx_; } | ||
| 471 | +}; | ||
| 472 | +``` | ||
| 473 | + | ||
| 474 | +**Boundary Constraints**: | ||
| 475 | + | ||
| 476 | +- `nicOps_` is set once at creation time and is read-only at runtime; it is immutable | ||
| 477 | +- `nicCtx_` is `this` for built-in, and a private opaque context allocated by the plugin for plugin | ||
| 478 | +- The new fields are located at the end of the class with a default value of `nullptr`; the memory layout of existing subclasses remains unchanged | ||
| 479 | +- The new interfaces are non-virtual and do not affect the vtable | ||
| 480 | + | ||
| 481 | +#### 5.1.3 New Fields in the Endpoint Base Class | ||
| 482 | + | ||
| 483 | +**File**: `src/base_comm/resources/endpoints/endpoint.h` (same pattern as Channel) | ||
| 484 | + | ||
| 485 | +```cpp | ||
| 486 | +class Endpoint { | ||
| 487 | +protected: | ||
| 488 | + // ==== New nicOps_ and nicCtx_ ==== | ||
| 489 | + HcommNicEndpointOps* nicOps_{nullptr}; | ||
| 490 | + void* nicCtx_{nullptr}; | ||
| 491 | +public: | ||
| 492 | + // ==== New set/get methods ==== | ||
| 493 | + void SetNicEndpointCtx(HcommNicEndpointOps* nicOps, void* nicCtx) | ||
| 494 | + { | ||
| 495 | + nicOps_ = nicOps; | ||
| 496 | + nicCtx_ = nicCtx; | ||
| 497 | + } | ||
| 498 | + HcommNicEndpointOps* GetNicOps() const { return nicOps_; } | ||
| 499 | + void* GetNicCtx() const { return nicCtx_; } | ||
| 500 | +}; | ||
| 501 | +``` | ||
| 502 | + | ||
| 503 | +#### 5.1.4 Plugin Registry and Plugin Subclasses | ||
| 504 | + | ||
| 505 | +```cpp | ||
| 506 | +struct NicPluginEntry { | ||
| 507 | + void *soHandle; // dlopen handle | ||
| 508 | + const HcommNicPluginInfo *info; // plugin metadata | ||
| 509 | + HcommNicPluginCreateEndpointFunc createEndpoint; // V1 creation function | ||
| 510 | + HcommNicPluginCreateChannelFunc createChannel; // V1 creation function | ||
| 511 | +}; | ||
| 512 | + | ||
| 513 | +// Global singleton, protocol number → plugin entry | ||
| 514 | +std::unordered_map<CommProtocol, const NicPluginEntry *> &ProtocolPlugins(); | ||
| 515 | + | ||
| 516 | +// Placeholder subclasses (new): pure virtual method implementations return NOT_SUPPORT (data plane does not go through vtable; the dispatch layer uses the ops table); used only to carry object lifetime in the Map | ||
| 517 | +class PluginEndpointHolder : public Endpoint { | ||
| 518 | + explicit PluginEndpointHolder(const EndpointDesc& endpointDesc, const NicPluginEntry* pluginEntry) | ||
| 519 | + : Endpoint(endpointDesc), | ||
| 520 | + pluginEntry_(pluginEntry) | ||
| 521 | + {} | ||
| 522 | + ~PluginEndpointHolder() override { DestroyNicPluginOpsAndCtx(nicOps_, nicCtx_); } | ||
| 523 | + | ||
| 524 | + const NicPluginEntry* GetPluginEntry() const { return pluginEntry_; } | ||
| 525 | + // Pure virtual method implementation returns NOT_SUPPORT | ||
| 526 | + HcclResult RegisterMemory(HcommMem mem, const char* memTag, void** memHandle) override | ||
| 527 | + { | ||
| 528 | + (void)mem; | ||
| 529 | + (void)memTag; | ||
| 530 | + (void)memHandle; | ||
| 531 | + return HCCL_E_NOT_SUPPORT; | ||
| 532 | + } | ||
| 533 | +}; | ||
| 534 | +class PluginChannelHolder : public Channel { | ||
| 535 | + explicit PluginChannelHolder(const NicPluginEntry* pluginEntry) : pluginEntry_(pluginEntry) {} | ||
| 536 | + ~PluginChannelHolder() override { DestroyNicPluginOpsAndCtx(nicOps_, nicCtx_); } | ||
| 537 | + | ||
| 538 | + const NicPluginEntry* GetPluginEntry() const { return pluginEntry_; } | ||
| 539 | + // Pure virtual method implementation returns NOT_SUPPORT | ||
| 540 | + HcclResult Write(void* dst, const void* src, uint64_t len) override | ||
| 541 | + { | ||
| 542 | + (void)dst; | ||
| 543 | + (void)src; | ||
| 544 | + (void)len; | ||
| 545 | + return HCCL_E_NOT_SUPPORT; | ||
| 546 | + } | ||
| 547 | +}; | ||
| 548 | + | ||
| 549 | +// When the plugin placeholder subclass PluginEndpointHolder/PluginChannelHolder is released, call the plugin destroy function to destroy the plugin nicCtx object | ||
| 550 | +template <typename Ops> | ||
| 551 | +void DestroyNicPluginOpsAndCtx(Ops*& nicOps, void* nicCtx) | ||
| 552 | +{ | ||
| 553 | + if (nicOps != nullptr) { | ||
| 554 | + if (nicOps->destroy != nullptr) { | ||
| 555 | + int32_t ret = nicOps->destroy(nicCtx); | ||
| 556 | + if (ret != HCCL_SUCCESS) { | ||
| 557 | + HCCL_WARNING("[%s] plugin destroy failed, ret[%d].", __func__, ret); | ||
| 558 | + } | ||
| 559 | + } | ||
| 560 | + delete nicOps; | ||
| 561 | + nicOps = nullptr; | ||
| 562 | + } | ||
| 563 | +} | ||
| 564 | +``` | ||
| 565 | + | ||
| 566 | +### 5.2 Built-in Ops Table Implementation | ||
| 567 | + | ||
| 568 | +#### 5.2.1 g_BuiltinChannelOps | ||
| 569 | + | ||
| 570 | +**File**: `src/base_comm/resources/endpoint_pairs/channels/builtin_channel_ops.h` | ||
| 571 | + | ||
| 572 | +```c | ||
| 573 | +// Built-in implementation wrapper function | ||
| 574 | +inline int32_t BuiltinWriteWithNotifyNbiOnThread( | ||
| 575 | + void* ctx, ThreadHandle thread, void* dst, const void* src, uint64_t len, uint32_t remoteNotifyIdx) | ||
| 576 | +{ | ||
| 577 | + HCCL_INFO( | ||
| 578 | + "[%s] START. thread[0x%llx], channel[0x%llx], dst[0x%llx], src[0x%llx], len[%llu], remoteNotifyIdx[%u].", | ||
| 579 | + __func__, thread, ctx, dst, src, len, remoteNotifyIdx); | ||
| 580 | + | ||
| 581 | + (void)thread; | ||
| 582 | + CHK_PTR_NULL(src); | ||
| 583 | + CHK_PTR_NULL(dst); | ||
| 584 | + HcclResult ret = HCCL_SUCCESS; | ||
| 585 | + DevType devType; | ||
| 586 | + CHK_RET(hrtGetDeviceType(devType)); | ||
| 587 | + if (devType == DevType::DEV_TYPE_950 || devType == DevType::DEV_TYPE_960 || thread == 0) { | ||
| 588 | + auto* const channelPtr = reinterpret_cast<hcomm::Channel*>(ctx); | ||
| 589 | + CHK_PTR_NULL(channelPtr); | ||
| 590 | + ret = channelPtr->WriteWithNotify(dst, src, len, remoteNotifyIdx); | ||
| 591 | + } else { | ||
| 592 | + ret = HCCL_E_NOT_SUPPORT; | ||
| 593 | + } | ||
| 594 | + CHK_PRT_RET( | ||
| 595 | + ret != HCCL_SUCCESS, | ||
| 596 | + HCCL_ERROR( | ||
| 597 | + "[%s] FAIL. thread[0x%llx], channel[0x%llx], dst[0x%llx], src[0x%llx], len[%llu], remoteNotifyIdx[%u].", | ||
| 598 | + __func__, thread, ctx, dst, src, len, remoteNotifyIdx), | ||
| 599 | + ret); | ||
| 600 | + HCCL_INFO("[%s] SUCCESS.", __func__); | ||
| 601 | + return HCCL_SUCCESS; | ||
| 602 | +} | ||
| 603 | +// ... Other interfaces for built-in implementation follow the same pattern | ||
| 604 | + | ||
| 605 | +inline HcommNicChannelOps g_BuiltinChannelOps = { | ||
| 606 | + {HCOMM_NIC_CHANNEL_OPS_VERSION, HCOMM_NIC_CHANNEL_OPS_MAGIC_WORD, sizeof(HcommNicChannelOps), 0}, | ||
| 607 | + BuiltinChannelInit, // init | ||
| 608 | + BuiltinChannelDestroy, // destroy | ||
| 609 | + BuiltinGetStatus, // getStatus | ||
| 610 | + BuiltinWriteNbi, // writeNbi | ||
| 611 | + BuiltinWriteNbiOnThread, // writeNbiOnThread | ||
| 612 | + hcomm::DefaultChannelWriteOnThread, // writeOnThread | ||
| 613 | + BuiltinWriteWithNotifyNbi, // writeWithNotifyNbi | ||
| 614 | + BuiltinWriteWithNotifyNbiOnThread, // writeWithNotifyNbiOnThread | ||
| 615 | + hcomm::DefaultChannelWriteWithNotifyOnThread, // writeWithNotifyOnThread | ||
| 616 | + hcomm::DefaultChannelWriteReduceOnThread, // writeReduceOnThread | ||
| 617 | + hcomm::DefaultChannelWriteReduceWithNotifyOnThread, // writeReduceWithNotifyOnThread | ||
| 618 | + BuiltinReadNbi, // readNbi | ||
| 619 | + BuiltinReadNbiOnThread, // readNbiOnThread | ||
| 620 | + hcomm::DefaultChannelReadOnThread, // readOnThread | ||
| 621 | + hcomm::DefaultChannelReadReduceOnThread, // readReduceOnThread | ||
| 622 | + BuiltinNotifyRecord, // notifyRecord | ||
| 623 | + BuiltinNotifyRecordOnThread, // notifyRecordOnThread | ||
| 624 | + BuiltinNotifyWait, // notifyWait | ||
| 625 | + BuiltinNotifyWaitOnThread, // notifyWaitOnThread | ||
| 626 | + hcomm::DefaultChannelNotifyWaitOnThreadWithDefaultTimeout, // notifyWaitOnThreadWithDefaultTimeout | ||
| 627 | + hcomm::DefaultChannelBatchTransferOnThread, // batchTransferOnThread | ||
| 628 | + BuiltinFence, // fence | ||
| 629 | + BuiltinFenceOnThread, // fenceOnThread | ||
| 630 | + hcomm::DefaultChannelDrainOnThread, // drainOnThread | ||
| 631 | +}; | ||
| 632 | +``` | ||
| 633 | + | ||
| 634 | +**Design Points**: | ||
| 635 | + | ||
| 636 | +- 1. The public APIs corresponding to Channel ops entries differ in product support. They are divided into two categories: APIs that support only the 950 series and APIs that also support non-950 series products. For APIs that support only the 950 series, wrapper functions must be implemented. For APIs that also support non-950 series products, unified dispatch is not possible due to flow differences, so wrapper functions are not implemented. For such interfaces, the dispatch point must distinguish between plugin and non-plugin scenarios. | ||
| 637 | +- 2. For interfaces without wrapper functions, default interfaces are populated to avoid null pointer call exceptions at the dispatch point. | ||
| 638 | + | ||
| 639 | +#### 5.2.2 g_BuiltinEndpointOps | ||
| 640 | + | ||
| 641 | +**File**: `src/base_comm/resources/endpoints/builtin_endpoint_ops.h` | ||
| 642 | + | ||
| 643 | +```c | ||
| 644 | +// Built-in implementation wrapper function | ||
| 645 | +inline int32_t BuiltinRegisterMemory(void* ctx, const CommMem* mem, const char* tag, void** handle) | ||
| 646 | +{ | ||
| 647 | + CHK_PTR_NULL(mem); | ||
| 648 | + CHK_PTR_NULL(handle); | ||
| 649 | + EXCEPTION_HANDLE_BEGIN(void) HcommResMgrInit(); | ||
| 650 | + EndpointHandle epHandle = reinterpret_cast<EndpointHandle>(ctx); | ||
| 651 | + HCCL_INFO("[%s] START. endpointHandle[0x%llx].", __func__, epHandle); | ||
| 652 | + auto endpoint = GetEndpointMap().GetEndpoint(epHandle); | ||
| 653 | + CHK_PRT_RET( | ||
| 654 | + endpoint == nullptr, HCCL_ERROR("[%s] endpoint not found, endpointHandle[0x%llx]", __func__, epHandle), | ||
| 655 | + HCCL_E_NOT_FOUND); | ||
| 656 | + CHK_RET(RefreshEndpointContext(endpoint->GetEndpointDesc())); | ||
| 657 | + CHK_RET(endpoint->RegisterMemory(*mem, tag, handle)); | ||
| 658 | + | ||
| 659 | + EXCEPTION_HANDLE_END | ||
| 660 | + return HCCL_SUCCESS; | ||
| 661 | +} | ||
| 662 | +// ... Other interfaces for built-in implementation follow the same pattern | ||
| 663 | + | ||
| 664 | +inline HcommNicEndpointOps g_BuiltinEndpointOps = { | ||
| 665 | + {HCOMM_NIC_ENDPOINT_OPS_VERSION, HCOMM_NIC_ENDPOINT_OPS_MAGIC_WORD, sizeof(HcommNicEndpointOps), 0}, | ||
| 666 | + BuiltinEndpointInit, // init | ||
| 667 | + BuiltinEndpointDestroy, // destroy | ||
| 668 | + BuiltinRegisterMemory, // registerMemory | ||
| 669 | + BuiltinUnregisterMemory, // unregisterMemory | ||
| 670 | + BuiltinMemoryExport, // memoryExport | ||
| 671 | + BuiltinMemoryImport, // memoryImport | ||
| 672 | + BuiltinMemoryUnimport, // memoryUnimport | ||
| 673 | + BuiltinGetListenPort, // getListenPort | ||
| 674 | +}; | ||
| 675 | +``` | ||
| 676 | + | ||
| 677 | +### 5.3 Creation Path | ||
| 678 | + | ||
| 679 | +#### 5.3.1 Endpoint Creation | ||
| 680 | + | ||
| 681 | +```c | ||
| 682 | +HcommResult HcommEndpointCreate(const EndpointDesc *endpoint, EndpointHandle *handle) { | ||
| 683 | +// Shows the distinction between plugin and built-in; context is omitted | ||
| 684 | + ... | ||
| 685 | + if (endpoint->loc.locType == ENDPOINT_LOC_TYPE_HOST) { | ||
| 686 | + const NicPluginEntry* pluginEntry = FindHostNicPlugin(endpoint->protocol); | ||
| 687 | + if (pluginEntry != nullptr) { | ||
| 688 | + return CreatePluginEndpointHolder(endpoint, pluginEntry, endpointHandle); | ||
| 689 | + } | ||
| 690 | + } | ||
| 691 | + ... | ||
| 692 | + CHK_RET(CreateBuiltinEndpoint(endpoint, endpointHandle)); | ||
| 693 | + ... | ||
| 694 | +} | ||
| 695 | +``` | ||
| 696 | + | ||
| 697 | +**Ops Population**: | ||
| 698 | + | ||
| 699 | +- Built-in path: `Endpoint::CreateEndpoint(*endpoint, ep)` → `ep->SetNicEndpointCtx(&g_BuiltinEndpointOps, ep.get())` → store into `g_EndpointMap` | ||
| 700 | +- Plugin path: `pluginEntry->createEndpoint(endpoint, &pluginCtx, &pluginOps)` → validate pluginOps → create `pluginHolderOps`, copy pluginOps and populate unimplemented interfaces → create `PluginEndpointHolder` → `holder->SetNicEndpointCtx(pluginHolderOps, pluginCtx)` → store into `g_EndpointMap` | ||
| 701 | + | ||
| 702 | +#### 5.3.2 Channel Creation | ||
| 703 | + | ||
| 704 | +```c | ||
| 705 | +HcommResult HcommChannelCreate( | ||
| 706 | + EndpointHandle endpointHandle, CommEngine engine, HcommChannelDesc* channelDescs, uint32_t channelNum, | ||
| 707 | + ChannelHandle* channels) | ||
| 708 | +{ | ||
| 709 | + // Shows the distinction between plugin and built-in; context is omitted | ||
| 710 | + ... | ||
| 711 | + auto endpoint = GetEndpointMap().GetEndpoint(endpointHandle); | ||
| 712 | + if (endpoint != nullptr && endpoint->GetNicOps() != nullptr && endpoint->GetNicOps() != &g_BuiltinEndpointOps) { | ||
| 713 | + CHK_RET( | ||
| 714 | + static_cast<HcclResult>(CreatePluginChannels(endpoint, channelDescFinals.data(), channelNum, channels))); | ||
| 715 | + return HCCL_SUCCESS; | ||
| 716 | + } | ||
| 717 | + ... | ||
| 718 | + CHK_RET(ChannelProcess::CreateChannelsLoop( | ||
| 719 | + endpointHandle, engine, channelDescFinals.data(), channelNum, targetChannels)); | ||
| 720 | + CHK_RET( | ||
| 721 | + ChannelProcess::PrepareUserChannels(targetChannels, channels, channelDescFinals.data(), channelNum, engine)); | ||
| 722 | + ... | ||
| 723 | +} | ||
| 724 | +``` | ||
| 725 | + | ||
| 726 | +**Ops Population**: | ||
| 727 | + | ||
| 728 | +- Built-in path: `Channel::CreateChannel` factory → `ch->SetNicChannelCtx(&g_BuiltinChannelOps, ch.get())` → store into `g_ChannelMap` | ||
| 729 | +- Plugin path: `pluginEntry->createChannel(epCtx, channelDesc, &pluginCtx, &pluginOps)` → validate pluginOps → create `pluginHolderOps`, copy pluginOps and populate unimplemented interfaces → create `PluginChannelHolder` → `holder->SetNicChannelCtx(pluginHolderOps, pluginCtx)` → store into `g_ChannelMap` | ||
| 730 | + | ||
| 731 | +#### 5.3.3 Decision Points in the Creation Path | ||
| 732 | + | ||
| 733 | +| Decision Point | Location | Frequency | Method | | ||
| 734 | +|---------------|----------|-----------|--------| | ||
| 735 | +| Whether to enable plugin | `HcommEndpointCreate` | Once per Endpoint creation | `NicPluginEntry* pluginEntry = FindHostNicPlugin(endpoint->protocol)`, whether the plugin supports the corresponding protocol | | ||
| 736 | +| Whether it is a plugin endpoint | `HcommChannelCreate` | Once per Channel creation | `endpoint->GetNicOps() != &g_BuiltinEndpointOps`, whether the endpoint was created by a plugin | | ||
| 737 | + | ||
| 738 | +The creation path is not a hot path, so the branching overhead is negligible. | ||
| 739 | + | ||
| 740 | +### 5.4 Dispatch Path | ||
| 741 | + | ||
| 742 | +> **Core Principle**: Among the public API interfaces corresponding to channel ops entries, interfaces that support non-950 series products require an additional plugin check. Other dispatch functions do not contain any `if/else`, `#ifdef`, or tag-bit checks. | ||
| 743 | + | ||
| 744 | +#### 5.4.1 Channel Dispatch Type 1 | ||
| 745 | + | ||
| 746 | +```c | ||
| 747 | +int32_t HcommWriteNbi(ChannelHandle channel, void* dst, const void* src, uint64_t len) | ||
| 748 | +{ | ||
| 749 | + auto* ch = CHANNEL_FROM_HANDLE(channel); | ||
| 750 | + CHK_PTR_NULL(ch); | ||
| 751 | + // No distinction between plugin and built-in | ||
| 752 | + return ch->GetNicOps()->writeNbi(ch->GetNicCtx(), dst, src, len); | ||
| 753 | +} | ||
| 754 | +``` | ||
| 755 | + | ||
| 756 | +#### 5.4.2 Channel Dispatch Type 2 | ||
| 757 | + | ||
| 758 | +```c | ||
| 759 | +int32_t HcommWriteOnThread(ThreadHandle thread, ChannelHandle channel, void* dst, const void* src, uint64_t len) | ||
| 760 | +{ | ||
| 761 | + if (IS_PLUGIN_HANDLE(channel)) { | ||
| 762 | + // Plugin scenario | ||
| 763 | + auto* ch = CHANNEL_FROM_HANDLE(channel); | ||
| 764 | + CHK_PTR_NULL(ch); | ||
| 765 | + return ch->GetNicOps()->writeOnThread(ch->GetNicCtx(), thread, dst, src, len); | ||
| 766 | + } | ||
| 767 | + // Built-in scenario | ||
| 768 | + ... | ||
| 769 | + HcclResult ret = HcclRemoteWrite(stream, reinterpret_cast<void*>(channel), &rmtBuf, &locBuf); | ||
| 770 | + ... | ||
| 771 | +} | ||
| 772 | +``` | ||
| 773 | + | ||
| 774 | +#### 5.4.3 Endpoint Dispatch | ||
| 775 | + | ||
| 776 | +```c | ||
| 777 | +HcommResult | ||
| 778 | +HcommMemReg(EndpointHandle endpointHandle, const char* memTag, const CommMem* mem, HcommMemHandle* memHandle) | ||
| 779 | +{ | ||
| 780 | + auto endpoint = GetEndpointMap().GetEndpoint(endpointHandle); | ||
| 781 | + CHK_PRT_RET( | ||
| 782 | + endpoint == nullptr, HCCL_ERROR("[%s] endpoint not found, endpointHandle[%p]", __func__, endpointHandle), | ||
| 783 | + HCCL_E_NOT_FOUND); | ||
| 784 | + // No distinction between plugin and built-in | ||
| 785 | + return static_cast<HcclResult>(endpoint->GetNicOps()->registerMemory( | ||
| 786 | + endpoint->GetNicCtx(), mem, memTag, reinterpret_cast<void**>(memHandle))); | ||
| 787 | +} | ||
| 788 | +``` | ||
| 789 | + | ||
| 790 | +### 5.5 Plugin Discovery and Loading | ||
| 791 | + | ||
| 792 | +- **Trigger Timing**: `std::call_once` is executed when the first endpoint is created | ||
| 793 | +- **Default Path**: `opendir($ASCEND_HOME_PATH/hcomm_plugin/)` → each `*.so` | ||
| 794 | + | ||
| 795 | +Loading flow for a single .so: | ||
| 796 | + | ||
| 797 | +```text | ||
| 798 | +dlopen(path, RTLD_NOW) | ||
| 799 | + ├─ Failure → WARNING log, skip | ||
| 800 | + └─ Success → | ||
| 801 | + dlsym("HcommNicPluginGetInfo") | ||
| 802 | + dlsym("HcommNicPluginCreateEndpoint") | ||
| 803 | + dlsym("HcommNicPluginCreateChannel") | ||
| 804 | + getInfo() → validate magic / version / size / protocols[] | ||
| 805 | + ├─ Pass → protocolPlugins[p] = entry (later load overwrites + WARNING) | ||
| 806 | + └─ Failure → dlclose, WARNING log | ||
| 807 | +``` | ||
| 808 | + | ||
| 809 | +**Lifecycle**: `dlclose` is not called after `dlopen`; the plugin has process-level singleton lifecycle. Runtime unloading is not supported. When the same protocol conflicts, the later-loaded plugin overwrites the earlier one, and a WARNING log is recorded. | ||
| 810 | + | ||
| 811 | +--- | ||
| 812 | + | ||
| 813 | +## 6. Key Interaction Flows | ||
| 814 | + | ||
| 815 | +### 6.1 Endpoint Creation | ||
| 816 | + | ||
| 817 | +```mermaid | ||
| 818 | +sequenceDiagram | ||
| 819 | + participant Caller as Caller | ||
| 820 | + participant EC as HcommEndpointCreate | ||
| 821 | + participant Mgr as NicPluginManager | ||
| 822 | + participant Map as g_EndpointMap | ||
| 823 | + participant SO as Plugin .so | ||
| 824 | + | ||
| 825 | + Caller->>EC: Create(&desc, &handle) | ||
| 826 | + | ||
| 827 | + EC->>Mgr: FindHostNicPlugin(desc.protocol) | ||
| 828 | + Mgr-->>EC: entry / null | ||
| 829 | + | ||
| 830 | + alt entry != null (Plugin Endpoint) | ||
| 831 | + EC->>SO: createEndpoint(&desc, &pluginCtx, &pluginOps) | ||
| 832 | + SO-->>EC: pluginCtx + pluginOps | ||
| 833 | + EC->>EC: Validate pluginOps, validate ABI header and required interfaces | ||
| 834 | + EC->>EC: Create pluginHolderOps, copy pluginOps and populate unimplemented interfaces | ||
| 835 | + EC->>EC: Initialize plugin object context pluginHolderOps->init(pluginCtx) | ||
| 836 | + EC->>EC: Create PluginEndpointHolder(desc, pluginEntry) | ||
| 837 | + EC->>EC: holder->SetNicEndpointCtx(pluginHolderOps, pluginCtx) | ||
| 838 | + EC->>Map: AddEndpoint(handle, ep) | ||
| 839 | + Map-->>EC: ok | ||
| 840 | + else entry == null (Built-in Endpoint) | ||
| 841 | + EC->>EC: ep = Endpoint::CreateEndpoint(*endpoint, ep) | ||
| 842 | + EC->>EC: ep->SetNicEndpointCtx(&g_BuiltinEndpointOps, ep.get()) | ||
| 843 | + EC->>Map: AddEndpoint(handle, ep) | ||
| 844 | + Map-->>EC: ok | ||
| 845 | + end | ||
| 846 | + | ||
| 847 | + EC-->>Caller: handle | ||
| 848 | +``` | ||
| 849 | + | ||
| 850 | +### 6.2 Channel Creation | ||
| 851 | + | ||
| 852 | +```mermaid | ||
| 853 | +sequenceDiagram | ||
| 854 | + participant Caller as Caller | ||
| 855 | + participant CC as HcommChannelCreate | ||
| 856 | + participant CP as ChannelProcess | ||
| 857 | + participant SO as Plugin .so | ||
| 858 | + | ||
| 859 | + Caller->>CC: Create(epHandle, engine, descs, num, &channels) | ||
| 860 | + | ||
| 861 | + alt ep->nicOps_ != &g_BuiltinEndpointOps (Plugin Endpoint) | ||
| 862 | + CC->>CC: entry = epHolder->GetPluginEntry() | ||
| 863 | + loop i in [0, num) | ||
| 864 | + CC->>SO: entry->createChannel(epCtx, channelDesc, &pluginCtx, &pluginOps) | ||
| 865 | + SO-->>CC: pluginCtx + pluginOps | ||
| 866 | + CC->>CC: Validate pluginOps, validate ABI header and required interfaces | ||
| 867 | + CC->>CC: Create pluginHolderOps, copy pluginOps and populate unimplemented interfaces | ||
| 868 | + CC->>CC: Initialize plugin object context pluginHolderOps->init(pluginCtx) | ||
| 869 | + CC->>CC: Create PluginChannelHolder(pluginEntry) | ||
| 870 | + CC->>CC: holder->SetNicChannelCtx(pluginHolderOps, pluginCtx) | ||
| 871 | + CC->>CC: ChannelHandle handle = (ChannelHandle)holder.get() | ||
| 872 | + CC->>Map: InsertPluginChannelToMap(handle, std::move(holder)) | ||
| 873 | + Map-->>CC: ok | ||
| 874 | + CC->>CC: Set high bit to mark plugin creation: channels[i] = MAKE_PLUGIN_CH_HANDLE(handle); | ||
| 875 | + end | ||
| 876 | + else ep->nicOps_ == &g_BuiltinEndpointOps (Built-in Endpoint) | ||
| 877 | + CC->>CP: CreateChannelsLoop(ep, eng, descs, num, channels) | ||
| 878 | + CP->>CP: Channel::CreateChannel factory → ch->SetNicChannelCtx(&g_BuiltinChannelOps, ch.get()) | ||
| 879 | + CP-->>CC: ok | ||
| 880 | + end | ||
| 881 | + | ||
| 882 | + CC-->>Caller: channels | ||
| 883 | +``` | ||
| 884 | + | ||
| 885 | +### 6.3 Data-Plane Invocation (Zero-Branch) | ||
| 886 | + | ||
| 887 | +```mermaid | ||
| 888 | +sequenceDiagram | ||
| 889 | + participant Caller as Upper-Layer Operator | ||
| 890 | + participant WN as HcommWriteNbi | ||
| 891 | + participant Ops as HcommNicChannelOps | ||
| 892 | + participant Impl as Implementation | ||
| 893 | + | ||
| 894 | + Caller->>WN: WriteNbi(ch, dst, src, len) | ||
| 895 | + | ||
| 896 | + WN->>WN: Obtain channel with restored bit: ch = CHANNEL_FROM_HANDLE(channel) | ||
| 897 | + WN->>Ops: ch->nicOps_->writeNbi(ch->nicCtx_, dst, src, len) | ||
| 898 | + | ||
| 899 | + alt Built-in (nicOps_ = &g_BuiltinChannelOps) | ||
| 900 | + Ops->>Impl: BuiltinWriteNbi(ctx, ...)<br/>→ ((Channel*)ctx)->Write(...) | ||
| 901 | + Impl->>Impl: HostCpuRoceChannel::Write (ibv_post_send) | ||
| 902 | + Impl-->>Ops: ret | ||
| 903 | + else Plugin (nicOps_ = pluginOps) | ||
| 904 | + Ops->>Impl: pluginWriteNbi(pluginCtx, ...) | ||
| 905 | + Impl->>Impl: Plugin data plane (custom RDMA/URMA) | ||
| 906 | + Impl-->>Ops: ret | ||
| 907 | + end | ||
| 908 | + | ||
| 909 | + Ops-->>WN: ret | ||
| 910 | + WN-->>Caller: ret | ||
| 911 | + | ||
| 912 | + Note over WN,Impl: Built-in and plugin follow the exact same path: handle decode + 2 loads + 1 indirect call | ||
| 913 | +``` | ||
| 914 | + | ||
| 915 | +### 6.4 Data-Plane Invocation (Branch Decision) | ||
| 916 | + | ||
| 917 | +```mermaid | ||
| 918 | +sequenceDiagram | ||
| 919 | + participant Caller as Upper-Layer Operator | ||
| 920 | + participant WO as HcommChannelNotifyRecordOnThread | ||
| 921 | + participant Ops as HcommNicChannelOps | ||
| 922 | + participant Ch as Channel | ||
| 923 | + participant HR as HcclRemoteWrite | ||
| 924 | + participant Impl as Plugin Implementation | ||
| 925 | + | ||
| 926 | + Caller->>WO: notifyRecordOnThread(thread, ch, remoteNotifyIdx) | ||
| 927 | + alt Plugin (IS_PLUGIN_HANDLE, ch high bit set) | ||
| 928 | + WO->>WO: Obtain channel with restored bit: ch = CHANNEL_FROM_HANDLE(channel) | ||
| 929 | + WO->>Ops: ch->GetNicOps()->notifyRecordOnThread(ch->GetNicCtx(), thread, remoteNotifyIdx) | ||
| 930 | + Ops->>Impl: pluginNotifyRecordOnThread(pluginCtx, ...) | ||
| 931 | + Impl->>Impl: Plugin data plane (custom RDMA/URMA) | ||
| 932 | + Impl-->>Ops: ret | ||
| 933 | + Ops-->>WO: ret | ||
| 934 | + else | ||
| 935 | + WO->>WO: hrtGetDeviceType(dt) | ||
| 936 | + alt DEV_TYPE_950 | ||
| 937 | + WO->>Ch: ((Channel*)ch)->Write(...) | ||
| 938 | + Ch-->>WO: ret | ||
| 939 | + else Non-950 | ||
| 940 | + WO->>HR: HcclRemoteNotifyRecord(stream, channel, remoteNotifyIdx) | ||
| 941 | + HR-->>WO: ret | ||
| 942 | + end | ||
| 943 | + end | ||
| 944 | + WO-->>Caller: ret | ||
| 945 | +``` | ||
| 946 | + | ||
| 947 | +### 6.5 Plugin Loading | ||
| 948 | + | ||
| 949 | +```mermaid | ||
| 950 | +sequenceDiagram | ||
| 951 | + participant Lib as libhcomm.so | ||
| 952 | + participant Loader as NicPluginLoader | ||
| 953 | + participant FS as File System | ||
| 954 | + participant DL as dl Runtime | ||
| 955 | + | ||
| 956 | + Lib->>Loader: First FindHostNicPlugin() triggers std::call_once | ||
| 957 | + | ||
| 958 | + Loader->>FS: opendir($ASCEND_HOME_PATH/hcomm_plugin/) | ||
| 959 | + FS-->>Loader: *.so list | ||
| 960 | + | ||
| 961 | + loop for each .so | ||
| 962 | + Loader->>DL: dlopen(path, RTLD_NOW) | ||
| 963 | + DL-->>Loader: soHandle | ||
| 964 | + | ||
| 965 | + alt dlopen Succeeded | ||
| 966 | + Loader->>DL: dlsym("HcommNicPluginGetInfo") | ||
| 967 | + Loader->>DL: dlsym("HcommNicPluginCreateEndpoint") | ||
| 968 | + Loader->>DL: dlsym("HcommNicPluginCreateChannel") | ||
| 969 | + | ||
| 970 | + alt Any Symbol Missing | ||
| 971 | + Loader->>DL: dlclose(soHandle) | ||
| 972 | + Note over Loader: WARNING log | ||
| 973 | + else All Symbols Available | ||
| 974 | + Loader->>SO: getInfo() | ||
| 975 | + SO-->>Loader: info{name, protocols[], magic, version} | ||
| 976 | + | ||
| 977 | + Loader->>Loader: Validate: magic/version/size/protocols | ||
| 978 | + | ||
| 979 | + alt Validation Passed | ||
| 980 | + Loader->>Loader: protocolPlugins[protocol] = entry | ||
| 981 | + Note over Loader: Same-protocol conflict: later load overwrites + WARNING | ||
| 982 | + else Validation Failed | ||
| 983 | + Loader->>DL: dlclose(soHandle) | ||
| 984 | + Note over Loader: WARNING log | ||
| 985 | + end | ||
| 986 | + end | ||
| 987 | + else dlopen Failed | ||
| 988 | + Note over Loader: WARNING log, skip | ||
| 989 | + end | ||
| 990 | + end | ||
| 991 | +``` | ||
| 992 | + | ||
| 993 | +--- | ||
| 994 | + | ||
| 995 | +## 7. Plugin Development Guide | ||
| 996 | + | ||
| 997 | +### 7.1 Directory Structure | ||
| 998 | + | ||
| 999 | +```text | ||
| 1000 | +experimental/base_comm/nic_plugin/<my_plugin>/ | ||
| 1001 | +├── CMakeLists.txt # Independent build; does not link libhcomm.so | ||
| 1002 | +└── src/ | ||
| 1003 | + └── my_plugin.c # 3 exported functions + ops table implementation | ||
| 1004 | +``` | ||
| 1005 | + | ||
| 1006 | +### 7.2 Three Required Exported Symbols | ||
| 1007 | + | ||
| 1008 | +```c | ||
| 1009 | +// 1. Return plugin metadata | ||
| 1010 | +const HcommNicPluginInfo *HcommNicPluginGetInfo(void) { | ||
| 1011 | + static const HcommNicPluginInfo info = { | ||
| 1012 | + .header = { | ||
| 1013 | + .magicWord = HCOMM_NIC_PLUGIN_INFO_MAGIC_WORD, | ||
| 1014 | + .version = HCOMM_NIC_PLUGIN_INFO_VERSION, | ||
| 1015 | + .size = sizeof(HcommNicPluginInfo), | ||
| 1016 | + .reserved = 0, | ||
| 1017 | + }, | ||
| 1018 | + .name = "my_plugin", | ||
| 1019 | + .protocolCount = 1, | ||
| 1020 | + .protocols = {COMM_PROTOCOL_ROCE}, // Replacement mode | ||
| 1021 | + // or .protocols = {COMM_PROTOCOL_CUSTOM_BASE + 0}, // Extension mode | ||
| 1022 | + }; | ||
| 1023 | + return &info; | ||
| 1024 | +} | ||
| 1025 | + | ||
| 1026 | +// 2. Create Endpoint | ||
| 1027 | +int32_t HcommNicPluginCreateEndpoint(const EndpointDesc *desc, | ||
| 1028 | + void **outCtx, HcommNicEndpointOps **outOps) { | ||
| 1029 | + MyEndpointCtx *ctx = malloc(sizeof(MyEndpointCtx)); | ||
| 1030 | + // Initialize ctx... | ||
| 1031 | + *outCtx = ctx; | ||
| 1032 | + *outOps = &kMyEndpointOps; // HcommNicEndpointOps table implemented by the plugin | ||
| 1033 | + return 0; | ||
| 1034 | +} | ||
| 1035 | + | ||
| 1036 | +// 3. Create Channel | ||
| 1037 | +int32_t HcommNicPluginCreateChannel(void *epCtx, const HcommChannelDesc *desc, | ||
| 1038 | + void **outCtx, HcommNicChannelOps **outOps) { | ||
| 1039 | + MyChannelCtx *ctx = malloc(sizeof(MyChannelCtx)); | ||
| 1040 | + // Initialize ctx (establish connections, exchange memory information, etc.)... | ||
| 1041 | + *outCtx = ctx; | ||
| 1042 | + *outOps = &kMyChannelOps; // HcommNicChannelOps table implemented by the plugin | ||
| 1043 | + return 0; | ||
| 1044 | +} | ||
| 1045 | +``` | ||
| 1046 | + | ||
| 1047 | +### 7.3 Implement the Ops Table | ||
| 1048 | + | ||
| 1049 | +```c | ||
| 1050 | +// ---- HcommNicEndpointOps ---- | ||
| 1051 | +static int32_t RegisterMemory(void *ctx, const CommMem *mem, | ||
| 1052 | + const char *tag, void **handle) { | ||
| 1053 | + // Implement memory registration (e.g., ibv_reg_mr / urma_reg_mr) | ||
| 1054 | + return 0; | ||
| 1055 | +} | ||
| 1056 | +static int32_t UnregisterMemory(void *ctx, void *handle) { | ||
| 1057 | + // Implement memory unregistration | ||
| 1058 | + return 0; | ||
| 1059 | +} | ||
| 1060 | +// ... memoryExport, memoryImport, memoryUnimport, init, destroy follow the same pattern | ||
| 1061 | + | ||
| 1062 | +static HcommNicEndpointOps kMyEndpointOps = { | ||
| 1063 | + {HCOMM_NIC_ENDPOINT_OPS_VERSION, HCOMM_NIC_ENDPOINT_OPS_MAGIC_WORD, sizeof(HcommNicEndpointOps), 0}, | ||
| 1064 | + InitEndpoint, // init | ||
| 1065 | + DestroyEndpoint, // destroy | ||
| 1066 | + RegisterMemory, // registerMemory | ||
| 1067 | + UnregisterMemory, // unregisterMemory | ||
| 1068 | + MemoryExport, // memoryExport | ||
| 1069 | + MemoryImport, // memoryImport | ||
| 1070 | + MemoryUnimport, // memoryUnimport | ||
| 1071 | + GetListenPort, // getListenPort | ||
| 1072 | +}; | ||
| 1073 | + | ||
| 1074 | +// ---- HcommNicChannelOps ---- | ||
| 1075 | +// Implement data-plane entries as needed; destroy must be implemented | ||
| 1076 | +static int32_t WriteNbi(void *ctx, void *dst, const void *src, uint64_t len) { | ||
| 1077 | + MyChannelCtx *ch = (MyChannelCtx*)ctx; | ||
| 1078 | + // Implement data write operations (e.g., ibv_post_send) | ||
| 1079 | + return 0; | ||
| 1080 | +} | ||
| 1081 | + | ||
| 1082 | +static HcommNicChannelOps kMyChannelOps = { | ||
| 1083 | + {HCOMM_NIC_CHANNEL_OPS_VERSION, HCOMM_NIC_CHANNEL_OPS_MAGIC_WORD, sizeof(HcommNicChannelOps), 0}, | ||
| 1084 | + InitChannel, // init | ||
| 1085 | + DestroyChannel, // destroy | ||
| 1086 | + GetStatus, // getStatus | ||
| 1087 | + WriteNbi, // writeNbi | ||
| 1088 | + WriteNbiOnThread, // writeNbiOnThread | ||
| 1089 | + nullptr, // writeOnThread | ||
| 1090 | + WriteWithNotifyNbi, // writeWithNotifyNbi | ||
| 1091 | + WriteWithNotifyNbiOnThread, // writeWithNotifyNbiOnThread | ||
| 1092 | + nullptr, // writeWithNotifyOnThread | ||
| 1093 | + nullptr, // writeReduceOnThread | ||
| 1094 | + nullptr, // writeReduceWithNotifyOnThread | ||
| 1095 | + ReadNbi, // readNbi | ||
| 1096 | + ReadNbiOnThread, // readNbiOnThread | ||
| 1097 | + nullptr, // readOnThread | ||
| 1098 | + nullptr, // readReduceOnThread | ||
| 1099 | + NotifyRecord, // notifyRecord | ||
| 1100 | + NotifyRecordOnThread, // notifyRecordOnThread | ||
| 1101 | + NotifyWait, // notifyWait | ||
| 1102 | + NotifyWaitOnThread, // notifyWaitOnThread | ||
| 1103 | + nullptr, // notifyWaitOnThreadWithDefaultTimeout | ||
| 1104 | + nullptr, // batchTransferOnThread | ||
| 1105 | + Fence, // fence | ||
| 1106 | + FenceOnThread, // fenceOnThread | ||
| 1107 | + nullptr, // drainOnThread | ||
| 1108 | +}; | ||
| 1109 | +``` | ||
| 1110 | + | ||
| 1111 | +### 7.4 Build and Deployment | ||
| 1112 | + | ||
| 1113 | +```bash | ||
| 1114 | +# 1. Build | ||
| 1115 | +cd experimental/base_comm/nic_plugin/<my_plugin> | ||
| 1116 | +mkdir build && cd build | ||
| 1117 | +cmake .. && make -j | ||
| 1118 | + | ||
| 1119 | +# CMakeLists.txt key points: | ||
| 1120 | +# - Do not link libhcomm.so | ||
| 1121 | +# - Align compilation options with the hcomm_nic_plugin.h header file path | ||
| 1122 | +# - Generate a .so file (e.g., libmy_plugin.so) | ||
| 1123 | + | ||
| 1124 | +# 2. Deploy | ||
| 1125 | +cp build/libmy_plugin.so ${ASCEND_HOME_PATH}/hcomm_plugin/ | ||
| 1126 | + | ||
| 1127 | +# 3. Verify | ||
| 1128 | +# Restart the process and check for the "[NicPlugin] protocol[X] is handled by plugin[my_plugin]" message in the log | ||
| 1129 | + | ||
| 1130 | +# 4. Debug (alternative path) | ||
| 1131 | +export HCOMM_NIC_PLUGIN_SO=/path/to/build/libmy_plugin.so | ||
| 1132 | +``` | ||
| 1133 | + | ||
| 1134 | +### 7.5 Protocol Number Selection | ||
| 1135 | + | ||
| 1136 | +| Category | Protocol Number Range | Description | | ||
| 1137 | +|----------|----------------------|-------------| | ||
| 1138 | +| Built-in Protocol | All non-RESERVED built-in enumeration values < `COMM_PROTOCOL_CUSTOM_BASE` (1000) | Replacement mode: overwrite the built-in implementation | | ||
| 1139 | +| Custom Protocol | ≥ `COMM_PROTOCOL_CUSTOM_BASE` (1000) | Extension mode: add a new protocol | | ||
| 1140 | + | ||
| 1141 | +The `protocols[]` array returned by `HcommNicPluginGetInfo()` can contain both built-in protocol numbers (replacement) and custom protocol numbers (extension), with a maximum of 4 entries. | ||
| 1142 | + | ||
| 1143 | +--- | ||
| 1144 | + | ||
| 1145 | +## 8. Performance Analysis | ||
| 1146 | + | ||
| 1147 | +### 8.1 Call Chain Overhead | ||
| 1148 | + | ||
| 1149 | +```text | ||
| 1150 | +Built-in and plugin share the same call path: | ||
| 1151 | + and rsi, ~HCOMM_PLUGIN_HANDLE_FLAG ; handle decode (1 cycle) | ||
| 1152 | + test rsi, rsi ; null check (parallel, 0 extra) | ||
| 1153 | + mov rax, [rsi+ nicOps_offs] ; load nicOps_ (4 cycles) | ||
| 1154 | + mov rdi, [rsi + nicCtx_offs] ; load nicCtx_ (parallel, 0 extra) | ||
| 1155 | + call [rax + writeNbi_offs] ; load writeNbi (4 cycles) + indirect call (1-2 cycles) | ||
| 1156 | + ; callee: BuiltinWriteNbi | ||
| 1157 | + ; Total: ~10-11 cycles (all loads hit L1, indirect call hits BTB) | ||
| 1158 | +``` | ||
| 1159 | + | ||
| 1160 | +--- | ||
| 1161 | + | ||
| 1162 | +## 9. Edge Cases | ||
| 1163 | + | ||
| 1164 | +| Scenario | Handling Strategy | | ||
| 1165 | +|----------|------------------| | ||
| 1166 | +| Some ops table entries are NULL | The interfaces to be implemented are divided into two categories: (1) Interfaces that the plugin must implement (destroy) — the framework performs strict validation. (2) Interfaces that the plugin is not required to implement — the plugin can fill NULL for such interfaces, and the framework populates default implementations. Default implementation return values have two types: a. For the init interface, return SUCCESS. b. For other interfaces, return NOT_SUPPORT. | | ||
| 1167 | +| Multiple plugins conflict on the same protocol | Load in filename lexicographic order; later load overwrites earlier load with a WARNING log | | ||
| 1168 | +| Plugin .so fails to load | `dlclose` releases the handle, WARNING log is recorded, and processing continues with the next .so | | ||
| 1169 | + | ||
| 1170 | +--- | ||
| 1171 | + | ||
| 1172 | +## 10. Constraints and Limitations | ||
| 1173 | + | ||
| 1174 | +| # | Constraint | Type | Description | | ||
| 1175 | +|---|-----------|------|-------------| | ||
| 1176 | +| C1 | `nicOps_` is immutable | Design Constraint | Set once at creation time; read-only at runtime | | ||
| 1177 | +| C2 | Plugin only for HOST endpoints | Runtime Constraint | `ENDPOINT_LOC_TYPE_HOST` | | ||
| 1178 | +| C3 | Process-level lifecycle | Runtime Constraint | No `dlclose`; the OS reclaims resources on process exit | | ||
| 1179 | +| C4 | One plugin per protocol | Design Constraint | `ProtocolPlugins` is a 1:1 mapping | | ||
| 1180 | +| C5 | No modifications to built-in subclasses | Design Constraint | 10+ subclasses require zero modifications | | ||
| 1181 | +| C6 | Public API unchanged | Compatibility Constraint | Zero changes to `include/` | | ||
| 1182 | +| C7 | Op thread safety guaranteed by the plugin | Design Constraint | The plugin must ensure that ops can be called concurrently by multiple threads; the framework does not perform serialization | | ||
| 1183 | + | ||
| 1184 | +--- | ||
| 1185 | + | ||
| 1186 | +## 11. Compatibility | ||
| 1187 | + | ||
| 1188 | +- **API**: `include/hcomm_primitives.h`, `include/hcomm_res.h`, `include/hcomm_res_defs.h` — zero changes | ||
| 1189 | +- **ABI**: `src/base_comm/primitives/api_c_adpt/nic_plugin/hcomm_nic_plugin.h` — this path is stable as part of the SDK; plugin builds must configure it as an include path | ||
| 1190 | +- **Build**: Behavior is equivalent to the original version when `ENABLE_EXPERIMENTAL=OFF` | ||
| 1191 | + | ||
| 1192 | +--- | ||
| 1193 | + | ||
| 1194 | +## 12. Test Plan | ||
| 1195 | + | ||
| 1196 | +| Level | Content | Method | | ||
| 1197 | +|-------|---------|--------| | ||
| 1198 | +| Unit | ABI signature stability | Verify that exported symbol types remain unchanged | | ||
| 1199 | +| Unit | `NicPluginLoader` loading/validation/mapping/conflict | Full coverage with mock .so | | ||
| 1200 | +| Unit | `g_BuiltinChannelOps` / `g_BuiltinEndpointOps` wrapper functions | Verify forwarding correctness | | ||
| 1201 | +| Integration | Plugin Endpoint → Channel → Write/Read/Notify/Fence | Experimental RoCE/UB plugin + UT | | ||
| 1202 | +| Integration | Extended protocol (custom protocol number) end-to-end | Mock new protocol plugin | | ||
| 1203 | +| Integration | Built-in path regression | Built-in channels function normally with `ENABLE_EXPERIMENTAL=ON` | | ||
| 1204 | + | ||
| 1205 | +--- | ||
| 1206 | + | ||
| 1207 | +## 13. Risk Assessment | ||
| 1208 | + | ||
| 1209 | +| Risk | Level | Mitigation | | ||
| 1210 | +|------|-------|-----------| | ||
| 1211 | +| New fields in the base class affect existing subclass ABI | Low | Non-virtual fields at the end of the protected section, default `nullptr`; UT covers all subclasses | | ||
| 1212 | +| `g_Builtin**Ops` duplicated across .o files | Low | Inline definition in header file | | ||
| 1213 | +| Plugin .so load failure affects built-in channels | Very Low | Early return within `std::call_once` | | ||
| 1214 | +| `nicOps_` is NULL when dispatched | Very Low | The creation flow ensures that every object is populated with an ops table; UT provides coverage | | ||
| 1215 | + | ||
| 1216 | +--- | ||
| 1217 | + | ||
| 1218 | +## 14. Alternatives | ||
| 1219 | + | ||
| 1220 | +| Approach | Advantages | Disadvantages | Conclusion | | ||
| 1221 | +|----------|-----------|---------------|-----------| | ||
| 1222 | +| Entry-level tag-bit branch dispatch | Minimal changes | 30+ hot-path branches | Not adopted | | ||
| 1223 | +| C++ virtual function adapter subclass | No base class changes | Virtual function overhead | Not adopted | | ||
| 1224 | +| C ops table embedded in base class | Zero branching | Requires base class modification (non-invasive) | Adopted | | ||
| 1225 | + | ||
| 1226 | +--- | ||
| 1227 | + | ||
| 1228 | +## 15. Open Questions | ||
| 1229 | + | ||
| 1230 | +| Number | Question | Description | | ||
| 1231 | +|--------|----------|-------------| | ||
| 1232 | +| O1 | Whether runtime unloading or updating of plugins is needed | Not currently supported | | ||
| 1233 | +| O2 | Conflict arbitration strategy for one plugin per protocol | Currently, the later-loaded plugin overwrites the earlier one | | ||
| 1234 | + | ||
| 1235 | +--- | ||
| 1236 | + | ||
| 1237 | +## 16. Review Records | ||
| 1238 | + | ||
| 1239 | +The review process takes place in the PR comment section. For detailed review comments, refer to the corresponding PR comments. | ||
| @@ -23,6 +23,7 @@ This file registers all assigned RFC numbers. Before adding a new RFC, claim the | |||
| 23 | 23 | ||
| 24 | | Number | Title | Author | Status | PR | | 24 | | Number | Title | Author | Status | PR | |
| 25 | |------|------|------|------|-----| | 25 | |------|------|------|------|-----| |
| 26 | +|0001|RFC: Topology-Based Cluster Communication Rapid Sub-Health Monitoring | Wang Yunhao (Xidian University)|`accepted`|#2491| | ||
| 26 | 27 | ||
| 27 | ## Numbering Rules | 28 | ## Numbering Rules |
| 28 | 29 | ||
| @@ -0,0 +1 @@ | |||
| 1 | +<svg width="3222" height="1801" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" overflow="hidden"><g transform="translate(-664 -325)"><g><rect x="835.5" y="1371.5" width="2104" height="715" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none"/><rect x="1318.5" y="1505.5" width="1588" height="545" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#E8E8E8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 868.401 1455)">Agent</text><rect x="706.5" y="455.5" width="1009" height="427" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><rect x="1768.5" y="455.5" width="1234" height="422" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><rect x="2435.5" y="1740.5" width="428" height="157" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#A6CAEC" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2520.48 1840)">Time Analysis</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 729.986 532)">Controller</text><rect x="765.5" y="580.5" width="432" height="228" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 788.838 716)">Topo Partition</text><rect x="1341" y="1629" width="857" height="198" fill="#4E95D9" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1521.93 1697)">Counter Collection</text><path d="M2412.34 1820.11 2187.12 1794.92C2185.23 1794.71 2183.87 1793 2184.08 1791.12 2184.3 1789.23 2186 1787.87 2187.88 1788.08L2413.1 1813.28C2414.99 1813.49 2416.35 1815.19 2416.14 1817.07 2415.93 1818.96 2414.23 1820.32 2412.34 1820.11ZM2409.69 1802.52 2435.5 1819.24 2406.64 1829.85ZM2185.97 1805.16C2178.42 1804.32 2172.99 1797.52 2173.84 1789.97 2174.68 1782.42 2181.48 1776.99 2189.03 1777.84 2196.58 1778.68 2202.01 1785.48 2201.17 1793.03 2200.32 1800.58 2193.52 1806.01 2185.97 1805.16Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2264.44 1939)">Latency,</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2264.44 2005)">Pass Rate</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2258.42 1679)">PFC, </text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2258.42 1745)">CNP,…</text><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="83" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 3505.66 1617)">③</text><rect x="869.5" y="1508.5" width="382" height="545" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 931.448 1725)">Topo Probing</text><rect x="2362.5" y="672.5" width="616" height="99" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#96DCF8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2477.85 732)">Link Solving</text><rect x="1808.5" y="643.5" width="507" height="165" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#96DCF8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1883.53 703)">Z-Score-Based</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1868.63 780)">Link Assessment</text><rect x="1341" y="1838" width="856" height="198" fill="#4E95D9" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1603.68 1906)">Active Probing</text><rect x="1032.5" y="1267.5" width="297" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1086.81 1352)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1242.07 1352)">0</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="83" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2107.46 1345)">…</text><rect x="1640.5" y="1277.5" width="296" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1694.49 1362)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1849.75 1362)">1</text><rect x="2356.5" y="1279.5" width="314" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2407.21 1364)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2562.47 1364)">M</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1800.96 543)">Analyzer</text><path d="M0 0 276.615 110.458" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1 0 0 -1 1180.5 1266.96)"/><path d="M0 0 19.2323 72.3369" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1 0 0 -1 1784.5 1279.84)"/><path d="M2392.16 1279.28 2117.5 1121.5" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><path d="M1474.9 1025.98C1465.57 994.312 1513.62 965.151 1582.23 960.844 1610.03 959.098 1638.3 961.707 1662.5 968.253 1688.15 945.937 1748.09 937.455 1796.38 949.309 1804.83 951.382 1812.58 954.01 1819.39 957.109 1839.37 938.61 1888.03 931.095 1928.08 940.323 1939.16 942.877 1948.83 946.57 1956.34 951.12 1988.54 933.626 2045.44 931.465 2083.43 946.294 2099.4 952.526 2110.16 961.122 2113.87 970.608 2166.63 977.253 2197.75 1002.42 2183.39 1026.82 2182.18 1028.87 2180.66 1030.88 2178.85 1032.84 2221.16 1058.27 2210.8 1094.71 2155.71 1114.24 2138.57 1120.32 2118.31 1124.25 2096.9 1125.67 2096.43 1153.02 2047.96 1175.02 1988.65 1174.8 1968.83 1174.73 1949.45 1172.13 1932.64 1167.29 1912.58 1197.96 1842.6 1215.29 1776.34 1206 1748.57 1202.11 1724.58 1193.88 1708.46 1182.72 1640.62 1201.61 1552.56 1191.45 1511.78 1160.03 1511.27 1159.63 1510.76 1159.23 1510.27 1158.83 1465.87 1161.23 1425.65 1146.6 1420.44 1126.17 1417.66 1115.27 1425.45 1104.37 1441.74 1096.36 1403.29 1085.91 1390.38 1062.98 1412.89 1045.14 1425.88 1034.85 1448.66 1028.05 1474.23 1026.82Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-rule="evenodd" fill-opacity="1"/><path d="M1489.62 1100.32C1473.21 1100.91 1456.8 1099.16 1442.6 1095.3M1531.12 1155.23C1524.52 1156.45 1517.6 1157.25 1510.54 1157.64M1708.41 1181.62C1703.45 1178.19 1699.29 1174.51 1696.01 1170.67M1937.67 1154.3C1936.94 1158.38 1935.28 1162.41 1932.72 1166.32M2036.1 1080.03C2073.31 1088.39 2096.79 1105.86 2096.46 1124.95M2178.47 1032.17C2172.45 1038.67 2163.25 1044.44 2151.59 1049.02M2113.98 969.664C2115 972.289 2115.48 974.953 2115.4 977.62M1942.33 960.383C1945.83 956.712 1950.47 953.294 1956.1 950.236M1813.54 965.218C1814.97 962.185 1817.21 959.245 1820.21 956.467M1662.41 968.19C1671.17 970.559 1679.27 973.41 1686.54 976.68M1479.12 1034.91C1477.21 1031.99 1475.8 1029 1474.91 1025.98" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="73" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1583.12 1090)">RoCE</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="73" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1781.08 1090)">Fabric</text><path d="M0 3.4375-27.5 3.4375-27.5-3.4375 0-3.4375ZM-48.125 3.4375-75.625 3.4375-75.625-3.4375-48.125-3.4375ZM-79.0617 13.7508-79.0617 41.2508-85.9367 41.2508-85.9367 13.7508ZM-79.0617 61.8758-79.0617 89.3757-85.9367 89.3757-85.9367 61.8758ZM-79.0617 110.001-79.0617 137.501-85.9367 137.501-85.9367 110.001ZM-79.0617 158.126-79.0617 185.626-85.9367 185.626-85.9367 158.126ZM-79.0617 206.251-79.0617 233.751-85.9367 233.751-85.9367 206.251ZM-79.0617 254.376-79.0617 281.876-85.9367 281.876-85.9367 254.376ZM-79.0617 302.501-79.0617 330.001-85.9367 330.001-85.9367 302.501ZM-79.0617 350.626-79.0617 378.126-85.9367 378.126-85.9367 350.626ZM-79.0617 398.751-79.0617 426.251-85.9367 426.251-85.9367 398.751ZM-79.0617 446.876-79.0617 474.376-85.9367 474.376-85.9367 446.876ZM-79.0617 495.001-79.0617 522.501-85.9367 522.501-85.9367 495.001ZM-79.0617 543.126-79.0617 570.626-85.9367 570.626-85.9367 543.126ZM-79.0617 591.251-79.0617 618.751-85.9367 618.751-85.9367 591.251ZM-79.0617 639.376-79.0617 666.876-85.9367 666.876-85.9367 639.376ZM-79.0617 687.501-79.0617 715.001-85.9367 715.001-85.9367 687.501ZM-79.0617 735.626-79.0617 763.126-85.9367 763.126-85.9367 735.626ZM-79.0617 783.751-79.0617 811.251-85.9367 811.251-85.9367 783.751ZM-79.0617 831.876-79.0617 842.294-82.4992 838.857-65.418 838.857-65.418 845.732-85.9367 845.732-85.9367 831.876ZM-44.793 838.857-17.293 838.857-17.293 845.732-44.793 845.732ZM3.33198 838.857 30.832 838.857 30.832 845.732 3.33198 845.732ZM51.457 838.857 78.957 838.857 78.957 845.732 51.457 845.732ZM99.582 838.857 115.806 838.857 115.806 857.008 108.931 857.008 108.931 842.294 112.368 845.732 99.582 845.732ZM115.806 877.633 115.806 905.133 108.931 905.133 108.931 877.633ZM115.806 925.758 115.806 949.418 108.931 949.418 108.931 925.758ZM126.118 944.835 112.368 972.335 98.6182 944.835Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 -1.22465e-16 -1.22465e-16 -1 869.5 1780.83)"/><path d="M0 3.4375-27.5 3.4375-27.5-3.4375 0-3.4375ZM-48.125 3.4375-75.625 3.4375-75.625-3.4375-48.125-3.4375ZM-96.25 3.4375-123.75 3.4375-123.75-3.4375-96.25-3.4375ZM-144.375 3.4375-150.117 3.4375-150.117-25.1958-143.242-25.1958-143.242 0-146.679-3.4375-144.375-3.4375ZM-150.117-45.8209-150.117-73.3208-143.242-73.3208-143.242-45.8209ZM-150.117-93.9458-150.117-121.446-143.242-121.446-143.242-93.9458ZM-150.117-142.071-150.117-169.571-143.242-169.571-143.242-142.071ZM-150.117-190.196-150.117-217.696-143.242-217.696-143.242-190.196ZM-150.117-238.321-150.117-265.821-143.242-265.821-143.242-238.321ZM-150.117-286.446-150.117-313.946-143.242-313.946-143.242-286.446ZM-150.117-334.571-150.117-362.071-143.242-362.071-143.242-334.571ZM-150.117-382.696-150.117-410.196-143.242-410.196-143.242-382.696ZM-150.117-430.821-150.117-458.321-143.242-458.321-143.242-430.821ZM-150.117-478.946-150.117-506.446-143.242-506.446-143.242-478.946ZM-150.117-527.071-150.117-554.571-143.242-554.571-143.242-527.071ZM-150.117-575.196-150.117-602.696-143.242-602.696-143.242-575.196ZM-150.117-623.321-150.117-650.821-143.242-650.821-143.242-623.321ZM-150.117-671.446-150.117-698.946-143.242-698.946-143.242-671.446ZM-150.117-719.571-150.117-747.071-143.242-747.071-143.242-719.571ZM-150.117-767.696-150.117-787.395-135.44-787.395-135.44-780.52-146.679-780.52-143.242-783.957-143.242-767.696ZM-114.815-787.395-87.3153-787.395-87.3153-780.52-114.815-780.52ZM-66.6903-787.395-39.1903-787.395-39.1903-780.52-66.6903-780.52ZM-18.5653-787.395 8.93467-787.395 8.93467-780.52-18.5653-780.52ZM29.5597-787.395 57.0597-787.395 57.0597-780.52 29.5597-780.52ZM77.6847-787.395 105.185-787.395 105.185-780.52 77.6847-780.52ZM125.81-787.395 153.31-787.395 153.31-780.52 125.81-780.52ZM173.935-787.395 201.435-787.395 201.435-780.52 173.935-780.52ZM222.06-787.395 249.56-787.395 249.56-780.52 222.06-780.52ZM270.185-787.395 297.685-787.395 297.685-780.52 270.185-780.52ZM318.31-787.395 345.81-787.395 345.81-780.52 318.31-780.52ZM366.435-787.395 393.935-787.395 393.935-780.52 366.435-780.52ZM414.56-787.395 442.06-787.395 442.06-780.52 414.56-780.52ZM462.685-787.395 490.185-787.395 490.185-780.52 462.685-780.52ZM510.81-787.395 538.31-787.395 538.31-780.52 510.81-780.52ZM558.935-787.395 586.435-787.395 586.435-780.52 558.935-780.52ZM607.06-787.395 634.56-787.395 634.56-780.52 607.06-780.52ZM655.185-787.395 682.685-787.395 682.685-780.52 655.185-780.52ZM703.31-787.395 730.81-787.395 730.81-780.52 703.31-780.52ZM751.435-787.395 778.935-787.395 778.935-780.52 751.435-780.52ZM799.56-787.395 827.06-787.395 827.06-780.52 799.56-780.52ZM847.685-787.395 875.185-787.395 875.185-780.52 847.685-780.52ZM895.81-787.395 923.31-787.395 923.31-780.52 895.81-780.52ZM943.935-787.395 971.435-787.395 971.435-780.52 943.935-780.52ZM992.06-787.395 1019.56-787.395 1019.56-780.52 992.06-780.52ZM1040.18-787.395 1067.68-787.395 1067.68-780.52 1040.18-780.52ZM1088.31-787.395 1115.81-787.395 1115.81-780.52 1088.31-780.52ZM1136.43-787.395 1163.93-787.395 1163.93-780.52 1136.43-780.52ZM1184.56-787.395 1212.06-787.395 1212.06-780.52 1184.56-780.52ZM1232.68-787.395 1260.18-787.395 1260.18-780.52 1232.68-780.52ZM1280.81-787.395 1308.31-787.395 1308.31-780.52 1280.81-780.52ZM1328.93-787.395 1356.43-787.395 1356.43-780.52 1328.93-780.52ZM1377.06-787.395 1404.56-787.395 1404.56-780.52 1377.06-780.52ZM1425.18-787.395 1452.68-787.395 1452.68-780.52 1425.18-780.52ZM1473.31-787.395 1500.81-787.395 1500.81-780.52 1473.31-780.52ZM1521.43-787.395 1542.1-787.395 1542.1-773.682 1535.22-773.682 1535.22-783.957 1538.66-780.52 1521.43-780.52ZM1542.1-753.057 1542.1-725.557 1535.22-725.557 1535.22-753.057ZM1542.1-704.932 1542.1-677.432 1535.22-677.432 1535.22-704.932ZM1542.1-656.807 1542.1-629.307 1535.22-629.307 1535.22-656.807ZM1542.1-608.682 1542.1-581.182 1535.22-581.182 1535.22-608.682ZM1542.1-560.557 1542.1-533.057 1535.22-533.057 1535.22-560.557ZM1542.1-512.432 1542.1-484.932 1535.22-484.932 1535.22-512.432ZM1542.1-464.307 1542.1-436.807 1535.22-436.807 1535.22-464.307ZM1542.1-416.182 1542.1-388.682 1535.22-388.682 1535.22-416.182ZM1542.1-368.057 1542.1-340.557 1535.22-340.557 1535.22-368.057ZM1542.1-319.932 1542.1-292.432 1535.22-292.432 1535.22-319.932ZM1542.1-271.807 1542.1-244.307 1535.22-244.307 1535.22-271.807ZM1542.1-223.682 1542.1-196.182 1535.22-196.182 1535.22-223.682ZM1542.1-175.557 1542.1-148.057 1535.22-148.057 1535.22-175.557ZM1542.1-127.432 1542.1-99.9321 1535.22-99.9321 1535.22-127.432ZM1542.1-79.3071 1542.1-51.8071 1535.22-51.8071 1535.22-79.3071ZM1542.1-31.1821 1542.1-3.68209 1535.22-3.68209 1535.22-31.1821ZM1542.1 16.9429 1542.1 44.4429 1535.22 44.4429 1535.22 16.9429ZM1542.1 65.0679 1542.1 92.5679 1535.22 92.5679 1535.22 65.0679ZM1542.1 113.193 1542.1 140.693 1535.22 140.693 1535.22 113.193ZM1542.1 161.318 1542.1 188.818 1535.22 188.818 1535.22 161.318ZM1542.1 209.443 1542.1 236.943 1535.22 236.943 1535.22 209.443ZM1542.1 257.568 1542.1 285.068 1535.22 285.068 1535.22 257.568ZM1537.99 308.458 1510.49 308.458 1510.49 301.583 1537.99 301.583ZM1489.86 308.458 1479.07 308.458 1479.07 301.583 1489.86 301.583ZM1483.65 318.77 1456.15 305.02 1483.65 291.27Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1.83697e-16 1 1 -1.83697e-16 1464.5 580.5)"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1361.81 1597)">Executor</text><path d="M0-3.4375 27.5-3.4375 27.5 3.4375 0 3.4375ZM48.125-3.4375 75.625-3.4375 75.625 3.4375 48.125 3.4375ZM96.25-3.4375 123.75-3.4375 123.75 3.4375 96.25 3.4375ZM144.375-3.4375 171.875-3.4375 171.875 3.4375 144.375 3.4375ZM192.5-3.4375 201.903-3.4375 201.903 21.5343 195.028 21.5343 195.028 0 198.466 3.4375 192.5 3.4375ZM201.903 42.1593 201.903 69.6593 195.028 69.6593 195.028 42.1593ZM201.903 90.2843 201.903 117.784 195.028 117.784 195.028 90.2843ZM201.903 138.409 201.903 165.909 195.028 165.909 195.028 138.409ZM201.903 186.534 201.903 214.034 195.028 214.034 195.028 186.534ZM201.903 234.659 201.903 262.159 195.028 262.159 195.028 234.659ZM201.903 282.784 201.903 310.284 195.028 310.284 195.028 282.784ZM201.903 330.909 201.903 358.409 195.028 358.409 195.028 330.909ZM201.903 379.034 201.903 406.534 195.028 406.534 195.028 379.034ZM201.903 427.159 201.903 454.659 195.028 454.659 195.028 427.159ZM201.903 475.284 201.903 502.784 195.028 502.784 195.028 475.284ZM201.903 523.409 201.903 550.909 195.028 550.909 195.028 523.409ZM201.903 571.534 201.903 599.034 195.028 599.034 195.028 571.534ZM201.903 619.659 201.903 647.159 195.028 647.159 195.028 619.659ZM201.903 667.784 201.903 695.284 195.028 695.284 195.028 667.784ZM201.903 715.909 201.903 743.409 195.028 743.409 195.028 715.909ZM201.903 764.034 201.903 791.534 195.028 791.534 195.028 764.034ZM201.903 812.159 201.903 839.659 195.028 839.659 195.028 812.159ZM201.903 860.284 201.903 887.784 195.028 887.784 195.028 860.284ZM201.903 908.409 201.903 935.909 195.028 935.909 195.028 908.409ZM201.903 956.534 201.903 984.034 195.028 984.034 195.028 956.534ZM201.903 1004.66 201.903 1032.16 195.028 1032.16 195.028 1004.66ZM201.903 1052.78 201.903 1080.28 195.028 1080.28 195.028 1052.78ZM194.789 1100.67 167.289 1100.67 167.289 1093.8 194.789 1093.8ZM146.664 1100.67 138.882 1100.67 138.882 1093.8 146.664 1093.8ZM143.466 1110.98 115.966 1097.23 143.466 1083.48Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 0 0 -1 2863.5 1818.73)"/><path d="M23.1157-1.6294 47.9136 0.32168C49.8062 0.470591 51.2197 2.12559 51.0708 4.01822 50.9219 5.91085 49.2669 7.32441 47.3743 7.1755L22.5764 5.22442C20.6838 5.07551 19.2702 3.42051 19.4192 1.52788 19.5681-0.364746 21.2231-1.77831 23.1157-1.6294ZM26.3368 15.8647 0 0 28.4938-11.5506ZM48.7224-9.95905C56.293-9.36341 61.9472-2.74342 61.3516 4.8271 60.7559 12.3976 54.1359 18.0519 46.5654 17.4562 38.9949 16.8606 33.3406 10.2406 33.9363 2.67008 34.5319-4.90044 41.1519-10.5547 48.7224-9.95905Z" fill="#000000" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 0 0 -1 2315.5 725.249)"/><path d="M2240.44 1135.69 2158.17 810.343C2157.7 808.502 2158.82 806.633 2160.66 806.167 2162.5 805.702 2164.37 806.817 2164.83 808.657L2247.1 1134C2247.57 1135.84 2246.46 1137.71 2244.61 1138.18 2242.77 1138.64 2240.91 1137.53 2240.44 1135.69ZM2255.98 1127.03 2249.39 1157.06 2229.32 1133.77ZM2148.17 812.871C2146.31 805.509 2150.77 798.031 2158.13 796.17 2165.49 794.308 2172.97 798.767 2174.83 806.129 2176.69 813.491 2172.23 820.969 2164.87 822.83 2157.51 824.692 2150.03 820.233 2148.17 812.871Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><path d="M2219 1185.5C2219 1169.21 2232.21 1156 2248.5 1156 2264.79 1156 2278 1169.21 2278 1185.5 2278 1201.79 2264.79 1215 2248.5 1215 2232.21 1215 2219 1201.79 2219 1185.5Z" stroke="#042433" stroke-width="13.75" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><path d="M2269 1207 2294 1235.94" stroke="#0E2841" stroke-width="13.75" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><rect x="2297.5" y="1009.5" width="622" height="169" stroke="#000000" stroke-width="4.58333" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#F6C6AD" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2443.01 1111)">Anomaly Output</text><path d="M1118 1501.87 1158.63 1461.16 1146.99 1449.54 1193.53 1449.5 1193.57 1496.04 1181.92 1484.41 1141.29 1525.12Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEBF8" fill-rule="evenodd" fill-opacity="1"/><path d="M1819.73 1529.21 1829.18 1472.48 1812.95 1469.77 1850.82 1442.72 1877.87 1480.59 1861.64 1477.89 1852.18 1534.62Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#D1D1D1" fill-rule="evenodd" fill-opacity="1"/><path d="M0 0 53.6921 0 53.6921 1332.72 12.4421 1332.72" stroke="#0E2841" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1.83697e-16 1 1 -1.83697e-16 1180.5 1392.5)"/><path d="M0 0 0.000360892 45.4724" stroke="#0E2841" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(-1 0 0 1 1783.5 1402.5)"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(6.12323e-17 1 -1 6.12323e-17 721.193 1156)">Topo Probing Results</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 839.244 409)">Probe Path List</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(6.12323e-17 1 -1 6.12323e-17 2990.24 985)">Network Status Results</text><rect x="1248.5" y="580.5" width="432" height="232" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1271.16 718)">Probe Plan</text><path d="M1242.76 697.124 1200.48 696.937C1198.59 696.929 1197.05 695.383 1197.06 693.485 1197.07 691.586 1198.62 690.054 1200.52 690.062L1242.79 690.249 1242.79 690.249C1244.69 690.258 1246.22 691.803 1246.21 693.702 1246.2 695.6 1244.66 697.132 1242.76 697.124ZM1238.25 679.917 1265.69 693.788 1238.13 707.416ZM1200.44 707.25C1192.85 707.216 1186.72 701.033 1186.75 693.439 1186.78 685.845 1192.97 679.716 1200.56 679.75 1208.15 679.784 1214.28 685.967 1214.25 693.561 1214.22 701.154 1208.03 707.283 1200.44 707.25Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><path d="M22.9023 3.53092 252.373 74.9825C254.186 75.5469 255.198 77.4739 254.633 79.2865 254.069 81.0992 252.142 82.1111 250.329 81.5466L20.8584 10.0951C19.0457 9.53065 18.0338 7.60367 18.5982 5.79103 19.1627 3.97839 21.0896 2.96651 22.9023 3.53092ZM22.1688 21.304 0 0 30.3444-4.95264ZM255.439 65.1363C262.689 67.3939 266.737 75.1018 264.479 82.3524 262.222 89.603 254.514 93.6505 247.263 91.3929 240.013 89.1352 235.965 81.4273 238.223 74.1767 240.48 66.9262 248.188 62.8786 255.439 65.1363Z" fill="#000000" fill-rule="nonzero" fill-opacity="1" transform="matrix(-1 0 0 1 2435.85 1819.5)"/><rect x="902.5" y="1791.5" width="315" height="172" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 932.581 1853)">hccn_tool</text><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 930.862 1930)">traceroute</text><rect x="1394.5" y="1919.5" width="723" height="89" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1609.33 1981)">HccnRping</text><rect x="1411.5" y="1718.5" width="723" height="89" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1593.27 1780)">hccn_tool</text><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1862.54 1780)">stat</text></g></g></svg> | ||
| @@ -52,7 +52,7 @@ CCU_IF(condExpr) { | |||
| 52 | ## 约束说明 | 52 | ## 约束说明 |
| 53 | 53 | ||
| 54 | - `CCU_IF(condExpr)`后必须紧跟{}包裹的then代码块 | 54 | - `CCU_IF(condExpr)`后必须紧跟{}包裹的then代码块 |
| 55 | -- A5代际的`CCU_IF`当前的`condExpr`只支持==和!=两种判断模式 | 55 | +- 针对Ascend 950PR/Ascend 950DT,`CCU_IF`当前的`condExpr`只支持==和!=两种判断模式 |
| 56 | - `CCU_IF`可以单独使用(无`CCU_ELSE`),也可以配合`CCU_ELSE`使用,两种写法均合法。 | 56 | - `CCU_IF`可以单独使用(无`CCU_ELSE`),也可以配合`CCU_ELSE`使用,两种写法均合法。 |
| 57 | - 支持嵌套:`CCU_IF`内部可以再嵌套`CCU_IF`。 | 57 | - 支持嵌套:`CCU_IF`内部可以再嵌套`CCU_IF`。 |
| 58 | - `CCU_IF`不建议在硬件Loop(`ccu::Loop`)的body lambda内部使用——硬件Loop body内不支持软件分支;框架不强制校验,但行为未定义。 | 58 | - `CCU_IF`不建议在硬件Loop(`ccu::Loop`)的body lambda内部使用——硬件Loop body内不支持软件分支;框架不强制校验,但行为未定义。 |
| @@ -1,17 +1,17 @@ | |||
| 1 | -# EI0002 Communication_Error_Timeout | 1 | +# EI0002 Communication_Error_Timeout |
| 2 | 2 | ||
| 3 | ## 错误信息 | 3 | ## 错误信息 |
| 4 | 4 | ||
| 5 | 报错格式如下,占位符%s的含义依次为对端Rank ID、任务信息、通信算子信息、通信域信息: | 5 | 报错格式如下,占位符%s的含义依次为对端Rank ID、任务信息、通信算子信息、通信域信息: |
| 6 | 6 | ||
| 7 | ```text | 7 | ```text |
| 8 | -An timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s. | 8 | +A timeout occurs when the Notify register waits for execution. Waiting peer rank: %s; task information: %s; communication operator information: %s; communicator: %s. |
| 9 | ``` | 9 | ``` |
| 10 | 10 | ||
| 11 | 报错示例如下: | 11 | 报错示例如下: |
| 12 | 12 | ||
| 13 | ```text | 13 | ```text |
| 14 | -An timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none. | 14 | +A timeout occurs when the Notify register waits for execution. Waiting peer rank: 4; task information: streamID:[90], taskID[686], taskType[Notify Wait], tag[AllReduce_80.48.9.154%enp48s3u1u1_60000_0_1779783710697217ringAllReduceMeshSmallCountExecutor_device], AlgType(level 0-1-2):[ring-ring-NHR].; communication operator information: notify id:[0x00000000000018fc], stage:[0], remote rank:[4]; communicator: none. |
| 15 | ``` | 15 | ``` |
| 16 | 16 | ||
| 17 | ## 可能原因 | 17 | ## 可能原因 |
| @@ -22,4 +22,4 @@ The Toolkit (cann-hcomm) and ops (cann-hccl) versions are inconsistent. The loca | |||
| 22 | 22 | ||
| 23 | ## 解决方法 | 23 | ## 解决方法 |
| 24 | 24 | ||
| 25 | -请安装相同的版本,本端的toolkit\(cann-hcomm\)版本、ops\(cann-hccl\)版本必须和远端对应的版本一致。 | 25 | +请安装相同的版本,本端的toolkit (cann-hcomm)版本、ops (cann-hccl)版本必须和远端对应的版本一致。 |
| @@ -11,7 +11,7 @@ Value %s for config %s is invalid. Expected value: %s. | |||
| 11 | 报错示例如下: | 11 | 报错示例如下: |
| 12 | 12 | ||
| 13 | ```text | 13 | ```text |
| 14 | -Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A." . | 14 | +Value Disable for config "tls" is invalid. Expected value: "All ranks are consistent. Current status: rankList for enabled tls:[80.48.25.34/0]; rankList for disabled tls:[80.48.25.34/1,2,3,4,5,6,7]; rankList for query failure tls:N/A". |
| 15 | ``` | 15 | ``` |
| 16 | 16 | ||
| 17 | ## 解决方法 | 17 | ## 解决方法 |
| @@ -16,6 +16,6 @@ Failed to enable listening for the host network adapter socket. Reason: The IP a | |||
| 16 | 16 | ||
| 17 | ## 解决方法 | 17 | ## 解决方法 |
| 18 | 18 | ||
| 19 | -1. 请确认是否已经有其他进程占用此端口,若已被占用可以通过环境变量HCCL_IF_BASE_PORT进行调整,并通过**sysctl -w net.ipv4.ip_local_reserved_ports=\*\*\*\*-\*\*\*\***调整预留端口范围。 | 19 | +1. 请确认是否已经有其他进程占用此端口,若已被占用可以通过环境变量HCCL_IF_BASE_PORT进行调整,并通过`sysctl -w net.ipv4.ip_local_reserved_ports=****-****`调整预留端口范围。 |
| 20 | 20 | ||
| 21 | 2. 请确认是否可能存在本次业务中一个device被多次拉起业务进程的情况。 | 21 | 2. 请确认是否可能存在本次业务中一个device被多次拉起业务进程的情况。 |
| @@ -59,6 +59,7 @@ | |||
| 59 | 本节只写模块内部实现。按模块分节,跨层功能按层拆分多个模块子节,不混写。 | 59 | 本节只写模块内部实现。按模块分节,跨层功能按层拆分多个模块子节,不混写。 |
| 60 | 60 | ||
| 61 | 每个模块子节包含: | 61 | 每个模块子节包含: |
| 62 | + | ||
| 62 | - 模块职责(一句话) | 63 | - 模块职责(一句话) |
| 63 | - 核心数据结构,可以使用UML展示 | 64 | - 核心数据结构,可以使用UML展示 |
| 64 | - 关键逻辑与算法 | 65 | - 关键逻辑与算法 |
| @@ -148,8 +148,7 @@ cmake --build build -j"$(nproc)" | |||
| 148 | 148 | ||
| 149 | ### 4.2 配置文件 | 149 | ### 4.2 配置文件 |
| 150 | 150 | ||
| 151 | -用户使用 JSON 文件同时描述部署拓扑、探测范围和运行参数。`schema_version=2` 为当前推荐格式;缺省 | 151 | +用户使用 JSON 文件同时描述部署拓扑、探测范围和运行参数。`schema_version=2` 为当前推荐格式;缺省`schema_version` 或 `schema_version=1` 继续兼容旧格式。 |
| 152 | -`schema_version` 或 `schema_version=1` 继续兼容旧格式。 | ||
| 153 | 152 | ||
| 154 | ```json | 153 | ```json |
| 155 | { | 154 | { |
| @@ -227,8 +226,7 @@ cmake --build build -j"$(nproc)" | |||
| 227 | | `probe.pingpong.payload_len` | int / `12` | 可选;HCCN Rping Payload 字节数,范围 `1~1500`。每个目标在 AddTarget 时独立生成指定长度的随机字节,不按字符串处理 | | 226 | | `probe.pingpong.payload_len` | int / `12` | 可选;HCCN Rping Payload 字节数,范围 `1~1500`。每个目标在 AddTarget 时独立生成指定长度的随机字节,不按字符串处理 | |
| 228 | | `probe.pingpong.interval_ms` | positive int / `1` | 可选;HCCN Rping 的探测周期,单位毫秒 | | 227 | | `probe.pingpong.interval_ms` | positive int / `1` | 可选;HCCN Rping 的探测周期,单位毫秒 | |
| 229 | 228 | ||
| 230 | -v1 兼容格式继续支持旧的 `deploy.host_to_user_pair`、`deploy.controller`、`probe_scope`、`probe_topo.tracert`、 | 229 | +v1 兼容格式继续支持旧的 `deploy.host_to_user_pair`、`deploy.controller`、`probe_scope`、`probe_topo.tracert`、`probe_controller.pingpong` 字段。兼容关系如下: |
| 231 | -`probe_controller.pingpong` 字段。兼容关系如下: | ||
| 232 | 230 | ||
| 233 | | v1 字段 | v2 字段 | 说明 | | 231 | | v1 字段 | v2 字段 | 说明 | |
| 234 | | --- | --- | --- | | 232 | | --- | --- | --- | |
| @@ -239,8 +237,7 @@ v1 兼容格式继续支持旧的 `deploy.host_to_user_pair`、`deploy.controlle | |||
| 239 | | `probe_topo.tracert` | `probe.topology` | Tracert 拓扑发现参数保持兼容,字段名前缀调整 | | 237 | | `probe_topo.tracert` | `probe.topology` | Tracert 拓扑发现参数保持兼容,字段名前缀调整 | |
| 240 | | `probe_controller.pingpong` | `probe.pingpong` | PingPong 探测参数保持兼容,字段名前缀调整 | | 238 | | `probe_controller.pingpong` | `probe.pingpong` | PingPong 探测参数保持兼容,字段名前缀调整 | |
| 241 | 239 | ||
| 242 | -v2 中 `probe.scope` 为必填;`probe.topology` 与 `probe.pingpong` 可缺省,缺省时使用上表默认参数。可用 | 240 | +v2 中 `probe.scope` 为必填;`probe.topology` 与 `probe.pingpong` 可缺省,缺省时使用上表默认参数。可用`./run.sh migrate-config old.json new.json` 生成 v2 配置模板。 |
| 243 | -`./run.sh migrate-config old.json new.json` 生成 v2 配置模板。 | ||
| 244 | 241 | ||
| 245 | ### 4.3 CLI 与执行顺序 | 242 | ### 4.3 CLI 与执行顺序 |
| 246 | 243 | ||
| @@ -600,7 +597,6 @@ BW_cluster = P × times × (B_req + B_rsp) × 8 / T_cycle | |||
| 600 | Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link) | 597 | Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link) |
| 601 | ``` | 598 | ``` |
| 602 | 599 | ||
| 603 | - | ||
| 604 | ### 8.5 安全与运维影响 | 600 | ### 8.5 安全与运维影响 |
| 605 | 601 | ||
| 606 | - Dispatcher 使用 SSH/SCP,配置中可能包含明文密码;生产环境优先使用私钥,并限制配置文件权限。 | 602 | - Dispatcher 使用 SSH/SCP,配置中可能包含明文密码;生产环境优先使用私钥,并限制配置文件权限。 |
| @@ -612,8 +608,7 @@ Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link) | |||
| 612 | ### 9.1 兼容性 | 608 | ### 9.1 兼容性 |
| 613 | 609 | ||
| 614 | - 工具以独立进程运行,不修改训练进程或 HCCL/HCOMM 接口。 | 610 | - 工具以独立进程运行,不修改训练进程或 HCCL/HCOMM 接口。 |
| 615 | -- 当前推荐配置以 `schema_version=2`、`hosts[]`、`probe.scope` 为必需契约,`probe.topology`、`probe.pingpong` 为可选参数组;缺省 | 611 | +- 当前推荐配置以 `schema_version=2`、`hosts[]`、`probe.scope` 为必需契约,`probe.topology`、`probe.pingpong` 为可选参数组;缺省 `schema_version` 或 `schema_version=1` 继续兼容 `probe_scope + probe_topo.tracert + probe_controller.pingpong` 格式,缺少整组探测配置时保留旧 ranktable 路径,但不新增能力。 |
| 616 | - `schema_version` 或 `schema_version=1` 继续兼容 `probe_scope + probe_topo.tracert + probe_controller.pingpong` 格式,缺少整组探测配置时保留旧 ranktable 路径,但不新增能力。 | ||
| 617 | - JSON/TXT 产物一旦被外部系统消费,新增字段应保持向后兼容;破坏性修改必须升级 Schema 或文件名。 | 612 | - JSON/TXT 产物一旦被外部系统消费,新增字段应保持向后兼容;破坏性修改必须升级 Schema 或文件名。 |
| 618 | 613 | ||
| 619 | ### 9.2 特性开关 | 614 | ### 9.2 特性开关 |
| @@ -633,7 +628,6 @@ Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link) | |||
| 633 | 628 | ||
| 634 | 实验室 2 Host/16 Device:验证接口、产物和故障注入。 | 629 | 实验室 2 Host/16 Device:验证接口、产物和故障注入。 |
| 635 | 630 | ||
| 636 | - | ||
| 637 | ## 10. 测试方案 | 631 | ## 10. 测试方案 |
| 638 | 632 | ||
| 639 | ### 10.1 目标—用例映射 | 633 | ### 10.1 目标—用例映射 |
| @@ -647,7 +641,6 @@ Util_d = P_d × times × (B_req + B_rsp) × 8 / (T_cycle × C_link) | |||
| 647 | | T5 方程求解 | Q3、定位准确率 | 可解、超定、秩不足和 NaN 输入 | UT | | 641 | | T5 方程求解 | Q3、定位准确率 | 可解、超定、秩不足和 NaN 输入 | UT | |
| 648 | | T6 计数器异常 | Q2/Q3 | Key 缺失、非法值、Device ID 归一化、计数器增长与重置/回绕 | UT | | 642 | | T6 计数器异常 | Q2/Q3 | Key 缺失、非法值、Device ID 归一化、计数器增长与重置/回绕 | UT | |
| 649 | 643 | ||
| 650 | - | ||
| 651 | ### 10.2 UT 接入 | 644 | ### 10.2 UT 接入 |
| 652 | 645 | ||
| 653 | 当前仓库使用 CMake `BUILD_TESTS` 和 CTest,没有可用的根目录 `build.sh -u/-s` 入口。RFC 以仓库现状为准: | 646 | 当前仓库使用 CMake `BUILD_TESTS` 和 CTest,没有可用的根目录 `build.sh -u/-s` 入口。RFC 以仓库现状为准: |
| @@ -36,4 +36,4 @@ | |||
| 36 | ## 相关 | 36 | ## 相关 |
| 37 | 37 | ||
| 38 | - [RFC 模板](./0000-template.md) | 38 | - [RFC 模板](./0000-template.md) |
| 39 | -- [RFC 流程说明](./README.md) | 39 | +- [RFC 流程说明](./README.md) |
| @@ -1,4 +1,5 @@ | |||
| 1 | # Topology based CCL Monitor | 1 | # Topology based CCL Monitor |
| 2 | + | ||
| 2 | 本工具面向 RoCE 无损集群的通信亚健康监测。通过拓扑发现生成低开销探测任务,结合 HCCN PingPong 时延/通过率与 RNIC PFC、CNP 计数器,输出 L1 接入链路、L2 跨域路径指标及异常链路候选,适用于训练前网络检查和训练期间持续巡检。 | 3 | 本工具面向 RoCE 无损集群的通信亚健康监测。通过拓扑发现生成低开销探测任务,结合 HCCN PingPong 时延/通过率与 RNIC PFC、CNP 计数器,输出 L1 接入链路、L2 跨域路径指标及异常链路候选,适用于训练前网络检查和训练期间持续巡检。 |
| 3 | 4 | ||
| 4 |  | 5 |  |
| @@ -136,5 +137,5 @@ python3 ./plot/status_plot.py --input-dir "output/<时间>" | |||
| 136 | ## 详细文档 | 137 | ## 详细文档 |
| 137 | 138 | ||
| 138 | - [方案设计与接口契约](../../docs/zh/rfcs/0001-topology-based-ccl-monitor.md) | 139 | - [方案设计与接口契约](../../docs/zh/rfcs/0001-topology-based-ccl-monitor.md) |
| 139 | -- [用户运行手册](docs/用户运行v1.0.md) | 140 | +- [用户运行手册](docs/user_execution.md) |
| 140 | -- [系统环境与构建](docs/系统环境v1.0.md) | 141 | +- [系统环境与构建](docs/system_environment.md) |
| @@ -0,0 +1,141 @@ | |||
| 1 | +# Topology-Based CCL Monitor | ||
| 2 | + | ||
| 3 | +This tool monitors communication sub-health in RoCE lossless clusters. It generates low-overhead probing tasks through topology discovery, combines HCCN PingPong latency and pass rates with RNIC PFC and CNP counters, and outputs L1 access link metrics, L2 cross-domain path metrics, and abnormal link candidates. It is suitable for pre-training network checks and continuous inspection during training. | ||
| 4 | + | ||
| 5 | + | ||
| 6 | + | ||
| 7 | +## Key Capabilities | ||
| 8 | + | ||
| 9 | +- Discover device, host, ToR/Leaf, and cross-domain topologies based on Tracert and LLDP. | ||
| 10 | +- Use layered, domain-based design with ring-path sets to reduce probing overhead in large-scale clusters. | ||
| 11 | +- Collect P90/P99/Mean latency, pass rates, and PFC/CNP counters. | ||
| 12 | +- Solve L1 link metrics and generate L2 path metrics and anomaly candidates. | ||
| 13 | +- Support topology charts, link status charts, and raw JSON/TXT/CSV result output. | ||
| 14 | + | ||
| 15 | +The system consists of two parts: the controller node runs `probe_topo`, `probe_controller`, and deployment scripts; each target host runs `rpc_host`, which calls HCCN interfaces to perform NPU-side probing. | ||
| 16 | + | ||
| 17 | +## Environment and Build | ||
| 18 | + | ||
| 19 | +The runtime environment must provide a matching version of Ascend CANN/HCCL/ACL and support Ubuntu or EulerOS/openEuler. First, set the installation paths: | ||
| 20 | + | ||
| 21 | +```bash | ||
| 22 | +export THIRDLIB_ROOT=/usr/local/third_lib | ||
| 23 | +export ASCEND_HOME_PATH=/usr/local/Ascend | ||
| 24 | +export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux | ||
| 25 | +``` | ||
| 26 | + | ||
| 27 | +Install system dependencies (choose one) and third-party libraries: | ||
| 28 | + | ||
| 29 | +```bash | ||
| 30 | +# Ubuntu | ||
| 31 | +sudo ./install.sh --install-system-packages --system ubuntu | ||
| 32 | + | ||
| 33 | +# EulerOS/openEuler | ||
| 34 | +sudo ./install.sh --install-system-packages --system euleros | ||
| 35 | + | ||
| 36 | +sudo -E ./install.sh --prefix "$THIRDLIB_ROOT" --jobs "$(nproc)" --strict-prereq | ||
| 37 | +``` | ||
| 38 | + | ||
| 39 | +Build the project: | ||
| 40 | + | ||
| 41 | +```bash | ||
| 42 | +source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" | ||
| 43 | + | ||
| 44 | +cmake -S . -B build \ | ||
| 45 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | ||
| 46 | + -DCMAKE_BUILD_TYPE=Release | ||
| 47 | +cmake --build build -j"$(nproc)" | ||
| 48 | +``` | ||
| 49 | + | ||
| 50 | +The main artifacts are `build/rpc_host`, `build/probe_topo`, and `build/probe_controller`. | ||
| 51 | + | ||
| 52 | +## Configuration | ||
| 53 | + | ||
| 54 | +Before running, edit `control_json/910b2_info.json` and focus on the following settings: | ||
| 55 | + | ||
| 56 | +- `schema_version`: Use `2` for new configurations; the default or `1` remains backward compatible with the legacy format. | ||
| 57 | +- `hosts`: An array of host identities. `id` is a stable reference name and `ip` is the connection address. For login, `ssh_key` is recommended; passwords can be read from the environment using `password_env`. | ||
| 58 | +- When `ssh_key` and `password_env` are configured together, SSH attempts key/agent/local key authentication first; the password read by `password_env` is used for password authentication, encrypted private key passphrase, and as the `su` fallback password when `su_password` is not configured. | ||
| 59 | +- `controller`: The controller node host ID, for example `"node-01"`. | ||
| 60 | +- `probe.scope`: Configure probe devices by host ID using `{"device_range": [0, 7]}` or `{"devices": [0, 1, 3, 5]}`. | ||
| 61 | +- `probe.topology` / `probe.pingpong`: Topology discovery and continuous probing parameters. | ||
| 62 | + | ||
| 63 | +Use the migration command to generate a v2 template from a legacy configuration: | ||
| 64 | + | ||
| 65 | +```bash | ||
| 66 | +./run.sh migrate-config ./control_json/old.json ./control_json/new.json | ||
| 67 | +``` | ||
| 68 | + | ||
| 69 | +The same configuration file must be distributed to all target hosts. The configuration file may contain passwords and private key paths and should be set to owner read/write only: | ||
| 70 | + | ||
| 71 | +```bash | ||
| 72 | +chmod 600 ./control_json/910b2_info.json | ||
| 73 | +``` | ||
| 74 | + | ||
| 75 | +For the configuration format and field constraints, refer to the authoritative RFC in this repository: [Configuration File](../../docs/zh/rfcs/0001-topology-based-ccl-monitor.md#42-配置文件). | ||
| 76 | + | ||
| 77 | +## Quick Start | ||
| 78 | + | ||
| 79 | +Before running, edit `control_json/910b2_info.json` as described in the "Configuration" section. | ||
| 80 | + | ||
| 81 | +Run the following commands from the project root directory. Terminal 0 is used for remote deployment and starting the host service. Terminal 1 is used for topology discovery and continuous monitoring. | ||
| 82 | + | ||
| 83 | +```bash | ||
| 84 | +# Terminal 0: Clean up residual processes, distribute binaries and configuration, start rpc_host | ||
| 85 | +./run.sh deploy | ||
| 86 | + | ||
| 87 | +# Optional: Enable NPU-side PingPong local logging when starting rpc_host | ||
| 88 | +./run.sh deploy --pingpong-log /root/output | ||
| 89 | +``` | ||
| 90 | + | ||
| 91 | +Open a new terminal, reset the variables from "Environment and Build", load `env.sh`, and then run: | ||
| 92 | + | ||
| 93 | +```bash | ||
| 94 | +# Terminal 1: Discover topology | ||
| 95 | +./run.sh topo | ||
| 96 | + | ||
| 97 | +# Optional: Generate topology charts immediately after discovery | ||
| 98 | +./run.sh topo --plot | ||
| 99 | + | ||
| 100 | +# Start HCCN PingPong, metric solving, and anomaly analysis | ||
| 101 | +./run.sh probe | ||
| 102 | +``` | ||
| 103 | + | ||
| 104 | +Common `probe_controller` options: | ||
| 105 | + | ||
| 106 | +| Option | Description | | ||
| 107 | +| --- | --- | | ||
| 108 | +| `--print-pingpong-plan` | Print the probe plan only; do not execute probing | | ||
| 109 | +| `--l1-only` | Probe and solve L1 only; disable L2 output | | ||
| 110 | +| `--no-metrics` | Disable PFC/CNP counter collection | | ||
| 111 | + | ||
| 112 | +## Output and Visualization | ||
| 113 | + | ||
| 114 | +Topology discovery results are located in `output/` and include `probe_topo.json`, `probe_topo_lldp.json`, and `allpath.json`. When L2 path discovery is enabled, `l2_fullmesh_path.json` is also generated. | ||
| 115 | + | ||
| 116 | +Continuous monitoring results are located in `output/<time>/`: | ||
| 117 | + | ||
| 118 | +- `link_lat.txt`, `link_pass_rate.txt`: L1 link latency and pass rate. | ||
| 119 | +- `l2_status/`: L2 path latency and pass rate. | ||
| 120 | +- `metrics/`: PFC/CNP counters. | ||
| 121 | +- `bad_link.txt`: Persistent anomaly alerts. | ||
| 122 | +- `bad_link_candidate.txt`: Latency candidate events. | ||
| 123 | + | ||
| 124 | +Generate topology charts and the most recent monitoring status charts: | ||
| 125 | + | ||
| 126 | +```bash | ||
| 127 | +python3 ./plot/topo_plot.py | ||
| 128 | +python3 ./plot/status_plot.py | ||
| 129 | +``` | ||
| 130 | + | ||
| 131 | +You can also specify a monitoring result directory: | ||
| 132 | + | ||
| 133 | +```bash | ||
| 134 | +python3 ./plot/status_plot.py --input-dir "output/<time>" | ||
| 135 | +``` | ||
| 136 | + | ||
| 137 | +## Detailed Documentation | ||
| 138 | + | ||
| 139 | +- [Design and Interface Contract](../../docs/zh/rfcs/0001-topology-based-ccl-monitor.md) | ||
| 140 | +- [User Execution Guide](docs/user_execution.md) | ||
| 141 | +- [System Environment and Build](docs/system_environment.md) | ||
| @@ -0,0 +1 @@ | |||
| 1 | +<svg width="3222" height="1801" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" overflow="hidden"><g transform="translate(-664 -325)"><g><rect x="835.5" y="1371.5" width="2104" height="715" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none"/><rect x="1318.5" y="1505.5" width="1588" height="545" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#E8E8E8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 868.401 1455)">Agent</text><rect x="706.5" y="455.5" width="1009" height="427" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><rect x="1768.5" y="455.5" width="1234" height="422" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><rect x="2435.5" y="1740.5" width="428" height="157" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#A6CAEC" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2520.48 1840)">Time Analysis</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 729.986 532)">Controller</text><rect x="765.5" y="580.5" width="432" height="228" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="40" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 788.838 716)">Topo Partition</text><rect x="1341" y="1629" width="857" height="198" fill="#4E95D9" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1521.93 1697)">Counter Collection</text><path d="M2412.34 1820.11 2187.12 1794.92C2185.23 1794.71 2183.87 1793 2184.08 1791.12 2184.3 1789.23 2186 1787.87 2187.88 1788.08L2413.1 1813.28C2414.99 1813.49 2416.35 1815.19 2416.14 1817.07 2415.93 1818.96 2414.23 1820.32 2412.34 1820.11ZM2409.69 1802.52 2435.5 1819.24 2406.64 1829.85ZM2185.97 1805.16C2178.42 1804.32 2172.99 1797.52 2173.84 1789.97 2174.68 1782.42 2181.48 1776.99 2189.03 1777.84 2196.58 1778.68 2202.01 1785.48 2201.17 1793.03 2200.32 1800.58 2193.52 1806.01 2185.97 1805.16Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2264.44 1939)">Latency,</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2264.44 2005)">Pass Rate</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2258.42 1679)">PFC, </text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="55" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2258.42 1745)">CNP,…</text><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="83" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 3505.66 1617)">③</text><rect x="869.5" y="1508.5" width="382" height="545" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 931.448 1725)">Topo Probing</text><rect x="2362.5" y="672.5" width="616" height="99" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#96DCF8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2477.85 732)">Link Solving</text><rect x="1808.5" y="643.5" width="507" height="165" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="round" stroke-miterlimit="10" stroke-opacity="1" fill="#96DCF8" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1883.53 703)">Z-Score-Based</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2011.86 703)"></text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2046.23 703)"></text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1868.63 780)">Link Assessment</text><rect x="1341" y="1838" width="856" height="198" fill="#4E95D9" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1603.68 1906)">Active Probing</text><rect x="1032.5" y="1267.5" width="297" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1086.81 1352)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1242.07 1352)">0</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="83" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2107.46 1345)">…</text><rect x="1640.5" y="1277.5" width="296" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1694.49 1362)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1849.75 1362)">1</text><rect x="2356.5" y="1279.5" width="314" height="125" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2407.21 1364)">NPU</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2562.47 1364)">M</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1800.96 543)">Analyzer</text><path d="M0 0 276.615 110.458" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1 0 0 -1 1180.5 1266.96)"/><path d="M0 0 19.2323 72.3369" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1 0 0 -1 1784.5 1279.84)"/><path d="M2392.16 1279.28 2117.5 1121.5" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><path d="M1474.9 1025.98C1465.57 994.312 1513.62 965.151 1582.23 960.844 1610.03 959.098 1638.3 961.707 1662.5 968.253 1688.15 945.937 1748.09 937.455 1796.38 949.309 1804.83 951.382 1812.58 954.01 1819.39 957.109 1839.37 938.61 1888.03 931.095 1928.08 940.323 1939.16 942.877 1948.83 946.57 1956.34 951.12 1988.54 933.626 2045.44 931.465 2083.43 946.294 2099.4 952.526 2110.16 961.122 2113.87 970.608 2166.63 977.253 2197.75 1002.42 2183.39 1026.82 2182.18 1028.87 2180.66 1030.88 2178.85 1032.84 2221.16 1058.27 2210.8 1094.71 2155.71 1114.24 2138.57 1120.32 2118.31 1124.25 2096.9 1125.67 2096.43 1153.02 2047.96 1175.02 1988.65 1174.8 1968.83 1174.73 1949.45 1172.13 1932.64 1167.29 1912.58 1197.96 1842.6 1215.29 1776.34 1206 1748.57 1202.11 1724.58 1193.88 1708.46 1182.72 1640.62 1201.61 1552.56 1191.45 1511.78 1160.03 1511.27 1159.63 1510.76 1159.23 1510.27 1158.83 1465.87 1161.23 1425.65 1146.6 1420.44 1126.17 1417.66 1115.27 1425.45 1104.37 1441.74 1096.36 1403.29 1085.91 1390.38 1062.98 1412.89 1045.14 1425.88 1034.85 1448.66 1028.05 1474.23 1026.82Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#FFFFFF" fill-rule="evenodd" fill-opacity="1"/><path d="M1489.62 1100.32C1473.21 1100.91 1456.8 1099.16 1442.6 1095.3M1531.12 1155.23C1524.52 1156.45 1517.6 1157.25 1510.54 1157.64M1708.41 1181.62C1703.45 1178.19 1699.29 1174.51 1696.01 1170.67M1937.67 1154.3C1936.94 1158.38 1935.28 1162.41 1932.72 1166.32M2036.1 1080.03C2073.31 1088.39 2096.79 1105.86 2096.46 1124.95M2178.47 1032.17C2172.45 1038.67 2163.25 1044.44 2151.59 1049.02M2113.98 969.664C2115 972.289 2115.48 974.953 2115.4 977.62M1942.33 960.383C1945.83 956.712 1950.47 953.294 1956.1 950.236M1813.54 965.218C1814.97 962.185 1817.21 959.245 1820.21 956.467M1662.41 968.19C1671.17 970.559 1679.27 973.41 1686.54 976.68M1479.12 1034.91C1477.21 1031.99 1475.8 1029 1474.91 1025.98" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="73" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1583.12 1090)">RoCE</text><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="73" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1781.08 1090)">Fabric</text><path d="M0 3.4375-27.5 3.4375-27.5-3.4375 0-3.4375ZM-48.125 3.4375-75.625 3.4375-75.625-3.4375-48.125-3.4375ZM-79.0617 13.7508-79.0617 41.2508-85.9367 41.2508-85.9367 13.7508ZM-79.0617 61.8758-79.0617 89.3757-85.9367 89.3757-85.9367 61.8758ZM-79.0617 110.001-79.0617 137.501-85.9367 137.501-85.9367 110.001ZM-79.0617 158.126-79.0617 185.626-85.9367 185.626-85.9367 158.126ZM-79.0617 206.251-79.0617 233.751-85.9367 233.751-85.9367 206.251ZM-79.0617 254.376-79.0617 281.876-85.9367 281.876-85.9367 254.376ZM-79.0617 302.501-79.0617 330.001-85.9367 330.001-85.9367 302.501ZM-79.0617 350.626-79.0617 378.126-85.9367 378.126-85.9367 350.626ZM-79.0617 398.751-79.0617 426.251-85.9367 426.251-85.9367 398.751ZM-79.0617 446.876-79.0617 474.376-85.9367 474.376-85.9367 446.876ZM-79.0617 495.001-79.0617 522.501-85.9367 522.501-85.9367 495.001ZM-79.0617 543.126-79.0617 570.626-85.9367 570.626-85.9367 543.126ZM-79.0617 591.251-79.0617 618.751-85.9367 618.751-85.9367 591.251ZM-79.0617 639.376-79.0617 666.876-85.9367 666.876-85.9367 639.376ZM-79.0617 687.501-79.0617 715.001-85.9367 715.001-85.9367 687.501ZM-79.0617 735.626-79.0617 763.126-85.9367 763.126-85.9367 735.626ZM-79.0617 783.751-79.0617 811.251-85.9367 811.251-85.9367 783.751ZM-79.0617 831.876-79.0617 842.294-82.4992 838.857-65.418 838.857-65.418 845.732-85.9367 845.732-85.9367 831.876ZM-44.793 838.857-17.293 838.857-17.293 845.732-44.793 845.732ZM3.33198 838.857 30.832 838.857 30.832 845.732 3.33198 845.732ZM51.457 838.857 78.957 838.857 78.957 845.732 51.457 845.732ZM99.582 838.857 115.806 838.857 115.806 857.008 108.931 857.008 108.931 842.294 112.368 845.732 99.582 845.732ZM115.806 877.633 115.806 905.133 108.931 905.133 108.931 877.633ZM115.806 925.758 115.806 949.418 108.931 949.418 108.931 925.758ZM126.118 944.835 112.368 972.335 98.6182 944.835Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 -1.22465e-16 -1.22465e-16 -1 869.5 1780.83)"/><path d="M0 3.4375-27.5 3.4375-27.5-3.4375 0-3.4375ZM-48.125 3.4375-75.625 3.4375-75.625-3.4375-48.125-3.4375ZM-96.25 3.4375-123.75 3.4375-123.75-3.4375-96.25-3.4375ZM-144.375 3.4375-150.117 3.4375-150.117-25.1958-143.242-25.1958-143.242 0-146.679-3.4375-144.375-3.4375ZM-150.117-45.8209-150.117-73.3208-143.242-73.3208-143.242-45.8209ZM-150.117-93.9458-150.117-121.446-143.242-121.446-143.242-93.9458ZM-150.117-142.071-150.117-169.571-143.242-169.571-143.242-142.071ZM-150.117-190.196-150.117-217.696-143.242-217.696-143.242-190.196ZM-150.117-238.321-150.117-265.821-143.242-265.821-143.242-238.321ZM-150.117-286.446-150.117-313.946-143.242-313.946-143.242-286.446ZM-150.117-334.571-150.117-362.071-143.242-362.071-143.242-334.571ZM-150.117-382.696-150.117-410.196-143.242-410.196-143.242-382.696ZM-150.117-430.821-150.117-458.321-143.242-458.321-143.242-430.821ZM-150.117-478.946-150.117-506.446-143.242-506.446-143.242-478.946ZM-150.117-527.071-150.117-554.571-143.242-554.571-143.242-527.071ZM-150.117-575.196-150.117-602.696-143.242-602.696-143.242-575.196ZM-150.117-623.321-150.117-650.821-143.242-650.821-143.242-623.321ZM-150.117-671.446-150.117-698.946-143.242-698.946-143.242-671.446ZM-150.117-719.571-150.117-747.071-143.242-747.071-143.242-719.571ZM-150.117-767.696-150.117-787.395-135.44-787.395-135.44-780.52-146.679-780.52-143.242-783.957-143.242-767.696ZM-114.815-787.395-87.3153-787.395-87.3153-780.52-114.815-780.52ZM-66.6903-787.395-39.1903-787.395-39.1903-780.52-66.6903-780.52ZM-18.5653-787.395 8.93467-787.395 8.93467-780.52-18.5653-780.52ZM29.5597-787.395 57.0597-787.395 57.0597-780.52 29.5597-780.52ZM77.6847-787.395 105.185-787.395 105.185-780.52 77.6847-780.52ZM125.81-787.395 153.31-787.395 153.31-780.52 125.81-780.52ZM173.935-787.395 201.435-787.395 201.435-780.52 173.935-780.52ZM222.06-787.395 249.56-787.395 249.56-780.52 222.06-780.52ZM270.185-787.395 297.685-787.395 297.685-780.52 270.185-780.52ZM318.31-787.395 345.81-787.395 345.81-780.52 318.31-780.52ZM366.435-787.395 393.935-787.395 393.935-780.52 366.435-780.52ZM414.56-787.395 442.06-787.395 442.06-780.52 414.56-780.52ZM462.685-787.395 490.185-787.395 490.185-780.52 462.685-780.52ZM510.81-787.395 538.31-787.395 538.31-780.52 510.81-780.52ZM558.935-787.395 586.435-787.395 586.435-780.52 558.935-780.52ZM607.06-787.395 634.56-787.395 634.56-780.52 607.06-780.52ZM655.185-787.395 682.685-787.395 682.685-780.52 655.185-780.52ZM703.31-787.395 730.81-787.395 730.81-780.52 703.31-780.52ZM751.435-787.395 778.935-787.395 778.935-780.52 751.435-780.52ZM799.56-787.395 827.06-787.395 827.06-780.52 799.56-780.52ZM847.685-787.395 875.185-787.395 875.185-780.52 847.685-780.52ZM895.81-787.395 923.31-787.395 923.31-780.52 895.81-780.52ZM943.935-787.395 971.435-787.395 971.435-780.52 943.935-780.52ZM992.06-787.395 1019.56-787.395 1019.56-780.52 992.06-780.52ZM1040.18-787.395 1067.68-787.395 1067.68-780.52 1040.18-780.52ZM1088.31-787.395 1115.81-787.395 1115.81-780.52 1088.31-780.52ZM1136.43-787.395 1163.93-787.395 1163.93-780.52 1136.43-780.52ZM1184.56-787.395 1212.06-787.395 1212.06-780.52 1184.56-780.52ZM1232.68-787.395 1260.18-787.395 1260.18-780.52 1232.68-780.52ZM1280.81-787.395 1308.31-787.395 1308.31-780.52 1280.81-780.52ZM1328.93-787.395 1356.43-787.395 1356.43-780.52 1328.93-780.52ZM1377.06-787.395 1404.56-787.395 1404.56-780.52 1377.06-780.52ZM1425.18-787.395 1452.68-787.395 1452.68-780.52 1425.18-780.52ZM1473.31-787.395 1500.81-787.395 1500.81-780.52 1473.31-780.52ZM1521.43-787.395 1542.1-787.395 1542.1-773.682 1535.22-773.682 1535.22-783.957 1538.66-780.52 1521.43-780.52ZM1542.1-753.057 1542.1-725.557 1535.22-725.557 1535.22-753.057ZM1542.1-704.932 1542.1-677.432 1535.22-677.432 1535.22-704.932ZM1542.1-656.807 1542.1-629.307 1535.22-629.307 1535.22-656.807ZM1542.1-608.682 1542.1-581.182 1535.22-581.182 1535.22-608.682ZM1542.1-560.557 1542.1-533.057 1535.22-533.057 1535.22-560.557ZM1542.1-512.432 1542.1-484.932 1535.22-484.932 1535.22-512.432ZM1542.1-464.307 1542.1-436.807 1535.22-436.807 1535.22-464.307ZM1542.1-416.182 1542.1-388.682 1535.22-388.682 1535.22-416.182ZM1542.1-368.057 1542.1-340.557 1535.22-340.557 1535.22-368.057ZM1542.1-319.932 1542.1-292.432 1535.22-292.432 1535.22-319.932ZM1542.1-271.807 1542.1-244.307 1535.22-244.307 1535.22-271.807ZM1542.1-223.682 1542.1-196.182 1535.22-196.182 1535.22-223.682ZM1542.1-175.557 1542.1-148.057 1535.22-148.057 1535.22-175.557ZM1542.1-127.432 1542.1-99.9321 1535.22-99.9321 1535.22-127.432ZM1542.1-79.3071 1542.1-51.8071 1535.22-51.8071 1535.22-79.3071ZM1542.1-31.1821 1542.1-3.68209 1535.22-3.68209 1535.22-31.1821ZM1542.1 16.9429 1542.1 44.4429 1535.22 44.4429 1535.22 16.9429ZM1542.1 65.0679 1542.1 92.5679 1535.22 92.5679 1535.22 65.0679ZM1542.1 113.193 1542.1 140.693 1535.22 140.693 1535.22 113.193ZM1542.1 161.318 1542.1 188.818 1535.22 188.818 1535.22 161.318ZM1542.1 209.443 1542.1 236.943 1535.22 236.943 1535.22 209.443ZM1542.1 257.568 1542.1 285.068 1535.22 285.068 1535.22 257.568ZM1537.99 308.458 1510.49 308.458 1510.49 301.583 1537.99 301.583ZM1489.86 308.458 1479.07 308.458 1479.07 301.583 1489.86 301.583ZM1483.65 318.77 1456.15 305.02 1483.65 291.27Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1.83697e-16 1 1 -1.83697e-16 1464.5 580.5)"/><text fill="#000000" fill-opacity="1" font-family="Goudy Old Style,Goudy Old Style_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="semi-expanded" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1361.81 1597)">Executor</text><path d="M0-3.4375 27.5-3.4375 27.5 3.4375 0 3.4375ZM48.125-3.4375 75.625-3.4375 75.625 3.4375 48.125 3.4375ZM96.25-3.4375 123.75-3.4375 123.75 3.4375 96.25 3.4375ZM144.375-3.4375 171.875-3.4375 171.875 3.4375 144.375 3.4375ZM192.5-3.4375 201.903-3.4375 201.903 21.5343 195.028 21.5343 195.028 0 198.466 3.4375 192.5 3.4375ZM201.903 42.1593 201.903 69.6593 195.028 69.6593 195.028 42.1593ZM201.903 90.2843 201.903 117.784 195.028 117.784 195.028 90.2843ZM201.903 138.409 201.903 165.909 195.028 165.909 195.028 138.409ZM201.903 186.534 201.903 214.034 195.028 214.034 195.028 186.534ZM201.903 234.659 201.903 262.159 195.028 262.159 195.028 234.659ZM201.903 282.784 201.903 310.284 195.028 310.284 195.028 282.784ZM201.903 330.909 201.903 358.409 195.028 358.409 195.028 330.909ZM201.903 379.034 201.903 406.534 195.028 406.534 195.028 379.034ZM201.903 427.159 201.903 454.659 195.028 454.659 195.028 427.159ZM201.903 475.284 201.903 502.784 195.028 502.784 195.028 475.284ZM201.903 523.409 201.903 550.909 195.028 550.909 195.028 523.409ZM201.903 571.534 201.903 599.034 195.028 599.034 195.028 571.534ZM201.903 619.659 201.903 647.159 195.028 647.159 195.028 619.659ZM201.903 667.784 201.903 695.284 195.028 695.284 195.028 667.784ZM201.903 715.909 201.903 743.409 195.028 743.409 195.028 715.909ZM201.903 764.034 201.903 791.534 195.028 791.534 195.028 764.034ZM201.903 812.159 201.903 839.659 195.028 839.659 195.028 812.159ZM201.903 860.284 201.903 887.784 195.028 887.784 195.028 860.284ZM201.903 908.409 201.903 935.909 195.028 935.909 195.028 908.409ZM201.903 956.534 201.903 984.034 195.028 984.034 195.028 956.534ZM201.903 1004.66 201.903 1032.16 195.028 1032.16 195.028 1004.66ZM201.903 1052.78 201.903 1080.28 195.028 1080.28 195.028 1052.78ZM194.789 1100.67 167.289 1100.67 167.289 1093.8 194.789 1093.8ZM146.664 1100.67 138.882 1100.67 138.882 1093.8 146.664 1093.8ZM143.466 1110.98 115.966 1097.23 143.466 1083.48Z" fill="#0E2841" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 0 0 -1 2863.5 1818.73)"/><path d="M23.1157-1.6294 47.9136 0.32168C49.8062 0.470591 51.2197 2.12559 51.0708 4.01822 50.9219 5.91085 49.2669 7.32441 47.3743 7.1755L22.5764 5.22442C20.6838 5.07551 19.2702 3.42051 19.4192 1.52788 19.5681-0.364746 21.2231-1.77831 23.1157-1.6294ZM26.3368 15.8647 0 0 28.4938-11.5506ZM48.7224-9.95905C56.293-9.36341 61.9472-2.74342 61.3516 4.8271 60.7559 12.3976 54.1359 18.0519 46.5654 17.4562 38.9949 16.8606 33.3406 10.2406 33.9363 2.67008 34.5319-4.90044 41.1519-10.5547 48.7224-9.95905Z" fill="#000000" fill-rule="nonzero" fill-opacity="1" transform="matrix(1 0 0 -1 2315.5 725.249)"/><path d="M2240.44 1135.69 2158.17 810.343C2157.7 808.502 2158.82 806.633 2160.66 806.167 2162.5 805.702 2164.37 806.817 2164.83 808.657L2247.1 1134C2247.57 1135.84 2246.46 1137.71 2244.61 1138.18 2242.77 1138.64 2240.91 1137.53 2240.44 1135.69ZM2255.98 1127.03 2249.39 1157.06 2229.32 1133.77ZM2148.17 812.871C2146.31 805.509 2150.77 798.031 2158.13 796.17 2165.49 794.308 2172.97 798.767 2174.83 806.129 2176.69 813.491 2172.23 820.969 2164.87 822.83 2157.51 824.692 2150.03 820.233 2148.17 812.871Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><path d="M2219 1185.5C2219 1169.21 2232.21 1156 2248.5 1156 2264.79 1156 2278 1169.21 2278 1185.5 2278 1201.79 2264.79 1215 2248.5 1215 2232.21 1215 2219 1201.79 2219 1185.5Z" stroke="#042433" stroke-width="13.75" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><path d="M2269 1207 2294 1235.94" stroke="#0E2841" stroke-width="13.75" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd"/><rect x="2297.5" y="1009.5" width="622" height="169" stroke="#000000" stroke-width="4.58333" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#F6C6AD" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="44" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 2443.01 1111)">Anomaly Output</text><path d="M1118 1501.87 1158.63 1461.16 1146.99 1449.54 1193.53 1449.5 1193.57 1496.04 1181.92 1484.41 1141.29 1525.12Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEBF8" fill-rule="evenodd" fill-opacity="1"/><path d="M1819.73 1529.21 1829.18 1472.48 1812.95 1469.77 1850.82 1442.72 1877.87 1480.59 1861.64 1477.89 1852.18 1534.62Z" stroke="#042433" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#D1D1D1" fill-rule="evenodd" fill-opacity="1"/><path d="M0 0 53.6921 0 53.6921 1332.72 12.4421 1332.72" stroke="#0E2841" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(1.83697e-16 1 1 -1.83697e-16 1180.5 1392.5)"/><path d="M0 0 0.000360892 45.4724" stroke="#0E2841" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="none" fill-rule="evenodd" transform="matrix(-1 0 0 1 1783.5 1402.5)"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(6.12323e-17 1 -1 6.12323e-17 721.193 1156)">Topo Probing Results</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 839.244 409)">Probe Path List</text><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(6.12323e-17 1 -1 6.12323e-17 2990.24 985)">Network Status Results</text><rect x="1248.5" y="580.5" width="432" height="232" stroke="#000000" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#DCEAF7" fill-opacity="1"/><text fill="#000000" fill-opacity="1" font-family="DengXian,DengXian_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="700" font-stretch="normal" font-size="48" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1271.16 718)">Probe Plan</text><path d="M1242.76 697.124 1200.48 696.937C1198.59 696.929 1197.05 695.383 1197.06 693.485 1197.07 691.586 1198.62 690.054 1200.52 690.062L1242.79 690.249 1242.79 690.249C1244.69 690.258 1246.22 691.803 1246.21 693.702 1246.2 695.6 1244.66 697.132 1242.76 697.124ZM1238.25 679.917 1265.69 693.788 1238.13 707.416ZM1200.44 707.25C1192.85 707.216 1186.72 701.033 1186.75 693.439 1186.78 685.845 1192.97 679.716 1200.56 679.75 1208.15 679.784 1214.28 685.967 1214.25 693.561 1214.22 701.154 1208.03 707.283 1200.44 707.25Z" fill="#000000" fill-rule="nonzero" fill-opacity="1"/><path d="M22.9023 3.53092 252.373 74.9825C254.186 75.5469 255.198 77.4739 254.633 79.2865 254.069 81.0992 252.142 82.1111 250.329 81.5466L20.8584 10.0951C19.0457 9.53065 18.0338 7.60367 18.5982 5.79103 19.1627 3.97839 21.0896 2.96651 22.9023 3.53092ZM22.1688 21.304 0 0 30.3444-4.95264ZM255.439 65.1363C262.689 67.3939 266.737 75.1018 264.479 82.3524 262.222 89.603 254.514 93.6505 247.263 91.3929 240.013 89.1352 235.965 81.4273 238.223 74.1767 240.48 66.9262 248.188 62.8786 255.439 65.1363Z" fill="#000000" fill-rule="nonzero" fill-opacity="1" transform="matrix(-1 0 0 1 2435.85 1819.5)"/><rect x="902.5" y="1791.5" width="315" height="172" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 932.581 1853)">hccn_tool</text><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 930.862 1930)">traceroute</text><rect x="1394.5" y="1919.5" width="723" height="89" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1609.33 1981)">HccnRping</text><rect x="1411.5" y="1718.5" width="723" height="89" stroke="#1C440D" stroke-width="6.875" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="8" stroke-opacity="1" fill="#4EA72E" fill-opacity="1"/><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1593.27 1780)">hccn_tool</text><text fill="#FFFFFF" fill-opacity="1" font-family="Times New Roman,Times New Roman_MSFontService,sans-serif" font-style="normal" font-variant="normal" font-weight="400" font-stretch="normal" font-size="64" text-anchor="start" direction="ltr" writing-mode="lr-tb" unicode-bidi="normal" text-decoration="none" transform="matrix(1 0 0 1 1862.54 1780)">stat</text></g></g></svg> | ||
Rexperimental/cluster_link_diag/docs/系统环境v1.0.md→experimental/cluster_link_diag/docs/system_environment.md+36-22
| @@ -1,28 +1,41 @@ | |||
| 1 | # 项目构建 | 1 | # 项目构建 |
| 2 | + | ||
| 2 | ## 安装系统环境和三方库 | 3 | ## 安装系统环境和三方库 |
| 3 | -1、指定目录 | 4 | + |
| 4 | -```bash | 5 | +1. 指定目录。 |
| 5 | -export THIRDLIB_ROOT=/usr/local/third_lib | 6 | + |
| 6 | -export ASCEND_HOME_PATH=/usr/local/Ascend | 7 | + ```bash |
| 7 | -export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux | 8 | + export THIRDLIB_ROOT=/usr/local/third_lib |
| 8 | -``` | 9 | + export ASCEND_HOME_PATH=/usr/local/Ascend |
| 9 | -2、安装系统环境 | 10 | + export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux |
| 10 | -Ubuntu | 11 | + ``` |
| 11 | -```bash | 12 | + |
| 12 | -sudo ./install.sh --install-system-packages --system ubuntu | 13 | +2. 安装系统环境。 |
| 13 | -``` | 14 | + |
| 14 | -EulerOS/openEuler | 15 | + Ubuntu: |
| 15 | -```bash | 16 | + |
| 16 | -sudo ./install.sh --install-system-packages --system euleros | 17 | + ```bash |
| 17 | -``` | 18 | + sudo ./install.sh --install-system-packages --system ubuntu |
| 18 | -3、安装第三方库 | 19 | + ``` |
| 19 | -C/C++:CLI11、nlohmann_json、Eigen、rpclib、可选 fmt/spdlog | 20 | + |
| 20 | -Python venv:paramiko、scp、matplotlib | 21 | + EulerOS/openEuler: |
| 21 | -```bash | 22 | + |
| 22 | -sudo -E ./install.sh --prefix "$THIRDLIB_ROOT" --jobs "$(nproc)" --strict-prereq | 23 | + ```bash |
| 23 | -``` | 24 | + sudo ./install.sh --install-system-packages --system euleros |
| 25 | + ``` | ||
| 26 | + | ||
| 27 | +3. 安装第三方库。 | ||
| 28 | + | ||
| 29 | + C/C++:CLI11、nlohmann_json、Eigen、rpclib、可选 fmt/spdlog。 | ||
| 30 | + | ||
| 31 | + Python venv:paramiko、scp、matplotlib。 | ||
| 32 | + | ||
| 33 | + ```bash | ||
| 34 | + sudo -E ./install.sh --prefix "$THIRDLIB_ROOT" --jobs "$(nproc)" --strict-prereq | ||
| 35 | + ``` | ||
| 24 | 36 | ||
| 25 | ## 编译 | 37 | ## 编译 |
| 38 | + | ||
| 26 | ```bash | 39 | ```bash |
| 27 | source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" | 40 | source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" |
| 28 | 41 | ||
| @@ -34,6 +47,7 @@ cmake --build build -j"$(nproc)" | |||
| 34 | ``` | 47 | ``` |
| 35 | 48 | ||
| 36 | ## 产物 | 49 | ## 产物 |
| 50 | + | ||
| 37 | ```bash | 51 | ```bash |
| 38 | ./build/rpc_host | 52 | ./build/rpc_host |
| 39 | ./build/probe_topo | 53 | ./build/probe_topo |
| @@ -45,4 +59,4 @@ cmake --build build -j"$(nproc)" | |||
| 45 | 59 | ||
| 46 | ```bash | 60 | ```bash |
| 47 | ./build/internal/ | 61 | ./build/internal/ |
| 48 | -``` | 62 | +``` |
| @@ -0,0 +1,62 @@ | |||
| 1 | +# Project Build | ||
| 2 | + | ||
| 3 | +## Install System Environment and Third-Party Libraries | ||
| 4 | + | ||
| 5 | +1. Specify directories. | ||
| 6 | + | ||
| 7 | + ```bash | ||
| 8 | + export THIRDLIB_ROOT=/usr/local/third_lib | ||
| 9 | + export ASCEND_HOME_PATH=/usr/local/Ascend | ||
| 10 | + export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux | ||
| 11 | + ``` | ||
| 12 | + | ||
| 13 | +2. Install the system environment. | ||
| 14 | + | ||
| 15 | + Ubuntu: | ||
| 16 | + | ||
| 17 | + ```bash | ||
| 18 | + sudo ./install.sh --install-system-packages --system ubuntu | ||
| 19 | + ``` | ||
| 20 | + | ||
| 21 | + EulerOS/openEuler: | ||
| 22 | + | ||
| 23 | + ```bash | ||
| 24 | + sudo ./install.sh --install-system-packages --system euleros | ||
| 25 | + ``` | ||
| 26 | + | ||
| 27 | +3. Install third-party libraries. | ||
| 28 | + | ||
| 29 | + C/C++: CLI11, nlohmann_json, Eigen, rpclib, and optional fmt/spdlog. | ||
| 30 | + | ||
| 31 | + Python venv: paramiko, scp, matplotlib. | ||
| 32 | + | ||
| 33 | + ```bash | ||
| 34 | + sudo -E ./install.sh --prefix "$THIRDLIB_ROOT" --jobs "$(nproc)" --strict-prereq | ||
| 35 | + ``` | ||
| 36 | + | ||
| 37 | +## Compilation | ||
| 38 | + | ||
| 39 | +```bash | ||
| 40 | +source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" | ||
| 41 | + | ||
| 42 | +cmake -S . -B build \ | ||
| 43 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | ||
| 44 | + -DCMAKE_BUILD_TYPE=Release | ||
| 45 | + | ||
| 46 | +cmake --build build -j"$(nproc)" | ||
| 47 | +``` | ||
| 48 | + | ||
| 49 | +## Artifacts | ||
| 50 | + | ||
| 51 | +```bash | ||
| 52 | +./build/rpc_host | ||
| 53 | +./build/probe_topo | ||
| 54 | +./build/probe_controller | ||
| 55 | +./build/controller | ||
| 56 | +``` | ||
| 57 | + | ||
| 58 | +Test and auxiliary executables are output to: | ||
| 59 | + | ||
| 60 | +```bash | ||
| 61 | +./build/internal/ | ||
| 62 | +``` | ||
Rexperimental/cluster_link_diag/docs/用户运行v1.0.md→experimental/cluster_link_diag/docs/user_execution.md+107-108
| @@ -1,28 +1,28 @@ | |||
| 1 | -# 用户运行 | 1 | +# 用户运行 |
| 2 | - | 2 | + |
| 3 | -## 环境变量配置 | 3 | +## 环境变量配置 |
| 4 | - | 4 | + |
| 5 | -```bash | 5 | +```bash |
| 6 | -cd .../disp_probe-main | 6 | +cd .../disp_probe-main |
| 7 | - | 7 | + |
| 8 | -export THIRDLIB_ROOT=/usr/local/third_lib | 8 | +export THIRDLIB_ROOT=/usr/local/third_lib |
| 9 | -export ASCEND_HOME_PATH=/usr/local/Ascend | 9 | +export ASCEND_HOME_PATH=/usr/local/Ascend |
| 10 | -export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux | 10 | +export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux |
| 11 | -source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" | 11 | +source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" |
| 12 | -``` | 12 | +``` |
| 13 | - | 13 | + |
| 14 | -## 编译 | 14 | +## 编译 |
| 15 | - | 15 | + |
| 16 | -```bash | 16 | +```bash |
| 17 | -cmake -S . -B build \ | 17 | +cmake -S . -B build \ |
| 18 | - -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | 18 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ |
| 19 | - -DCMAKE_BUILD_TYPE=Release | 19 | + -DCMAKE_BUILD_TYPE=Release |
| 20 | - | 20 | + |
| 21 | -cmake --build build -j"$(nproc)" | 21 | +cmake --build build -j"$(nproc)" |
| 22 | -``` | 22 | +``` |
| 23 | - | 23 | + |
| 24 | -## 运行 | 24 | +## 运行 |
| 25 | - | 25 | + |
| 26 | ```bash | 26 | ```bash |
| 27 | # 清理残留进程、分发 rpc_host 和配置文件、启动 dev 侧程序 | 27 | # 清理残留进程、分发 rpc_host 和配置文件、启动 dev 侧程序 |
| 28 | ./run.sh deploy | 28 | ./run.sh deploy |
| @@ -30,98 +30,97 @@ cmake --build build -j"$(nproc)" | |||
| 30 | # 启动新终端, 启动监控系统(网络状态探测) | 30 | # 启动新终端, 启动监控系统(网络状态探测) |
| 31 | ./build/probe_controller -f ./control_json/910b2_info.json | 31 | ./build/probe_controller -f ./control_json/910b2_info.json |
| 32 | ``` | 32 | ``` |
| 33 | - | 33 | + |
| 34 | -### 开启 npu 侧探测日志: | 34 | +### 开启NPU侧探测日志 |
| 35 | - | 35 | + |
| 36 | - `--pingpong-local-log` 开启每个 NPU 侧结果日志 | 36 | +`--pingpong-local-log`:开启每个NPU侧结果日志。 |
| 37 | - `--pingpong-log-dir <PATH>` NPU 侧日志根目录,默认 /root/output | 37 | + |
| 38 | - | 38 | +`--pingpong-log-dir <PATH>`:NPU侧日志根目录,默认为`/root/output`。 |
| 39 | + | ||
| 39 | ```bash | 40 | ```bash |
| 40 | ./run.sh deploy --pingpong-log /root/output | 41 | ./run.sh deploy --pingpong-log /root/output |
| 41 | ``` | 42 | ``` |
| 42 | - | 43 | + |
| 43 | `rpc_host` 默认读取远端部署目录下的 `control_json/910b2_info.json`,也可以用 `-f` 指定其他 control_json;v2 配置会将 `probe.scope` 中的 host id 转换为对应 IP,再根据本机 IP 对应的卡号初始化 HCCN,未在探测范围中使用的卡不会初始化。 | 44 | `rpc_host` 默认读取远端部署目录下的 `control_json/910b2_info.json`,也可以用 `-f` 指定其他 control_json;v2 配置会将 `probe.scope` 中的 host id 转换为对应 IP,再根据本机 IP 对应的卡号初始化 HCCN,未在探测范围中使用的卡不会初始化。 |
| 44 | 45 | ||
| 45 | `probe.pingpong.payload_len` 用于设置 HCCN Rping Payload 长度,默认 `12` 字节,范围 `1~1500`。Payload 在每个目标 AddTarget 时使用随机字节填充;修改配置后需要重启对应 Host 上的 `rpc_host` 才会生效。 | 46 | `probe.pingpong.payload_len` 用于设置 HCCN Rping Payload 长度,默认 `12` 字节,范围 `1~1500`。Payload 在每个目标 AddTarget 时使用随机字节填充;修改配置后需要重启对应 Host 上的 `rpc_host` 才会生效。 |
| 46 | 47 | ||
| 47 | `probe.pingpong.interval_ms` 用于设置传给 `HccnRpingBatchPingStart` 的 Ping 报文发送间隔,默认 `1` 毫秒,必须为正整数;修改配置后需要重启对应 Host 上的 `rpc_host`。 | 48 | `probe.pingpong.interval_ms` 用于设置传给 `HccnRpingBatchPingStart` 的 Ping 报文发送间隔,默认 `1` 毫秒,必须为正整数;修改配置后需要重启对应 Host 上的 `rpc_host`。 |
| 48 | - | 49 | + |
| 49 | -### 全网络拓扑生成 | 50 | +### 全网络拓扑生成 |
| 50 | - | 51 | + |
| 51 | -```bash | 52 | +```bash |
| 52 | -./build/probe_topo -f ./control_json/910b2_info.json | 53 | +./build/probe_topo -f ./control_json/910b2_info.json |
| 53 | - | 54 | +``` |
| 54 | -``` | 55 | + |
| 55 | - | ||
| 56 | 拓扑探测默认分为四轮;`probe.topology.l2_path_aware` 默认为 `true`,如设置为 `false` 则跳过第 4 轮: | 56 | 拓扑探测默认分为四轮;`probe.topology.l2_path_aware` 默认为 `true`,如设置为 `false` 则跳过第 4 轮: |
| 57 | - | 57 | + |
| 58 | -1. batch 1:环状 tracert,用于获取网络分层,输出基础分层拓扑到 `output/probe_topo.json`。 | 58 | +1. batch 1:环状 tracert,用于获取网络分层,输出基础分层拓扑到 `output/probe_topo.json`。 |
| 59 | -2. batch 2:对每张卡执行 LLDP 探测,通过交换机管理面 IP 合并第 0 层分组,输出到 `output/probe_topo_lldp.json`。 | 59 | +2. batch 2:对每张卡执行 LLDP 探测,通过交换机管理面 IP 合并第 0 层分组,输出到 `output/probe_topo_lldp.json`。 |
| 60 | 3. batch 3:对同层不同交换机分组之间做多路径覆盖;当 `probe.topology.topology_optimized=true` 时,按 ToR 域环形探测同编号设备,即 `TOPO[0,X]` 的第 N 个设备探测 `TOPO[0,X+1]` 的第 N 个设备,最后一个域再探测 `TOPO[0,0]`;当 `topology_optimized=false` 时,探测不同 ToR 域间所有设备两两 directed pair。每条 directed pair 的覆盖次数等于 `control_json/910b2_info.json` 中的 `probe.topology.sport_count`,输出到 `output/allpath.json`。 | 60 | 3. batch 3:对同层不同交换机分组之间做多路径覆盖;当 `probe.topology.topology_optimized=true` 时,按 ToR 域环形探测同编号设备,即 `TOPO[0,X]` 的第 N 个设备探测 `TOPO[0,X+1]` 的第 N 个设备,最后一个域再探测 `TOPO[0,0]`;当 `topology_optimized=false` 时,探测不同 ToR 域间所有设备两两 directed pair。每条 directed pair 的覆盖次数等于 `control_json/910b2_info.json` 中的 `probe.topology.sport_count`,输出到 `output/allpath.json`。 |
| 61 | 4. batch 4:L2 域间环形 FullMesh 路径探测。开启 `probe.topology.l2_path_aware=true` 时执行,固定使用 `srcPort=49152`,由 `TOPO[0,X]` 的每个设备依次探测 `TOPO[0,X+1]` 的所有设备,最后一个 `TOPO[0,X+1]` 再探测 `TOPO[0,0]`,用于获取 L2 pair 的具体多跳路径,输出到 `output/l2_fullmesh_path.json`。 | 61 | 4. batch 4:L2 域间环形 FullMesh 路径探测。开启 `probe.topology.l2_path_aware=true` 时执行,固定使用 `srcPort=49152`,由 `TOPO[0,X]` 的每个设备依次探测 `TOPO[0,X+1]` 的所有设备,最后一个 `TOPO[0,X+1]` 再探测 `TOPO[0,0]`,用于获取 L2 pair 的具体多跳路径,输出到 `output/l2_fullmesh_path.json`。 |
| 62 | - | 62 | + |
| 63 | -输出在: `output/probe_topo.json` 、`output/probe_topo_lldp.json`(LLDP聚合后) 、`output/allpath.json`;开启 `l2_path_aware` 时额外输出 `output/l2_fullmesh_path.json`。 | 63 | +输出在: `output/probe_topo.json` 、`output/probe_topo_lldp.json`(LLDP聚合后) 、`output/allpath.json`;开启 `l2_path_aware` 时额外输出 `output/l2_fullmesh_path.json`。 |
| 64 | - | 64 | + |
| 65 | -拓扑探测完成后,读取`output/probe_topo_lldp.json` 、`output/allpath.json`,进行拓扑图绘制。 | 65 | +拓扑探测完成后,读取`output/probe_topo_lldp.json` 、`output/allpath.json`,进行拓扑图绘制。 |
| 66 | - | 66 | + |
| 67 | -```bash | 67 | +```bash |
| 68 | -python3 ./plot/topo_plot.py | 68 | +python3 ./plot/topo_plot.py |
| 69 | -``` | 69 | +``` |
| 70 | -最终输出的拓扑图保存在output/topo_plot/topo.png。 | 70 | + |
| 71 | - | 71 | +最终输出的拓扑图保存在output/topo_plot/topo.png。 |
| 72 | - | 72 | + |
| 73 | - | 73 | +### 执行pinglist |
| 74 | -### 执行pinglist | 74 | + |
| 75 | - | 75 | +`probe_controller` 默认读取 `output/probe_topo_lldp.json` 作为 L1 pinglist 拓扑输入。默认会额外添加 L2 域间 环形FullMesh 探测:固定使用 `srcPort=49152`,由 `TOPO[0,X]` 的每个设备依次探测 `TOPO[0,X+1]` 的所有设备,最后一个 `TOPO[0,X+1]` 再探测 `TOPO[0,0]`。 |
| 76 | -`probe_controller` 默认读取 `output/probe_topo_lldp.json` 作为 L1 pinglist 拓扑输入。默认会额外添加 L2 域间 环形FullMesh 探测:固定使用 `srcPort=49152`,由 `TOPO[0,X]` 的每个设备依次探测 `TOPO[0,X+1]` 的所有设备,最后一个 `TOPO[0,X+1]` 再探测 `TOPO[0,0]`。 | 76 | + |
| 77 | - | 77 | +```bash |
| 78 | -```bash | 78 | +./build/probe_controller -f ./control_json/910b2_info.json |
| 79 | -./build/probe_controller -f ./control_json/910b2_info.json | 79 | +``` |
| 80 | -``` | 80 | + |
| 81 | - | 81 | +`probe_controller` 支持的运行选项如下: |
| 82 | -`probe_controller` 支持的运行选项如下: | 82 | + |
| 83 | - | 83 | +| 选项 | 说明 | |
| 84 | -| 选项 | 说明 | | 84 | +| --- | --- | |
| 85 | -| --- | --- | | 85 | +| `-h, --help` | 输出帮助信息并退出。 | |
| 86 | -| `-h, --help` | 输出帮助信息并退出。 | | 86 | +| `-f, --file <PATH>` | 指定 control_json 配置文件路径。 | |
| 87 | -| `-f, --file <PATH>` | 指定 control_json 配置文件路径。 | | 87 | +| `--print-pingpong-plan` | 只打印 pingpong 探测计划,打印后退出,不下发 pinglist、不执行 pingpong。 | |
| 88 | -| `--print-pingpong-plan` | 只打印 pingpong 探测计划,打印后退出,不下发 pinglist、不执行 pingpong。 | | 88 | +| `--l1-only` | 只构建并求解 L1(ToR 下侧链路) 探测任务,关闭 L2 域间 FullMesh 探测和 L2 输出。 | |
| 89 | -| `--l1-only` | 只构建并求解 L1(ToR 下侧链路) 探测任务,关闭 L2 域间 FullMesh 探测和 L2 输出。 | | 89 | +| `--no-metrics` | 关闭 PFC/CNP 计数器采集。默认开启计数器采集。 | |
| 90 | -| `--no-metrics` | 关闭 PFC/CNP 计数器采集。默认开启计数器采集。 | | 90 | + |
| 91 | - | ||
| 92 | 默认会每秒采集一次探测范围(v2 `probe.scope` 或 v1 `probe_scope`)涉及 NPU 的 PFC/CNP 计数器并输出到 `output/<时间>/metrics/`。如需关闭计数器采集: | 91 | 默认会每秒采集一次探测范围(v2 `probe.scope` 或 v1 `probe_scope`)涉及 NPU 的 PFC/CNP 计数器并输出到 `output/<时间>/metrics/`。如需关闭计数器采集: |
| 93 | - | 92 | + |
| 94 | -```bash | 93 | +```bash |
| 95 | -./build/probe_controller -f ./control_json/910b2_info.json --no-metrics | 94 | +./build/probe_controller -f ./control_json/910b2_info.json --no-metrics |
| 96 | -``` | 95 | +``` |
| 97 | - | 96 | + |
| 98 | -如果只需要构建 L1(ToR 下侧链路) 的探测计划并进行求解,关闭 L2 FullMesh 探测计划和 L2 扣减输出: | 97 | +如果只需要构建 L1(ToR 下侧链路) 的探测计划并进行求解,关闭 L2 FullMesh 探测计划和 L2 扣减输出: |
| 99 | - | 98 | + |
| 100 | -```bash | 99 | +```bash |
| 101 | -./build/probe_controller -f ./control_json/910b2_info.json --l1-only | 100 | +./build/probe_controller -f ./control_json/910b2_info.json --l1-only |
| 102 | -./build/probe_controller -f ./control_json/910b2_info.json --print-pingpong-plan --l1-only | 101 | +./build/probe_controller -f ./control_json/910b2_info.json --print-pingpong-plan --l1-only |
| 103 | -``` | 102 | +``` |
| 104 | - | 103 | + |
| 105 | -### 结果分析 | 104 | +### 结果分析 |
| 106 | - | 105 | + |
| 107 | -网络状态分析默认读取 `output/<时间>/` 下最近一次运行结果,并提取最近 100 轮数据: | 106 | +网络状态分析默认读取 `output/<时间>/` 下最近一次运行结果,并提取最近 100 轮数据: |
| 108 | - | 107 | + |
| 109 | -```bash | 108 | +```bash |
| 110 | -python3 ./plot/status_plot.py | 109 | +python3 ./plot/status_plot.py |
| 111 | -``` | 110 | +``` |
| 112 | - | 111 | + |
| 113 | -也可以指定读取某一次运行结果: | 112 | +也可以指定读取某一次运行结果: |
| 114 | - | 113 | + |
| 115 | -```bash | 114 | +```bash |
| 116 | -python3 ./plot/status_plot.py --input-dir "output/2026-07-02 21:10:19" | 115 | +python3 ./plot/status_plot.py --input-dir "output/2026-07-02 21:10:19" |
| 117 | -# 或 | 116 | +# 或 |
| 118 | -python3 ./plot/status_plot.py "output/2026-07-02 21:10:19" | 117 | +python3 ./plot/status_plot.py "output/2026-07-02 21:10:19" |
| 119 | -``` | 118 | +``` |
| 120 | - | 119 | + |
| 121 | -状态图输出到 `output/<时间>/status_plot/`: | 120 | +状态图输出到 `output/<时间>/status_plot/`: |
| 122 | - | 121 | + |
| 123 | -- `l1_latency.png`:L1 latency 最近 100 轮变化。 | 122 | +- `l1_latency.png`:L1 latency 最近 100 轮变化。 |
| 124 | -- `l1_passrate.png`:L1 pass rate 最近 100 轮变化。 | 123 | +- `l1_passrate.png`:L1 pass rate 最近 100 轮变化。 |
| 125 | -- `l2_passrate.png`:L2 pass rate 最近 100 轮变化。 | 124 | +- `l2_passrate.png`:L2 pass rate 最近 100 轮变化。 |
| 126 | -- `l2_latency_top10.png`:按最近 100 轮平均 L2 latency 选出的前 10 个 pair 的变化。 | 125 | +- `l2_latency_top10.png`:按最近 100 轮平均 L2 latency 选出的前 10 个 pair 的变化。 |
| 127 | -- `l2_latency_top10.txt`:前 10 个 L2 latency pair 的平均值列表;如果存在 `output/l2_fullmesh_path.json`,会同时输出该 pair 的 `srcPort=49152` 多跳路径。 | 126 | +- `l2_latency_top10.txt`:前 10 个 L2 latency pair 的平均值列表;如果存在 `output/l2_fullmesh_path.json`,会同时输出该 pair 的 `srcPort=49152` 多跳路径。 |
| @@ -0,0 +1,126 @@ | |||
| 1 | +# User Execution | ||
| 2 | + | ||
| 3 | +## Environment Variable Configuration | ||
| 4 | + | ||
| 5 | +```bash | ||
| 6 | +cd .../disp_probe-main | ||
| 7 | + | ||
| 8 | +export THIRDLIB_ROOT=/usr/local/third_lib | ||
| 9 | +export ASCEND_HOME_PATH=/usr/local/Ascend | ||
| 10 | +export ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux | ||
| 11 | +source "$THIRDLIB_ROOT/share/disp_probe/third_party/env.sh" | ||
| 12 | +``` | ||
| 13 | + | ||
| 14 | +## Compilation | ||
| 15 | + | ||
| 16 | +```bash | ||
| 17 | +cmake -S . -B build \ | ||
| 18 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | ||
| 19 | + -DCMAKE_BUILD_TYPE=Release | ||
| 20 | + | ||
| 21 | +cmake --build build -j"$(nproc)" | ||
| 22 | +``` | ||
| 23 | + | ||
| 24 | +## Execution | ||
| 25 | + | ||
| 26 | +```bash | ||
| 27 | +# Clean up residual processes, distribute rpc_host and configuration files, start device-side programs | ||
| 28 | +./run.sh deploy | ||
| 29 | + | ||
| 30 | +# Open a new terminal and start the monitoring system (network status probing) | ||
| 31 | +./build/probe_controller -f ./control_json/910b2_info.json | ||
| 32 | +``` | ||
| 33 | + | ||
| 34 | +### Enable NPU-Side Probing Logs | ||
| 35 | + | ||
| 36 | +`--pingpong-local-log`: Enable result logging on each NPU side. | ||
| 37 | + | ||
| 38 | +`--pingpong-log-dir <PATH>`: NPU-side log root directory, default `/root/output`. | ||
| 39 | + | ||
| 40 | +```bash | ||
| 41 | +./run.sh deploy --pingpong-log /root/output | ||
| 42 | +``` | ||
| 43 | + | ||
| 44 | +`rpc_host` reads `control_json/910b2_info.json` from the remote deployment directory by default. You can also use `-f` to specify a different control_json. The v2 configuration converts the host ID in `probe.scope` to the corresponding IP and initializes HCCN based on the device IDs mapped to the local IP. Devices not in the probe scope are not initialized. | ||
| 45 | + | ||
| 46 | +`probe.pingpong.payload_len` sets the HCCN Rping payload length, with a default of `12` bytes and a range of `1-1500`. The payload is filled with random bytes during AddTarget for each target. After modifying the configuration, restart `rpc_host` on the corresponding host for the changes to take effect. | ||
| 47 | + | ||
| 48 | +`probe.pingpong.interval_ms` sets the ping packet sending interval passed to `HccnRpingBatchPingStart`, with a default of `1` millisecond. The value must be a positive integer. After modifying the configuration, restart `rpc_host` on the corresponding host. | ||
| 49 | + | ||
| 50 | +### Full Network Topology Generation | ||
| 51 | + | ||
| 52 | +```bash | ||
| 53 | +./build/probe_topo -f ./control_json/910b2_info.json | ||
| 54 | +``` | ||
| 55 | + | ||
| 56 | +Topology probing runs in four batches by default. `probe.topology.l2_path_aware` defaults to `true`; if set to `false`, batch 4 is skipped: | ||
| 57 | + | ||
| 58 | +1. Batch 1: Ring tracert to obtain network layering. Outputs the base layered topology to `output/probe_topo.json`. | ||
| 59 | +2. Batch 2: LLDP probing for each device. Merges layer-0 groups by switch management IP. Outputs to `output/probe_topo_lldp.json`. | ||
| 60 | +3. Batch 3: Multi-path coverage between different switch groups at the same layer. When `probe.topology.topology_optimized=true`, ring probing is performed on same-index devices across ToR domains: the Nth device in `TOPO[0,X]` probes the Nth device in `TOPO[0,X+1]`, and the last domain probes back to `TOPO[0,0]`. When `topology_optimized=false`, all directed pairs between devices in different ToR domains are probed. The coverage count per directed pair equals `probe.topology.sport_count` in `control_json/910b2_info.json`. Outputs to `output/allpath.json`. | ||
| 61 | +4. Batch 4: L2 inter-domain ring FullMesh path probing. Executed when `probe.topology.l2_path_aware=true`. Uses a fixed `srcPort=49152`. Each device in `TOPO[0,X]` probes all devices in `TOPO[0,X+1]` sequentially, and the last `TOPO[0,X+1]` probes back to `TOPO[0,0]`. This obtains the specific multi-hop paths for L2 pairs. Outputs to `output/l2_fullmesh_path.json`. | ||
| 62 | + | ||
| 63 | +Output files: `output/probe_topo.json`, `output/probe_topo_lldp.json` (after LLDP aggregation), and `output/allpath.json`. When `l2_path_aware` is enabled, `output/l2_fullmesh_path.json` is also generated. | ||
| 64 | + | ||
| 65 | +After topology probing completes, read `output/probe_topo_lldp.json` and `output/allpath.json` to generate the topology chart. | ||
| 66 | + | ||
| 67 | +```bash | ||
| 68 | +python3 ./plot/topo_plot.py | ||
| 69 | +``` | ||
| 70 | + | ||
| 71 | +The final topology chart is saved to `output/topo_plot/topo.png`. | ||
| 72 | + | ||
| 73 | +### Execute PingList | ||
| 74 | + | ||
| 75 | +`probe_controller` reads `output/probe_topo_lldp.json` as the L1 pinglist topology input by default. It also adds L2 inter-domain ring FullMesh probing by default: uses a fixed `srcPort=49152`, where each device in `TOPO[0,X]` probes all devices in `TOPO[0,X+1]` sequentially, and the last `TOPO[0,X+1]` probes back to `TOPO[0,0]`. | ||
| 76 | + | ||
| 77 | +```bash | ||
| 78 | +./build/probe_controller -f ./control_json/910b2_info.json | ||
| 79 | +``` | ||
| 80 | + | ||
| 81 | +The following runtime options are supported by `probe_controller`: | ||
| 82 | + | ||
| 83 | +| Option | Description | | ||
| 84 | +| --- | --- | | ||
| 85 | +| `-h, --help` | Print help information and exit. | | ||
| 86 | +| `-f, --file <PATH>` | Specify the control_json configuration file path. | | ||
| 87 | +| `--print-pingpong-plan` | Print the pingpong probe plan only and exit; do not distribute the pinglist or execute pingpong. | | ||
| 88 | +| `--l1-only` | Build and solve L1 (below-ToR link) probe tasks only; disable L2 inter-domain FullMesh probing and L2 output. | | ||
| 89 | +| `--no-metrics` | Disable PFC/CNP counter collection. Counter collection is enabled by default. | | ||
| 90 | + | ||
| 91 | +By default, PFC/CNP counters for NPUs in the probe scope (v2 `probe.scope` or v1 `probe_scope`) are collected every second and output to `output/<time>/metrics/`. To disable counter collection: | ||
| 92 | + | ||
| 93 | +```bash | ||
| 94 | +./build/probe_controller -f ./control_json/910b2_info.json --no-metrics | ||
| 95 | +``` | ||
| 96 | + | ||
| 97 | +To build only the L1 (below-ToR link) probe plan and perform solving, disabling L2 FullMesh probe plans and L2 subtraction output: | ||
| 98 | + | ||
| 99 | +```bash | ||
| 100 | +./build/probe_controller -f ./control_json/910b2_info.json --l1-only | ||
| 101 | +./build/probe_controller -f ./control_json/910b2_info.json --print-pingpong-plan --l1-only | ||
| 102 | +``` | ||
| 103 | + | ||
| 104 | +### Result Analysis | ||
| 105 | + | ||
| 106 | +Network status analysis reads the most recent run results from `output/<time>/` by default and extracts the most recent 100 turns of data: | ||
| 107 | + | ||
| 108 | +```bash | ||
| 109 | +python3 ./plot/status_plot.py | ||
| 110 | +``` | ||
| 111 | + | ||
| 112 | +You can also specify a particular run result: | ||
| 113 | + | ||
| 114 | +```bash | ||
| 115 | +python3 ./plot/status_plot.py --input-dir "output/2026-07-02 21:10:19" | ||
| 116 | +# Or | ||
| 117 | +python3 ./plot/status_plot.py "output/2026-07-02 21:10:19" | ||
| 118 | +``` | ||
| 119 | + | ||
| 120 | +Status charts are output to `output/<time>/status_plot/`: | ||
| 121 | + | ||
| 122 | +- `l1_latency.png`: L1 latency changes over the most recent 100 turns. | ||
| 123 | +- `l1_passrate.png`: L1 pass rate changes over the most recent 100 turns. | ||
| 124 | +- `l2_passrate.png`: L2 pass rate changes over the most recent 100 turns. | ||
| 125 | +- `l2_latency_top10.png`: Changes for the top 10 pairs selected by average L2 latency over the most recent 100 turns. | ||
| 126 | +- `l2_latency_top10.txt`: A list of average values for the top 10 L2 latency pairs. If `output/l2_fullmesh_path.json` exists, the `srcPort=49152` multi-hop path for each pair is also output. | ||
| @@ -0,0 +1,23 @@ | |||
| 1 | +# Unit Tests | ||
| 2 | + | ||
| 3 | +This directory implements T1, T2, T3, T4, T5, and T6 from Section 10 of the RFC. All tests are offline unit tests and do not connect to NPUs, `rpc_host`, or switches. System tests are not in scope for this directory. | ||
| 4 | + | ||
| 5 | +| Test | File | Coverage | | ||
| 6 | +| --- | --- | --- | | ||
| 7 | +| T1 | `t1_config_test.cpp` | Configuration defaults, duplicate/illegal device IDs, pairing sections, positive integer and output path validation | | ||
| 8 | +| T2 | `t2_topology_test.cpp` | ControlTopo routing, device count completeness, LLDP domain partitioning, unknown neighbors, MeshTopo JSON round-trip, and exception access | | ||
| 9 | +| T3 | `t3_probe_plan_test.cpp` | L1 ring, L2 adjacent-domain FullMesh, task scale, deduplication, and single-domain degradation | | ||
| 10 | +| T4 | `t4_metric_reduction_test.cpp` | P90/P99/Mean/Pass mapping, total packet loss, out-of-range Pass, missing fields, and illegal times | | ||
| 11 | +| T5 | `t5_link_solver_test.cpp` | Exact solution, overdetermined solution, rank deficiency, and NaN input | | ||
| 12 | +| T6 | `t6_metrics_collector_test.cpp` | Counter text parsing, missing key/illegal value, `valid` flag, genuine zero value, device ID normalization, and reset/wraparound delta | | ||
| 13 | + | ||
| 14 | +How to run: | ||
| 15 | + | ||
| 16 | +```bash | ||
| 17 | +cmake -S . -B build-test \ | ||
| 18 | + -DTHIRDLIB_ROOT="$THIRDLIB_ROOT" \ | ||
| 19 | + -DBUILD_TESTS=ON \ | ||
| 20 | + -DCMAKE_BUILD_TYPE=Debug | ||
| 21 | +cmake --build build-test -j"$(nproc)" | ||
| 22 | +ctest --test-dir build-test --output-on-failure -L UT | ||
| 23 | +``` | ||
| @@ -512,7 +512,7 @@ classDiagram | |||
| 512 | 5. **Null pointer protection**: All reporting interfaces check for non-null pointers before calling the Reporter to avoid null pointer dereferences. | 512 | 5. **Null pointer protection**: All reporting interfaces check for non-null pointers before calling the Reporter to avoid null pointer dereferences. |
| 513 | 6. **EXCEPTION_CATCH macro**: `ReportKernel` uses the `EXCEPTION_CATCH` macro to catch exceptions during `ProfilingHandler` reporting, returning `HCCL_E_PTR` on failure. | 513 | 6. **EXCEPTION_CATCH macro**: `ReportKernel` uses the `EXCEPTION_CATCH` macro to catch exceptions during `ProfilingHandler` reporting, returning `HCCL_E_PTR` on failure. |
| 514 | 7. **MC2 Stream group reporting**: `ReportMc2CommInfo` groups every 8 streamIds into a group and reports them through `ProfilingDeviceCommResInfo`. The `commStreamIds` array size is fixed at 8. | 514 | 7. **MC2 Stream group reporting**: `ReportMc2CommInfo` groups every 8 streamIds into a group and reports them through `ProfilingDeviceCommResInfo`. The `commStreamIds` array size is fixed at 8. |
| 515 | -8. **Device-side device type restriction**: `HcommProfilingReportDeviceOp`, `HcommProfilingReportKernelStartTask`, and `HcommProfilingReportKernelEndTask` only execute on `DEV_TYPE_950` devices. For other device types, they directly return success. | 515 | +8. **Device-side device type restriction**: `HcommProfilingReportDeviceOp`, `HcommProfilingReportKernelStartTask`, and `HcommProfilingReportKernelEndTask` only execute on `DEV_TYPE_950` and `DEV_TYPE_960` devices. For other device types, they directly return success. |
| 516 | 516 | ||
| 517 | ### Known Limitations | 517 | ### Known Limitations |
| 518 | 518 | ||
| @@ -1,6 +1,6 @@ | |||
| 1 | # topo addr info | 1 | # topo addr info |
| 2 | 2 | ||
| 3 | -# 简介 | 3 | +## 简介 |
| 4 | 4 | ||
| 5 | 昇腾芯片在950代际中, 超平面使用Unified Bus总线组网,在不同产品形态中使用多种不同的拓扑组网方式, 本模块用于在发现在不同拓扑下每条边的端点地址 | 5 | 昇腾芯片在950代际中, 超平面使用Unified Bus总线组网,在不同产品形态中使用多种不同的拓扑组网方式, 本模块用于在发现在不同拓扑下每条边的端点地址 |
| 6 | 6 | ||
| @@ -26,10 +26,9 @@ chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run | |||
| 26 | ./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend | 26 | ./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend |
| 27 | ``` | 27 | ``` |
| 28 | 28 | ||
| 29 | - | ||
| 30 | ### 2.2 hccl_test编译 | 29 | ### 2.2 hccl_test编译 |
| 31 | 30 | ||
| 32 | -hccl_test是昇腾官方提供的HCCL性能测试工具,详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html),HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。 | 31 | +hccl_test是昇腾官方提供的HCCL性能测试工具,详见[HCCL性能测试工具](https://www.hiascend.com/document/redirect/CANNCommunityToolHcclTest),HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。 |
| 33 | 32 | ||
| 34 | 备注:可选,未来支持Pytorch用例。 | 33 | 备注:可选,未来支持Pytorch用例。 |
| 35 | 34 | ||
| @@ -56,6 +55,7 @@ curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/master/test/hccl_vm/hccl_vm_in | |||
| 56 | **hccl_test**:默认一并编译 OpenMPI 与 hccl_test 性能测试工具,`--skip-hccl-test` 可关闭。 | 55 | **hccl_test**:默认一并编译 OpenMPI 与 hccl_test 性能测试工具,`--skip-hccl-test` 可关闭。 |
| 57 | 56 | ||
| 58 | **常用参数**: | 57 | **常用参数**: |
| 58 | + | ||
| 59 | - `--profile <名>`:配套方案(默认 `main`,`--list-profiles` 列全部) | 59 | - `--profile <名>`:配套方案(默认 `main`,`--list-profiles` 列全部) |
| 60 | - `--workspace <路径>`:工作目录,源码/编译/产物所在(默认当前目录) | 60 | - `--workspace <路径>`:工作目录,源码/编译/产物所在(默认当前目录) |
| 61 | - `--ascend-path <路径>`:指定 CANN 目录,有则复用、无则装到此处 | 61 | - `--ascend-path <路径>`:指定 CANN 目录,有则复用、无则装到此处 |
| @@ -102,44 +102,44 @@ bash build_pkg.sh | |||
| 102 | 102 | ||
| 103 | #### 3.3.2 CCU模式 | 103 | #### 3.3.2 CCU模式 |
| 104 | 104 | ||
| 105 | -1. 环境变量配置。 | 105 | +1. 配置环境变量。 |
| 106 | 106 | ||
| 107 | -```bash | 107 | + ```bash |
| 108 | -# 进入工具安装目录 | 108 | + # 进入工具安装目录 |
| 109 | -cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install | 109 | + cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install |
| 110 | -source /home/workspace/Ascend/cann/set_env.sh | 110 | + source /home/workspace/Ascend/cann/set_env.sh |
| 111 | -export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | 111 | + export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH |
| 112 | -export RANK_TABLE_FILE=$(pwd)/data/ranktable.json | 112 | + export RANK_TABLE_FILE=$(pwd)/data/ranktable.json |
| 113 | -export HCCL_OP_EXPANSION_MODE="CCU_SCHED" | 113 | + export HCCL_OP_EXPANSION_MODE="CCU_SCHED" |
| 114 | -``` | 114 | + ``` |
| 115 | 115 | ||
| 116 | -2. 执行 | 116 | +2. 执行。 |
| 117 | 117 | ||
| 118 | + ```bash | ||
| 119 | + # 需要进入到新的bin文件目录下执行hccl-vm | ||
| 120 | + cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | ||
| 121 | + | ||
| 122 | + # 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行 | ||
| 123 | + ./hccl-vm start ascend950_cluster_32_server_normal.yaml | ||
| 124 | + | ||
| 125 | + # 如需启用runner插件(可选) | ||
| 126 | + (hvm)$> hccl-vm plugin install @runner | ||
| 127 | + | ||
| 128 | + # 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例) | ||
| 129 | + (hvm)$> hccl-vm mock-comm 112 | ||
| 130 | + (hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | ||
| 131 | + | ||
| 132 | + # 执行checker校验 | ||
| 133 | + (hvm)$> hccl-vm plugin run @checker | ||
| 134 | + | ||
| 135 | + # 退出工具终端 | ||
| 136 | + (hvm)$> exit | ||
| 137 | + ``` | ||
| 118 | 138 | ||
| 119 | -```bash | 139 | +3. 验证hccl_test用例运行结果。 |
| 120 | -# 需要进入到新的bin文件目录下执行hccl-vm | ||
| 121 | -cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | ||
| 122 | 140 | ||
| 123 | -# 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行 | 141 | + [Runner结果查看](#491-runner插件结果) |
| 124 | -./hccl-vm start ascend950_cluster_32_server_normal.yaml | 142 | + [Checker结果查看](#492-checker插件结果) |
| 125 | - | ||
| 126 | -# 如需启用runner插件(可选) | ||
| 127 | -(hvm)$> hccl-vm plugin install @runner | ||
| 128 | - | ||
| 129 | -# 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例) | ||
| 130 | -(hvm)$> hccl-vm mock-comm 112 | ||
| 131 | -(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | ||
| 132 | - | ||
| 133 | -# 执行checker校验 | ||
| 134 | -(hvm)$> hccl-vm plugin run @checker | ||
| 135 | - | ||
| 136 | -# 退出工具终端 | ||
| 137 | -(hvm)$> exit | ||
| 138 | -``` | ||
| 139 | - | ||
| 140 | -3. 验证hccl_test用例运行结果 | ||
| 141 | -[Runner结果查看](#491-runner插件结果) | ||
| 142 | -[Checker结果查看](#492-checker插件结果) | ||
| 143 | 143 | ||
| 144 | #### 3.3.3 AICPU模式 | 144 | #### 3.3.3 AICPU模式 |
| 145 | 145 | ||
| @@ -147,37 +147,37 @@ AICPU展开模式需要将算法展开步骤放到设备侧执行,因此hccl-v | |||
| 147 | 147 | ||
| 148 | 1. 环境变量配置。 | 148 | 1. 环境变量配置。 |
| 149 | 149 | ||
| 150 | -```bash | 150 | + ```bash |
| 151 | -# 进入工具安装目录 | 151 | + # 进入工具安装目录 |
| 152 | -cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install | 152 | + cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install |
| 153 | -source /home/workspace/Ascend/cann/set_env.sh | 153 | + source /home/workspace/Ascend/cann/set_env.sh |
| 154 | -export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | 154 | + export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH |
| 155 | -export RANK_TABLE_FILE=$(pwd)/data/ranktable.json | 155 | + export RANK_TABLE_FILE=$(pwd)/data/ranktable.json |
| 156 | -export HCCL_OP_EXPANSION_MODE="AI_CPU" | 156 | + export HCCL_OP_EXPANSION_MODE="AI_CPU" |
| 157 | -``` | 157 | + ``` |
| 158 | 158 | ||
| 159 | 2. 执行 | 159 | 2. 执行 |
| 160 | 160 | ||
| 161 | -```bash | 161 | + ```bash |
| 162 | -# 需要进入到新的bin文件目录下执行hccl-vm | 162 | + # 需要进入到新的bin文件目录下执行hccl-vm |
| 163 | -cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | 163 | + cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin |
| 164 | - | 164 | + |
| 165 | -# 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行 | 165 | + # 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行 |
| 166 | -./hccl-vm start ascend950_cluster_32_server_normal.yaml | 166 | + ./hccl-vm start ascend950_cluster_32_server_normal.yaml |
| 167 | - | 167 | + |
| 168 | -# 如需启用runner插件(可选) | 168 | + # 如需启用runner插件(可选) |
| 169 | -(hvm)$> hccl-vm plugin install @runner | 169 | + (hvm)$> hccl-vm plugin install @runner |
| 170 | - | 170 | + |
| 171 | -# 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例) | 171 | + # 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例) |
| 172 | -(hvm)$> hccl-vm mock-comm 112 | 172 | + (hvm)$> hccl-vm mock-comm 112 |
| 173 | -(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | 173 | + (hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt |
| 174 | - | 174 | + |
| 175 | -# 执行checker校验 | 175 | + # 执行checker校验 |
| 176 | -(hvm)$> hccl-vm plugin run @checker | 176 | + (hvm)$> hccl-vm plugin run @checker |
| 177 | - | 177 | + |
| 178 | -# 退出工具终端 | 178 | + # 退出工具终端 |
| 179 | -(hvm)$> exit | 179 | + (hvm)$> exit |
| 180 | -``` | 180 | + ``` |
| 181 | 181 | ||
| 182 | 3. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果) | 182 | 3. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果) |
| 183 | 183 | ||
| @@ -185,37 +185,37 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | |||
| 185 | 185 | ||
| 186 | 1. 环境变量配置。 | 186 | 1. 环境变量配置。 |
| 187 | 187 | ||
| 188 | -```bash | 188 | + ```bash |
| 189 | -# 进入工具安装目录 | 189 | + # 进入工具安装目录 |
| 190 | -cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install | 190 | + cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install |
| 191 | -source /home/workspace/Ascend/cann/set_env.sh | 191 | + source /home/workspace/Ascend/cann/set_env.sh |
| 192 | -export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | 192 | + export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH |
| 193 | -export RANK_TABLE_FILE=$(pwd)/data/ranktable.json | 193 | + export RANK_TABLE_FILE=$(pwd)/data/ranktable.json |
| 194 | -export HCCL_OP_EXPANSION_MODE="AIV" | 194 | + export HCCL_OP_EXPANSION_MODE="AIV" |
| 195 | -``` | 195 | + ``` |
| 196 | 196 | ||
| 197 | 2. 执行 | 197 | 2. 执行 |
| 198 | 198 | ||
| 199 | -```bash | 199 | + ```bash |
| 200 | -# 需要进入到新的bin文件目录下执行hccl-vm | 200 | + # 需要进入到新的bin文件目录下执行hccl-vm |
| 201 | -cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | 201 | + cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin |
| 202 | - | 202 | + |
| 203 | -# 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行 | 203 | + # 选择昇腾集群拓扑配置文件,启动工具,初始化集群环境,进入工具命令行 |
| 204 | -./hccl-vm start ascend950_cluster_32_server_normal.yaml | 204 | + ./hccl-vm start ascend950_cluster_32_server_normal.yaml |
| 205 | - | 205 | + |
| 206 | -# 如需启用runner插件(可选) | 206 | + # 如需启用runner插件(可选) |
| 207 | -(hvm)$> hccl-vm plugin install @runner | 207 | + (hvm)$> hccl-vm plugin install @runner |
| 208 | - | 208 | + |
| 209 | -# 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例) | 209 | + # 选择本次算子执行的通信域配置文件(在1个超节点1个Server1个NPU的集群环境运行hccl_test用例) |
| 210 | -(hvm)$> hccl-vm mock-comm 112 | 210 | + (hvm)$> hccl-vm mock-comm 112 |
| 211 | -(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | 211 | + (hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt |
| 212 | - | 212 | + |
| 213 | -# 执行checker校验 | 213 | + # 执行checker校验 |
| 214 | -(hvm)$> hccl-vm plugin run @checker | 214 | + (hvm)$> hccl-vm plugin run @checker |
| 215 | - | 215 | + |
| 216 | -# 退出工具终端 | 216 | + # 退出工具终端 |
| 217 | -(hvm)$> exit | 217 | + (hvm)$> exit |
| 218 | -``` | 218 | + ``` |
| 219 | 219 | ||
| 220 | 3. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果) | 220 | 3. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果) |
| 221 | 221 | ||
| @@ -263,31 +263,31 @@ hccl_test用例源码在CANN包安装目录下,支持OpenMPI和MPICH两种环 | |||
| 263 | 263 | ||
| 264 | 1. 安装OpenMPI | 264 | 1. 安装OpenMPI |
| 265 | 265 | ||
| 266 | -```bash | 266 | + ```bash |
| 267 | -sudo apt-get update | 267 | + sudo apt-get update |
| 268 | -sudo apt install openmpi-bin libopenmpi-dev | 268 | + sudo apt install openmpi-bin libopenmpi-dev |
| 269 | -``` | 269 | + ``` |
| 270 | 270 | ||
| 271 | 2. 编译hccl_test | 271 | 2. 编译hccl_test |
| 272 | 272 | ||
| 273 | -```bash | 273 | + ```bash |
| 274 | -# 修改CANN安装目录权限 | 274 | + # 修改CANN安装目录权限 |
| 275 | -chmod -R 755 /home/workspace/Ascend | 275 | + chmod -R 755 /home/workspace/Ascend |
| 276 | - | 276 | + |
| 277 | -# 进入hccl_test用例源码目录 | 277 | + # 进入hccl_test用例源码目录 |
| 278 | -cd /home/workspace/Ascend/cann/tools/hccl_test | 278 | + cd /home/workspace/Ascend/cann/tools/hccl_test |
| 279 | - | 279 | + |
| 280 | -# 设置CANN环境变量 | 280 | + # 设置CANN环境变量 |
| 281 | -source /home/workspace/Ascend/cann/set_env.sh | 281 | + source /home/workspace/Ascend/cann/set_env.sh |
| 282 | - | 282 | + |
| 283 | -# 临时修改Makefile脚本 | 283 | + # 临时修改Makefile脚本 |
| 284 | -if ! grep -q '\-lmpi_cxx' Makefile; then | 284 | + if ! grep -q '\-lmpi_cxx' Makefile; then |
| 285 | - sed -i 's/-lmpi/-lmpi -lmpi_cxx/g' Makefile | 285 | + sed -i 's/-lmpi/-lmpi -lmpi_cxx/g' Makefile |
| 286 | -fi | 286 | + fi |
| 287 | - | 287 | + |
| 288 | -# 编译hccl_test用例 | 288 | + # 编译hccl_test用例 |
| 289 | -MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH} | 289 | + MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH} |
| 290 | -``` | 290 | + ``` |
| 291 | 291 | ||
| 292 | #### 4.2.2 MPICH环境编译 | 292 | #### 4.2.2 MPICH环境编译 |
| 293 | 293 | ||
| @@ -369,8 +369,8 @@ links: | |||
| 369 | - **port_group**: 描述哪些port合并为一个portGroup,相同portGroup的port对应IP地址相同。没有配置的port,则默认每个port为一个portGroup。 | 369 | - **port_group**: 描述哪些port合并为一个portGroup,相同portGroup的port对应IP地址相同。没有配置的port,则默认每个port为一个portGroup。 |
| 370 | - **links**: 链路配置表,描述Server/Pod内的所有NPU卡之间,以及NPU与交换机之间的连接关系。 | 370 | - **links**: 链路配置表,描述Server/Pod内的所有NPU卡之间,以及NPU与交换机之间的连接关系。 |
| 371 | - **NPU直连关系**: 工具提供了两种方式配置NPU直连关系: | 371 | - **NPU直连关系**: 工具提供了两种方式配置NPU直连关系: |
| 372 | - - **link_mode == "fullmesh"**: 表示所有Device基于一个Die的Port进行全连接。后续有新增典型的连接方式,可以新增link_mode类型,如"ring"。 | 372 | + - **link_mode == "fullmesh"**: 表示所有Device基于一个Die的Port进行全连接。后续有新增典型的连接方式,可以新增link_mode类型,如"ring"。 |
| 373 | - - **link_mode == "enum"**: 枚举法。当Server/Pod内的NPU连接方式比较复杂时,可以通过枚举所有的链路关系来描述。 | 373 | + - **link_mode == "enum"**: 枚举法。当Server/Pod内的NPU连接方式比较复杂时,可以通过枚举所有的链路关系来描述。 |
| 374 | - **NPU与交换机连接关系**: 用户可以通过枚举法配置NPU与交换机之间的连接关系。 | 374 | - **NPU与交换机连接关系**: 用户可以通过枚举法配置NPU与交换机之间的连接关系。 |
| 375 | - **device_to_device_links**: 描述NPU与NPU之间的连接关系。 | 375 | - **device_to_device_links**: 描述NPU与NPU之间的连接关系。 |
| 376 | - **device_to_switch_links**: 描述NPU与交换机之间的连接关系。 | 376 | - **device_to_switch_links**: 描述NPU与交换机之间的连接关系。 |
| @@ -770,6 +770,7 @@ data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result: | |||
| 770 | ``` | 770 | ``` |
| 771 | 771 | ||
| 772 | --- | 772 | --- |
| 773 | + | ||
| 773 | ### 4.10 大块内存复用(仅校验模式) | 774 | ### 4.10 大块内存复用(仅校验模式) |
| 774 | 775 | ||
| 775 | 仅校验模式用于大规模集群仅运行 Checker 校验的场景。开启后,单块 200MB 到 4GB 的大内存申请复用同一块 4GB 共享区 `HcclCommPool`,各 rank 共享、允许互相覆盖,以此大幅降低 `/dev/shm` 占用。此时大块内容不保证正确,仅适用于不读取缓冲区数据的 Checker V3 校验链路,需要数值正确的结果时请勿开启。 | 776 | 仅校验模式用于大规模集群仅运行 Checker 校验的场景。开启后,单块 200MB 到 4GB 的大内存申请复用同一块 4GB 共享区 `HcclCommPool`,各 rank 共享、允许互相覆盖,以此大幅降低 `/dev/shm` 占用。此时大块内容不保证正确,仅适用于不读取缓冲区数据的 Checker V3 校验链路,需要数值正确的结果时请勿开启。 |
| @@ -782,7 +783,7 @@ data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result: | |||
| 782 | ./hccl-vm start ascend950_cluster_32_server_normal.yaml --check-only | 783 | ./hccl-vm start ascend950_cluster_32_server_normal.yaml --check-only |
| 783 | ``` | 784 | ``` |
| 784 | 785 | ||
| 785 | -*** | 786 | +--- |
| 786 | 787 | ||
| 787 | ## 5 附录 | 788 | ## 5 附录 |
| 788 | 789 | ||