已合并
【fix docs】增加英文文档&修复失效超链接 #3597
zangyan创建于 7月9日
【fix docs】增加英文文档&修复失效超链接 #3597
已合并
共 65 个文件变更+13479-45
| @@ -0,0 +1,47 @@ | |||
| 1 | +## Description | ||
| 2 | +<!-- Describe your changes in detail here, including the reason for the changes and the approach taken. --> | ||
| 3 | + | ||
| 4 | +## Type of Change | ||
| 5 | +Select the type of change introduced by this PR: | ||
| 6 | +<!-- [x] indicates selection --> | ||
| 7 | +- [ ] Bug fix | ||
| 8 | +- [ ] New feature | ||
| 9 | +- [ ] Performance optimization | ||
| 10 | +- [ ] Documentation update | ||
| 11 | +- [ ] Other, please describe: | ||
| 12 | + | ||
| 13 | +## Related Issue | ||
| 14 | +<!-- If this PR addresses a specific Issue, provide the Issue link here. --> | ||
| 15 | +<!-- If this PR does not involve an Issue, enter "NA". --> | ||
| 16 | + | ||
| 17 | +## Testing | ||
| 18 | +<!-- Describe what tests were performed to verify your changes. Include but are not limited to constructing corresponding test cases, secondary smoke tests, operator generalization, and so on. --> | ||
| 19 | +Completed test cases and scenarios: | ||
| 20 | +1. | ||
| 21 | +2. | ||
| 22 | + | ||
| 23 | +Supplementary UT cases: | ||
| 24 | + | ||
| 25 | +## Documentation Update | ||
| 26 | +<!-- If this PR includes documentation updates, specify them here. For example: Updated the README.md file. --> | ||
| 27 | + | ||
| 28 | +## Pre-Merge Checklist | ||
| 29 | +<!-- Before merging, ensure necessary code testing, case supplementation, software code style checks, and so on are completed to improve merge efficiency. --> | ||
| 30 | +<!-- [x] indicates selection --> | ||
| 31 | +- [ ] I have read the Contribution Guide (CONTRIBUTING.md) and followed all its rules, including commit message format and squashing invalid commits. | ||
| 32 | +- [ ] Necessary checks before requesting a committer to comment `/lgtm` | ||
| 33 | + - [ ] An appropriate type label is used in the title (for example, `[feat]`, `[fix]`) | ||
| 34 | + - [ ] Code changes are briefly described and related documentation is updated | ||
| 35 | + - [ ] Code comments are updated and the code follows the project's overall code style | ||
| 36 | + - [ ] UT tests are updated and coverage meets the requirements | ||
| 37 | + - [ ] Verification methods are updated in the Testing section | ||
| 38 | + - [ ] Code has passed static analysis tools with no errors | ||
| 39 | + - [ ] Code review and necessary code walkthroughs have been conducted to ensure code quality | ||
| 40 | + - [ ] All review comments have been addressed or responded to, with no unresolved feedback | ||
| 41 | +- [ ] Necessary checks before scheduling pre-smoke test cases | ||
| 42 | + - [ ] Code has received `/lgtm` comments from the responsible committer and module committer | ||
| 43 | + - [ ] Code compiles without errors or warnings | ||
| 44 | + - [ ] Code has passed basic functional local or online testing to ensure normal functionality | ||
| 45 | +- [ ] Necessary checks before requesting an approver to comment `/approve` for formal merge | ||
| 46 | + - [ ] All pre-smoke test cases have passed | ||
| 47 | + - [ ] New features have been supplemented with basic functional test cases in the pre-smoke tests | ||
| @@ -0,0 +1,107 @@ | |||
| 1 | +# HCOMM Agent Rules | ||
| 2 | + | ||
| 3 | +This file serves as the main entry point for AI Agent governance of the HCOMM repository. It is intended for AI programming tools that support the AGENTS.md standard. If a subdirectory has its own `AGENTS.md`, that file supplements this one. In case of conflicts, follow the user's explicit requirements first, and then the rules in the nearest subdirectory. | ||
| 4 | + | ||
| 5 | +> This file provides only the **essential hard constraints and entry points**. Detailed content is progressively disclosed through links. Before modifying code, read the architecture constraints in Section 3. | ||
| 6 | + | ||
| 7 | +## 1. Repository Positioning | ||
| 8 | + | ||
| 9 | +HCOMM (Huawei Communication) is the underlying communication library of HCCL. It provides communication domain and communication resource management capabilities and serves as the base layer of the CANN collective communication stack. HCCL consists of two parts: the HCCL operator library (`cann/hccl`) and the HCOMM communication base library (this repository, `cann/hcomm`). The two repositories are decoupled through dynamic loading via `dlsym`. For details, see the [`README.md`](./README_en.md) and the [Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md). | ||
| 10 | + | ||
| 11 | +## 2. Directory Structure | ||
| 12 | + | ||
| 13 | +```text | ||
| 14 | +src/ | ||
| 15 | +├── base_comm/ # Basic communication layer (L3, lowest layer, must not have reverse dependencies on upper layers) | ||
| 16 | +├── coll_communicator_mgr/ # Collective communication domain management HCCM (L2, depends on base_comm) | ||
| 17 | +└── legacy/ # Historical compatibility directory, does not accept new features | ||
| 18 | +include/ # External header files (hccl/, ccu/, hcomm_primitives.h, and so on) | ||
| 19 | +pkg_inc/ # Inter-package interface header files (hccl/, hcomm/, not guaranteed to be stable externally) | ||
| 20 | +test/ # ut / st / stub | ||
| 21 | +docs/ # Documentation | ||
| 22 | +build.sh # One-click build script | ||
| 23 | +``` | ||
| 24 | + | ||
| 25 | +For details on the target directory structure and layer responsibilities, see [Section 3.2 of the Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md). | ||
| 26 | + | ||
| 27 | +## 3. Software Architecture and Architecture Constraints (Core) | ||
| 28 | + | ||
| 29 | +> **Authoritative Architecture Source (Chinese)**: [`docs/zh/architecture/architecture-brief.md`](./docs/zh/architecture/architecture-brief.md). Before modifying `src/`, `include/`, or `pkg_inc/`, read Section 3 - Software Layering Logic and the Software Architecture Constraint Description at the end. | ||
| 30 | + | ||
| 31 | +### Layers | ||
| 32 | + | ||
| 33 | +| Software Layer | Repository Location | | ||
| 34 | +|----|--------| | ||
| 35 | +| HCCL collective communication operators | `cann/hccl` | | ||
| 36 | +| HCOMM collective communication domain management (HCCM) | This repository `src/coll_communicator_mgr` | | ||
| 37 | +| HCOMM basic communication | This repository `src/base_comm` | | ||
| 38 | + | ||
| 39 | +Dependency direction is top-down: `coll_comm_ops` (hccl) → `coll_communicator_mgr` → `base_comm`. | ||
| 40 | + | ||
| 41 | +### Architecture Constraints (Hard, Must Not Violate) | ||
| 42 | + | ||
| 43 | +| Constraint | AI Agent Behavior Requirements | | ||
| 44 | +|------|------------------| | ||
| 45 | +| **Layer dependency direction**: Upper layers depend on lower layers. Lower layers must not have reverse dependencies on upper layers (`base_comm` ↛ `coll_communicator_mgr`; both ↛ `coll_comm_ops`) | Lower layers must not `#include` upper-layer header files or call upper-layer interfaces. When adding new classes or functions, first determine the layer and only call within the same layer or a lower layer. | | ||
| 46 | +| **Control plane and data plane separation**: Resource management and topology query (control plane) and data transfer or synchronization (data plane) interfaces evolve independently and are not coupled. | Do not introduce strong control plane coupling in data plane primitives. The control plane must not depend on specific data plane operator implementations. | | ||
| 47 | +| **HCCL and HCOMM decoupling**: HCCL dynamically loads HCOMM interfaces through `dlsym`. The two repositories compile and version independently. | HCOMM must not `#include` HCCL private headers. Do not introduce compile-time hard dependencies on `cann/hccl`. Cross-repository calls go through symbol tables and `dlsym`. | | ||
| 48 | +| **Legacy does not continuously evolve**: `legacy/` is for historical compatibility only and does not accept new features. | **Do not add new functions, operators, or interfaces in `src/legacy/`**. Legacy is for bug fixes and compatibility maintenance only. Place new features in `base_comm/` or `coll_communicator_mgr/`. | | ||
| 49 | + | ||
| 50 | +### External API Layers | ||
| 51 | + | ||
| 52 | +| Layer | Header Files | Audience | | ||
| 53 | +|------|------|------| | ||
| 54 | +| L2-comm | `include/hccl/hccl_comm.h` | AI framework layer (communication domain creation) | | ||
| 55 | +| L2-res-rank_graph | `include/hccl/hccl_res.h`, `include/hccl/hccl_rank_graph.h` | Operator developers (topology query, resource acquisition) | | ||
| 56 | +| L3-prim | `include/hcomm_primitives.h` | Operator or communication library developers (data transfer and synchronization) | | ||
| 57 | +| L3-res | `include/hcomm_res.h`, `include/hcomm_res_defs.h` | Communication library developers (device, channel, and memory resources) | | ||
| 58 | +| CCU | `include/ccu/` (`ccu_primitives.hpp`, `ccu_res.h`, `ccu_launch.h`) | CCU operator developers | | ||
| 59 | + | ||
| 60 | +Changes to `include/` must be backward compatible. `pkg_inc/` is for inter-package use between HCOMM, HCCL, GE, and so on, and is not externally stable. For the complete API layer relationships, see [Section 3.3 of the Architecture Overview (Chinese)](./docs/zh/architecture/architecture-brief.md). | ||
| 61 | + | ||
| 62 | +## 4. Build and Test | ||
| 63 | + | ||
| 64 | +```bash | ||
| 65 | +bash build.sh --pkg # Build the host package (default) | ||
| 66 | +bash build.sh --pkg --full # Build the host and device full package | ||
| 67 | +bash build.sh -u # Build and run UT (equivalent to --ut) | ||
| 68 | +bash build.sh -s # Build and run ST | ||
| 69 | +bash build.sh --ut --noexec # Build UT without running test cases (--noexec must be used with --ut or --st) | ||
| 70 | +bash build.sh -j64 # Parallel compilation | ||
| 71 | +``` | ||
| 72 | + | ||
| 73 | +Output: `build_out/cann-hcomm_<version>_linux-<arch>.run`. For a complete list of options, environment setup, and on-board testing, see [`docs/en/build/build.md`](./docs/en/build/build.md). Before pushing, verify locally with `--pkg` + `--ut --noexec` + `--st --noexec`. | ||
| 74 | + | ||
| 75 | +## 5. Coding Standards | ||
| 76 | + | ||
| 77 | +- Naming: Classes and functions use PascalCase; member variables use `camelCase_` (lower camelCase with a trailing underscore); constants and macros use `UPPER_SNAKE_CASE`. | ||
| 78 | +- Style: Follow the `.clang-format` in the root directory (120 columns, 4 spaces, pointer right-aligned, K&R braces). Use C++14. | ||
| 79 | +- Static warnings: Code must pass CANN static check requirements (verified during the CI codecheck stage) and compile without warnings. | ||
| 80 | +- pre-commit: clang-format v16 + OAT compliance check. New source files must include the CANN-2.0 license header. | ||
| 81 | + | ||
| 82 | +References: [CANN Coding Standards](https://gitcode.com/cann/community/tree/master/contributor/coding-standards), [CANN CI Guide](https://gitcode.com/cann/community/blob/master/contributor/repository/ci-guide.md), [pre-commit Guide](./docs/en/build/pre-commit-guide.md), `.clang-format`, `OAT.xml`. | ||
| 83 | + | ||
| 84 | +## 6. Contribution Process | ||
| 85 | + | ||
| 86 | +- Simple issues: Issue → Claim → PR → Committer Review → `/lgtm` + `/approve` merge. | ||
| 87 | +- New features: Requirement Issue → SIG Decision → `docs/en/rfcs/` RFC Review → Implementation (including UT and ST) → Review and Merge. | ||
| 88 | +- All PRs must be associated with an Issue. Fill in the description according to `.gitcode/PULL_REQUEST_TEMPLATE.en-US.md`. | ||
| 89 | + | ||
| 90 | +For details, see [`CONTRIBUTING.md`](./CONTRIBUTING_en.md). | ||
| 91 | + | ||
| 92 | +## 7. Agent Work Principles | ||
| 93 | + | ||
| 94 | +- Prioritize small and reviewable changes. Avoid large-scale refactoring unless the user explicitly requests it. | ||
| 95 | +- Before editing, locate the file and describe the plan in 3 to 6 statements. | ||
| 96 | +- When unsure about APIs, configurations, paths, or facts, search the repository or verify. Do not fabricate information. | ||
| 97 | +- Before modifying `src/`, check against the architecture constraints in Section 3: Does it violate layer dependencies? Does it add new features to `legacy/`? Does it break control plane and data plane separation? Does it introduce compile-time hard dependencies on `cann/hccl`? | ||
| 98 | +- Never write keys, tokens, passwords, private keys, `.env` values, or credentials into code, logs, or replies. | ||
| 99 | +- Unless the user requests it, do not add telemetry, analytics reporting, or additional network calls. | ||
| 100 | +- For behavioral changes, supplement or update tests under the project's existing test framework. Prioritize running the fastest relevant verification. | ||
| 101 | +- When renaming or moving directories in `src/`, simultaneously check `CMakeLists.txt`, test include paths, `#include` relative paths, `classify_rule.yaml`, `blacklist.txt`, and clean the build directory before re-verifying. | ||
| 102 | +- Destructive commands, `git commit`, and `git push` require explicit user approval. | ||
| 103 | +- By default, use Chinese for explanations. Keep output concise, specific, and reproducible. | ||
| 104 | + | ||
| 105 | +--- | ||
| 106 | + | ||
| 107 | +*Architecture constraints are based on [`docs/zh/architecture/architecture-brief.md (Chinese)`](./docs/zh/architecture/architecture-brief.md). The contribution process follows [`CONTRIBUTING_en.md`](./CONTRIBUTING_en.md). Personal temporary preferences go in `AGENTS.local.md` (gitignored).* | ||
| @@ -0,0 +1,125 @@ | |||
| 1 | +# Contribution Guide | ||
| 2 | + | ||
| 3 | +Thank you for your interest in HCCL. This project welcomes developers to explore and participate in its development. Before participating in community contributions, refer to the [cann-community](https://gitcode.com/cann/community) to understand the code of conduct, sign the CLA agreement, and learn about the source repository contribution process. | ||
| 4 | + | ||
| 5 | +## Expected Contributions | ||
| 6 | + | ||
| 7 | +- Bug fixes: Fix bugs you discover or find in the Issue list, such as logic errors, memory leaks, or crashes in the code. | ||
| 8 | +- Community tasks: Claim tasks published by the HCCL community. | ||
| 9 | +- Performance optimization: Optimize performance for specific operators or specific architectures. | ||
| 10 | +- New feature support: Add platform features, new operators, or support for new business scenarios. | ||
| 11 | +- Documentation improvement: Improve documentation, comments, or usage examples. | ||
| 12 | + | ||
| 13 | +## Prerequisites | ||
| 14 | + | ||
| 15 | +### Coding Standards | ||
| 16 | + | ||
| 17 | +Follow the [CANN Community Coding Standards](https://gitcode.com/cann/community/tree/master/contributor/coding-standards). | ||
| 18 | + | ||
| 19 | +### PR Standards | ||
| 20 | + | ||
| 21 | +1. When submitting a PR, fill in the business background, purpose, and solution details carefully according to the PR template. | ||
| 22 | +2. **All PRs must be associated with an Issue**. Reference the corresponding Issue number in the PR description. | ||
| 23 | +3. Before committing code with Git, refer to the [pre-commit tool guide](./docs/en/build/pre-commit-guide.md) to maintain consistent code style and compliance. | ||
| 24 | +4. If your changes involve new features, new interfaces, new configuration parameters, or code flow modifications (rather than simple bug fixes), discuss the plan through an Issue first to avoid rejection of your code. If you are unsure whether your changes qualify as simple bug fixes, you can also submit an Issue for discussion. | ||
| 25 | + | ||
| 26 | +## Contribution Process | ||
| 27 | + | ||
| 28 | +Contributions fall into two categories: | ||
| 29 | + | ||
| 30 | +- Simple issue handling: Bug fixes, simple code modifications, documentation changes, and so on. | ||
| 31 | +- New features or capabilities: Adding new features, capabilities, interfaces, or supporting new business scenarios. | ||
| 32 | + | ||
| 33 | +**Overall Flow** | ||
| 34 | + | ||
| 35 | +```mermaid | ||
| 36 | +flowchart TD | ||
| 37 | + A1[Start Contribution] --> A2{Type?} | ||
| 38 | + A2 -->|Simple Issue| A3[Search Issue List] | ||
| 39 | + A2 -->|New Feature| A4[Submit Requirement Issue] | ||
| 40 | + | ||
| 41 | + A3 --> A5{Existing Issue?} | ||
| 42 | + A5 -->|Yes| A6[Claim the Issue] | ||
| 43 | + A5 -->|No| A7[Create and Claim an Issue] | ||
| 44 | + A7 --> A6 | ||
| 45 | + A6 --> A19 | ||
| 46 | + | ||
| 47 | + A4 --> A10{SIG Decision} | ||
| 48 | + A10 -->|Rejected| A11[Close Issue] | ||
| 49 | + A10 -->|Accepted| A12[Add accepted Label] | ||
| 50 | + A12 --> A13[Design System Solution] | ||
| 51 | + A13 --> A14[Write or Modify RFC] | ||
| 52 | + A14 --> A15[Submit RFC Document PR] | ||
| 53 | + A15 --> A16{Maintainer Review} | ||
| 54 | + A16 -->|Feedback| A14 | ||
| 55 | + A16 -->|Approved| A17[Merge RFC] | ||
| 56 | + A17 --> A19[Modify or Implement Code, Submit PR] | ||
| 57 | + | ||
| 58 | + A19 --> A20{Committer Review} | ||
| 59 | + A20 -->|Review Feedback| A21[Modify] | ||
| 60 | + A21 --> A20 | ||
| 61 | + A20 -->|Approved| A24[Merge PR] | ||
| 62 | +``` | ||
| 63 | + | ||
| 64 | +### Simple Issue Handling | ||
| 65 | + | ||
| 66 | +1. Search for and claim an Issue | ||
| 67 | + | ||
| 68 | + - Check the Issue list to see if a corresponding Issue exists for the problem. | ||
| 69 | + - **If an Issue exists**: Claim the Issue directly. | ||
| 70 | + - **If no Issue exists**: Create a new Issue and claim it. | ||
| 71 | + | ||
| 72 | +2. Modify code and submit a PR | ||
| 73 | + | ||
| 74 | + - Meet the coding standards and PR standards. | ||
| 75 | + - Include regression tests that trigger the bug. | ||
| 76 | + | ||
| 77 | +3. Code review and merge | ||
| 78 | + | ||
| 79 | + - The Committer responsible for the corresponding module or component reviews the code and provides feedback. Modify the code based on the feedback. Once approved, add the `/lgtm` and `/approve` labels and merge. | ||
| 80 | + | ||
| 81 | +### Adding New Features or Capabilities | ||
| 82 | + | ||
| 83 | +1. Submit a Requirement Issue | ||
| 84 | + | ||
| 85 | + - Submit a Requirement type Issue in the repository. | ||
| 86 | + - Provide a detailed description including the usage scenario, business value, and high-level technical solution. | ||
| 87 | + - Initiate a discussion in the community. The SIG group decides whether to accept the requirement. If accepted, add the `accepted` label. | ||
| 88 | + | ||
| 89 | +2. Submit a number reservation PR | ||
| 90 | + | ||
| 91 | + - After the requirement is accepted, add a reservation row in the [RFC Number Registry](./docs/en/rfcs/INDEX.md) following the smallest unused number rule (status = reserved). | ||
| 92 | + - Submit a **number reservation PR** (containing only the one-line update to INDEX.md). Merging this PR indicates that the number is available for use. | ||
| 93 | + | ||
| 94 | +3. System solution design | ||
| 95 | + | ||
| 96 | + - Create an RFC document in markdown format in the `docs/en/rfcs` directory (filename starting with the registered number) and write the system solution following the [RFC template](./docs/en/rfcs/0000-template.md). | ||
| 97 | + - Submit an **RFC document PR**. | ||
| 98 | + | ||
| 99 | +4. System solution review | ||
| 100 | + | ||
| 101 | + - The detailed design solution is reviewed through the RFC document PR. | ||
| 102 | + - Modify the solution based on the review feedback. | ||
| 103 | + | ||
| 104 | +5. RFC merge | ||
| 105 | + | ||
| 106 | + - After all Maintainers agree on the solution, the Maintainer adds the `/lgtm` and `/approve` labels and merges. | ||
| 107 | + - The merged RFC solution serves as the contract for subsequent code implementation. The code implementation must follow the RFC solution. | ||
| 108 | + - After the RFC document PR is merged, update the corresponding row status in the [RFC Number Registry](./docs/en/rfcs/INDEX.md) from `reserved` to `accepted`. | ||
| 109 | + | ||
| 110 | +6. Software implementation | ||
| 111 | + | ||
| 112 | + - Implement the code according to the RFC solution and submit a PR. | ||
| 113 | + - Include corresponding test code (both unit tests and system tests). | ||
| 114 | + | ||
| 115 | +7. Code review and merge | ||
| 116 | + | ||
| 117 | + - The Committer responsible for the corresponding module or component reviews the code and provides feedback. Modify the code based on the feedback. Once approved, add the `/lgtm` and `/approve` labels and merge. | ||
| 118 | + | ||
| 119 | +--- | ||
| 120 | + | ||
| 121 | +## Dispute Resolution | ||
| 122 | + | ||
| 123 | +Disputed Issues, PRs, or RFCs can be submitted as agenda items at the [SIG Working Meeting](https://etherpad-cann.meeting.osinfra.cn/p/sig-hccl) for the SIG group to decide. | ||
| 124 | + | ||
| 125 | +*This document is maintained by the community. For suggestions on changes, submit them in an Issue.* | ||
| @@ -1,6 +1,6 @@ | |||
| 1 | # HCOMM | 1 | # HCOMM |
| 2 | 2 | ||
| 3 | -## 🔥Latest News | 3 | +## 🔥 Latest News |
| 4 | 4 | ||
| 5 | - [2025/11/30] HCOMM项目开源。 | 5 | - [2025/11/30] HCOMM项目开源。 |
| 6 | 6 | ||
| @@ -67,7 +67,7 @@ HCOMM通信基础库采用分层解耦的设计思路，将通信能力划分为 | |||
| 67 | └── build.sh # 编译构建脚本 | 67 | └── build.sh # 编译构建脚本 |
| 68 | ``` | 68 | ``` |
| 69 | 69 | ||
| 70 | -## 📝版本配套 | 70 | +## 📝 版本配套 |
| 71 | 71 | ||
| 72 | 本项目源码会跟随CANN软件版本发布，关于CANN软件版本与本项目标签的对应关系请参阅[release仓库](https://gitcode.com/cann/release-management)中的相应版本说明。 | 72 | 本项目源码会跟随CANN软件版本发布，关于CANN软件版本与本项目标签的对应关系请参阅[release仓库](https://gitcode.com/cann/release-management)中的相应版本说明。 |
| 73 | 请注意，为确保您的源码定制开发顺利进行，请选择配套的CANN版本与GitCode标签源码，使用master分支可能存在版本不匹配的风险。 | 73 | 请注意，为确保您的源码定制开发顺利进行，请选择配套的CANN版本与GitCode标签源码，使用master分支可能存在版本不匹配的风险。 |
| @@ -0,0 +1,89 @@ | |||
| 1 | +# HCOMM | ||
| 2 | + | ||
| 3 | +## 🔥 Latest News | ||
| 4 | + | ||
| 5 | +- [2025/11/30] The HCOMM project is now open source. | ||
| 6 | + | ||
| 7 | +## 🚀 Overview | ||
| 8 | + | ||
| 9 | +HCOMM (Huawei Communication) is the underlying communication library of HCCL. It provides communication domain and communication resource management capabilities. | ||
| 10 | + | ||
| 11 | +HCOMM offers standardized communication programming interfaces with the following key features: | ||
| 12 | + | ||
| 13 | +- Supports multiple communication engines on Ascend devices, fully leveraging hardware capabilities. | ||
| 14 | +- Supports multiple communication protocols, including PCIe, HCCS, RDMA, and more. | ||
| 15 | +- Decouples the communication platform from communication operator development, enabling independent development, building, and deployment of communication operators. | ||
| 16 | + | ||
| 17 | +<img src="./docs/en/build/figures/architecture.png" alt="hccl-architecture" style="width: 65%; height:65%;" /> | ||
| 18 | + | ||
| 19 | +The HCOMM communication library adopts a layered and decoupled design approach, dividing communication capabilities into a control plane and a data plane. | ||
| 20 | + | ||
| 21 | +- Control plane: Provides topology information query and communication resource management. | ||
| 22 | +- Data plane: Provides data transfer and computation capabilities such as local operations, inter-operator synchronization, and communication operations. | ||
| 23 | + | ||
| 24 | +The control plane provides communication resources, and the data plane provides methods for operating on resources. The communication programming interfaces allow communication operator developers to focus on business innovation without concerning themselves with the complex implementation details at the chip level. | ||
| 25 | + | ||
| 26 | +## 🔍 Directory Structure | ||
| 27 | + | ||
| 28 | +The key directories of this project are as follows: | ||
| 29 | + | ||
| 30 | +```text | ||
| 31 | +├── src # Source code directory | ||
| 32 | +│ ├── base_comm # Basic communication layer | ||
| 33 | +│ │ ├── common # Common basic functionality for the basic communication layer | ||
| 34 | +│ │ ├── primitives # Basic communication primitives | ||
| 35 | +│ │ └── resources # Basic communication resources | ||
| 36 | +│ ├── coll_communicator_mgr # Collective communication domain management | ||
| 37 | +│ │ ├── api_c_adpt # C interface adaptation | ||
| 38 | +│ │ ├── common # Common basic functionality for the collective communication layer | ||
| 39 | +│ │ ├── communicator # Communication domain | ||
| 40 | +│ │ ├── dfx # Diagnostics and tracing | ||
| 41 | +│ │ ├── rank_graphs # Topology management | ||
| 42 | +│ │ └── resource_mgr # Resource management | ||
| 43 | +│ └── legacy # Historical version compatibility directory | ||
| 44 | +│ ├── ascend910 # A2 and A3 compatibility code | ||
| 45 | +│ │ ├── algorithm # Communication algorithm source code | ||
| 46 | +│ │ ├── common # Common basic functionality | ||
| 47 | +│ │ ├── framework # Communication framework source code | ||
| 48 | +│ │ ├── hccd # Inter-process point-to-point communication | ||
| 49 | +│ │ ├── platform # Communication platform source code | ||
| 50 | +│ │ └── pub_inc # Platform interface header files | ||
| 51 | +│ └── ascend950 # A5 legacy flow compatibility code | ||
| 52 | +│ ├── common # Common basic components | ||
| 53 | +│ ├── framework # Framework core implementation | ||
| 54 | +│ ├── include # Public interface header files | ||
| 55 | +│ ├── interface # Interface adaptation layer | ||
| 56 | +│ ├── local_build # Local build tools | ||
| 57 | +│ ├── service # Service layer | ||
| 58 | +│ └── unified_platform # Unified platform layer | ||
| 59 | +├── python # Python package | ||
| 60 | +├── include # External header files | ||
| 61 | +├── pkg_inc # Inter-package interface header files | ||
| 62 | +├── test # Test code directory | ||
| 63 | +│ ├── ut # Unit test code directory | ||
| 64 | +│ └── st # System test code directory | ||
| 65 | +├── docs # Documentation directory | ||
| 66 | +├── examples # Sample code directory | ||
| 67 | +└── build.sh # Build script | ||
| 68 | +``` | ||
| 69 | + | ||
| 70 | +## 📝 Version Compatibility | ||
| 71 | + | ||
| 72 | +The source code of this project is released alongside the CANN software version. For the mapping between CANN software versions and this project's tags, refer to the corresponding version descriptions in the [release repository](https://gitcode.com/cann/release-management). To ensure a smooth custom source code development experience, select a compatible CANN version and GitCode tag. Using the master branch may pose a version mismatch risk. | ||
| 73 | + | ||
| 74 | +## ⚡️ Quick Start | ||
| 75 | + | ||
| 76 | +To quickly build and experience this project, refer to the following simple guides. | ||
| 77 | + | ||
| 78 | +- [Source Code Build](./docs/en/build/build.md): Learn how to compile and install this project, and perform basic test verification. | ||
| 79 | +- [Sample Execution](./examples/README_en.md): Follow detailed sample code and step-by-step instructions for a quick experience. | ||
| 80 | + | ||
| 81 | +## 📖 Learning Tutorials | ||
| 82 | + | ||
| 83 | +HCCL provides usage guides, communication operator development guides, technical articles, and training videos. For details, see the [HCCL References](./docs/README_en.md). HCCL also offers QuickStart guides, common FAQs, and other wiki resources. For details, see the [WIKI](https://gitcode.com/cann/hcomm/wiki). | ||
| 84 | + | ||
| 85 | +## 📝 Related Information | ||
| 86 | + | ||
| 87 | +- [Contribution Guide](CONTRIBUTING_en.md) | ||
| 88 | +- [Security Statement](SECURITY_en.md) | ||
| 89 | +- [License](LICENSE) | ||
| @@ -0,0 +1,68 @@ | |||
| 1 | +# Security Statement | ||
| 2 | + | ||
| 3 | +## Recommended User Account | ||
| 4 | + | ||
| 5 | +For security reasons, avoid using administrator accounts such as root to execute commands. Follow the principle of least privilege. | ||
| 6 | + | ||
| 7 | +## File Permission Control | ||
| 8 | + | ||
| 9 | +- It is recommended to set the umask value to 0027 or higher on the host machine (including the host machine and containers) to ensure that the default maximum permissions for new folders are 750 and for new files are 640. | ||
| 10 | +- Apply security measures such as permission control to sensitive content, including personal private data, business assets, and source files. For example, control the permissions of the installation directory and input public data files. For recommended permission settings, refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios). | ||
| 11 | +- Control permissions during installation and usage. Refer to [A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios](#a---recommended-maximum-permissions-for-files-folders-in-various-scenarios) for file permission references. | ||
| 12 | + | ||
| 13 | +## Build Security Statement | ||
| 14 | + | ||
| 15 | +- When compiling and installing this project from source, the compilation process generates intermediate files. After compilation, control the permissions of these intermediate files to ensure file security. | ||
| 16 | + | ||
| 17 | +## Runtime Security Statement | ||
| 18 | + | ||
| 19 | +- When a runtime exception occurs, the process exits and prints error information. Locate the specific error cause based on the error message. | ||
| 20 | + | ||
| 21 | +## Public URL Statement | ||
| 22 | + | ||
| 23 | +The public URLs included in the code of this project are declared as follows: | ||
| 24 | + | ||
| 25 | +| Type | Open Source Code Address | File Name | Public IP Address or Public URL or Domain or Email or Archive Address | Purpose | | ||
| 26 | +| :--: | :----------: | :----- | :-------------------------------------------------------------- | :-------------------------------------------- | | ||
| 27 | +| Dependency | Not Applicable | cmake/third_party/makeself-fetch.cmake | https://gitcode.com/cann-src-third-party/makeself/releases/download/release-2.5.0-patch1.0/makeself-release-2.5.0-patch1.tar.gz | Download makeself source code from gitcode as a build dependency | | ||
| 28 | +| Dependency | Not Applicable | cmake/third_party/json.cmake | https://gitcode.com/cann-src-third-party/json/releases/download/v3.11.3/include.zip | Download json source code from gitcode as a build dependency | | ||
| 29 | +| Dependency | Not Applicable | cmake/third_party/openssl.cmake | https://gitcode.com/cann-src-third-party/openssl/releases/download/openssl-3.0.9/openssl-openssl-3.0.9.tar.gz | Download openssl source code from gitcode as a build dependency | | ||
| 30 | +| Dependency | Not Applicable | cmake/third_party/gtest.cmake | https://gitcode.com/cann-src-third-party/googletest/releases/download/v1.14.0/googletest-1.14.0.tar.gz | Download googletest source code from gitcode as a build dependency | | ||
| 31 | +| Dependency | Not Applicable | cmake/third_party/mockcpp.cmake | https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h4/mockcpp-2.7.tar.gz | Download mockcpp source code from gitcode as a build dependency | | ||
| 32 | +| Dependency | Not Applicable | cmake/third_party/protobuf.cmake | https://gitcode.com/cann-src-third-party/protobuf/releases/download/v25.1/protobuf-25.1.tar.gz | Download protobuf source code from gitcode as a build dependency | | ||
| 33 | +| Dependency | Not Applicable | rdma-core | https://gitcode.com/cann-src-third-party/rdma-core/releases/download/v42.7-h1/rdma-core-42.7.tar.gz | Download rdma-core source code from gitcode as a build dependency | | ||
| 34 | +| Dependency | Not Applicable | rdma-core-patch | https://gitcode.com/cann-src-third-party/rdma-core/releases/download/v42.7-h1/rdma-core-42.7.patch | Download rdma-core-patch source code from gitcode as a build dependency | | ||
| 35 | + | ||
| 36 | +--- | ||
| 37 | + | ||
| 38 | +## Port Statement | ||
| 39 | + | ||
| 40 | +For information about the ports opened by HCCL, the transport layer protocols used, authentication methods, and purposes, refer to the HCCL tab in the [CANN Communication Matrix](https://hiascend.com/document/redirect/CannCommunityCommMatrix). | ||
| 41 | + | ||
| 42 | +## Vulnerability Mechanism | ||
| 43 | + | ||
| 44 | +[Vulnerability Management](https://gitcode.com/cann/community/blob/master/security/security.md) | ||
| 45 | + | ||
| 46 | +## Appendix | ||
| 47 | + | ||
| 48 | +### A - Recommended Maximum Permissions for Files (Folders) in Various Scenarios | ||
| 49 | + | ||
| 50 | +| Type | Linux Recommended Maximum Permission | | ||
| 51 | +| ---------------------------------- | -------------------- | | ||
| 52 | +| User home directory | 750 (rwxr-x---) | | ||
| 53 | +| Program files (including scripts, library files, and so on) | 550 (r-xr-x---) | | ||
| 54 | +| Program file directory | 550 (r-xr-x---) | | ||
| 55 | +| Configuration files | 640 (rw-r-----) | | ||
| 56 | +| Configuration file directory | 750 (rwxr-x---) | | ||
| 57 | +| Log files (completed or archived) | 440 (r--r-----) | | ||
| 58 | +| Log files (being recorded) | 640 (rw-r-----) | | ||
| 59 | +| Log file directory | 750 (rwxr-x---) | | ||
| 60 | +| Debug files | 640 (rw-r-----) | | ||
| 61 | +| Debug file directory | 750 (rwxr-x---) | | ||
| 62 | +| Temporary file directory | 750 (rwxr-x---) | | ||
| 63 | +| Maintenance and upgrade file directory | 770 (rwxrwx---) | | ||
| 64 | +| Business data files | 640 (rw-r-----) | | ||
| 65 | +| Business data file directory | 750 (rwxr-x---) | | ||
| 66 | +| Key components, private keys, certificates, encrypted file directory | 700 (rwx------) | | ||
| 67 | +| Key components, private keys, certificates, encrypted ciphertext | 600 (rw-------) | | ||
| 68 | +| Encryption and decryption interfaces, encryption and decryption scripts | 500 (r-x------) | | ||
| @@ -0,0 +1,20 @@ | |||
| 1 | +# HCCL Document Library Overview | ||
| 2 | + | ||
| 3 | +## Documents | ||
| 4 | + | ||
| 5 | +- [Collective Communication Library User Guide](https://www.hiascend.com/document/detail/en/canncommercial/850/commlib/hcclug/hcclug_000001.html): Describes HCCL basic concepts, how to use HCCL APIs for collective communication development, how to develop communication operators, and fault handling with common configurations. | ||
| 6 | +- [Communication Operator Development Guide (Chinese)](./zh/comm_op_dev_guide/README.md): Describes how to develop communication operators using the interfaces provided by the HCOMM communication base library. | ||
| 7 | + | ||
| 8 | +## Technical Articles | ||
| 9 | + | ||
| 10 | +- [HCCL—Introduction to the Ascend High-Performance Collective Communication Library (Chinese)](https://www.hiascend.com/zh/developer/techArticles/20240809-1) | ||
| 11 | +- [HCCL Collective Communication Troubleshooting (Chinese)](https://www.hiascend.com/zh/developer/techArticles/20240930-1) | ||
| 12 | +- [Deep Learning Distributed Training and Collective Communication (1) (Chinese)](https://www.hiascend.com/zh/developer/techArticles/20241111-1) | ||
| 13 | +- [Deep Learning Distributed Training and Collective Communication (2) (Chinese)](https://www.hiascend.com/zh/developer/techArticles/20241122-1) | ||
| 14 | + | ||
| 15 | +## Training Videos | ||
| 16 | + | ||
| 17 | +- [Ascend Collective Communication Tutorials — Common Collective Communication Primitives (Chinese)](https://www.bilibili.com/video/BV1YtUWYQEq2/?spm_id_from=333.999.0.0) | ||
| 18 | +- [Ascend Collective Communication Tutorials — Typical Collective Communication Algorithms (Chinese)](https://www.bilibili.com/video/BV1XEz5YeE3A/?spm_id_from=333.999.0.0) | ||
| 19 | +- [Ascend Collective Communication Tutorials — Collective Communication Service Development (Chinese)](https://www.bilibili.com/video/BV1rfzoYTE3W/?spm_id_from=333.999.0.0) | ||
| 20 | + | ||
| @@ -0,0 +1,234 @@ | |||
| 1 | +# Source Code Build | ||
| 2 | + | ||
| 3 | +## Environment Preparation | ||
| 4 | + | ||
| 5 | +This project supports source code building. Before compiling and running, complete the basic environment setup and source code download by following the steps below. Ensure that the NPU driver, firmware, and CANN software are installed. | ||
| 6 | + | ||
| 7 | +### Prerequisites | ||
| 8 | + | ||
| 9 | +The software dependencies required for compiling this project are listed below. Ensure that the version requirements are met. | ||
| 10 | + | ||
| 11 | +- python >= 3.7.0 | ||
| 12 | +- pip3 >= 20.3.0 | ||
| 13 | +- setuptools >= 45.0.0 | ||
| 14 | +- wheel >= 0.34.0 | ||
| 15 | +- gcc and g++: 7.3.0 to 13.3.x | ||
| 16 | +- cmake >= 3.16.0 | ||
| 17 | +- pkg-config >= 0.29.1 (for compiling rdma-core) | ||
| 18 | +- patch >= 2.7.0 (for applying patch files) | ||
| 19 | +- ccache (optional, for improving incremental compilation speed) | ||
| 20 | +- lcov (optional, for generating UT or ST coverage reports) | ||
| 21 | + | ||
| 22 | +### Installing the CANN Software Package | ||
| 23 | + | ||
| 24 | +1. **Install the driver and firmware (runtime dependency)** | ||
| 25 | + | ||
| 26 | + For downloading and installing the driver and firmware, refer to the "Prepare Software Package" and "Install NPU Driver and Firmware" sections in the [CANN Software Installation Guide](https://www.hiascend.com/document/redirect/CannCommunityInstWizard). The driver and firmware are runtime dependencies. If you are only compiling the source code of this project, they do not need to be installed. | ||
| 27 | + | ||
| 28 | +2. **Install the CANN software package** | ||
| 29 | + | ||
| 30 | + - **Scenario 1: Experience or develop based on the master version** | ||
| 31 | + | ||
| 32 | + Click the [download link](https://ascend.devcloud.huaweicloud.com/artifactory/cann-run-mirror/software/master/), select the latest version, and download the corresponding package based on the product model and environment architecture. The installation commands are as follows. For more guidance, refer to the [CANN Software Installation Guide](https://www.hiascend.com/document/redirect/CannCommunityInstWizard). | ||
| 33 | + | ||
| 34 | + 1. Install the CANN Toolkit development kit package. | ||
| 35 | + | ||
| 36 | + ```bash | ||
| 37 | + # Ensure the installation package has executable permissions | ||
| 38 | + chmod +x Ascend-cann-toolkit_${cann_version}_linux-${arch}.run | ||
| 39 | + # Installation command | ||
| 40 | + ./Ascend-cann-toolkit_${cann_version}_linux-${arch}.run --install --install-path=${install_path} | ||
| 41 | + ``` | ||
| 42 | + | ||
| 43 | + 2. Install the CANN ops operator package (runtime dependency). | ||
| 44 | + | ||
| 45 | + The ops operator package is a runtime dependency. If you are only compiling the source code of this project, this package does not need to be installed. | ||
| 46 | + | ||
| 47 | + ```bash | ||
| 48 | + # Ensure the installation package has executable permissions | ||
| 49 | + chmod +x Ascend-cann-${soc_name}-ops_${cann_version}_linux-${arch}.run | ||
| 50 | + # Installation command | ||
| 51 | + ./Ascend-cann-${soc_name}-ops_${cann_version}_linux-${arch}.run --install --install-path=${install_path} | ||
| 52 | + ``` | ||
| 53 | + | ||
| 54 | + - \$\{cann\_version\}: Indicates the CANN software package version number. | ||
| 55 | + - \$\{arch\}$: Indicates the CPU architecture, for example, aarch64 or x86_64. | ||
| 56 | + - \$\{soc\_name\}: Indicates the NPU model name. | ||
| 57 | + - \$\{install\_path\}: Indicates the specified installation path. The CANN ops operator package must be installed in the same path as the CANN Toolkit development kit package. The default installation path for the root user is `/usr/local/Ascend`. | ||
| 58 | + | ||
| 59 | + - **Scenario 2: Experience or develop based on a released version** | ||
| 60 | + | ||
| 61 | + Visit the [CANN official download center](https://www.hiascend.com/cann/download), select a released version (only CANN 8.5.0 and later versions are supported), and download the corresponding package based on the product model and environment architecture. Finally, follow the commands provided on the webpage to complete the installation. | ||
| 62 | + | ||
| 63 | +### Environment Verification | ||
| 64 | + | ||
| 65 | +After installing the CANN software package, verify that the environment is functioning correctly. | ||
| 66 | + | ||
| 67 | +- **Check the NPU device**: | ||
| 68 | + | ||
| 69 | + ```bash | ||
| 70 | + # Run npu-smi. If device information is displayed normally, the driver is working correctly. | ||
| 71 | + npu-smi info | ||
| 72 | + ``` | ||
| 73 | + | ||
| 74 | +- **Check the CANN software**: | ||
| 75 | + | ||
| 76 | + ```bash | ||
| 77 | + # View the version information provided by the version field of the CANN Toolkit development kit package (default installation path). <arch> indicates the CPU architecture (aarch64 or x86_64). | ||
| 78 | + cat /usr/local/Ascend/cann/<arch>-linux/ascend_toolkit_install.info | ||
| 79 | + # View the version information provided by the version field of the CANN ops operator package (default installation path). | ||
| 80 | + cat /usr/local/Ascend/cann/<arch>-linux/ascend_ops_install.info | ||
| 81 | + ``` | ||
| 82 | + | ||
| 83 | +### Environment Variable Configuration | ||
| 84 | + | ||
| 85 | +Select the appropriate command to apply the environment variables. | ||
| 86 | + | ||
| 87 | +```bash | ||
| 88 | +# Default installation path, using the root user as an example (for non-root users, replace /usr/local with ${HOME}) | ||
| 89 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 90 | +# Specified installation path | ||
| 91 | +# source ${install_path}/cann/set_env.sh | ||
| 92 | +``` | ||
| 93 | + | ||
| 94 | +## Compilation and Installation | ||
| 95 | + | ||
| 96 | +### Downloading the Source Code | ||
| 97 | + | ||
| 98 | +Use the following command to download the source code. Replace \$\{tag\_version\} with the target branch tag name. For the mapping between source branch tags and CANN versions, refer to the [release repository](https://gitcode.com/cann/release-management). | ||
| 99 | + | ||
| 100 | +```shell | ||
| 101 | +# Download the source code for the corresponding project branch | ||
| 102 | +git clone -b ${tag_version} https://gitcode.com/cann/hcomm.git | ||
| 103 | +``` | ||
| 104 | + | ||
| 105 | +### Compiling the Source Code | ||
| 106 | + | ||
| 107 | +This project provides a one-click build capability. Navigate to the root directory of the repository and execute the following commands: | ||
| 108 | + | ||
| 109 | +```shell | ||
| 110 | +# Compile the host package | ||
| 111 | +bash build.sh --pkg | ||
| 112 | +# Compile the host and device package | ||
| 113 | +bash build.sh --pkg --full | ||
| 114 | +``` | ||
| 115 | + | ||
| 116 | +During compilation, the dependency packages listed in [Open Source Third-Party Software Dependencies](#open-source-third-party-software-dependencies) are automatically downloaded. If the compilation environment does not have network access, download the required dependency packages in a networked environment, manually upload them to the compilation environment, and specify the dependency package storage path using the `--cann_3rd_lib_path` parameter. | ||
| 117 | + | ||
| 118 | +```shell | ||
| 119 | +# Specify the dependency package storage path. Default: ./third_party | ||
| 120 | +bash build.sh --cann_3rd_lib_path={your_3rd_party_path} | ||
| 121 | +``` | ||
| 122 | + | ||
| 123 | +After compilation, a `cann-hcomm_<version>_linux-<arch>.run` software package is generated in the `./build_out` directory. | ||
| 124 | + | ||
| 125 | +`<version>` indicates the software version number, and `<arch>` indicates the operating system architecture. The values include x86_64 and aarch64. | ||
| 126 | + | ||
| 127 | +### Installation | ||
| 128 | + | ||
| 129 | +Install the compiled HCOMM software package: | ||
| 130 | + | ||
| 131 | +```shell | ||
| 132 | +bash ./build_out/cann-hcomm_<version>_linux-<arch>.run --full | ||
| 133 | +``` | ||
| 134 | + | ||
| 135 | +During installation, replace the package name in the command with the actual software package name. | ||
| 136 | + | ||
| 137 | +After installation, the HCOMM software package generated by the user replaces the HCOMM-related software in the installed CANN Toolkit development kit package. | ||
| 138 | + | ||
| 139 | +### Uninstallation | ||
| 140 | + | ||
| 141 | +To uninstall the compiled HCOMM software package and restore it to the state after installing the CANN Toolkit development kit package, use the following command: | ||
| 142 | + | ||
| 143 | +```shell | ||
| 144 | +bash ./build_out/cann-hcomm_<version>_linux-<arch>.run --uninstall | ||
| 145 | +``` | ||
| 146 | + | ||
| 147 | +During uninstallation, replace the package name in the command with the actual software package name. | ||
| 148 | + | ||
| 149 | +## Testing | ||
| 150 | + | ||
| 151 | +### LLT Testing | ||
| 152 | + | ||
| 153 | +After installing the compiled HCOMM software package, execute the following command to run LLT test cases: | ||
| 154 | + | ||
| 155 | +```shell | ||
| 156 | +bash build.sh --ut | ||
| 157 | +``` | ||
| 158 | + | ||
| 159 | +### On-Board Testing | ||
| 160 | + | ||
| 161 | +> **Note** | ||
| 162 | +> Before on-board testing, ensure that the driver, firmware, CANN Toolkit development kit package, and CANN ops operator package are installed. | ||
| 163 | + | ||
| 164 | +Developers can use the HCCL Test tool for collective communication function and performance testing on the board. The workflow for using the HCCL Test tool is as follows: | ||
| 165 | + | ||
| 166 | +1. Tool compilation | ||
| 167 | + | ||
| 168 | + Before using the HCCL Test tool, install the MPI dependency and compile the HCCL Test tool. For detailed instructions, refer to the "MPI Installation and Configuration" and "Tool Compilation" sections in the corresponding version of the [Ascend Documentation Center - HCCL Performance Test Tool Guide](https://hiascend.com/document/redirect/CannCommunityToolHcclTest). | ||
| 169 | + | ||
| 170 | +2. Disable signature verification | ||
| 171 | + | ||
| 172 | + The `cann-hcomm_<version>_linux-<arch>.run` software package generated from this source repository contains the following tar.gz subpackages: | ||
| 173 | + - `cann-hcomm-compat.tar.gz`: HCOMM compatibility upgrade package. | ||
| 174 | + - `cann-hccd-compat.tar.gz`: DataFlow compatibility upgrade package. | ||
| 175 | + - `aicpu_hcomm.tar.gz`: AI CPU communication base package. | ||
| 176 | + | ||
| 177 | + These tar.gz packages are loaded to the Device when the service starts. During the loading process, the driver performs security signature verification by default to ensure the package is trusted. Because the tar.gz packages compiled from this source repository do not contain a signature header, the driver security signature verification mechanism needs to be disabled. | ||
| 178 | + | ||
| 179 | + **Method to disable signature verification:** | ||
| 180 | + | ||
| 181 | + Use Ascend HDK 25.5.T2.B001 or later, and use the npu-smi tool provided with the Ascend HDK to disable signature verification. The following are reference commands. Execute them as the root user on the physical machine (using device 0 as an example). | ||
| 182 | + | ||
| 183 | + ```shell | ||
| 184 | + npu-smi set -t custom-op-secverify-enable -i 0 -d 1 # Enable signature verification configuration | ||
| 185 | + npu-smi set -t custom-op-secverify-mode -i 0 -d 0 # Disable custom signature verification | ||
| 186 | + ``` | ||
| 187 | + | ||
| 188 | +3. Execute the HCCL Test command to test the function and performance of collective communication. | ||
| 189 | + | ||
| 190 | + Using one compute node, 8 NPU devices, and testing the AllReduce operator performance as an example: | ||
| 191 | + | ||
| 192 | + ```shell | ||
| 193 | + # /usr/local/Ascend is the CANN software installation path for the root user under the default installation path. Replace it with the actual path. | ||
| 194 | + cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test | ||
| 195 | + | ||
| 196 | + # Data size (-b) from 8KB to 64MB, increment factor (-f) of 2, number of NPUs participating in training is 8 | ||
| 197 | + mpirun -n 8 ./bin/all_reduce_test -b 8K -e 64M -f 2 -d fp32 -o sum -p 8 | ||
| 198 | + ``` | ||
| 199 | + | ||
| 200 | + For detailed usage instructions of the tool, refer to the "Tool Execution" section in the [Ascend Documentation Center - HCCL Performance Test Tool Guide](https://hiascend.com/document/redirect/CannCommunityToolHcclTest). | ||
| 201 | + | ||
| 202 | +4. View the results | ||
| 203 | + | ||
| 204 | + After executing the HCCL Test tool, the display output is as follows: | ||
| 205 | + | ||
| 206 | +  | ||
| 207 | + | ||
| 208 | + - "check_result" shows success, indicating that the communication operator executed successfully and the AllReduce operator function is correct. | ||
| 209 | + - "aveg_time": The execution time of the collective communication operator, in us. | ||
| 210 | + - "alg_bandwidth": The execution bandwidth of the collective communication operator, in GB/s. | ||
| 211 | + - "data_size": The amount of data participating in collective communication on a single NPU, in Bytes. | ||
| 212 | + | ||
| 213 | +## Appendix | ||
| 214 | + | ||
| 215 | +### Open Source Third-Party Software Dependencies | ||
| 216 | + | ||
| 217 | +When compiling this project, the following third-party open source software dependencies are required: | ||
| 218 | + | ||
| 219 | +| Open Source Software | Version | Download URL | | ||
| 220 | +| ------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| 221 | +| json | 3.11.3 | [include.zip](https://gitcode.com/cann-src-third-party/json/releases/download/v3.11.3/include.zip) | | ||
| 222 | +| makeself | 2.5.0 | [makeself-release-2.5.0-patch1.tar.gz](https://gitcode.com/cann-src-third-party/makeself/releases/download/release-2.5.0-patch1.0/makeself-release-2.5.0-patch1.tar.gz) | | ||
| 223 | +| openssl | 3.0.9 | [openssl-openssl-3.0.9.tar.gz](https://gitcode.com/cann-src-third-party/openssl/releases/download/openssl-3.0.9/openssl-openssl-3.0.9.tar.gz) | | ||
| 224 | +| hcomm_utils | 9.0.0 (aarch64) | [cann-hcomm-utils_9.0.0_linux-aarch64.tar.gz](https://ascend-cann.obs.cn-north-4.myhuaweicloud.com/CANN/20260330_newest/cann-hcomm-utils_9.0.0_linux-aarch64.tar.gz) | | ||
| 225 | +| hcomm_utils | 9.0.0 (x86_64) | [cann-hcomm-utils_9.0.0_linux-x86_64.tar.gz](https://ascend-cann.obs.cn-north-4.myhuaweicloud.com/CANN/20260330_newest/cann-hcomm-utils_9.0.0_linux-x86_64.tar.gz) | | ||
| 226 | +| googletest | 1.14.0 | [googletest-1.14.0.tar.gz](https://gitcode.com/cann-src-third-party/googletest/releases/download/v1.14.0/googletest-1.14.0.tar.gz) | | ||
| 227 | +| boost | 1.87.0 | [boost_1_87_0.tar.gz](https://gitcode.com/cann-src-third-party/boost/releases/download/v1.87.0/boost_1_87_0.tar.gz) | | ||
| 228 | +| mockcpp | 2.7-h4 | [mockcpp-2.7.tar.gz](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h4/mockcpp-2.7.tar.gz) | | ||
| 229 | +| mockcpp-patch | 2.7-h4 | [mockcpp-2.7_py3.patch](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h4/mockcpp-2.7_py3.patch) | | ||
| 230 | +| abseil-cpp | 20250127.0 | [abseil-cpp-20250127.0.tar.gz](https://gitcode.com/cann-src-third-party/abseil-cpp/releases/download/20250127.0/abseil-cpp-20250127.0.tar.gz) | | ||
| 231 | +| protobuf | 25.1 | [protobuf-25.1.tar.gz](https://gitcode.com/cann-src-third-party/protobuf/releases/download/v25.1/protobuf-25.1.tar.gz) | | ||
| 232 | +| rdma-core | v42.7-h1 | [rdma-core-42.7.tar.gz](https://gitcode.com/cann-src-third-party/rdma-core/releases/download/v42.7-h1/rdma-core-42.7.tar.gz) | | ||
| 233 | +| rdma-core-patch | v42.7-h1 | [rdma-core-42.7.patch](https://gitcode.com/cann-src-third-party/rdma-core/releases/download/v42.7-h1/rdma-core-42.7.patch) | | ||
| 234 | +| cann-cmake | master-025 | [cmake-master-025.tar.gz](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cmake/cmake-master-025.tar.gz) | | ||
| @@ -0,0 +1,3 @@ | |||
| 1 | +version https://git-lfs.github.com/spec/v1 | ||
| 2 | +oid sha256:aceca6af04af869c6187064a24b6072d70db91f8cc097296811a3f707928c138 | ||
| 3 | +size 226102 | ||
| @@ -0,0 +1,3 @@ | |||
| 1 | +version https://git-lfs.github.com/spec/v1 | ||
| 2 | +oid sha256:25d13a896e69d492299d5c09623db58c971d9c243cfa6f098fbe3e7a98182bac | ||
| 3 | +size 28606 | ||
| @@ -0,0 +1,131 @@ | |||
| 1 | +# pre-commit Tool Guide | ||
| 2 | + | ||
| 3 | +## Overview | ||
| 4 | + | ||
| 5 | +pre-commit is a Git Hooks framework that automatically runs code checking and formatting tools during `git commit`. This project is configured with the following checks: | ||
| 6 | + | ||
| 7 | +| Hook | Function | Description | | ||
| 8 | +| ---------------- | ---------------- | -------------------------------- | | ||
| 9 | +| **clang-format** | C and C++ code formatting | Automatically formats code to maintain consistent style | | ||
| 10 | +| **OAT Check** | Open source compliance check | Detects license headers and prevents binary file submission | | ||
| 11 | + | ||
| 12 | +## Requirements | ||
| 13 | + | ||
| 14 | +- **Git**: 2.0+ | ||
| 15 | +- **Python**: 3.8+ | ||
| 16 | +- **clang-format**: 14.0+ (code formatting tool) | ||
| 17 | +- **Java**: 17+ (required for OAT tool, can be installed automatically) | ||
| 18 | +- **Maven**: 3.6+ (required for OAT tool, can be installed automatically) | ||
| 19 | + | ||
| 20 | +## Installation Steps | ||
| 21 | + | ||
| 22 | +### 1. Install pre-commit | ||
| 23 | + | ||
| 24 | +```bash | ||
| 25 | +# Option 1: Using pip | ||
| 26 | +pip install pre-commit | ||
| 27 | + | ||
| 28 | +# Option 2: Using system package manager (Ubuntu or Debian) | ||
| 29 | +sudo apt install pre-commit | ||
| 30 | +``` | ||
| 31 | + | ||
| 32 | +### 2. Install Dependency Tools | ||
| 33 | + | ||
| 34 | +```bash | ||
| 35 | +# Ubuntu or Debian | ||
| 36 | +sudo apt install clang-format openjdk-17-jre maven | ||
| 37 | + | ||
| 38 | +# macOS | ||
| 39 | +brew install clang-format openjdk@17 maven | ||
| 40 | +``` | ||
| 41 | + | ||
| 42 | +### 3. Install Git Hooks in the Project Path | ||
| 43 | + | ||
| 44 | +```bash | ||
| 45 | +# Navigate to the repository root directory | ||
| 46 | +cd /path/to/hcomm | ||
| 47 | +pre-commit install | ||
| 48 | +``` | ||
| 49 | + | ||
| 50 | +A successful installation displays: | ||
| 51 | + | ||
| 52 | +```bash | ||
| 53 | +pre-commit installed at .git/hooks/pre-commit | ||
| 54 | +``` | ||
| 55 | + | ||
| 56 | +## Usage | ||
| 57 | + | ||
| 58 | +### Automatic Checking (Recommended) | ||
| 59 | + | ||
| 60 | +Each time you execute `git commit`, pre-commit automatically runs checks: | ||
| 61 | + | ||
| 62 | +```bash | ||
| 63 | +git add . | ||
| 64 | +git commit -m "your commit message" | ||
| 65 | +``` | ||
| 66 | + | ||
| 67 | +Example output: | ||
| 68 | + | ||
| 69 | +```text | ||
| 70 | +clang-format.............................................................Passed | ||
| 71 | +OAT Compliance Check.....................................................Passed | ||
| 72 | +``` | ||
| 73 | + | ||
| 74 | +### Running Checks Manually | ||
| 75 | + | ||
| 76 | +```bash | ||
| 77 | +# Run all checks | ||
| 78 | +pre-commit run | ||
| 79 | + | ||
| 80 | +# Run a specific type of check | ||
| 81 | +pre-commit run clang-format | ||
| 82 | +pre-commit run oat-check | ||
| 83 | + | ||
| 84 | +# Check all files (not limited to the staging area) | ||
| 85 | +pre-commit run --all-files | ||
| 86 | +``` | ||
| 87 | + | ||
| 88 | +### Skipping Checks (Emergency) | ||
| 89 | + | ||
| 90 | +```bash | ||
| 91 | +git commit --no-verify -m "emergency fix" | ||
| 92 | +``` | ||
| 93 | + | ||
| 94 | +> **Note**: Use this only in emergencies. During normal development, ensure all checks pass. | ||
| 95 | + | ||
| 96 | +## Check Descriptions | ||
| 97 | + | ||
| 98 | +### 1. clang-format | ||
| 99 | + | ||
| 100 | +Automatically formats C and C++ code according to the [.clang-format](../../../.clang-format) configuration in the project root directory. | ||
| 101 | + | ||
| 102 | +### 2. OAT Compliance Check | ||
| 103 | + | ||
| 104 | +OAT (Open Source Audit Tool) checks open source compliance: | ||
| 105 | + | ||
| 106 | +| Check Item | Description | | ||
| 107 | +| -------------- | ------------------------------ | | ||
| 108 | +| License header check | Ensures source files contain the CANN License header | | ||
| 109 | +| Binary file check | Prevents submission of binary files | | ||
| 110 | +| Archive file check | Prevents submission of archive files such as zip and tar | | ||
| 111 | + | ||
| 112 | +The first time the OAT check script runs, it automatically: | ||
| 113 | + | ||
| 114 | +1. Detects or installs Java 17 | ||
| 115 | +2. Detects or installs Maven | ||
| 116 | +3. Clones and compiles the tools_oat tool (approximately 1 to 2 minutes) | ||
| 117 | + | ||
| 118 | +## Frequently Asked Questions | ||
| 119 | + | ||
| 120 | +### Q1: The OAT check is slow during the first commit | ||
| 121 | + | ||
| 122 | +**Cause**: The first run needs to clone and compile the OAT tool. | ||
| 123 | + | ||
| 124 | +**Solution**: This is normal. Subsequent commits use the cached JAR and are much faster. | ||
| 125 | + | ||
| 126 | +## Related Documents | ||
| 127 | + | ||
| 128 | +- [pre-commit Official Documentation](https://pre-commit.com/) | ||
| 129 | +- [clang-format Configuration](https://clang.llvm.org/docs/ClangFormatStyleOptions.html) | ||
| 130 | +- [OAT Tool](https://gitcode.com/openharmony-sig/tools_oat) | ||
| 131 | +- [pre-commit Integration Guide for Code Repositories (Chinese)](https://gitcode.com/cann/infrastructure/blob/main/docs/SC/pre-commit/pre-commit%E9%85%8D%E7%BD%AE%E6%8C%87%E5%AF%BC%E4%B9%A6.md) | ||
| @@ -0,0 +1,64 @@ | |||
| 1 | +# RFC Template | ||
| 2 | + | ||
| 3 | +- Start Date: (Fill in the date in YYYY-MM-DD format) | ||
| 4 | +- RFC PR Number: (Associated PR number) | ||
| 5 | +- Related Issue: (Associated Requirement Issue number) | ||
| 6 | + | ||
| 7 | +--- | ||
| 8 | + | ||
| 9 | +## Summary | ||
| 10 | + | ||
| 11 | +A one-sentence summary of the core content of this RFC. | ||
| 12 | + | ||
| 13 | +## Background and Motivation | ||
| 14 | + | ||
| 15 | +- Why is this feature or modification needed? | ||
| 16 | +- What problem does it solve? | ||
| 17 | +- What are the expected use cases? | ||
| 18 | + | ||
| 19 | +## Detailed Design | ||
| 20 | + | ||
| 21 | +### 1. Overall Architecture | ||
| 22 | + | ||
| 23 | +Describe the overall architecture design, including module division, data flow, and so on. | ||
| 24 | + | ||
| 25 | +### 2. Interface Design | ||
| 26 | + | ||
| 27 | +Describe the new or modified interfaces, configuration parameters, and so on. | ||
| 28 | + | ||
| 29 | +### 3. Data Structures | ||
| 30 | + | ||
| 31 | +Describe the new core data structures. | ||
| 32 | + | ||
| 33 | +### 4. Key Logic | ||
| 34 | + | ||
| 35 | +Describe the key processing logic and algorithms. | ||
| 36 | + | ||
| 37 | +### 5. Compatibility Considerations | ||
| 38 | + | ||
| 39 | +- Does it affect backward compatibility? | ||
| 40 | +- Is a feature switch required? | ||
| 41 | +- What is the phased rollout strategy? | ||
| 42 | + | ||
| 43 | +### 6. Test Plan | ||
| 44 | + | ||
| 45 | +Describe how to verify the correctness of the feature. | ||
| 46 | + | ||
| 47 | +## Risk Assessment | ||
| 48 | + | ||
| 49 | +- Potential risks | ||
| 50 | +- Risk mitigation measures | ||
| 51 | + | ||
| 52 | +## Alternative Solutions | ||
| 53 | + | ||
| 54 | +Describe other solutions considered and their advantages and disadvantages. | ||
| 55 | + | ||
| 56 | +## Open Issues | ||
| 57 | + | ||
| 58 | +Issues that have not been resolved during the design phase or require further discussion. | ||
| 59 | + | ||
| 60 | +--- | ||
| 61 | + | ||
| 62 | +## Review Records | ||
| 63 | + | ||
| 64 | +The review process takes place in the PR comment section. For detailed review feedback, refer to the corresponding PR comments. | ||
| @@ -0,0 +1,38 @@ | |||
| 1 | +# RFC Number Registry | ||
| 2 | + | ||
| 3 | +This file registers all assigned RFC numbers. Before adding a new RFC, claim the **smallest unused number** from this table. | ||
| 4 | + | ||
| 5 | +## Number Reservation Process (Independent of RFC Document Creation) | ||
| 6 | + | ||
| 7 | +1. Check the Allocated Numbers table to find the smallest unused N (typically the last row number plus 1) | ||
| 8 | +2. Append a row for N in this table (fill in the status as `reserved`; the title and author can be placeholders) | ||
| 9 | +3. Submit a **number reservation PR** (containing only the one-line update to this INDEX.md) | ||
| 10 | +4. After the number reservation PR is merged, number N is occupied (status remains `reserved`), and you can start writing the RFC document | ||
| 11 | +5. The RFC document `NNNN-xxx-xxx.md` is submitted in a subsequent independent PR | ||
| 12 | +6. After the RFC document PR is merged, the status changes to `accepted`, and the RFC officially takes effect | ||
| 13 | + | ||
| 14 | +## Status Description | ||
| 15 | + | ||
| 16 | +| Status | Meaning | | ||
| 17 | +|------|------| | ||
| 18 | +| `reserved` | Number is reserved (number reservation PR merged), RFC document pending submission or review | | ||
| 19 | +| `accepted` | RFC document PR merged, RFC officially takes effect | | ||
| 20 | +| `superseded` | Replaced by a subsequent RFC. See the `Superseded by` note at the end of the original document. | | ||
| 21 | + | ||
| 22 | +## Allocated Numbers | ||
| 23 | + | ||
| 24 | +| Number | Title | Author | Status | PR | | ||
| 25 | +|------|------|------|------|-----| | ||
| 26 | + | ||
| 27 | +## Numbering Rules | ||
| 28 | + | ||
| 29 | +- **Format**: 4-digit zero-padded (0001 to 9999) | ||
| 30 | +- **Never reused**: Merged numbers are not recycled even if the RFC is subsequently replaced | ||
| 31 | +- **Sequential allocation**: In principle, do not skip numbers. The next number is the largest used number plus 1 | ||
| 32 | +- **Replacement relationship**: Add `> Superseded by 00NN` at the end of the original RFC document and update the status column in this table to `superseded` | ||
| 33 | +- **Conflict resolution**: If two people claim the same number simultaneously, the latter must rebase and change to the new smallest number | ||
| 34 | + | ||
| 35 | +## Related | ||
| 36 | + | ||
| 37 | +- [RFC Template](./0000-template.md) | ||
| 38 | +- [RFC Process Description](./README.md) | ||
| @@ -0,0 +1,56 @@ | |||
| 1 | +# RFC Document Directory | ||
| 2 | + | ||
| 3 | +This directory contains technical design documents (RFC - Request for Comments) for the HCOMM repository. These documents are used to align on solutions and document design decisions before code implementation. | ||
| 4 | + | ||
| 5 | +## Directory Structure | ||
| 6 | + | ||
| 7 | +- `0000-template.md` - RFC writing template | ||
| 8 | +- `INDEX.md` - RFC number registry, a complete list of all assigned RFC numbers | ||
| 9 | +- `NNNN-xxx-xxx.md` - RFC documents (4-digit number plus a brief description) | ||
| 10 | + | ||
| 11 | +## Naming Convention | ||
| 12 | + | ||
| 13 | +RFC file naming format: `{4-digit-number}-{brief-description}.md` | ||
| 14 | + | ||
| 15 | +For example: `0001-add-new-feature.md` | ||
| 16 | + | ||
| 17 | +- Number: 4-digit zero-padded (0001 to 9999) | ||
| 18 | +- Description: English lowercase, hyphen-separated, concise | ||
| 19 | + | ||
| 20 | +## Numbering Mechanism (Core) | ||
| 21 | + | ||
| 22 | +1. **Number reservation PR** (lightweight): Modify only [INDEX.md](./INDEX.md) to append a reservation row | ||
| 23 | +2. **RFC document PR** (heavyweight): Write the RFC document and submit it for review. The number is already locked through the number reservation PR. | ||
| 24 | + | ||
| 25 | +**Number reservation rules**: | ||
| 26 | + | ||
| 27 | +- Sequential allocation, prioritize the **smallest unused number** | ||
| 28 | +- Never reused. Merged numbers are not recycled even if the RFC is subsequently replaced | ||
| 29 | +- For details, see [INDEX.md](./INDEX.md) | ||
| 30 | + | ||
| 31 | +## RFC Lifecycle | ||
| 32 | + | ||
| 33 | +1. **Requirement phase**: Submit the requirement as a Requirement type Issue and wait for the SIG group to accept it | ||
| 34 | +2. **Number reservation PR**: Append a reservation row in [INDEX.md](./INDEX.md) (status `reserved`) and submit a number reservation PR | ||
| 35 | +3. **Number reservation PR merged**: The number is occupied. You can start writing the RFC. | ||
| 36 | +4. **Writing phase**: Write the system solution following the [RFC template](./0000-template.md) | ||
| 37 | +5. **Review phase**: The RFC document PR is reviewed. Modify the solution based on feedback during the process. | ||
| 38 | +6. **Decision phase**: | ||
| 39 | + - **Merge**: The Maintainer approves the review. Add `/lgtm` and `/approve` to merge. Update the status in INDEX.md to `accepted`. | ||
| 40 | + - **Close**: The review is not approved. Close the PR. The number remains `reserved` and is not recycled (the author can restart the review process). | ||
| 41 | +7. **Implementation phase**: The merged RFC serves as the implementation contract. Code PRs must follow the RFC solution. | ||
| 42 | + | ||
| 43 | +## Replacement Relationship | ||
| 44 | + | ||
| 45 | +When the implementation of an RFC is replaced by a subsequent RFC: | ||
| 46 | + | ||
| 47 | +- Add the following note at the end of the replaced RFC document: `> Superseded by 00NN` | ||
| 48 | +- Update the status of the corresponding row in [INDEX.md](./INDEX.md) from `accepted` to `superseded` | ||
| 49 | +- Do not modify the original number | ||
| 50 | + | ||
| 51 | +## Related Links | ||
| 52 | + | ||
| 53 | +- [Contribution Guide](../../../CONTRIBUTING_en.md) | ||
| 54 | +- [RFC Template](./0000-template.md) | ||
| 55 | +- [RFC Number Registry](./INDEX.md) | ||
| 56 | +- [SIG Meeting](https://etherpad-cann.meeting.osinfra.cn/p/sig-hccl) | ||
| @@ -35,7 +35,7 @@ CCU_DO { | |||
| 35 | 35 | ||
| 36 | ## 参数说明 | 36 | ## 参数说明 |
| 37 | 37 | ||
| 38 | -`CCU_DO`无参数。`CCU_WHILE(condExpr)`的`condExpr`定义与生成方式请参见[CCU_IF](CCU_IF.md#condexpr)。 | 38 | +`CCU_DO`无参数。`CCU_WHILE(condExpr)`的`condExpr`定义与生成方式请参见[CCU_IF](CCU_IF.md#参数说明)。 |
| 39 | 39 | ||
| 40 | ## 返回值 | 40 | ## 返回值 |
| 41 | 41 | ||
| @@ -41,7 +41,7 @@ CCU_DO { | |||
| 41 | | --- | --- | | 41 | | --- | --- | |
| 42 | | condExpr | 条件表达式，类型为`AscendC::ccu::CondExpr`。通过`ccu::Variable`的`operator==(uint64_t)`或`operator!=(uint64_t)`产生。条件在运行期由CCU硬件计算。 | | 42 | | condExpr | 条件表达式，类型为`AscendC::ccu::CondExpr`。通过`ccu::Variable`的`operator==(uint64_t)`或`operator!=(uint64_t)`产生。条件在运行期由CCU硬件计算。 | |
| 43 | 43 | ||
| 44 | -`CondExpr`的定义与生成方式请参见[CCU_IF](CCU_IF.md#condexpr)。 | 44 | +`CondExpr`的定义与生成方式请参见[CCU_IF](CCU_IF.md#参数说明)。 |
| 45 | 45 | ||
| 46 | ## 返回值 | 46 | ## 返回值 |
| 47 | 47 | ||
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
| 2 | -oid sha256:06cd6ddcd38ca91d9b04afcab63d968d3caf92a06a8cd450c28768997f9502ab | 2 | +oid sha256:3f0be9ffc8cce5c88337b00d8bd8b3d3c7a54e91d6d6edbf73c530ba3b4fb5b2 |
| 3 | -size 36127 | 3 | +size 14884 |
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
| 2 | -oid sha256:cd690072097016335d25262acfb54a48923714945dd17a34d16970316f447f42 | 2 | +oid sha256:947c726b269299b039fe06a58694a1ba76cd3bcb95e6bad8eaf7438fac78718c |
| 3 | -size 43184 | 3 | +size 10388 |
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
W | |||
| 2 | -oid sha256:4525180a7117ac154080028e17785019cfb041182878c1df09d1359458f4454e | 2 | +oid sha256:0fec6a304d8df7c1fc15a5530786f3614a8051f5fc29ce9918685c5600862bd1 |
| 3 | -size 206208 | 3 | +size 8577 |
| @@ -23,6 +23,7 @@ RFC 文件命名格式：`{4位编号}-{简短描述}.md` | |||
| 23 | 2. **RFC 文档 PR**（重量）：撰写 RFC 文档并提交评审，编号已通过取号 PR 锁定 | 23 | 2. **RFC 文档 PR**（重量）：撰写 RFC 文档并提交评审，编号已通过取号 PR 锁定 |
| 24 | 24 | ||
| 25 | **取号规则**： | 25 | **取号规则**： |
| 26 | + | ||
| 26 | - 顺序分配，**最小未使用编号**优先 | 27 | - 顺序分配，**最小未使用编号**优先 |
| 27 | - 永不重用，已合入的编号即使 RFC 被后续替代也不回收 | 28 | - 永不重用，已合入的编号即使 RFC 被后续替代也不回收 |
| 28 | - 详见 [INDEX.md](./INDEX.md) | 29 | - 详见 [INDEX.md](./INDEX.md) |
| @@ -47,6 +47,7 @@ | |||
| 47 | ### 关闭验签 | 47 | ### 关闭验签 |
| 48 | 48 | ||
| 49 | 本源码仓编译生成的`cann-hcomm_<version>_linux-<arch>.run`软件包中包含如下tar.gz子包： | 49 | 本源码仓编译生成的`cann-hcomm_<version>_linux-<arch>.run`软件包中包含如下tar.gz子包： |
| 50 | + | ||
| 50 | - `cann-hcomm-compat.tar.gz`: HCOMM兼容升级包。 | 51 | - `cann-hcomm-compat.tar.gz`: HCOMM兼容升级包。 |
| 51 | - `cann-hccd-compat.tar.gz`: DataFlow兼容升级包。 | 52 | - `cann-hccd-compat.tar.gz`: DataFlow兼容升级包。 |
| 52 | - `aicpu_hcomm.tar.gz`: AI CPU通信基础包。 | 53 | - `aicpu_hcomm.tar.gz`: AI CPU通信基础包。 |
| @@ -0,0 +1,98 @@ | |||
| 1 | +# Communicator Management - One NPU Device per Process (Initialize Communicator Based on Root Node Information) | ||
| 2 | + | ||
| 3 | +## Sample Overview | ||
| 4 | + | ||
| 5 | +This sample demonstrates how to use the `HcclCommInitRootInfoConfig()` API to initialize a communicator based on root node information. It covers the following features: | ||
| 6 | + | ||
| 7 | +- Use rank0 as the root node and generate rootinfo identification information through the `HcclGetRootInfo()` API. | ||
| 8 | + | ||
| 9 | + > The rootinfo identification information mainly includes the Device IP, Device ID, and so on. This information must be broadcast to all ranks in the cluster to initialize the communicator. | ||
| 10 | + | ||
| 11 | +- Launch multiple processes through MPI and broadcast the rootinfo identification information to all processes. | ||
| 12 | +- Initialize the communicator in each process based on the rootinfo identification information using the `HcclCommInitRootInfoConfig()` API. | ||
| 13 | +- Call the `HcclAllReduce()` operator and print the results. | ||
| 14 | + | ||
| 15 | +## Directory Structure | ||
| 16 | + | ||
| 17 | +```text | ||
| 18 | +├── main.cc # Sample source file | ||
| 19 | +├── Makefile # Build or configuration file | ||
| 20 | +└── one_device_per_process # Compiled executable | ||
| 21 | +``` | ||
| 22 | + | ||
| 23 | +## Environment Preparation | ||
| 24 | + | ||
| 25 | +### Requirements | ||
| 26 | + | ||
| 27 | +This sample supports the following products with a single-node 2-card cluster topology: | ||
| 28 | + | ||
| 29 | +- <term>Ascend 950PR</term> / <term>Ascend 950DT</term> | ||
| 30 | + | ||
| 31 | +This sample supports the following products with a single-node 8-card cluster topology: | ||
| 32 | + | ||
| 33 | +- <term>Atlas A3 training series products</term> / <term>Atlas A3 inference series products</term> | ||
| 34 | +- <term>Atlas A2 training series products</term> | ||
| 35 | +- <term>Atlas training series products</term> / <term>Atlas inference series products</term> | ||
| 36 | + | ||
| 37 | +### Software Dependencies | ||
| 38 | + | ||
| 39 | +Running this sample requires the CANN ops operator package to be installed. For detailed installation steps, see the "Install the CANN Software Package" section in [Source Code Build](../../../docs/en/build/build.md). | ||
| 40 | + | ||
| 41 | +### Installing MPI | ||
| 42 | + | ||
| 43 | +This sample depends on MPI to launch processes on each Device. Before running this sample, install MPI. For detailed installation steps, see the "MPI Installation and Configuration" section in the corresponding version of the [HCCL Performance Test Tool User Guide][1]. | ||
| 44 | + | ||
| 45 | +[1]: https://hiascend.com/document/redirect/CannCommunityToolHcclTest | ||
| 46 | + | ||
| 47 | +### Disabling Signature Verification | ||
| 48 | + | ||
| 49 | +The `cann-hcomm_<version>_linux-<arch>.run` software package generated from this source repository contains the following tar.gz subpackages: | ||
| 50 | + | ||
| 51 | + - `cann-hcomm-compat.tar.gz`: HCOMM compatibility upgrade package. | ||
| 52 | + - `cann-hccd-compat.tar.gz`: DataFlow compatibility upgrade package. | ||
| 53 | + - `aicpu_hcomm.tar.gz`: AI CPU communication base package. | ||
| 54 | + | ||
| 55 | +These tar.gz packages are loaded to the Device when the service starts. During the loading process, the driver performs security signature verification by default to ensure the package is trusted. Because the tar.gz packages compiled from this source repository do not contain a signature header, the driver security signature verification mechanism needs to be disabled. For the method to disable signature verification, refer to the "Disable Signature Verification" section in [Source Code Build](../../../docs/en/build/build.md). | ||
| 56 | + | ||
| 57 | +### Configuring Environment Variables | ||
| 58 | + | ||
| 59 | +```bash | ||
| 60 | +# Set CANN environment variables, using the root user default installation path as an example | ||
| 61 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 62 | +# Set the MPI installation directory. Adjust it based on the actual situation. | ||
| 63 | +export MPI_HOME=/usr/local/mpich | ||
| 64 | +``` | ||
| 65 | + | ||
| 66 | +## Compiling and Running the Sample | ||
| 67 | + | ||
| 68 | +Execute the following commands in the sample code directory: | ||
| 69 | + | ||
| 70 | +```bash | ||
| 71 | +make | ||
| 72 | +make test N=${RANK_SIZE} | ||
| 73 | +``` | ||
| 74 | + | ||
| 75 | +`RANK_SIZE` is the number of cluster devices. For the <term>Ascend 950PR</term> and <term>Ascend 950DT</term> product series, `RANK_SIZE` is 2. For other product series, it is 8. | ||
| 76 | + | ||
| 77 | +> Note: You can set the `HCCL_OP_EXPANSION_MODE` environment variable to configure the expansion mode of communication operators. For the range supported by different product models, refer to the usage of this environment variable in the [Environment Variable List](https://hiascend.com/document/redirect/CannCommunityEnvRef). | ||
| 78 | +> | ||
| 79 | +> ```bash | ||
| 80 | +> # Set the expansion mode of communication operators to AI CPU communication engine | ||
| 81 | +> export HCCL_OP_EXPANSION_MODE=AI_CPU | ||
| 82 | +> ``` | ||
| 83 | + | ||
| 84 | +## Sample Output | ||
| 85 | + | ||
| 86 | +The data for each rank is initialized to 0 to 7. After the AllReduce operation, the result for each rank is the sum of the data at the corresponding positions across all ranks (the data from 8 ranks is summed). | ||
| 87 | + | ||
| 88 | +```text | ||
| 89 | +Found 8 NPU device(s) available | ||
| 90 | +rankId: 0, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 91 | +rankId: 1, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 92 | +rankId: 2, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 93 | +rankId: 3, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 94 | +rankId: 4, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 95 | +rankId: 5, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 96 | +rankId: 6, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 97 | +rankId: 7, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 98 | +``` | ||
| @@ -43,6 +43,7 @@ | |||
| 43 | ### 关闭验签 | 43 | ### 关闭验签 |
| 44 | 44 | ||
| 45 | 本源码仓编译生成的`cann-hcomm_<version>_linux-<arch>.run`软件包中包含如下tar.gz子包： | 45 | 本源码仓编译生成的`cann-hcomm_<version>_linux-<arch>.run`软件包中包含如下tar.gz子包： |
| 46 | + | ||
| 46 | - `cann-hcomm-compat.tar.gz`: HCOMM兼容升级包。 | 47 | - `cann-hcomm-compat.tar.gz`: HCOMM兼容升级包。 |
| 47 | - `cann-hccd-compat.tar.gz`: DataFlow兼容升级包。 | 48 | - `cann-hccd-compat.tar.gz`: DataFlow兼容升级包。 |
| 48 | - `aicpu_hcomm.tar.gz`: AI CPU通信基础包。 | 49 | - `aicpu_hcomm.tar.gz`: AI CPU通信基础包。 |
| @@ -0,0 +1,94 @@ | |||
| 1 | +# Communicator Management - One NPU Device per Process (Initialize Communicator Based on Rank Table) | ||
| 2 | + | ||
| 3 | +## Sample Overview | ||
| 4 | + | ||
| 5 | +This sample demonstrates how to use the `HcclCommInitClusterInfoConfig()` API to initialize a communicator based on the `rank_table.json` configuration file. It covers the following features: | ||
| 6 | + | ||
| 7 | +- Launch multiple processes through MPI. Each process reads the `rank_table.json` file and initializes the communicator using the `HcclCommInitClusterInfoConfig()` API. | ||
| 8 | +- Call the `HcclAllReduce()` operator and print the results. | ||
| 9 | + | ||
| 10 | +## Directory Structure | ||
| 11 | + | ||
| 12 | +```text | ||
| 13 | +├── main.cc # Sample source file | ||
| 14 | +├── Makefile # Build or configuration file | ||
| 15 | +├── rank_table.json # Cluster information configuration file | ||
| 16 | +└── one_device_per_process_rank_table # Compiled executable | ||
| 17 | +``` | ||
| 18 | + | ||
| 19 | +## Environment Preparation | ||
| 20 | + | ||
| 21 | +### Requirements | ||
| 22 | + | ||
| 23 | +This sample supports the following products with a single-node 2-card cluster topology: | ||
| 24 | + | ||
| 25 | +- <term>Ascend 950PR</term> / <term>Ascend 950DT</term> | ||
| 26 | + | ||
| 27 | +This sample supports the following products with a single-node 8-card cluster topology: | ||
| 28 | + | ||
| 29 | +- <term>Atlas A3 training series products</term> / <term>Atlas A3 inference series products</term> | ||
| 30 | +- <term>Atlas A2 training series products</term> | ||
| 31 | +- <term>Atlas training series products</term> / <term>Atlas inference series products</term> | ||
| 32 | + | ||
| 33 | +### Software Dependencies | ||
| 34 | + | ||
| 35 | +Running this sample requires the CANN ops operator package to be installed. For detailed installation steps, see the "Install the CANN Software Package" section in [Source Code Build](../../../docs/en/build/build.md). | ||
| 36 | + | ||
| 37 | +### Installing MPI | ||
| 38 | + | ||
| 39 | +This sample depends on MPI to launch processes on each Device. Before running this sample, install MPI. For detailed installation steps, see the "MPI Installation and Configuration" section in the corresponding version of the [HCCL Performance Test Tool User Guide][1]. | ||
| 40 | + | ||
| 41 | +[1]: https://hiascend.com/document/redirect/CannCommunityToolHcclTest | ||
| 42 | + | ||
| 43 | +### Disabling Signature Verification | ||
| 44 | + | ||
| 45 | +The `cann-hcomm_<version>_linux-<arch>.run` software package generated from this source repository contains the following tar.gz subpackages: | ||
| 46 | + | ||
| 47 | + - `cann-hcomm-compat.tar.gz`: HCOMM compatibility upgrade package. | ||
| 48 | + - `cann-hccd-compat.tar.gz`: DataFlow compatibility upgrade package. | ||
| 49 | + - `aicpu_hcomm.tar.gz`: AI CPU communication base package. | ||
| 50 | + | ||
| 51 | +These tar.gz packages are loaded to the Device when the service starts. During the loading process, the driver performs security signature verification by default to ensure the package is trusted. Because the tar.gz packages compiled from this source repository do not contain a signature header, the driver security signature verification mechanism needs to be disabled. For the method to disable signature verification, refer to the "Disable Signature Verification" section in [Source Code Build](../../../docs/en/build/build.md). | ||
| 52 | + | ||
| 53 | +### Configuring Environment Variables | ||
| 54 | + | ||
| 55 | +```bash | ||
| 56 | +# Set CANN environment variables, using the root user default installation path as an example | ||
| 57 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 58 | +# Set the MPI installation directory. Adjust it based on the actual situation. | ||
| 59 | +export MPI_HOME=/usr/local/mpich | ||
| 60 | +``` | ||
| 61 | + | ||
| 62 | +## Compiling and Running the Sample | ||
| 63 | + | ||
| 64 | +Execute the following commands in the sample code directory: | ||
| 65 | + | ||
| 66 | +```bash | ||
| 67 | +make | ||
| 68 | +make test N=${RANK_SIZE} | ||
| 69 | +``` | ||
| 70 | + | ||
| 71 | +`RANK_SIZE` is the number of cluster devices. For the <term>Ascend 950PR</term> and <term>Ascend 950DT</term> product series, `RANK_SIZE` is 2. For other product series, it is 8. | ||
| 72 | + | ||
| 73 | +> Note: You can set the `HCCL_OP_EXPANSION_MODE` environment variable to configure the expansion mode of communication operators. For the range supported by different product models, refer to the usage of this environment variable in the [Environment Variable List](https://hiascend.com/document/redirect/CannCommunityEnvRef). | ||
| 74 | +> | ||
| 75 | +> ```bash | ||
| 76 | +> # Set the expansion mode of communication operators to AI CPU communication engine | ||
| 77 | +> export HCCL_OP_EXPANSION_MODE=AI_CPU | ||
| 78 | +> ``` | ||
| 79 | + | ||
| 80 | +## Sample Output | ||
| 81 | + | ||
| 82 | +The data for each rank is initialized to 0 to 7. After the AllReduce operation, the result for each rank is the sum of the data at the corresponding positions across all ranks (the data from 8 ranks is summed). | ||
| 83 | + | ||
| 84 | +```text | ||
| 85 | +Found 8 NPU device(s) available | ||
| 86 | +rankId: 0, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 87 | +rankId: 1, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 88 | +rankId: 2, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 89 | +rankId: 3, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 90 | +rankId: 4, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 91 | +rankId: 5, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 92 | +rankId: 6, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 93 | +rankId: 7, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 94 | +``` | ||
| @@ -38,6 +38,7 @@ | |||
| 38 | ### 关闭验签 | 38 | ### 关闭验签 |
| 39 | 39 | ||
| 40 | 本源码仓编译生成的`cann-hcomm_<version>_linux-<arch>.run`软件包中包含如下tar.gz子包： | 40 | 本源码仓编译生成的`cann-hcomm_<version>_linux-<arch>.run`软件包中包含如下tar.gz子包： |
| 41 | + | ||
| 41 | - `cann-hcomm-compat.tar.gz`: HCOMM兼容升级包。 | 42 | - `cann-hcomm-compat.tar.gz`: HCOMM兼容升级包。 |
| 42 | - `cann-hccd-compat.tar.gz`: DataFlow兼容升级包。 | 43 | - `cann-hccd-compat.tar.gz`: DataFlow兼容升级包。 |
| 43 | - `aicpu_hcomm.tar.gz`: AI CPU通信基础包。 | 44 | - `aicpu_hcomm.tar.gz`: AI CPU通信基础包。 |
| @@ -0,0 +1,85 @@ | |||
| 1 | +# Communicator Management - One NPU Device per Thread | ||
| 2 | + | ||
| 3 | +## Sample Overview | ||
| 4 | + | ||
| 5 | +This sample demonstrates how to use the `HcclCommInitRootInfoConfig()` API to initialize a communicator in a single process, with each thread managing one NPU device. It covers the following features: | ||
| 6 | + | ||
| 7 | +- Device detection: Query the number of available devices using the `aclrtGetDeviceCount()` API. | ||
| 8 | +- Use rank0 as the root node and generate rootinfo identification information through the `HcclGetRootInfo()` API. | ||
| 9 | + | ||
| 10 | + > The rootinfo identification information mainly includes the Device IP, Device ID, and so on. This information must be broadcast to all ranks in the cluster to initialize the communicator. | ||
| 11 | + | ||
| 12 | +- Initialize the communicator in each thread based on the rootinfo identification information using the `HcclCommInitRootInfoConfig()` API. | ||
| 13 | +- Call the `HcclAllReduce()` operator and print the results. | ||
| 14 | + | ||
| 15 | +## Directory Structure | ||
| 16 | + | ||
| 17 | +```text | ||
| 18 | +├── main.cc # Sample source file | ||
| 19 | +├── Makefile # Build or configuration file | ||
| 20 | +└── one_device_per_pthread # Compiled executable | ||
| 21 | +``` | ||
| 22 | + | ||
| 23 | +## Environment Preparation | ||
| 24 | + | ||
| 25 | +### Requirements | ||
| 26 | + | ||
| 27 | +This sample supports the following products: | ||
| 28 | + | ||
| 29 | +- <term>Ascend 950PR</term> / <term>Ascend 950DT</term> | ||
| 30 | +- <term>Atlas A3 training series products</term> / <term>Atlas A3 inference series products</term> | ||
| 31 | +- <term>Atlas A2 training series products</term> | ||
| 32 | +- <term>Atlas training series products</term> / <term>Atlas inference series products</term> | ||
| 33 | + | ||
| 34 | +### Software Dependencies | ||
| 35 | + | ||
| 36 | +Running this sample requires the CANN ops operator package to be installed. For detailed installation steps, see the "Install the CANN Software Package" section in [Source Code Build](../../../docs/en/build/build.md). | ||
| 37 | + | ||
| 38 | +### Disabling Signature Verification | ||
| 39 | + | ||
| 40 | +The `cann-hcomm_<version>_linux-<arch>.run` software package generated from this source repository contains the following tar.gz subpackages: | ||
| 41 | + | ||
| 42 | + - `cann-hcomm-compat.tar.gz`: HCOMM compatibility upgrade package. | ||
| 43 | + - `cann-hccd-compat.tar.gz`: DataFlow compatibility upgrade package. | ||
| 44 | + - `aicpu_hcomm.tar.gz`: AI CPU communication base package. | ||
| 45 | + | ||
| 46 | +These tar.gz packages are loaded to the Device when the service starts. During the loading process, the driver performs security signature verification by default to ensure the package is trusted. Because the tar.gz packages compiled from this source repository do not contain a signature header, the driver security signature verification mechanism needs to be disabled. For the method to disable signature verification, refer to the "Disable Signature Verification" section in [Source Code Build](../../../docs/en/build/build.md). | ||
| 47 | + | ||
| 48 | +### Configuring Environment Variables | ||
| 49 | + | ||
| 50 | +```bash | ||
| 51 | +# Set CANN environment variables, using the root user default installation path as an example | ||
| 52 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 53 | +``` | ||
| 54 | + | ||
| 55 | +## Compiling and Running the Sample | ||
| 56 | + | ||
| 57 | +Execute the following commands in the sample code directory: | ||
| 58 | + | ||
| 59 | +```bash | ||
| 60 | +make | ||
| 61 | +make test | ||
| 62 | +``` | ||
| 63 | + | ||
| 64 | +> Note: You can set the `HCCL_OP_EXPANSION_MODE` environment variable to configure the expansion mode of communication operators. For the range supported by different product models, refer to the usage of this environment variable in the [Environment Variable List](https://hiascend.com/document/redirect/CannCommunityEnvRef). | ||
| 65 | +> | ||
| 66 | +> ```bash | ||
| 67 | +> # Set the expansion mode of communication operators to AI CPU communication engine | ||
| 68 | +> export HCCL_OP_EXPANSION_MODE=AI_CPU | ||
| 69 | +> ``` | ||
| 70 | + | ||
| 71 | +## Sample Output | ||
| 72 | + | ||
| 73 | +The data for each rank is initialized to 0 to 7. After the AllReduce operation, the result for each rank is the sum of the data at the corresponding positions across all ranks (the data from 8 ranks is summed). | ||
| 74 | + | ||
| 75 | +```text | ||
| 76 | +Found 8 NPU device(s) available | ||
| 77 | +rankId: 0, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 78 | +rankId: 1, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 79 | +rankId: 2, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 80 | +rankId: 3, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 81 | +rankId: 4, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 82 | +rankId: 5, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 83 | +rankId: 6, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 84 | +rankId: 7, output: [ 0 8 16 24 32 40 48 56 ] | ||
| 85 | +``` | ||
| @@ -4,6 +4,6 @@ | |||
| 4 | 4 | ||
| 5 | ## 通信域管理 | 5 | ## 通信域管理 |
| 6 | 6 | ||
| 7 | -- [每个进程管理一个 NPU 设备（基于 root 节点信息初始化通信域）](./01_communicators/01_one_device_per_process/) | 7 | +- [每个进程管理一个 NPU 设备（基于 root 节点信息初始化通信域）](./01_communicators/01_one_device_per_process) |
| 8 | -- [每个进程管理一个 NPU 设备（基于 rank table 初始化通信域）](./01_communicators/02_one_device_per_process_rank_table/) | 8 | +- [每个进程管理一个 NPU 设备（基于 rank table 初始化通信域）](./01_communicators/02_one_device_per_process_rank_table) |
| 9 | -- [每个线程管理一个 NPU 设备（基于 root 节点信息初始化通信域）](./01_communicators/03_one_device_per_pthread/) | 9 | +- [每个线程管理一个 NPU 设备（基于 root 节点信息初始化通信域）](./01_communicators/03_one_device_per_pthread) |
| @@ -0,0 +1,9 @@ | |||
| 1 | +# HCCL Code Examples | ||
| 2 | + | ||
| 3 | +This directory provides example code for implementing collective communication using HCCL interfaces in different scenarios. | ||
| 4 | + | ||
| 5 | +## Communicator Management | ||
| 6 | + | ||
| 7 | +- [One NPU Device per Process (Initialize Communicator Based on Root Node Information)](./01_communicators/01_one_device_per_process) | ||
| 8 | +- [One NPU Device per Process (Initialize Communicator Based on Rank Table)](./01_communicators/02_one_device_per_process_rank_table) | ||
| 9 | +- [One NPU Device per Thread (Initialize Communicator Based on Root Node Information)](./01_communicators/03_one_device_per_pthread) | ||
| @@ -2,9 +2,7 @@ | |||
| 2 | 2 | ||
| 3 | ## 功能简介 | 3 | ## 功能简介 |
| 4 | 4 | ||
| 5 | -HCOMM Experimental NIC Plugin 用于在通用服务器场景下扩展 host 侧网卡通信实现。插件以独立 `.so` | 5 | +HCOMM Experimental NIC Plugin 用于在通用服务器场景下扩展 host 侧网卡通信实现。插件以独立`.so`形式构建和部署，HCOMM 在创建 HOST endpoint/channel 时根据 `EndpointDesc.protocol` 查找已注册插件；命中插件协议后走插件实现，未命中时继续走原有内置实现。 |
| 6 | -形式构建和部署，HCOMM 在创建 HOST endpoint/channel 时根据 `EndpointDesc.protocol` 查找已注册插件； | ||
| 7 | -命中插件协议后走插件实现，未命中时继续走原有内置实现。 | ||
| 8 | 6 | ||
| 9 | 当前目录提供两个示例插件： | 7 | 当前目录提供两个示例插件： |
| 10 | 8 | ||
| @@ -29,8 +27,7 @@ bash build.sh --pkg --experimental | |||
| 29 | bash build.sh --pkg --full --experimental | 27 | bash build.sh --pkg --full --experimental |
| 30 | ``` | 28 | ``` |
| 31 | 29 | ||
| 32 | -如果编译环境无法访问网络，可在联网环境下载第三方依赖包后上传到编译环境，并通过 | 30 | +如果编译环境无法访问网络，可在联网环境下载第三方依赖包后上传到编译环境，并通过`--cann_3rd_lib_path` 指定依赖包路径： |
| 33 | -`--cann_3rd_lib_path` 指定依赖包路径： | ||
| 34 | 31 | ||
| 35 | ```bash | 32 | ```bash |
| 36 | bash build.sh --pkg --experimental --cann_3rd_lib_path={your_3rd_party_path} | 33 | bash build.sh --pkg --experimental --cann_3rd_lib_path={your_3rd_party_path} |
| @@ -63,8 +60,7 @@ ${ASCEND_HOME_PATH}/hcomm_plugin/libhcomm_cpu_ub_plugin.so | |||
| 63 | 60 | ||
| 64 | ## 验签说明 | 61 | ## 验签说明 |
| 65 | 62 | ||
| 66 | -host-only 插件使用不涉及 device 包验签；如果编译安装 `--full` 包并运行上板测试，关闭验签步骤参考 | 63 | +host-only 插件使用不涉及 device 包验签；如果编译安装 `--full` 包并运行上板测试，关闭验签步骤参考`docs/zh/build/build.md`。 |
| 67 | -`docs/zh/build/build.md`。 | ||
| 68 | 64 | ||
| 69 | ## 运行启用 | 65 | ## 运行启用 |
| 70 | 66 | ||
| @@ -92,9 +88,7 @@ source ${install_path}/cann/set_env.sh | |||
| 92 | export HCOMM_NIC_PLUGIN_SO=/path/to/libhcomm_cpu_roce_plugin.so:/path/to/libhcomm_cpu_ub_plugin.so | 88 | export HCOMM_NIC_PLUGIN_SO=/path/to/libhcomm_cpu_roce_plugin.so:/path/to/libhcomm_cpu_ub_plugin.so |
| 93 | ``` | 89 | ``` |
| 94 | 90 | ||
| 95 | -业务代码无需直接调用插件接口。HCOMM 在 `HcommEndpointCreate` 创建 HOST endpoint 时，会根据 | 91 | +业务代码无需直接调用插件接口。HCOMM 在 `HcommEndpointCreate` 创建 HOST endpoint 时，会根据`EndpointDesc.protocol` 查找插件；如果协议由插件注册，则后续 endpoint、channel 和数据面接口会分发到插件`ops`；如果没有匹配插件，则保持原有内置路径。 |
| 96 | -`EndpointDesc.protocol` 查找插件；如果协议由插件注册，则后续 endpoint、channel 和数据面接口会分发到插件 | ||
| 97 | -ops；如果没有匹配插件，则保持原有内置路径。 | ||
| 98 | 92 | ||
| 99 | ## 使用条件和限制 | 93 | ## 使用条件和限制 |
| 100 | 94 | ||
| @@ -102,8 +96,7 @@ ops；如果没有匹配插件，则保持原有内置路径。 | |||
| 102 | - 当前示例插件面向通用服务器 host-only 场景。 | 96 | - 当前示例插件面向通用服务器 host-only 场景。 |
| 103 | - 当前加载器在检测到 runtime device 数量非 0 时会跳过插件加载。 | 97 | - 当前加载器在检测到 runtime device 数量非 0 时会跳过插件加载。 |
| 104 | - 多个插件注册同一协议时，后加载的插件会覆盖先加载插件。 | 98 | - 多个插件注册同一协议时，后加载的插件会覆盖先加载插件。 |
| 105 | -- 插件 ABI 需要匹配 `HcommNicPluginInfo`、`HcommNicEndpointOps`、`HcommNicChannelOps` 中定义的 | 99 | +- 插件 ABI 需要匹配 `HcommNicPluginInfo`、`HcommNicEndpointOps`、`HcommNicChannelOps` 中定义的magic word、version和size。 |
| 106 | - magic word、version 和 size。 | ||
| 107 | 100 | ||
| 108 | ## 自定义插件开发入口 | 101 | ## 自定义插件开发入口 |
| 109 | 102 | ||
| @@ -0,0 +1,136 @@ | |||
| 1 | +# HCOMM Experimental NIC Plugin Guide | ||
| 2 | + | ||
| 3 | +## Overview | ||
| 4 | + | ||
| 5 | +The HCOMM Experimental NIC Plugin extends the host-side NIC communication implementation in general server scenarios. The plugin is built and deployed as an independent `.so` file. When creating a HOST endpoint or channel, HCOMM searches for registered plugins based on `EndpointDesc.protocol`. If a matching plugin protocol is found, the plugin implementation is used. Otherwise, the original built-in implementation is used. | ||
| 6 | + | ||
| 7 | +This directory provides two example plugins: | ||
| 8 | + | ||
| 9 | +| Plugin | Artifact | Registered Protocol | | ||
| 10 | +| --- | --- | --- | | ||
| 11 | +| HOST RoCE plugin | `libhcomm_cpu_roce_plugin.so` | `COMM_PROTOCOL_ROCE` | | ||
| 12 | +| HOST UB plugin | `libhcomm_cpu_ub_plugin.so` | `COMM_PROTOCOL_UBC_TP`, `COMM_PROTOCOL_UBC_CTP` | | ||
| 13 | + | ||
| 14 | +## Building and Packaging | ||
| 15 | + | ||
| 16 | +Before compiling, complete the dependency installation, CANN software package installation, and environment variable configuration as described in `docs/en/build/build.md`. | ||
| 17 | + | ||
| 18 | +After navigating to the repository root directory, execute the following command to compile the host package with experimental plugins enabled: | ||
| 19 | + | ||
| 20 | +```bash | ||
| 21 | +bash build.sh --pkg --experimental | ||
| 22 | +``` | ||
| 23 | + | ||
| 24 | +To compile both the host and device packages: | ||
| 25 | + | ||
| 26 | +```bash | ||
| 27 | +bash build.sh --pkg --full --experimental | ||
| 28 | +``` | ||
| 29 | + | ||
| 30 | +If the compilation environment does not have network access, download the third-party dependency packages in a networked environment, upload them to the compilation environment, and specify the dependency package path using `--cann_3rd_lib_path`: | ||
| 31 | + | ||
| 32 | +```bash | ||
| 33 | +bash build.sh --pkg --experimental --cann_3rd_lib_path={your_3rd_party_path} | ||
| 34 | +``` | ||
| 35 | + | ||
| 36 | +After compilation, the HCOMM software package is generated in the `build_out` directory at the repository root: | ||
| 37 | + | ||
| 38 | +```text | ||
| 39 | +./build_out/cann-hcomm_<version>_linux-<arch>.run | ||
| 40 | +``` | ||
| 41 | + | ||
| 42 | +`<version>` is the software version number, and `<arch>` is the system architecture, for example, `x86_64` or `aarch64`. | ||
| 43 | + | ||
| 44 | +## Installation and Deployment | ||
| 45 | + | ||
| 46 | +Install the compiled HCOMM software package: | ||
| 47 | + | ||
| 48 | +```bash | ||
| 49 | +bash ./build_out/cann-hcomm_<version>_linux-<arch>.run --full | ||
| 50 | +``` | ||
| 51 | + | ||
| 52 | +Replace the package name in the command with the actual generated file name. After installation, the experimental plugins are installed in the HCOMM plugin directory: | ||
| 53 | + | ||
| 54 | +```text | ||
| 55 | +${ASCEND_HOME_PATH}/hcomm_plugin/libhcomm_cpu_roce_plugin.so | ||
| 56 | +${ASCEND_HOME_PATH}/hcomm_plugin/libhcomm_cpu_ub_plugin.so | ||
| 57 | +``` | ||
| 58 | + | ||
| 59 | +For manual debugging, you can also copy the plugin `.so` files directly to the `${ASCEND_HOME_PATH}/hcomm_plugin/` directory. | ||
| 60 | + | ||
| 61 | +## Signature Verification Notes | ||
| 62 | + | ||
| 63 | +Host-only plugin usage does not involve device package signature verification. If you compile and install the `--full` package and run on-board tests, refer to `docs/en/build/build.md` for the steps to disable signature verification. | ||
| 64 | + | ||
| 65 | +## Enabling at Runtime | ||
| 66 | + | ||
| 67 | +Before running, load the CANN or HCOMM environment variables. For the default installation path, execute: | ||
| 68 | + | ||
| 69 | +```bash | ||
| 70 | +source /usr/local/Ascend/cann/set_env.sh | ||
| 71 | +``` | ||
| 72 | + | ||
| 73 | +For a specified installation path, execute: | ||
| 74 | + | ||
| 75 | +```bash | ||
| 76 | +source ${install_path}/cann/set_env.sh | ||
| 77 | +``` | ||
| 78 | + | ||
| 79 | +The plugin loading rules are as follows: | ||
| 80 | + | ||
| 81 | +- When `ASCEND_HOME_PATH` is not empty, HCOMM scans `${ASCEND_HOME_PATH}/hcomm_plugin/*.so`. | ||
| 82 | +- When `ASCEND_HOME_PATH` is empty, HCOMM reads the plugin path specified by `HCOMM_NIC_PLUGIN_SO`. | ||
| 83 | +- `HCOMM_NIC_PLUGIN_SO` supports multiple `.so` paths separated by colons. | ||
| 84 | + | ||
| 85 | +Example: | ||
| 86 | + | ||
| 87 | +```bash | ||
| 88 | +export HCOMM_NIC_PLUGIN_SO=/path/to/libhcomm_cpu_roce_plugin.so:/path/to/libhcomm_cpu_ub_plugin.so | ||
| 89 | +``` | ||
| 90 | + | ||
| 91 | +Business code does not need to call plugin interfaces directly. When HCOMM creates a HOST endpoint through `HcommEndpointCreate`, it searches for plugins based on `EndpointDesc.protocol`. If the protocol is registered by a plugin, subsequent endpoint, channel, and data plane interfaces are dispatched to the plugin `ops`. If no matching plugin is found, the original built-in path is used. | ||
| 92 | + | ||
| 93 | +## Usage Conditions and Limitations | ||
| 94 | + | ||
| 95 | +- The plugin only applies to endpoints where `EndpointDesc.loc.locType == ENDPOINT_LOC_TYPE_HOST`. | ||
| 96 | +- The current example plugins target general server host-only scenarios. | ||
| 97 | +- The current loader skips plugin loading when it detects that the number of runtime devices is non-zero. | ||
| 98 | +- When multiple plugins register the same protocol, the later-loaded plugin overrides the earlier one. | ||
| 99 | +- The plugin ABI must match the magic word, version, and size defined in `HcommNicPluginInfo`, `HcommNicEndpointOps`, and `HcommNicChannelOps`. | ||
| 100 | + | ||
| 101 | +## Custom Plugin Development Entry Points | ||
| 102 | + | ||
| 103 | +Custom plugins need to export the following C ABI symbols: | ||
| 104 | + | ||
| 105 | +```cpp | ||
| 106 | +const HcommNicPluginInfo *HcommNicPluginGetInfo(void); | ||
| 107 | +int32_t HcommNicPluginCreateEndpoint( | ||
| 108 | + const EndpointDesc *endpointDesc, void **outCtx, HcommNicEndpointOps **outOps); | ||
| 109 | +int32_t HcommNicPluginCreateChannel( | ||
| 110 | + void *epCtx, const HcommChannelDesc *channelDesc, void **outCtx, HcommNicChannelOps **outOps); | ||
| 111 | +``` | ||
| 112 | + | ||
| 113 | +Development requirements: | ||
| 114 | + | ||
| 115 | +- Declare the plugin name and supported `CommProtocol` in `HcommNicPluginInfo`. | ||
| 116 | +- Fill `HcommNicEndpointOps` for the endpoint, implementing interfaces such as initialization, memory registration, export, import, and destroy. | ||
| 117 | +- Fill `HcommNicChannelOps` for the channel, implementing interfaces such as initialization, status query, read, write, notify, fence, and destroy. | ||
| 118 | +- Ensure the magic word, version, and size in the ABI header match the definitions in `hcomm_nic_plugin.h`. | ||
| 119 | + | ||
| 120 | +For reference, see the following files: | ||
| 121 | + | ||
| 122 | +- `host_roce_plugin.cc`: HOST RoCE plugin entry point. | ||
| 123 | +- `host_ub_plugin.cc`: HOST UB plugin entry point. | ||
| 124 | +- `nic_plugin_ops.h`: Example plugin ops adaptation template. | ||
| 125 | +- `plugin_core.cc`: Plugin common auxiliary logic. | ||
| 126 | + | ||
| 127 | +## Troubleshooting | ||
| 128 | + | ||
| 129 | +If the plugin does not take effect, check the following items in order: | ||
| 130 | + | ||
| 131 | +1. Confirm that `--experimental` was added during the build and that the plugin `.so` is included in the software package. | ||
| 132 | +2. Confirm that the plugin `.so` is installed in `${ASCEND_HOME_PATH}/hcomm_plugin/`. | ||
| 133 | +3. Confirm that `ASCEND_HOME_PATH` or `HCOMM_NIC_PLUGIN_SO` is set according to the current loading rules. | ||
| 134 | +4. Confirm that the business creates a HOST endpoint, that is, `EndpointDesc.loc.locType == ENDPOINT_LOC_TYPE_HOST`. | ||
| 135 | +5. Confirm that `EndpointDesc.protocol` is registered in the plugin `HcommNicPluginInfo.protocols`. | ||
| 136 | +6. Check the runtime log for the `[NicPlugin]` keyword to see whether the plugin was scanned, loaded, and its protocol registered. | ||
| @@ -21,7 +21,7 @@ | |||
| 21 | 21 | ||
| 22 | ## 目录描述 | 22 | ## 目录描述 |
| 23 | 23 | ||
| 24 | -``` | 24 | +```text |
| 25 | cluster_monitor/ | 25 | cluster_monitor/ |
| 26 | ├── CMakeLists.txt # 构建脚本，仅将 cluster_monitor.cc 加入 hcomm 目标 | 26 | ├── CMakeLists.txt # 构建脚本，仅将 cluster_monitor.cc 加入 hcomm 目标 |
| 27 | ├── cluster_monitor.h # 类与数据结构定义（ClusterMonitor、Frame、SockCtx、UID 等） | 27 | ├── cluster_monitor.h # 类与数据结构定义（ClusterMonitor、Frame、SockCtx、UID 等） |
| @@ -373,4 +373,4 @@ ClusterMonitorCallBackInit() { | |||
| 373 | | **错误传播路径** | 异常帧通过Ring链路逐步扩散，传播延迟 ≈ `BROADCAST_INTERVAL` × Ring跳数；非即时同步。 | | 373 | | **错误传播路径** | 异常帧通过Ring链路逐步扩散，传播延迟 ≈ `BROADCAST_INTERVAL` × Ring跳数；非即时同步。 | |
| 374 | | **资源释放** | `DeInit` / `UnRegister` 中均会 `SocketDestroy` 并清空引用映射；多次调用安全（`isDeInit_` 守护、`while(uid2SocketRefMap_.erase(rem)) {}` 自旋）。 | | 374 | | **资源释放** | `DeInit` / `UnRegister` 中均会 `SocketDestroy` 并清空引用映射；多次调用安全（`isDeInit_` 守护、`while(uid2SocketRefMap_.erase(rem)) {}` 自旋）。 | |
| 375 | 375 | ||
| 376 | ---- | 376 | +--- |
| @@ -0,0 +1,376 @@ | |||
| 1 | +# ClusterMonitor Cluster Heartbeat Detection Function Description | ||
| 2 | + | ||
| 3 | +--- | ||
| 4 | + | ||
| 5 | +## Function Description | ||
| 6 | + | ||
| 7 | +`ClusterMonitor` is the cluster-level heartbeat monitoring module of HCCL (Huawei Collective Communication Library) under the `coll_communicator_mgr/dfx` path, mainly used for **continuous detection of reachability and liveness of all participating nodes and abnormal propagation after the establishment of a collective communication domain (Communicator)**. | ||
| 8 | + | ||
| 9 | +Core capabilities include: | ||
| 10 | + | ||
| 11 | +1. **Node UID identification**: Each Rank is encoded as a unique `ClusterUIDType` (up to 2048 bytes) via `netInstId + localId`, used for cross-layer (device / server / pod / superPod) node identification. | ||
| 12 | +2. **Dual Ring link establishment**: Within each `netLayer` plane, nodes are sorted by `localId` or `netInstId` to form a Ring topology; each Rank establishes heartbeat socket connections with the "left" and "right" neighbors on the ring (only one neighbor when there are only 2 nodes). | ||
| 13 | +3. **Async link establishment**: Each remote UID starts an independent thread `CreateLinkWithRemotePonit` for `SocketCreate` + status polling, controlled by `HCCL_CONNECT_TIMEOUT`. | ||
| 14 | +4. **Periodic heartbeat send/receive**: The background `MonitorThread` traverses all sockets at `BROADCAST_INTERVAL` intervals, first `SocketSendNb` to send heartbeats then `SocketRecvNb` to receive heartbeats; `lostNum` accumulates every `HEARTBEAT_COUNT` cycles, and when `HCCL_LOST_THRESHOLD` (30s) is reached, the status is determined as `LOST`. | ||
| 15 | +5. **Abnormal status propagation**: Node `LOST` or `CQE_ERR` status is broadcast to other neighbors via the Ring link (`SetStatus` → `errRankQueue_` → `ProcessExceptionEvent` → `SendFrame`). | ||
| 16 | +6. **Error information reporting**: Through the `GetCqeErrInfoFromTaskException` callback registered via `__attribute__((constructor))`, CQE errors from AICPU/CCU tasks are collected into `cqeErrInfo_`, and triggered via `SetStatus(..., CLUSTER_MONITOR_CQE_ERR, true)` to broadcast; the query interface `GetErrStatusVecFromCluserMonitor` formats error descriptions by priority (CQE_ERR > LOST) and returns them to the upper layer. | ||
| 17 | + | ||
| 18 | +The module belongs to the DFX (Design For X) category and is a key component of HCCL for providing "network disconnection / peer CoreDump" observability at the cluster level. | ||
| 19 | + | ||
| 20 | +--- | ||
| 21 | + | ||
| 22 | +## Directory Description | ||
| 23 | + | ||
| 24 | +```text | ||
| 25 | +cluster_monitor/ | ||
| 26 | +├── CMakeLists.txt # Build script, only adds cluster_monitor.cc to the hcomm target | ||
| 27 | +├── cluster_monitor.h # Class and data structure definitions (ClusterMonitor, Frame, SockCtx, UID, etc.) | ||
| 28 | +└── cluster_monitor.cc # Module implementation, including heartbeat main thread, async link establishment, frame send/receive, exception handling | ||
| 29 | +``` | ||
| 30 | + | ||
| 31 | +**Directory features**: | ||
| 32 | + | ||
| 33 | +- Located in `coll_communicator_mgr/dfx/`, belongs to DFX monitoring functionality; | ||
| 34 | +- Module autonomous: singleton (`GetInstance(u32 deviceId)`), does not depend on other files in the directory; | ||
| 35 | +- External dependencies: `ring_buffer`, `reference_map`, `hcclCommSocket`, `hccl_communicator`, `hcclCommDfx`, `coll_comm`, `log`, `comm_addr_logger`, etc. | ||
| 36 | + | ||
| 37 | +--- | ||
| 38 | + | ||
| 39 | +## Flow Description (Mermaid Sequence Diagram) | ||
| 40 | + | ||
| 41 | +### Overall Registration / Link Establishment / Heartbeat / Deregistration Main Flow | ||
| 42 | + | ||
| 43 | +```mermaid | ||
| 44 | +sequenceDiagram | ||
| 45 | + autonumber | ||
| 46 | + actor Caller as External Caller | ||
| 47 | + participant Mgr as CollCommMgr | ||
| 48 | + participant CM as ClusterMonitor(Singleton) | ||
| 49 | + participant MT as MonitorThread<br/>(Hccl_HeartBeat) | ||
| 50 | + participant LT as LinkThread<br/>(hb<UID>) | ||
| 51 | + participant Sock as Socket Layer | ||
| 52 | + participant Peer as Peer Rank | ||
| 53 | + | ||
| 54 | + %% ===== Registration Phase ===== | ||
| 55 | + rect rgb(230, 245, 255) | ||
| 56 | + Note over Caller, CM: Registration Phase | ||
| 57 | + Caller->>Mgr: RegisterToClusterMonitor(comm) | ||
| 58 | + Mgr->>CM: GetInstance(deviceId) | ||
| 59 | + CM->>CM: GetRemEndpointDescs<br/>(traverse netLayer, collect UID contexts) | ||
| 60 | + CM->>CM: GetConnectRank<br/>(sort + dual Ring) | ||
| 61 | + CM->>CM: clusterLinkContext_[commId].push(...) | ||
| 62 | + alt First registration | ||
| 63 | + CM->>MT: RunMonitorThread() | ||
| 64 | + end | ||
| 65 | + end | ||
| 66 | + | ||
| 67 | + %% ===== Async Link Establishment ===== | ||
| 68 | + rect rgb(255, 248, 230) | ||
| 69 | + Note over MT, Sock: Async Link Establishment Phase | ||
| 70 | + MT->>LT: CreateHBLinksAsync<br/>→ new thread(CreateLinkWithRemotePonit) | ||
| 71 | + LT->>Sock: SocketCreate(socketDesc) | ||
| 72 | + loop Polling until complete / timeout | ||
| 73 | + LT->>Sock: SocketGetStatus | ||
| 74 | + Sock-->>LT: CONNECTING / OK / TIMEOUT | ||
| 75 | + end | ||
| 76 | + LT->>Sock: recvBuffer.Init(frameSize*BASE_NUMBER) | ||
| 77 | + LT->>CM: monitorLinkStatusMap_[rem]=COMPLETED<br/>uid2SocketRefMap_.insert(rem, ctx)<br/>commIdMap_[commId][rem]=true | ||
| 78 | + end | ||
| 79 | + | ||
| 80 | + %% ===== Heartbeat Phase ===== | ||
| 81 | + rect rgb(230, 255, 230) | ||
| 82 | + Note over MT, Peer: Heartbeat Phase(BROADCAST_INTERVAL cycle) | ||
| 83 | + loop Each cycle | ||
| 84 | + MT->>Sock: SocketSendNb(heartbeat frame) | ||
| 85 | + MT->>Sock: SocketRecvNb(parse frame) | ||
| 86 | + Sock-->>MT: compSize / frame | ||
| 87 | + alt Received and not OK | ||
| 88 | + MT->>CM: ParseFrame → SetStatus(crimer, informer, status) | ||
| 89 | + CM->>CM: errRankQueue_.push(crimer) | ||
| 90 | + end | ||
| 91 | + alt lostNum >= threshold | ||
| 92 | + MT->>CM: SetStatus(rem, myRank, LOST) | ||
| 93 | + end | ||
| 94 | + MT->>CM: ProcessExceptionEvent() | ||
| 95 | + CM->>Peer: SendFrame(broadcast to neighbors except informer) | ||
| 96 | + end | ||
| 97 | + end | ||
| 98 | + | ||
| 99 | + %% ===== Deregistration / Destruction ===== | ||
| 100 | + rect rgb(255, 230, 230) | ||
| 101 | + Note over Caller, CM: Deregistration / Destruction Phase | ||
| 102 | + Caller->>CM: UnRegisterToClusterMonitor(collComm) | ||
| 103 | + CM->>CM: clusterLinkContext_.erase(commId) | ||
| 104 | + CM->>Sock: SocketDestroy / erase refMap | ||
| 105 | + alt commIdMap_.empty() | ||
| 106 | + CM->>MT: DeInit() → clusterMonitorThreadFlag_=false<br/>→ join monitor thread | ||
| 107 | + end | ||
| 108 | + end | ||
| 109 | +``` | ||
| 110 | + | ||
| 111 | +### CQE Exception Reporting and Broadcast Sequence | ||
| 112 | + | ||
| 113 | +```mermaid | ||
| 114 | +sequenceDiagram | ||
| 115 | + autonumber | ||
| 116 | + participant Cb as Task Exception Callback<br/>(AICPU/CCU) | ||
| 117 | + participant Init as ClusterMonitorCallBackInit | ||
| 118 | + participant CM as ClusterMonitor | ||
| 119 | + participant MT as MonitorThread | ||
| 120 | + participant Peer as Ring Neighbor | ||
| 121 | + | ||
| 122 | + rect rgb(240, 240, 255) | ||
| 123 | + Note over Init: At module load time(__attribute__((constructor))) | ||
| 124 | + Init->>Init: Register GetCqeErrInfoFromTaskException<br/>and GetErrStatusVecFromCluserMonitor callbacks | ||
| 125 | + end | ||
| 126 | + | ||
| 127 | + rect rgb(255, 245, 230) | ||
| 128 | + Note over Cb, CM: CQE Exception Entry | ||
| 129 | + Cb->>CM: GetCqeErrInfoFromTaskException(remoteLocalId, status, eid, insId) | ||
| 130 | + CM->>CM: Assign cqeErrInfo_ | ||
| 131 | + CM->>CM: SetStatus(myUID, remoteUID, CQE_ERR, true) | ||
| 132 | + CM->>CM: errRankQueue_.push(myUID)<br/>errStatusQueue_.push(frame) | ||
| 133 | + end | ||
| 134 | + | ||
| 135 | + rect rgb(230, 255, 230) | ||
| 136 | + Note over MT, Peer: MonitorThread Periodic Broadcast | ||
| 137 | + MT->>CM: ProcessExceptionEvent() | ||
| 138 | + loop errRankQueue_ is not empty | ||
| 139 | + CM->>Peer: SendFrame(rem, crimer=myUID,<br/>informer=remoteUID, CQE_ERR) | ||
| 140 | + end | ||
| 141 | + end | ||
| 142 | + | ||
| 143 | + rect rgb(245, 245, 245) | ||
| 144 | + Note over CM: Upper Layer Error Query | ||
| 145 | + Note right of CM: GetErrStatusVecFromCluserMonitor()<br/>Formats string by CQE_ERR > LOST priority | ||
| 146 | + end | ||
| 147 | +``` | ||
| 148 | + | ||
| 149 | +--- | ||
| 150 | + | ||
| 151 | +## Data Description (Mermaid Class Diagram) | ||
| 152 | + | ||
| 153 | +```mermaid | ||
| 154 | +classDiagram | ||
| 155 | + direction LR | ||
| 156 | + | ||
| 157 | + %% ===== Enums and Helpers ===== | ||
| 158 | + class ClusterMonitorStatus { | ||
| 159 | + <<enum>> | ||
| 160 | + CLUSTER_MONITOR_OK | ||
| 161 | + CLUSTER_MONITOR_LOST | ||
| 162 | + CLUSTER_MONITOR_NOTIFY | ||
| 163 | + CLUSTER_MONITOR_CQE_ERR | ||
| 164 | + CLUSTER_MONITOR_OPRETRY_NOT_SUPPORT | ||
| 165 | + CLUSTER_MONITOR_STUCK | ||
| 166 | + CLUSTER_MONITOR_INCONSISTENT | ||
| 167 | + } | ||
| 168 | + | ||
| 169 | + class MonitorLinkStatus { | ||
| 170 | + <<enum>> | ||
| 171 | + MONITOR_LINK_NOT_START | ||
| 172 | + MONITOR_LINK_BUILDING | ||
| 173 | + MONITOR_LINK_COMPLETED | ||
| 174 | + } | ||
| 175 | + | ||
| 176 | + %% ===== UID Identification ===== | ||
| 177 | + class HcclClusterMonitorUID { | ||
| 178 | + +char id[2048] | ||
| 179 | + +operator==() | ||
| 180 | + +operator!=() | ||
| 181 | + +operator<() | ||
| 182 | + } | ||
| 183 | + | ||
| 184 | + class ClusterUIDCxt { | ||
| 185 | + +string netInstId | ||
| 186 | + +uint32_t localId | ||
| 187 | + } | ||
| 188 | + | ||
| 189 | + class UIDContext { | ||
| 190 | + +ClusterUIDType uid | ||
| 191 | + +uint32_t netLayer | ||
| 192 | + +uint32_t rankId | ||
| 193 | + +uint32_t localId | ||
| 194 | + +string netInstId | ||
| 195 | + } | ||
| 196 | + | ||
| 197 | + %% ===== Frame and Socket ===== | ||
| 198 | + class ClusterMonitorFrame { | ||
| 199 | + +ClusterUIDType src | ||
| 200 | + +ClusterUIDType dst | ||
| 201 | + +ClusterUIDType crimer | ||
| 202 | + +ClusterUIDType informer | ||
| 203 | + +ClusterMonitorStatus status | ||
| 204 | + +HcclUs TOARelative | ||
| 205 | + +HcclSystemTime TOASystem | ||
| 206 | + +char reserved[256] | ||
| 207 | + } | ||
| 208 | + | ||
| 209 | + class ClusterMonitorSocketCtx { | ||
| 210 | + +SocketDesc socketDesc | ||
| 211 | + +SocketHandle socketHandler | ||
| 212 | + +queue~ClusterMonitorFrame~ sendBuffer | ||
| 213 | + +u32 restSize | ||
| 214 | + +RingBuffer recvBuffer | ||
| 215 | + +u32 lostNum | ||
| 216 | + +bool newConn | ||
| 217 | + } | ||
| 218 | + | ||
| 219 | + class ErrorCqeInfo { | ||
| 220 | + +u32 cqeLocalId | ||
| 221 | + +u32 cqeRemoteLocalId | ||
| 222 | + +uint16_t cqeStatus | ||
| 223 | + +string cqeLocalEid | ||
| 224 | + +string cqeRemoteEid | ||
| 225 | + +string cqeRemoteInsId | ||
| 226 | + +string cqeLocalInsId | ||
| 227 | + } | ||
| 228 | + | ||
| 229 | + %% ===== FrameStatus(nested) ===== | ||
| 230 | + class FrameStatus { | ||
| 231 | + +ClusterMonitorStatus status | ||
| 232 | + +ClusterUIDType informer | ||
| 233 | + +bool needBroadcast | ||
| 234 | + } | ||
| 235 | + | ||
| 236 | + %% ===== ClusterMonitor Class ===== | ||
| 237 | + class ClusterMonitor { | ||
| 238 | + +RegisterToClusterMonitor(comm) | ||
| 239 | + +UnRegisterToClusterMonitor(collComm) | ||
| 240 | + +FormatUID(ClusterUIDCxt) | ||
| 241 | + +GetUID(const ClusterUIDType&) | ||
| 242 | + +FormatConnTag(role, uidPair) | ||
| 243 | + +InsertClusterMonitorCxt(comm, remoteCtx, needConnectRank) | ||
| 244 | + +GetSamePlaneRank(comm, singlePlaneCtx, needConnectRank) | ||
| 245 | + +GetConnectRank(comm, needConnectRank, uidCtxs, netLayersVector) | ||
| 246 | + +CreateHBLinksAsync() | ||
| 247 | + +SetStatus(crimer, informer, status, needBroadcast) | ||
| 248 | + +MonitorThread() | ||
| 249 | + +RunMonitorThread() | ||
| 250 | + +SendFrame(dst, crimer, informer, status) | ||
| 251 | + +RecvFrame(rem) | ||
| 252 | + +ParseFrame(cmFrame, src) | ||
| 253 | + +DelErrorSocket() | ||
| 254 | + +ProcessExceptionEvent() | ||
| 255 | + +DeInit() | ||
| 256 | + +GetInstance(deviceId)$ static | ||
| 257 | + +GetCqeErrInfoFromTaskException(remoteLocalId, status, localEid, remoteEid, remoteInsId) | ||
| 258 | + +GetErrStatusVecFromCluserMonitor() | ||
| 259 | + +PrintEvents(keyEvents) | ||
| 260 | + +MakeErrMsg(keyEvents, errStatusVec) | ||
| 261 | + -GetRemEndpointDescs(comm, uidCtxs, netLayersVector) | ||
| 262 | + -GetRemEndpointDescsPerLayer(netLayer, comm, rankGraph, collComm, uidCtxs, rankIdsSet) | ||
| 263 | + -CreateTransportHandle(info) | ||
| 264 | + -CreateLinkWithRemotePonit(group, rem, ctx) | ||
| 265 | + -uid2SocketRefMap_ ReferenceMap~ClusterUIDType,ClusterMonitorSocketCtx~ | ||
| 266 | + -uid2FrameStatusMap_ ReferenceMap~ClusterUIDType,FrameStatus~ | ||
| 267 | + -monitorLinkStatusMap_ map~ClusterUIDType,MonitorLinkStatus~ | ||
| 268 | + -commIdMap_ map~string, map~ClusterUIDType,bool~~ | ||
| 269 | + -clusterLinkContext_ map~string, queue~pair~ | ||
| 270 | + -linkThreadMap_ map~ClusterUIDType, thread~ | ||
| 271 | + -errRankQueue_ queue~ClusterUIDType~ | ||
| 272 | + -errStatusQueue_ queue~ClusterMonitorFrame~ | ||
| 273 | + -cqeErrInfo_ ErrorCqeInfo | ||
| 274 | + } | ||
| 275 | + | ||
| 276 | + %% ===== Relationships ===== | ||
| 277 | + ClusterMonitorFrame --> HcclClusterMonitorUID : src/dst/crimer/informer | ||
| 278 | + ClusterMonitorFrame --> ClusterMonitorStatus : status | ||
| 279 | + ClusterMonitorSocketCtx --> ClusterMonitorFrame : sendBuffer | ||
| 280 | + UIDContext --> HcclClusterMonitorUID : uid | ||
| 281 | + HcclClusterMonitorUID ..> ClusterUIDCxt : constructed from | ||
| 282 | + FrameStatus --> ClusterMonitorStatus : status | ||
| 283 | + | ||
| 284 | + ClusterMonitor "1" o-- "*" FrameStatus : uid2FrameStatusMap_ | ||
| 285 | + ClusterMonitor "1" o-- "*" ClusterMonitorSocketCtx : uid2SocketRefMap_ | ||
| 286 | + ClusterMonitor "1" o-- "*" MonitorLinkStatus : monitorLinkStatusMap_ | ||
| 287 | + ClusterMonitor "1" o-- "*" HcclClusterMonitorUID : commIdMap_/clusterLinkContext_/linkThreadMap_ | ||
| 288 | + ClusterMonitor "1" o-- "*" ClusterMonitorFrame : errStatusQueue_ | ||
| 289 | + ClusterMonitor "1" *-- "1" ErrorCqeInfo : cqeErrInfo_ | ||
| 290 | + ClusterMonitor ..> MonitorLinkStatus : nested enum | ||
| 291 | + ClusterMonitor ..> FrameStatus : nested struct | ||
| 292 | +``` | ||
| 293 | + | ||
| 294 | +--- | ||
| 295 | + | ||
| 296 | +## Interface Description | ||
| 297 | + | ||
| 298 | +### Public API (External) | ||
| 299 | + | ||
| 300 | +| Interface | Description | | ||
| 301 | +|------|------| | ||
| 302 | +| `static ClusterMonitor& GetInstance(u32 deviceId)` | Get module singleton by device (held indirectly via `CollCommMgr`). | | ||
| 303 | +| `HcclResult RegisterToClusterMonitor(HcclComm comm)` | Register a communicator: build UID context, calculate Ring connection set, push into `clusterLinkContext_` waiting for background link establishment; first registration starts `MonitorThread`. | | ||
| 304 | +| `HcclResult UnRegisterToClusterMonitor(hccl::CollComm* collComm)` | Deregister a communicator: clear reference counts for that commId in `clusterLinkContext_`, `commIdMap_`, `monitorLinkStatusMap_`, `uid2SocketRefMap_`; triggers `DeInit` when the last commId is deregistered. | | ||
| 305 | +| `void GetCqeErrInfoFromTaskException(u32 remoteLocalId, uint16_t status, std::string localEid, std::string remoteEid, std::string remoteInsId)` | Called by the AICPU/CCU CQE error callback, records CQE errors and propagates them via broadcast. | | ||
| 306 | +| `std::vector<std::string> GetErrStatusVecFromCluserMonitor()` | Drain `errStatusQueue_`, format error descriptions by priority (CQE_ERR > LOST), for upper layer queries. | | ||
| 307 | +| `HcclResult RunMonitorThread()` | Explicitly start the background heartbeat thread (`MonitorThread`). | | ||
| 308 | +| `HcclResult DeInit()` | Stop the heartbeat thread, destroy sockets, clean up all internal containers; idempotent. | | ||
| 309 | +| `void SetStatus(crimer, informer, status, needBroadcast=true)` | Set / update a node's status, push into `errRankQueue_` if broadcast is needed. | | ||
| 310 | +| `ClusterUIDType FormatUID(ClusterUIDCxt cxt)` | Assemble UID using `netInstId/localId`. | | ||
| 311 | +| `std::string FormatConnTag(role, uidPair)` | Generate a socket tag in the format `HeartBeat_<src>_to_<dst>`. | | ||
| 312 | + | ||
| 313 | +### Internal / Private Interfaces | ||
| 314 | + | ||
| 315 | +| Interface | Description | | ||
| 316 | +|------|------| | ||
| 317 | +| `GetRemEndpointDescs / GetRemEndpointDescsPerLayer` | Enumerate all ranks in each `netLayer` from `RankGraph`, generate `UIDContext`, initialize `uid2FrameStatusMap_`, `commIdMap_`. | | ||
| 318 | +| `InsertClusterMonitorCxt` | Given a peer UID, query RankGraph to obtain link and devicePort, decide SERVER/CLIENT role and construct `SocketDesc`. | | ||
| 319 | +| `GetSamePlaneRank` | Select left and right neighbors within the same plane according to Ring topology (only one neighbor when size==2). | | ||
| 320 | +| `GetConnectRank` | Merge netLayer=0 plane (sorted by `localId`) with `>=1` plane (sorted by `netInstId`), form rings respectively. | | ||
| 321 | +| `CreateHBLinksAsync` | Background thread entry, traverse `clusterLinkContext_` to start / restart independent link establishment threads for each remUID. | | ||
| 322 | +| `CreateLinkWithRemotePonit` | Link establishment thread entry for a single remUID: `SocketCreate` → poll `SocketGetStatus` → initialize `recvBuffer` → register into `uid2SocketRefMap_`. | | ||
| 323 | +| `CreateTransportHandle` | Wrap `SocketCreate` to avoid duplicate creation. | | ||
| 324 | +| `SendFrame` / `RecvFrame` / `ParseFrame` | Non-blocking heartbeat frame send (with partial send resume), non-blocking receive (with ring buffer), validity check and status update. | | ||
| 325 | +| `MonitorThread` | Background main loop: `CreateHBLinksAsync` → send heartbeat + accumulate `lostNum` every `HEARTBEAT_COUNT` cycles → receive heartbeat → handle `lostNum` threshold exceeded → `ProcessExceptionEvent`. | | ||
| 326 | +| `DelErrorSocket` | Destroy sockets marked as abnormal in `errorSocket_`. | | ||
| 327 | +| `ProcessExceptionEvent` | Consume `errRankQueue_`, broadcast abnormal frames to all neighbors where "rem != informer and status == OK". | | ||
| 328 | +| `PrintEvents / MakeErrMsg` | Format `ClusterMonitorFrame` queue into readable string vectors. | | ||
| 329 | + | ||
| 330 | +### Cross-module Callback Registration (`__attribute__((constructor))`) | ||
| 331 | + | ||
| 332 | +```cpp | ||
| 333 | +ClusterMonitorCallBackInit() { | ||
| 334 | + RegisterGetAicpuCqeErrInfoCallBackHcomm(GetCqeErrInfoFromTaskException); | ||
| 335 | + RegisterGetCcuCqeErrInfoCallBackHcomm(GetCqeErrInfoFromTaskException); | ||
| 336 | + RegisterAicpuGetErrStatusVecCallBack(GetErrStatusVecFromCluserMonitor); | ||
| 337 | + RegisterCcuGetErrStatusVecCallBack(GetErrStatusVecFromCluserMonitor); | ||
| 338 | +} | ||
| 339 | +``` | ||
| 340 | + | ||
| 341 | +The module registers two callbacks with the AICPU/CCU framework at load time: **Exception Entry** (error reporting) and **Error Query** (error formatting export), which are the only coupling points between the module and the upper-layer Task exception system. | ||
| 342 | + | ||
| 343 | +--- | ||
| 344 | + | ||
| 345 | +## Usage Limitations (Supported Scenarios and Constraint Specifications) | ||
| 346 | + | ||
| 347 | +### Supported Scenarios | ||
| 348 | + | ||
| 349 | +1. **Multi-Rank communicator**: rankSize >= 2 communicators; when rankSize == 1, `RegisterToClusterMonitor` directly returns `HCCL_SUCCESS` with a `WARNING` log and no link establishment. | ||
| 350 | +2. **Multi-plane topology**: Supports dual Ring link establishment for `netLayer = 0` (within same server/device, sorted by `localId`) and `netLayer >= 1` (cross-server/pod/superPod, sorted by `netInstId`). | ||
| 351 | +3. **Cross-communicator shared socket**: Via `hccl::ReferenceMap` counting, only one socket is created when multiple communicators connect to the same remote node; reference count `--` on deregistration, truly destroyed only when it reaches zero. | ||
| 352 | +4. **Ring fault propagation**: Single node `LOST` / `CQE_ERR` status can spread to other nodes along the Ring link, enabling querying cluster-wide exceptions from any node. | ||
| 353 | +5. **CQE error capture**: Receives `Task` exceptions via AICPU/CCU callbacks, formatted as readable logs with local / remote `instanceId / localId / Eid`. | ||
| 354 | +6. **Toggle switch**: The environment variable `HCCL_DFS_CONFIG.cluster_heartbeat` can disable heartbeat registration / CQE error capture chain. | ||
| 355 | +7. **HCCL v2 communicator**: Depends on `HcclCommunicator::GetRankGraphV2` and `RankGraph`, only effective when `CommunicatorV2` exists; otherwise fails with `CHK_PTR_NULL` in `GetRemEndpointDescs`. | ||
| 356 | + | ||
| 357 | +### Constraint Specifications | ||
| 358 | + | ||
| 359 | +| Category | Constraint | | ||
| 360 | +|------|------| | ||
| 361 | +| **Thread model** | 1 `MonitorThread` (Hccl_HeartBeat) + N `LinkThread`s (hb<UID>); `threadLock_` protects `commIdMap_ / uid2SocketRefMap_ / uid2FrameStatusMap_ / monitorLinkStatusMap_ / errRankQueue_ / errStatusQueue_`; `clusertMonitorLinkMtx_` protects `clusterLinkContext_`. | | ||
| 362 | +| **Lifecycle** | Module singleton is held by `CollCommMgr`; `DeInit` is triggered by the last commId deregistration or `~ClusterMonitor`, idempotent (guarded by `isDeInit_`). | | ||
| 363 | +| **Timeout control** | Link establishment timeout taken from `EnvConfig::GetSocketConfig().GetLinkTimeOut()` (i.e., `HCCL_CONNECT_TIMEOUT`); heartbeat loss threshold `lostThreshold_ = HCCL_LOST_THRESHOLD` (30s). | | ||
| 364 | +| **Port validity** | `devicePort` / `rmtPort` must be `<= Hccl::MAX_VALUE_TCPPORT`, otherwise returns `HCCL_E_PARA`. | | ||
| 365 | +| **Frame size** | `ClusterMonitorFrame` contains 4 UIDs of 2048 bytes each + status + dual timestamps + 256 bytes reserved, fixed total length (`sizeof(ClusterMonitorFrame)`); `recvBuffer.Init` capacity is `BASE_NUMBER * frameSize` (approximately 2x). | | ||
| 366 | +| **UID length** | `HcclClusterMonitorUID.id` is fixed at 2048 bytes, requiring `netInstId + "/" + localId` to not exceed 2048 bytes. | | ||
| 367 | +| **Role decision** | SERVER/CLIENT is determined by `localIpAddr < remoteIpAddr`; when local is SERVER, fill local `listenPort`, otherwise fill peer `rmtPort`, must be consistent with SocketConfig's listener strategy. | | ||
| 368 | +| **Broadcast strategy** | `ProcessExceptionEvent` only broadcasts to neighbors where "`rem != informer` and `status == OK`", avoiding loop storms; non-OK neighbors are already self-aware. | | ||
| 369 | +| **Status priority** | Error query order: CQE_ERR > LOST (call order within `PrintEvents`). | | ||
| 370 | +| **Device scope** | Singleton maintained per device; each device has independent heartbeats in multi-device scenarios without mutual interference. | | ||
| 371 | +| **Platform dependencies** | Depends on `HcclCommunicator` (v2), `RankGraph`, and Socket abstraction layer (`SocketCreate/SendNb/RecvNb/GetStatus/Destroy`); v1 communicator path is not supported. | | ||
| 372 | +| **Environment switch** | When `clusterHeartBeatEnable = false`, no new sockets are created during registration (commIdMap_ markers are retained), CQE error callbacks directly `return`. | | ||
| 373 | +| **Error propagation path** | Abnormal frames spread gradually through the Ring link; propagation delay ≈ `BROADCAST_INTERVAL` × Ring hops; not instantaneous synchronization. | | ||
| 374 | +| **Resource release** | `DeInit` / `UnRegister` both perform `SocketDestroy` and clear reference mappings; multiple calls are safe (guarded by `isDeInit_`, `while(uid2SocketRefMap_.erase(rem)) {}` spin). | | ||
| 375 | + | ||
| 376 | +--- | ||
| @@ -0,0 +1,523 @@ | |||
| 1 | +# Profiling Module Code Analysis | ||
| 2 | + | ||
| 3 | +## Feature Description | ||
| 4 | + | ||
| 5 | +The Profiling module is responsible for collecting and reporting performance data of HCCL collective communication tasks. It is a core component of the HCCL DFX (Diagnostics and Observability) system. The module is deployed on both the Host side and the Device side, providing unified profiling capabilities. | ||
| 6 | + | ||
| 7 | +Core capabilities: | ||
| 8 | + | ||
| 9 | +1. **Report communication tasks**: Report execution information of collective communication tasks (AllReduce, Broadcast, and so on) to the Profiling framework. | ||
| 10 | +2. **Report operator information**: Report key metrics such as start and end times and operator types for communication operators (Host side only). | ||
| 11 | +3. **Report MC2 communication information**: Report Stream, Rank, and other metadata of the MC2 communication domain (Host side only). | ||
| 12 | +4. **Report Kernel**: Report AICPU or AIV Kernel execution timeline information. The Device side reports kernel start and end task events. | ||
| 13 | +5. **Update Profiling status**: Update Profiling statistics based on task queue consumption progress. | ||
| 14 | +6. **Manage Profiling switches**: Respond to subscription and unsubscription commands from the Profiling framework to control data collection start and stop. | ||
| 15 | + | ||
| 16 | +### External Interfaces | ||
| 17 | + | ||
| 18 | +| Header File | Interface | Side | Description | | ||
| 19 | +|--------|------|------|----------| | ||
| 20 | +| `hccl_diag.h` | `HcclDfxRegOpInfoByCommId` | Host + Device | Registers operator information with the communication domain, records `beginTime`, and stores it in `MirrorTaskManager`. | | ||
| 21 | +| `hccl_diag.h` | `HcclProfilingReportOp` | Host | Reports operator execution events: first `ReportAllTasks`, then `ReportOp`. | | ||
| 22 | +| `hccl_diag.h` | `HcclReportAicpuKernel` | Host | Reports AICPU Kernel execution events, records taskId and streamId, and adds task information. | | ||
| 23 | +| `hccl_diag.h` | `HcclReportAivKernel` | Host | Reports AIV Kernel execution events, records taskId and streamId, and adds task information. | | ||
| 24 | +| `hccl_diag.h` | `HcommGetProfilingSysCycleTime` | Host | Obtains the Profiling system cycle time. | | ||
| 25 | +| `hcomm_diag.h` | `HcommProfilingReportDeviceOp` | Device | Reports Device operator execution events: first `ReportAllTasks`, then reports OP information through `ProfilingHandlerLite`. | | ||
| 26 | +| `hcomm_diag.h` | `HcommProfilingReportKernelStartTask` | Device | Reports kernel start task events, reports HEAD type `FlagTaskInfo` through `ProfilingHandlerLite`. | | ||
| 27 | +| `hcomm_diag.h` | `HcommProfilingReportKernelEndTask` | Device | Reports kernel end task events, reports TAIL type `FlagTaskInfo` through `ProfilingHandlerLite`. | | ||
| 28 | + | ||
| 29 | +## Directory Description | ||
| 30 | + | ||
| 31 | +```text | ||
| 32 | +profiling/ | ||
| 33 | +├── CMakeLists.txt # Top-level build, includes aicpu and host subdirectories | ||
| 34 | +├── host/ | ||
| 35 | +│ ├── CMakeLists.txt # Host-side build, compiles hcclCommProfiling.cc | ||
| 36 | +│ ├── hcclCommProfiling.h # Host-side Profiling facade class definition | ||
| 37 | +│ └── hcclCommProfiling.cc # Host-side Profiling facade class implementation | ||
| 38 | +└── aicpu/ | ||
| 39 | + ├── CMakeLists.txt # AICPU-side build, compiles hcclCommProfilingLite.cc | ||
| 40 | + ├── hcclCommProfilingLite.h # AICPU-side Profiling facade class definition | ||
| 41 | + ├── hcclCommProfilingLite.cc # AICPU-side Profiling facade class implementation | ||
| 42 | + ├── aicpu_ts_urma_dfx_kernel.h # [Deprecated] URMA DFX Kernel | ||
| 43 | + └── aicpu_ts_urma_dfx_kernel.cc # [Deprecated] URMA DFX Kernel | ||
| 44 | +``` | ||
| 45 | + | ||
| 46 | +### File Relationships | ||
| 47 | + | ||
| 48 | +| File | Function | Dependencies | | ||
| 49 | +|------|------|----------| | ||
| 50 | +| `host/hcclCommProfiling.h` | Host-side Profiling facade class declaration | Depends on `MirrorTaskManager`, `ProfilingReporter` | | ||
| 51 | +| `host/hcclCommProfiling.cc` | Host-side Profiling facade class implementation | Depends on `profiling_reporter.h`, `profiling_handler.h`, `dlprof_function.h` | | ||
| 52 | +| `aicpu/hcclCommProfilingLite.h` | AICPU-side Profiling facade class declaration | Depends on `MirrorTaskManagerLite`, `ProfilingReporterLite` | | ||
| 53 | +| `aicpu/hcclCommProfilingLite.cc` | AICPU-side Profiling facade class implementation | Depends on `profiling_reporter_lite.h`, `mirror_task_manager_lite.h` | | ||
| 54 | + | ||
| 55 | +### Profiling File Interaction | ||
| 56 | + | ||
| 57 | +```mermaid | ||
| 58 | +graph TB | ||
| 59 | + subgraph External Callers | ||
| 60 | + HCCL[hccl] | ||
| 61 | + end | ||
| 62 | + | ||
| 63 | + subgraph Host Side | ||
| 64 | + HCD[HcclCommDfx] | ||
| 65 | + HCP[HcclCommProfiling] | ||
| 66 | + PR[ProfilingReporter] | ||
| 67 | + PH[ProfilingHandler] | ||
| 68 | + MTM[MirrorTaskManager] | ||
| 69 | + end | ||
| 70 | + | ||
| 71 | + subgraph Device Side | ||
| 72 | + AIP[AicpuIndopProcess] | ||
| 73 | + HCDL[HcclCommDfxLite] | ||
| 74 | + HCPL[HcclCommProfilingLite] | ||
| 75 | + PRL[ProfilingReporterLite] | ||
| 76 | + PHL[ProfilingHandlerLite] | ||
| 77 | + MTML[MirrorTaskManagerLite] | ||
| 78 | + end | ||
| 79 | + | ||
| 80 | + subgraph External Components | ||
| 81 | + Profapi[profapi component] | ||
| 82 | + end | ||
| 83 | + | ||
| 84 | + HCCL -->|hccl_diag.h| HCD | ||
| 85 | + HCCL -->|hcomm_diag.h| AIP | ||
| 86 | + HCD --> HCP | ||
| 87 | + HCP --> PR | ||
| 88 | + PR --> MTM | ||
| 89 | + PR --> PH | ||
| 90 | + PH -->|dlMsprofReportApi, etc.| Profapi | ||
| 91 | + AIP --> HCDL | ||
| 92 | + AIP --> PHL | ||
| 93 | + HCDL --> HCPL | ||
| 94 | + HCPL --> PRL | ||
| 95 | + PRL --> MTML | ||
| 96 | + PRL --> PHL | ||
| 97 | + PHL -->|MsprofReportAdditionalInfo| Profapi | ||
| 98 | + HCP -->|dlMsprofSysCycleTime/dlMsprofStr2Id| Profapi | ||
| 99 | +``` | ||
| 100 | + | ||
| 101 | +## Flow Description | ||
| 102 | + | ||
| 103 | +### Host Profiling Flow | ||
| 104 | + | ||
| 105 | +#### Register Operator Information (HcclDfxRegOpInfoByCommId) | ||
| 106 | + | ||
| 107 | +```mermaid | ||
| 108 | +sequenceDiagram | ||
| 109 | + participant HCCL as hccl | ||
| 110 | + participant HCD as HcclCommDfx | ||
| 111 | + participant PH as ProfilingHandler | ||
| 112 | + participant MTM as MirrorTaskManager | ||
| 113 | + participant Profapi as profapi component | ||
| 114 | + | ||
| 115 | + HCCL->>HCD: HcclDfxRegOpInfoByCommId | ||
| 116 | + Note right of HCD: Convert HcclDfxOpInfo to DfxOpInfo | ||
| 117 | + HCD->>Profapi: hrtMsprofSysCycleTime | ||
| 118 | + Note right of Profapi: Get beginTime | ||
| 119 | + HCD->>HCD: UpdateProfStat | ||
| 120 | + Note right of HCD: Update switch status | ||
| 121 | + HCD->>MTM: SetCurrDfxOpInfo | ||
| 122 | + Note right of MTM: Store current operator info for later reporting | ||
| 123 | + HCD->>PH: SetIsOpbase | ||
| 124 | + Note right of PH: Set operator mode flag | ||
| 125 | +``` | ||
| 126 | + | ||
| 127 | +#### Report Operator (HcclProfilingReportOp) | ||
| 128 | + | ||
| 129 | +```mermaid | ||
| 130 | +sequenceDiagram | ||
| 131 | + participant HCCL as hccl | ||
| 132 | + participant HCD as HcclCommDfx | ||
| 133 | + participant HCP as HcclCommProfiling | ||
| 134 | + participant PR as ProfilingReporter | ||
| 135 | + participant PH as ProfilingHandler | ||
| 136 | + participant Profapi as profapi component | ||
| 137 | + | ||
| 138 | + HCCL->>HCD: HcclProfilingReportOp | ||
| 139 | + alt currDfxOpInfo is empty | ||
| 140 | + Note right of HCD: Skip reporting, return success | ||
| 141 | + else currDfxOpInfo is not empty | ||
| 142 | + HCD->>HCD: IsOpBase | ||
| 143 | + HCD->>HCP: ReportAllTasks | ||
| 144 | + HCP->>PR: ReportAllTasks | ||
| 145 | + Note right of PR: Traverse the task queue in MirrorTaskManager | ||
| 146 | + PR->>PH: ReportHcclTaskDetails | ||
| 147 | + PH->>Profapi: dlMsprofReportAdditionalInfo | ||
| 148 | + Note right of Profapi: Report Task details | ||
| 149 | + HCD->>HCP: ReportOp | ||
| 150 | + HCP->>PR: ReportOp | ||
| 151 | + PR->>PH: ReportHostApi | ||
| 152 | + PH->>Profapi: dlMsprofReportApi | ||
| 153 | + Note right of Profapi: level=ACL_LEVEL, report Host API timestamps | ||
| 154 | + end | ||
| 155 | +``` | ||
| 156 | + | ||
| 157 | +#### Report AICPU Kernel (HcclReportAicpuKernel) | ||
| 158 | + | ||
| 159 | +```mermaid | ||
| 160 | +sequenceDiagram | ||
| 161 | + participant HCCL as hccl | ||
| 162 | + participant HCD as HcclCommDfx | ||
| 163 | + participant HCP as HcclCommProfiling | ||
| 164 | + participant PH as ProfilingHandler | ||
| 165 | + participant DPF as DlProfFunction | ||
| 166 | + participant Profapi as profapi component | ||
| 167 | + | ||
| 168 | + HCCL->>HCD: HcclReportAicpuKernel | ||
| 169 | + alt currDfxOpInfo is empty | ||
| 170 | + Note right of HCD: Skip reporting, return success | ||
| 171 | + else currDfxOpInfo is not empty | ||
| 172 | + HCD->>HCP: ReportKernel | ||
| 173 | + HCP->>DPF: dlMsprofSysCycleTime | ||
| 174 | + Note right of DPF: Get endTime | ||
| 175 | + HCP->>DPF: dlMsprofStr2Id | ||
| 176 | + Note right of DPF: Hash kernelName to cmdItemId | ||
| 177 | + HCP->>PH: ReportNodeApi | ||
| 178 | + PH->>Profapi: dlMsprofReportApi | ||
| 179 | + Note right of Profapi: level=NODE_LEVEL, type=NODE_LAUNCH_TYPE | ||
| 180 | + HCP->>PH: ReportNodeBasicInfo | ||
| 181 | + PH->>Profapi: dlMsprofReportCompactInfo | ||
| 182 | + Note right of Profapi: level=NODE_LEVEL, type=NODE_BASIC_INFO_TYPE | ||
| 183 | + DPF->>Profapi: dlMsprofSysCycleTime | ||
| 184 | + Note right of Profapi: Get endTime for taskParam | ||
| 185 | + HCD->>HCD: AddTaskInfoCallback | ||
| 186 | + Note right of HCD: Record AICPU Kernel task info | ||
| 187 | + end | ||
| 188 | +``` | ||
| 189 | + | ||
| 190 | +#### Report AIV Kernel (HcclReportAivKernel) | ||
| 191 | + | ||
| 192 | +```mermaid | ||
| 193 | +sequenceDiagram | ||
| 194 | + participant HCCL as hccl | ||
| 195 | + participant HCD as HcclCommDfx | ||
| 196 | + participant DPF as DlProfFunction | ||
| 197 | + participant Profapi as profapi component | ||
| 198 | + | ||
| 199 | + HCCL->>HCD: HcclReportAivKernel | ||
| 200 | + DPF->>Profapi: dlMsprofSysCycleTime | ||
| 201 | + Note right of Profapi: Get endTime | ||
| 202 | + HCD->>HCD: AddTaskInfoCallback | ||
| 203 | + Note right of HCD: taskType=TASK_AIV, isMaster=true | ||
| 204 | +``` | ||
| 205 | + | ||
| 206 | +#### Report MC2 Communication Information | ||
| 207 | + | ||
| 208 | +```mermaid | ||
| 209 | +sequenceDiagram | ||
| 210 | + participant HCD as HcclCommDfx | ||
| 211 | + participant HCP as HcclCommProfiling | ||
| 212 | + participant PR as ProfilingReporter | ||
| 213 | + participant PH as ProfilingHandler | ||
| 214 | + participant Profapi as profapi component | ||
| 215 | + | ||
| 216 | + HCD->>HCP: ReportMc2CommInfo | ||
| 217 | + HCP->>PR: CallReportMc2CommInfo | ||
| 218 | + PR->>PH: ReportHcclMC2CommInfo | ||
| 219 | + Note right of PH: Assemble ProfilingDeviceCommResInfo, every 8 streamIds as a group | ||
| 220 | + PH->>Profapi: dlMsprofReportAdditionalInfo | ||
| 221 | + Note right of Profapi: level=NODE_LEVEL, type=MC2_COMMINFO | ||
| 222 | +``` | ||
| 223 | + | ||
| 224 | +#### Manage Host-Side Profiling Switch | ||
| 225 | + | ||
| 226 | +```mermaid | ||
| 227 | +sequenceDiagram | ||
| 228 | + participant Profapi as profapi component | ||
| 229 | + participant PH as ProfilingHandler | ||
| 230 | + participant DPF as DlProfFunction | ||
| 231 | + | ||
| 232 | + Profapi->>PH: CommandHandleWrapper | ||
| 233 | + Note right of PH: profapi component notifies switch status changes through callback | ||
| 234 | + PH->>PH: CommandHandle | ||
| 235 | + Note right of PH: Parse the switch mask based on rtType, update enableHostApi_/enableHcclNode_/enableHcclL0_/enableHcclL1_ | ||
| 236 | + PH->>DPF: dlMsprofRegTypeInfo | ||
| 237 | + Note right of DPF: Register type mapping with the profapi component | ||
| 238 | +``` | ||
| 239 | + | ||
| 240 | +### Device Profiling Flow | ||
| 241 | + | ||
| 242 | +#### Register Operator Information (HcclDfxRegOpInfoByCommId) | ||
| 243 | + | ||
| 244 | +```mermaid | ||
| 245 | +sequenceDiagram | ||
| 246 | + participant HCCL as hccl | ||
| 247 | + participant AIP as AicpuIndopProcess | ||
| 248 | + participant HCDL as HcclCommDfxLite | ||
| 249 | + participant MTML as MirrorTaskManagerLite | ||
| 250 | + | ||
| 251 | + HCCL->>AIP: HcclDfxRegOpInfoByCommId | ||
| 252 | + AIP->>AIP: HcommThreadGetNotifyId | ||
| 253 | + Note right of AIP: Get cpuWaitAicpuNotifyId | ||
| 254 | + AIP->>AIP: AicpuDfxOpInfoInit | ||
| 255 | + Note right of AIP: Convert HcclDfxOpInfo to DfxOpInfo | ||
| 256 | + AIP->>HCDL: UpdateProfStat | ||
| 257 | + AIP->>MTML: SetCurrDfxOpInfo | ||
| 258 | + Note right of MTML: Store current operator info for later reporting | ||
| 259 | +``` | ||
| 260 | + | ||
| 261 | +#### Report Device Operator (HcommProfilingReportDeviceOp) | ||
| 262 | + | ||
| 263 | +```mermaid | ||
| 264 | +sequenceDiagram | ||
| 265 | + participant HCCL as hccl | ||
| 266 | + participant AIP as AicpuIndopProcess | ||
| 267 | + participant HCDL as HcclCommDfxLite | ||
| 268 | + participant HCPL as HcclCommProfilingLite | ||
| 269 | + participant PRL as ProfilingReporterLite | ||
| 270 | + participant PHL as ProfilingHandlerLite | ||
| 271 | + participant Profapi as profapi component | ||
| 272 | + | ||
| 273 | + HCCL->>AIP: HcommProfilingReportDeviceOp | ||
| 274 | + alt currDfxOpInfo is empty | ||
| 275 | + Note right of AIP: Skip reporting, return success | ||
| 276 | + else currDfxOpInfo is not empty | ||
| 277 | + AIP->>HCDL: ReportAllTasks | ||
| 278 | + HCDL->>HCPL: ReportAllTasks | ||
| 279 | + HCPL->>PRL: ReportAllTasks | ||
| 280 | + Note right of PRL: Traverse the task queue in MirrorTaskManagerLite | ||
| 281 | + PRL->>PHL: ReportHcclTaskDetails | ||
| 282 | + PHL->>Profapi: MsprofReportAdditionalInfo | ||
| 283 | + Note right of Profapi: Report Task details | ||
| 284 | + AIP->>PHL: ReportHcclOpInfo | ||
| 285 | + PHL->>Profapi: MsprofReportAdditionalInfo | ||
| 286 | + Note right of Profapi: Report HCCL OP info | ||
| 287 | + end | ||
| 288 | +``` | ||
| 289 | + | ||
| 290 | +#### Report Kernel Start Task (HcommProfilingReportKernelStartTask) | ||
| 291 | + | ||
| 292 | +```mermaid | ||
| 293 | +sequenceDiagram | ||
| 294 | + participant HCCL as hccl | ||
| 295 | + participant AIP as AicpuIndopProcess | ||
| 296 | + participant PHL as ProfilingHandlerLite | ||
| 297 | + participant Profapi as profapi component | ||
| 298 | + | ||
| 299 | + HCCL->>AIP: HcommProfilingReportKernelStartTask | ||
| 300 | + AIP->>AIP: UpdateTask | ||
| 301 | + Note right of AIP: Update Profiling status | ||
| 302 | + AIP->>PHL: ReportMainStreamTask | ||
| 303 | + Note right of PHL: type=HEAD, record the first task of the main stream | ||
| 304 | + PHL->>Profapi: MsprofReportAdditionalInfo | ||
| 305 | + Note right of Profapi: Report FlagTaskInfo | ||
| 306 | +``` | ||
| 307 | + | ||
| 308 | +#### Report Kernel End Task (HcommProfilingReportKernelEndTask) | ||
| 309 | + | ||
| 310 | +```mermaid | ||
| 311 | +sequenceDiagram | ||
| 312 | + participant HCCL as hccl | ||
| 313 | + participant AIP as AicpuIndopProcess | ||
| 314 | + participant PHL as ProfilingHandlerLite | ||
| 315 | + participant Profapi as profapi component | ||
| 316 | + | ||
| 317 | + HCCL->>AIP: HcommProfilingReportKernelEndTask | ||
| 318 | + AIP->>PHL: ReportMainStreamTask | ||
| 319 | + Note right of PHL: type=TAIL, record the last task of the main stream | ||
| 320 | + PHL->>Profapi: MsprofReportAdditionalInfo | ||
| 321 | + Note right of Profapi: Report FlagTaskInfo | ||
| 322 | +``` | ||
| 323 | + | ||
| 324 | +#### Manage Device-Side Profiling Switch | ||
| 325 | + | ||
| 326 | +```mermaid | ||
| 327 | +sequenceDiagram | ||
| 328 | + participant PHL as ProfilingHandlerLite | ||
| 329 | + participant Profapi as profapi component | ||
| 330 | + | ||
| 331 | + PHL->>PHL: UpdateProfSwitch | ||
| 332 | + Note right of PHL: Actively query the profapi component switch status | ||
| 333 | + PHL->>Profapi: AdprofCheckFeatureIsOn | ||
| 334 | + Note right of Profapi: Check whether L0/L1 switches are enabled | ||
| 335 | + PHL->>PHL: SetProL0On/SetProL1On | ||
| 336 | + Note right of PHL: Update enableHcclL0_/enableHcclL1_ | ||
| 337 | +``` | ||
| 338 | + | ||
| 339 | +### Report Level and Type Constants Summary | ||
| 340 | + | ||
| 341 | +| Constant Name | Value | Description | | ||
| 342 | +|--------|-----|------| | ||
| 343 | +| `MSPROF_REPORT_ACL_LEVEL` | 20000 | ACL level, used for Host API reporting | | ||
| 344 | +| `MSPROF_REPORT_NODE_LEVEL` | 10000 | Node level, used for Node BasicInfo, HCCL OP, and MC2 CommInfo reporting | | ||
| 345 | +| `MSPROF_REPORT_HCCL_NODE_LEVEL` | 5500 | HCCL Node level, used for Task details and CCU information reporting | | ||
| 346 | +| `MSPROF_REPORT_ACL_HOST_HCCL_BASE_TYPE` | 0x070000 | ACL Host HCCL base type | | ||
| 347 | +| `MSPROF_REPORT_NODE_LAUNCH_TYPE` | 5 | Node Launch type, used for `ReportNodeApi` | | ||
| 348 | +| `MSPROF_REPORT_NODE_BASIC_INFO_TYPE` | 0 | Node basic information type, used for `ReportNodeBasicInfo` | | ||
| 349 | +| `MSPROF_REPORT_NODE_HCCL_OP_INFO_TYPE` | 10 | Node HCCL OP information type | | ||
| 350 | +| `MSPROF_REPORT_NODE_MC2_COMMINFO_TYPE` | 12 | Node MC2 communication resource information type | | ||
| 351 | +| `MSPROF_REPORT_HCCL_MASTER_TYPE` | 0x010001 | HCCL main stream type | | ||
| 352 | +| `MSPROF_REPORT_HCCL_SLAVE_TYPE` | 0x010002 | HCCL slave stream type | | ||
| 353 | +| `MSPROF_REPORT_CCU_TASK_INFO` | 14 | CCU Task information type | | ||
| 354 | +| `MSPROF_REPORT_CCU_WAIT_SIGNAL_INFO` | 15 | CCU Wait Signal information type | | ||
| 355 | +| `MSPROF_REPORT_CCU_GROUP_INFO` | 16 | CCU Group information type | | ||
| 356 | + | ||
| 357 | +### Switch Control and Report Content Mapping | ||
| 358 | + | ||
| 359 | +| Switch | Corresponding Mask | Controlled Report Content | | ||
| 360 | +|------|----------|---------------| | ||
| 361 | +| `enableHostApi_` | `PROF_ACL_API_MASK` = 0x1 | Host API timestamp reporting (`ReportAclApi`, `ReportNodeApi`, `ReportHcclOpApi`, `ReportHcclOpInfo`, `ReportMc2AdditionInfo`) | | ||
| 362 | +| `enableHcclL0_` | `PROF_TASK_TIME_MASK` = 0x800 | HCCL operator granularity tracing (`ReportHcclOpApi`) | | ||
| 363 | +| `enableHcclNode_` | `PROF_TASK_TIME_L1_MASK` = 0x2 | Task granularity tracing (`ReportHcclTaskApi`) | | ||
| 364 | +| `enableHcclL1_` | `PROF_TASK_TIME_L1_MASK` = 0x2 | Task details reporting (`CallAddtionInfo`, `ReportNodeBasicInfo`) | | ||
| 365 | +| `enableHcclL2_` | `PROF_TASK_TIME_L2_MASK` = 0x2000 | CCU details reporting (`ReportCcuInfo`) | | ||
| 366 | + | ||
| 367 | +## Interface Description (Class Diagram) | ||
| 368 | + | ||
| 369 | +```mermaid | ||
| 370 | +classDiagram | ||
| 371 | + class HcclCommProfiling { | ||
| 372 | + -MirrorTaskManager* mirrorTaskManager_ | ||
| 373 | + -unique_ptr~ProfilingReporter~ profilingReporter_ | ||
| 374 | + +HcclCommProfiling(u32 deviceId, MirrorTaskManager* mirrorTaskManager) | ||
| 375 | + +ReportAllTasks(bool cachedReq) void | ||
| 376 | + +ReportOp(uint64_t beginTime, bool cachedReq, bool opbased) void | ||
| 377 | + +ReportMc2CommInfo(Mc2CommInfo& mc2CommInfo) void | ||
| 378 | + +UpdateProfStat() void | ||
| 379 | + +GetMirrorTaskManager() MirrorTaskManager* | ||
| 380 | + +ReportKernel(uint64_t beginTime, string& commTag, string& kernelName, uint32_t threadId, bool cachedReq) HcclResult | ||
| 381 | + } | ||
| 382 | + | ||
| 383 | + class HcclCommProfilingLite { | ||
| 384 | + -MirrorTaskManagerLite* mirrorTaskManagerLite_ | ||
| 385 | + -unique_ptr~ProfilingReporterLite~ profilingReporterLite_ | ||
| 386 | + +HcclCommProfilingLite(DevId deviceId, MirrorTaskManagerLite* mirrorTaskManagerLite) | ||
| 387 | + +ReportAllTasks() void | ||
| 388 | + +UpdateProfStat() void | ||
| 389 | + +GetMirrorTaskManagerLite() MirrorTaskManagerLite* | ||
| 390 | + } | ||
| 391 | + | ||
| 392 | + class ProfilingReporter { | ||
| 393 | + -MirrorTaskManager* mirrorTaskMgr_ | ||
| 394 | + -bool enableHcclL1_ | ||
| 395 | + -ProfilingHandler* profilingHandler_ | ||
| 396 | + +ProfilingReporter(MirrorTaskManager*, ProfilingHandler*) | ||
| 397 | + +Init() void | ||
| 398 | + +ReportOp(uint64_t beginTime, bool cachedReq, bool opbased) void | ||
| 399 | + +ReportAllTasks(bool cachedReq) void | ||
| 400 | + +UpdateProfStat() void | ||
| 401 | + +CallReportMc2CommInfo(u32 kfcStreamId, vector~u32~ aicpuStreamsId, string id, RankId myRank, u32 rankSize, RankId rankInParentComm) void | ||
| 402 | + } | ||
| 403 | + | ||
| 404 | + class ProfilingReporterLite { | ||
| 405 | + -MirrorTaskManagerLite* mirrorTaskMgrLite_ | ||
| 406 | + -ProfilingHandlerLite* profilingHandlerLite_ | ||
| 407 | + -map lastPoses_ | ||
| 408 | + +ProfilingReporterLite(MirrorTaskManagerLite*, ProfilingHandlerLite*, bool isIndop) | ||
| 409 | + +Init() void | ||
| 410 | + +ReportAllTasks() void | ||
| 411 | + +UpdateProfStat() void | ||
| 412 | + } | ||
| 413 | + | ||
| 414 | + class ProfilingHandler { | ||
| 415 | + -static ProfilingHandler instance_ | ||
| 416 | + -bool enableHostApi_ | ||
| 417 | + -bool enableHcclNode_ | ||
| 418 | + -bool enableHcclL0_ | ||
| 419 | + -bool enableHcclL1_ | ||
| 420 | + +GetInstance() ProfilingHandler&$ | ||
| 421 | + +ReportNodeApi(uint64_t beginTime, uint64_t endTime, uint64_t cmdItemId, uint32_t threadId, bool cachedReq) void | ||
| 422 | + +ReportNodeBasicInfo(uint64_t timeStamp, uint64_t cmdItemId, uint32_t threadId, bool cachedReq) void | ||
| 423 | + +ReportHostApi(OpType opType, uint64_t beginTime, uint64_t endTime, bool cachedReq, bool isAiCpu) void | ||
| 424 | + +ReportHcclOp(DfxOpInfo& opInfo, bool cachedReq) void | ||
| 425 | + } | ||
| 426 | + | ||
| 427 | + class ProfilingHandlerLite { | ||
| 428 | + -static ProfilingHandlerLite instance_ | ||
| 429 | + -bool enableHcclL0_ | ||
| 430 | + -bool enableHcclL1_ | ||
| 431 | + +GetInstance() ProfilingHandlerLite&$ | ||
| 432 | + +ReportHcclOpInfo(DfxOpInfo& opInfo) void | ||
| 433 | + +ReportHcclTaskDetails(vector~TaskInfo~& taskInfo) void | ||
| 434 | + +ReportMainStreamTask(FlagTaskInfo& flagTaskInfo) void | ||
| 435 | + +UpdateProfSwitch() void | ||
| 436 | + } | ||
| 437 | + | ||
| 438 | + class Mc2CommInfo { | ||
| 439 | + +u32 FreeStreamId | ||
| 440 | + +vector~u32~ streamsId | ||
| 441 | + +string groupname | ||
| 442 | + +u32 myRankId | ||
| 443 | + +u32 rankSize | ||
| 444 | + +u32 parentRankId | ||
| 445 | + } | ||
| 446 | + | ||
| 447 | + HcclCommProfiling o-- ProfilingReporter : holds | ||
| 448 | + HcclCommProfiling o-- MirrorTaskManager : holds pointer | ||
| 449 | + HcclCommProfiling ..> ProfilingHandler : accessed indirectly through Reporter | ||
| 450 | + HcclCommProfiling ..> profapi component : dlMsprofSysCycleTime/dlMsprofStr2Id | ||
| 451 | + HcclCommProfiling --> Mc2CommInfo : uses | ||
| 452 | + | ||
| 453 | + HcclCommProfilingLite o-- ProfilingReporterLite : holds | ||
| 454 | + HcclCommProfilingLite o-- MirrorTaskManagerLite : holds pointer | ||
| 455 | + HcclCommProfilingLite ..> ProfilingHandlerLite : accessed indirectly through Reporter | ||
| 456 | + | ||
| 457 | + ProfilingReporter o-- MirrorTaskManager : holds pointer | ||
| 458 | + ProfilingReporter ..> ProfilingHandler : calls reporting interface | ||
| 459 | + | ||
| 460 | + ProfilingReporterLite o-- MirrorTaskManagerLite : holds pointer | ||
| 461 | + ProfilingReporterLite ..> ProfilingHandlerLite : calls reporting interface | ||
| 462 | + | ||
| 463 | + ProfilingHandler ..> profapi component : dlMsprofReportApi/dlMsprofReportCompactInfo/dlMsprofReportAdditionalInfo | ||
| 464 | + ProfilingHandler ..> rts component : orion_adapter_rts.h | ||
| 465 | + ProfilingHandlerLite ..> profapi component : MsprofReportAdditionalInfo/AdprofReportAdditionalInfo | ||
| 466 | +``` | ||
| 467 | + | ||
| 468 | +## Interface Description | ||
| 469 | + | ||
| 470 | +### HcclCommProfiling (Host Side) | ||
| 471 | + | ||
| 472 | +| Interface | Type | Parameters | Return Value | Description | | ||
| 473 | +|--------|------|------|--------|----------| | ||
| 474 | +| `HcclCommProfiling` | Public | `[in] u32 deviceId`, `[in] MirrorTaskManager* mirrorTaskManager` | - | Constructor, saves the task manager pointer, creates a `ProfilingReporter` instance (associated with `ProfilingHandler::GetInstance()`). | | ||
| 475 | +| `ReportAllTasks` | Public | `[in] bool cachedReq = false` | void | Reports all communication tasks. When `cachedReq=true`, indicates cache request mode. Delegates to `ProfilingReporter::ReportAllTasks` to traverse the task queue and report through `ProfilingHandler`. | | ||
| 476 | +| `ReportOp` | Public | `[in] uint64_t beginTime`, `[in] bool cachedReq`, `[in] bool opbased` | void | Reports operator information. Delegates to `ProfilingReporter::ReportOp`, which ultimately calls the profapi component `dlMsprofReportApi` through `ProfilingHandler::ReportHostApi`. | | ||
| 477 | +| `ReportMc2CommInfo` | Public | `[in] const Mc2CommInfo& mc2CommInfo` | void | Reports MC2 communication domain information. Splits the `Mc2CommInfo` fields and calls `ProfilingReporter::CallReportMc2CommInfo`, which ultimately reports through `dlMsprofReportAdditionalInfo`. | | ||
| 478 | +| `UpdateProfStat` | Public | - | void | Updates Profiling statistics. Delegates to `ProfilingReporter::UpdateProfStat` to update the switch status. | | ||
| 479 | +| `GetMirrorTaskManager` | Public | - | `MirrorTaskManager*` | Returns the internally held `MirrorTaskManager` pointer. | | ||
| 480 | +| `ReportKernel` | Public | `[in] uint64_t beginTime`, `[in] const string& commTag`, `[in] const string& kernelName`, `[in] uint32_t threadId`, `[in] bool cachedReq` | HcclResult | Reports CCU Kernel information. Calls the profapi component `dlMsprofSysCycleTime` to get endTime and `dlMsprofStr2Id` to get cmdItemId, then calls `ProfilingHandler` to report `ReportNodeApi` and `ReportNodeBasicInfo`. The `EXCEPTION_CATCH` macro catches exceptions and returns `HCCL_E_PTR` on failure. | | ||
| 481 | + | ||
| 482 | +### HcclCommProfilingLite (Device Side) | ||
| 483 | + | ||
| 484 | +| Interface | Type | Parameters | Return Value | Description | | ||
| 485 | +|--------|------|------|--------|----------| | ||
| 486 | +| `HcclCommProfilingLite` | Public | `[in] DevId deviceId`, `[in] MirrorTaskManagerLite* mirrorTaskManagerLite` | - | Constructor, saves the task manager pointer, creates a `ProfilingReporterLite` instance (`isIndop=true`, associated with `ProfilingHandlerLite::GetInstance()`). | | ||
| 487 | +| `ReportAllTasks` | Public | - | void | Reports all communication tasks. Delegates to `ProfilingReporterLite::ReportAllTasks`, which ultimately calls the profapi component `MsprofReportAdditionalInfo` through a weak symbol. | | ||
| 488 | +| `UpdateProfStat` | Public | - | void | Updates Profiling statistics. Delegates to `ProfilingReporterLite::UpdateProfStat` to update the switch status. | | ||
| 489 | +| `GetMirrorTaskManagerLite` | Public | - | `MirrorTaskManagerLite*` | Returns the internally held `MirrorTaskManagerLite` pointer. | | ||
| 490 | + | ||
| 491 | +## Usage Limitations | ||
| 492 | + | ||
| 493 | +### Supported Scenarios | ||
| 494 | + | ||
| 495 | +| Scenario | Host Side | Device Side | Description | | ||
| 496 | +|------|---------|----------|------| | ||
| 497 | +| Register operator information | Supported | Supported | Registered through `HcclDfxRegOpInfoByCommId`. | | ||
| 498 | +| Report operator | Supported | Supported | Host reports through `HcclProfilingReportOp`; Device reports through `HcommProfilingReportDeviceOp`. | | ||
| 499 | +| Report Kernel | Supported | Supported | Host supports AICPU and AIV Kernel reporting; Device supports kernel start and end task reporting. | | ||
| 500 | +| Report MC2 communication info | Supported | Not supported | Only the Host side supports `ReportMc2CommInfo`. | | ||
| 501 | +| Update Profiling status | Supported | Supported | Both sides support this. | | ||
| 502 | +| Multi-device task management | Supported | Not supported | The Host-side `MirrorTaskManager` supports multi-device queue mapping. | | ||
| 503 | +| Report CCU information | Supported | Not supported | Only the Host side supports CCU Task, WaitSignal, and Group information reporting. | | ||
| 504 | +| Get system cycle time | Supported | Not supported | Only the Host side supports `HcommGetProfilingSysCycleTime`. | | ||
| 505 | + | ||
| 506 | +### Constraint Specifications | ||
| 507 | + | ||
| 508 | +1. **Maximum device count**: The Host-side `ProfilingReporter` static array `allLastPoses_` has a size of `REPORTER_MAX_MODULE_DEVICE_NUM` = 65, supporting profiling position records for up to 65 devices. | ||
| 509 | +2. **ProfilingHandler singleton pattern**: Both Host-side and Device-side `ProfilingHandler` and `ProfilingHandlerLite` are singletons, globally unique. Copying and assignment are prohibited. | ||
| 510 | +3. **DlProfFunction dynamic loading**: The Host side dynamically loads `libprofapi.so` through `DlProfFunction` using `dlopen`. If the SDK is unavailable, it falls back to stub functions (printing a WARNING log and skipping). | ||
| 511 | +4. **Device-side weak symbol linking**: The Device side declares profapi component functions through `__attribute__((weak))` (such as `MsprofReportAdditionalInfo` and `AdprofReportAdditionalInfo`). At runtime, the selection priority is: `MsprofReportBatchAdditionalInfo` > `AdprofReportAdditionalInfo` > `MsprofReportAdditionalInfo`. | ||
| 512 | +5. **Null pointer protection**: All reporting interfaces check for non-null pointers before calling the Reporter to avoid null pointer dereferences. | ||
| 513 | +6. **EXCEPTION_CATCH macro**: `ReportKernel` uses the `EXCEPTION_CATCH` macro to catch exceptions during `ProfilingHandler` reporting, returning `HCCL_E_PTR` on failure. | ||
| 514 | +7. **MC2 Stream group reporting**: `ReportMc2CommInfo` groups every 8 streamIds into a group and reports them through `ProfilingDeviceCommResInfo`. The `commStreamIds` array size is fixed at 8. | ||
| 515 | +8. **Device-side device type restriction**: `HcommProfilingReportDeviceOp`, `HcommProfilingReportKernelStartTask`, and `HcommProfilingReportKernelEndTask` only execute on `DEV_TYPE_950` devices. For other device types, they directly return success. | ||
| 516 | + | ||
| 517 | +### Known Limitations | ||
| 518 | + | ||
| 519 | +1. **`aicpu_ts_urma_dfx_kernel` is deprecated**: The Device-side `aicpu_ts_urma_dfx_kernel.h` and `.cc` files are deprecated and no longer maintained, but the build entry is still retained in `CMakeLists.txt`. | ||
| 520 | +2. **`Mc2CommInfo` has no validation**: The `ReportMc2CommInfo` interface does not validate the length of the `streamsId` vector in `mc2CommInfo`. This is handled by the underlying `CallReportMc2CommInfo`. | ||
| 521 | +3. **Thread safety**: The Host-side `ProfilingHandler` uses multiple mutexes (`cacheTaskInfosMutex_`, `cachedTaskApiInfoMutex_`, `cacheHcclOpInfoMutex_`, `cacheHcclAdditionInfoMutex_`) to protect cached data. The Device-side `ProfilingHandlerLite` does not use locks and relies on a single-threaded execution environment. | ||
| 522 | +4. **Device-side switch query mode**: Unlike the Host side, which receives switch status passively through callbacks, the Device-side `ProfilingHandlerLite` must actively call `UpdateProfSwitch()` to query the switch status from the profapi component. | ||
| 523 | +5. **Non-V2 communication domain handling**: On `DEV_TYPE_910B` devices, `HcclProfilingReportOp` directly returns success and skips reporting for non-`CommunicatorV2` communication domains. For other device types, `HCCL_E_NOT_SUPPORT` is returned for non-V2 domains. | ||
| @@ -8,6 +8,7 @@ taskException 模块是 HCCL 集合通信库中的 **DFX（Design for eXcellence | |||
| 8 | - **AICPU 侧**：以守护线程方式定期检测各通信域流上的 CQE（Completion Queue Entry）异常，完成异常 CQE 解析、错误信息组织、通过 HDC 通道上报到 Host 侧、通过 Mailbox 通知 TSFW。 | 8 | - **AICPU 侧**：以守护线程方式定期检测各通信域流上的 CQE（Completion Queue Entry）异常，完成异常 CQE 解析、错误信息组织、通过 HDC 通道上报到 Host 侧、通过 Mailbox 通知 TSFW。 |
| 9 | 9 | ||
| 10 | 核心能力包括： | 10 | 核心能力包括： |
| 11 | + | ||
| 11 | 1. 向 Runtime 注册/注销异常回调函数 | 12 | 1. 向 Runtime 注册/注销异常回调函数 |
| 12 | 2. 基于 GlobalMirrorTasks 查找异常 TaskInfo | 13 | 2. 基于 GlobalMirrorTasks 查找异常 TaskInfo |
| 13 | 3. 通过 HDC 通道从 AICPU 侧读取 ErrorMessageReport | 14 | 3. 通过 HDC 通道从 AICPU 侧读取 ErrorMessageReport |
| @@ -20,7 +21,7 @@ taskException 模块是 HCCL 集合通信库中的 **DFX（Design for eXcellence | |||
| 20 | 21 | ||
| 21 | ## 目录描述 | 22 | ## 目录描述 |
| 22 | 23 | ||
| 23 | -``` | 24 | +```text |
| 24 | taskException/ | 25 | taskException/ |
| 25 | ├── host/ # Host 侧异常处理 | 26 | ├── host/ # Host 侧异常处理 |
| 26 | │ ├── hcclCommTaskException.h # TaskExceptionHost / TaskExceptionHostManager 类声明 | 27 | │ ├── hcclCommTaskException.h # TaskExceptionHost / TaskExceptionHostManager 类声明 |
| @@ -0,0 +1,666 @@ | |||
| 1 | +# taskException Module Code Analysis | ||
| 2 | + | ||
| 3 | +## Feature Description | ||
| 4 | + | ||
| 5 | +The taskException module is the **DFX (Design for eXcellence) exception diagnosis subsystem** in the HCCL collective communication library. It is responsible for exception capture, error information parsing, diagnostic log printing, and error reporting when collective communication tasks fail to execute. This module spans two runtime environments: the Host side and the AICPU side. | ||
| 6 | + | ||
| 7 | +- **Host side**: Receives exception callbacks from the Runtime, distributes them to different exception handling flows based on task type (general task exception or CCU task exception), reads AICPU error information, prints UB DFX register information, and reports cluster monitoring errors. | ||
| 8 | +- **AICPU side**: Runs as a daemon thread that periodically checks for CQE (Completion Queue Entry) exceptions on each communication domain stream. It parses exception CQEs, organizes error information, reports to the Host side through the HDC channel, and notifies TSFW through Mailbox. | ||
| 9 | + | ||
| 10 | +Core capabilities include: | ||
| 11 | + | ||
| 12 | +1. Registering and unregistering exception callback functions with the Runtime | ||
| 13 | +2. Searching for abnormal TaskInfo based on GlobalMirrorTasks | ||
| 14 | +3. Reading ErrorMessageReport from the AICPU side through the HDC channel | ||
| 15 | +4. Parsing CCU (Cube Compute Unit) task exceptions and restoring the error instruction context based on the CcuRep instruction representation system | ||
| 16 | +5. Printing UB DFX register information for hardware diagnostics | ||
| 17 | +6. Reporting CQE error information to cluster monitoring | ||
| 18 | +7. Printing the context of up to 50 preceding tasks before the exception task | ||
| 19 | + | ||
| 20 | +--- | ||
| 21 | + | ||
| 22 | +## Directory Description | ||
| 23 | + | ||
| 24 | +```text | ||
| 25 | +taskException/ | ||
| 26 | +├── host/ # Host-side exception handling | ||
| 27 | +│ ├── hcclCommTaskException.h # TaskExceptionHost / TaskExceptionHostManager class declarations | ||
| 28 | +│ ├── hcclCommTaskException.cc # Host-side general exception handling implementation + callback registration management | ||
| 29 | +│ ├── ccuTaskException.h # CcuTaskException class declaration | ||
| 30 | +│ ├── ccuTaskException.cc # CCU task exception handling implementation (instruction parsing, register reading, error information generation) | ||
| 31 | +│ └── ccu_error_info_v1.h # CCU error information data structure definitions (CcuErrorInfo, CcuLoopContext, CcuMissionContext, and so on) | ||
| 32 | +└── aicpu/ # AICPU-side exception handling | ||
| 33 | + ├── hcclCommTaskExceptionLite.h # HcclCommTaskExceptionLite class declaration | ||
| 34 | + └── hcclCommTaskExceptionLite.cc # AICPU-side exception detection, CQE parsing, and error reporting implementation | ||
| 35 | +``` | ||
| 36 | + | ||
| 37 | +### File Relationships | ||
| 38 | + | ||
| 39 | +| File | Function | Dependencies | | ||
| 40 | +|------|------|----------| | ||
| 41 | +| `hcclCommTaskException.h/.cc` | Host-side main entry point, registers Runtime exception callbacks, distributes to general or CCU exception handling | Depends on `ccuTaskException.h` for CCU type exception handling; depends on `global_mirror_tasks.h` for finding TaskInfo; obtains ErrorMessageReport from the AICPU side through callbacks | | ||
| 42 | +| `ccuTaskException.h/.cc` | CCU task exception-specific handling, parses CcuRep instruction representation, reads hardware registers | Depends on data structures in `ccu_error_info_v1.h`; depends on `ccu_kernel_mgr.h` for CcuRepContext; depends on `ccu_urma_channel.h` for channel information | | ||
| 43 | +| `ccu_error_info_v1.h` | CCU error information data structure definitions | Referenced by `ccuTaskException.h/.cc`; depends on `ccu_rep_type_v1.h` for the CcuRepType enum definition | | ||
| 44 | +| `hcclCommTaskExceptionLite.h/.cc` | AICPU-side daemon thread, detects CQE exceptions and reports to Host | Depends on `coll_comm_aicpu.h` for the AICPU communication domain; depends on `error_message_v2.h` for organizing ErrorMessageReport and reporting to the Host through HDC | | ||
| 45 | + | ||
| 46 | +### TaskException File Interaction | ||
| 47 | + | ||
| 48 | +```mermaid | ||
| 49 | +graph TB | ||
| 50 | + subgraph Registration Entry | ||
| 51 | + HCCL[HCCL] | ||
| 52 | + end | ||
| 53 | + | ||
| 54 | + subgraph Exception Trigger Source | ||
| 55 | + Runtime[Runtime] | ||
| 56 | + end | ||
| 57 | + | ||
| 58 | + subgraph Host Side | ||
| 59 | + hcomm_c_adpt_H[hcomm_c_adpt] | ||
| 60 | + CollComm[CollComm] | ||
| 61 | + HcclCommDfx[HcclCommDfx] | ||
| 62 | + MirrorTaskManager[MirrorTaskManager] | ||
| 63 | + TaskExceptionHost[TaskExceptionHost] | ||
| 64 | + CcuTaskException[CcuTaskException] | ||
| 65 | + CcuKernelMgr[CcuKernelMgr] | ||
| 66 | + end | ||
| 67 | + | ||
| 68 | + subgraph AICPU Side | ||
| 69 | + hcomm_c_adpt_D[hcomm_c_adpt] | ||
| 70 | + AicpuIndopProcess[AicpuIndopProcess] | ||
| 71 | + CollCommAicpu[CollCommAicpu] | ||
| 72 | + HcclCommDfxLite[HcclCommDfxLite] | ||
| 73 | + MirrorTaskManagerLite[MirrorTaskManagerLite] | ||
| 74 | + BGThread[AICPU background thread] | ||
| 75 | + Lite[HcclCommTaskExceptionLite] | ||
| 76 | + StreamLite[StreamLite] | ||
| 77 | + end | ||
| 78 | + | ||
| 79 | + subgraph External Components | ||
| 80 | + hccp[hccp component] | ||
| 81 | + rts[rts component] | ||
| 82 | + error_manager[error_manager component] | ||
| 83 | + hal[hal component] | ||
| 84 | + end | ||
| 85 | + | ||
| 86 | + HCCL -->|HcclDfxRegOpInfoByCommId| hcomm_c_adpt_H | ||
| 87 | + HCCL -->|HcclDfxRegOpInfoByCommId| hcomm_c_adpt_D | ||
| 88 | + hcomm_c_adpt_H -->|RegisterAicpuTaskExceptionCallback| CollComm | ||
| 89 | + CollComm -->|Register| TaskExceptionHost | ||
| 90 | + hcomm_c_adpt_H -->|GetMirrorTaskManager| HcclCommDfx | ||
| 91 | + HcclCommDfx -->|SetCurrDfxOpInfo/AddTaskInfo| MirrorTaskManager | ||
| 92 | + hcomm_c_adpt_D -->|AicpuDfxOpInfoInit| AicpuIndopProcess | ||
| 93 | + AicpuIndopProcess -->|GetHcclCommDfxLite| CollCommAicpu | ||
| 94 | + CollCommAicpu -->|GetMirrorTaskManagerLite| HcclCommDfxLite | ||
| 95 | + HcclCommDfxLite -->|SetCurrDfxOpInfo/AddTaskInfo| MirrorTaskManagerLite | ||
| 96 | + Runtime -->|Process| TaskExceptionHost | ||
| 97 | + BGThread -->|Call| Lite | ||
| 98 | + TaskExceptionHost -->|ProcessCcuException| CcuTaskException | ||
| 99 | + CcuTaskException --> CcuKernelMgr | ||
| 100 | + Lite -->|GetThreadCqe| StreamLite | ||
| 101 | + Lite -->|HDC channel| TaskExceptionHost | ||
| 102 | + Lite -->|Mailbox| Runtime | ||
| 103 | + TaskExceptionHost -->|adapter_error_manager_pub| error_manager | ||
| 104 | + CcuTaskException -->|HccpRaCustomChannel| hccp | ||
| 105 | + TaskExceptionHost -->|orion_adapter_hccp| hccp | ||
| 106 | + TaskExceptionHost -->|orion_adapter_rts| rts | ||
| 107 | + Lite -->|dlHalEschedSubmitEvent| hal | ||
| 108 | +``` | ||
| 109 | + | ||
| 110 | +--- | ||
| 111 | + | ||
| 112 | +## Flow Description | ||
| 113 | + | ||
| 114 | +### Registration Flow | ||
| 115 | + | ||
| 116 | +#### Aicpu Mode Host-Side Registration Flow | ||
| 117 | + | ||
| 118 | +```mermaid | ||
| 119 | +sequenceDiagram | ||
| 120 | + participant HCCL | ||
| 121 | + participant hcomm_c_adpt | ||
| 122 | + participant CollComm | ||
| 123 | + participant TaskExceptionHost | ||
| 124 | + participant HcclCommDfx | ||
| 125 | + participant MirrorTaskManager | ||
| 126 | + | ||
| 127 | + Note over HCCL,MirrorTaskManager: Operator registration phase | ||
| 128 | + | ||
| 129 | + HCCL->>hcomm_c_adpt: HcclDfxRegOpInfoByCommId | ||
| 130 | + Note right of hcomm_c_adpt: Register operator DFX info | ||
| 131 | + | ||
| 132 | + hcomm_c_adpt->>hcomm_c_adpt: ConvertToDfxOpInfo | ||
| 133 | + Note right of hcomm_c_adpt: Convert to internal DfxOpInfo, set isIndop_=true | ||
| 134 | + | ||
| 135 | + hcomm_c_adpt->>CollComm: RegisterAicpuTaskExceptionCallback | ||
| 136 | + Note right of CollComm: Register AICPU exception callback | ||
| 137 | + | ||
| 138 | + CollComm->>TaskExceptionHost: Register | ||
| 139 | + Note right of TaskExceptionHost: Write to CommRegisterMap_. The first comm registration calls aclrtSetExceptionInfoCallback. | ||
| 140 | + | ||
| 141 | + hcomm_c_adpt->>HcclCommDfx: GetMirrorTaskManager | ||
| 142 | + HcclCommDfx-->>hcomm_c_adpt: mirrorTaskManager | ||
| 143 | + hcomm_c_adpt->>MirrorTaskManager: SetCurrDfxOpInfo | ||
| 144 | + Note right of MirrorTaskManager: Set current operator info | ||
| 145 | + | ||
| 146 | + Note over HCCL,MirrorTaskManager: Task dispatch phase | ||
| 147 | + | ||
| 148 | + HcclCommDfx->>HcclCommDfx: AddTaskInfoCallback | ||
| 149 | + Note right of HcclCommDfx: Thread or Channel callback triggers TaskInfo creation associated with DfxOpInfo | ||
| 150 | + HcclCommDfx->>MirrorTaskManager: AddTaskInfo | ||
| 151 | + Note right of MirrorTaskManager: Register with GlobalMirrorTasks | ||
| 152 | +``` | ||
| 153 | + | ||
| 154 | +#### Aicpu Mode Device-Side Registration Flow | ||
| 155 | + | ||
| 156 | +```mermaid | ||
| 157 | +sequenceDiagram | ||
| 158 | + participant HCCL | ||
| 159 | + participant hcomm_c_adpt | ||
| 160 | + participant AicpuIndopProcess | ||
| 161 | + participant CollCommAicpu | ||
| 162 | + participant HcclCommDfxLite | ||
| 163 | + participant MirrorTaskManagerLite | ||
| 164 | + | ||
| 165 | + Note over HCCL,MirrorTaskManagerLite: Operator registration phase | ||
| 166 | + | ||
| 167 | + HCCL->>hcomm_c_adpt: HcclDfxRegOpInfoByCommId | ||
| 168 | + Note right of hcomm_c_adpt: aicpu_ts_primitives_c_adpt, get cpuWaitAicpuNotifyId | ||
| 169 | + | ||
| 170 | + hcomm_c_adpt->>AicpuIndopProcess: AicpuDfxOpInfoInit | ||
| 171 | + Note right of AicpuIndopProcess: Initialize AICPU-side operator info | ||
| 172 | + | ||
| 173 | + AicpuIndopProcess->>AicpuIndopProcess: ConvertToDfxOpInfo | ||
| 174 | + Note right of AicpuIndopProcess: Set isIndop_=true, opIndex_ auto-increment | ||
| 175 | + | ||
| 176 | + AicpuIndopProcess->>CollCommAicpu: GetHcclCommDfxLite | ||
| 177 | + CollCommAicpu-->>AicpuIndopProcess: hcclCommDfxLite | ||
| 178 | + AicpuIndopProcess->>HcclCommDfxLite: GetMirrorTaskManagerLite | ||
| 179 | + HcclCommDfxLite-->>AicpuIndopProcess: mirrorTaskManagerLite | ||
| 180 | + AicpuIndopProcess->>MirrorTaskManagerLite: SetCurrDfxOpInfo | ||
| 181 | + Note right of MirrorTaskManagerLite: Set current operator info | ||
| 182 | + | ||
| 183 | + Note over HCCL,MirrorTaskManagerLite: Task dispatch phase | ||
| 184 | + | ||
| 185 | + HcclCommDfxLite->>HcclCommDfxLite: AddTaskInfoCallback | ||
| 186 | + Note right of HcclCommDfxLite: Thread callback triggers TaskInfo creation associated with DfxOpInfo | ||
| 187 | + HcclCommDfxLite->>MirrorTaskManagerLite: AddTaskInfo | ||
| 188 | + Note right of MirrorTaskManagerLite: Register with MirrorTaskManagerLite | ||
| 189 | +``` | ||
| 190 | + | ||
| 191 | +#### CCU Mode Registration Flow | ||
| 192 | + | ||
| 193 | +```mermaid | ||
| 194 | +sequenceDiagram | ||
| 195 | + participant HCCL | ||
| 196 | + participant hcomm_c_adpt | ||
| 197 | + participant HcclCommDfx | ||
| 198 | + participant MirrorTaskManager | ||
| 199 | + | ||
| 200 | + Note over HCCL,MirrorTaskManager: CCU and Aicpu share the HcclDfxRegOpInfoByCommId entry | ||
| 201 | + | ||
| 202 | + HCCL->>hcomm_c_adpt: HcclDfxRegOpInfoByCommId | ||
| 203 | + Note right of hcomm_c_adpt: Skip RegAicpuTaskException when not AICPU_TS engine | ||
| 204 | + | ||
| 205 | + hcomm_c_adpt->>hcomm_c_adpt: ConvertToDfxOpInfo | ||
| 206 | + Note right of hcomm_c_adpt: Set isIndop_=true | ||
| 207 | + | ||
| 208 | + hcomm_c_adpt->>HcclCommDfx: GetMirrorTaskManager | ||
| 209 | + HcclCommDfx-->>hcomm_c_adpt: mirrorTaskManager | ||
| 210 | + hcomm_c_adpt->>MirrorTaskManager: SetCurrDfxOpInfo | ||
| 211 | + | ||
| 212 | + Note over HCCL,MirrorTaskManager: CCU task dispatch phase | ||
| 213 | + | ||
| 214 | + HcclCommDfx->>HcclCommDfx: AddTaskInfoCallback | ||
| 215 | + Note right of HcclCommDfx: Thread or Channel callback triggers. When taskType==TASK_CCU, traverse ccuDetailInfo to fill remoteRankId. | ||
| 216 | + HcclCommDfx->>HcclCommDfx: Create TaskInfo associated with DfxOpInfo | ||
| 217 | + HcclCommDfx->>MirrorTaskManager: AddTaskInfo | ||
| 218 | + Note right of MirrorTaskManager: Register with GlobalMirrorTasks | ||
| 219 | +``` | ||
| 220 | + | ||
| 221 | +### Exception Handling Flow | ||
| 222 | + | ||
| 223 | +#### Aicpu Mode Host-Side Exception Handling Flow | ||
| 224 | + | ||
| 225 | +```mermaid | ||
| 226 | +sequenceDiagram | ||
| 227 | + participant Runtime | ||
| 228 | + participant TaskExceptionHost | ||
| 229 | + participant MirrorTaskManager | ||
| 230 | + participant CollComm | ||
| 231 | + participant CcuTaskException | ||
| 232 | + participant Legacy | ||
| 233 | + participant hccp component | ||
| 234 | + participant error_manager component | ||
| 235 | + | ||
| 236 | + Runtime->>TaskExceptionHost: ProcessCallback | ||
| 237 | + Note right of TaskExceptionHost: Runtime exception callback, forwarded to Process | ||
| 238 | + | ||
| 239 | + alt IsMC2Exception | ||
| 240 | + TaskExceptionHost->>Legacy: TaskExceptionHandler::Process | ||
| 241 | + Note right of Legacy: Fall back to legacy flow | ||
| 242 | + else Non-MC2 exception | ||
| 243 | + TaskExceptionHost->>MirrorTaskManager: FindTaskInfo | ||
| 244 | + MirrorTaskManager-->>TaskExceptionHost: curTask | ||
| 245 | + | ||
| 246 | + alt Non-Indop task or commHandle not registered | ||
| 247 | + TaskExceptionHost->>Legacy: TaskExceptionHandler::Process | ||
| 248 | + Note right of Legacy: Fall back to legacy flow | ||
| 249 | + else Indop task and commHandle registered | ||
| 250 | + alt taskType==TASK_CCU | ||
| 251 | + TaskExceptionHost->>CcuTaskException: ProcessCcuException | ||
| 252 | + Note right of CcuTaskException: Forward to CCU exception handling flow | ||
| 253 | + else Non-CCU task | ||
| 254 | + TaskExceptionHost->>TaskExceptionHost: ProcessException | ||
| 255 | + Note right of TaskExceptionHost: Check hasAicpuReport_ | ||
| 256 | + | ||
| 257 | + alt hasAicpuReport_ is false | ||
| 258 | + TaskExceptionHost->>CollComm: GetAicpuTaskException | ||
| 259 | + CollComm-->>TaskExceptionHost: ErrorMessageReport | ||
| 260 | + end | ||
| 261 | + | ||
| 262 | + alt AICPU has reported error (tag is not empty) | ||
| 263 | + TaskExceptionHost->>TaskExceptionHost: HandleAicpuErrorReport | ||
| 264 | + Note right of TaskExceptionHost: hasAicpuReport_=true | ||
| 265 | + TaskExceptionHost->>TaskExceptionHost: Print error log | ||
| 266 | + Note right of TaskExceptionHost: BaseInfo/ParaInfo/GroupInfo/OpDataInfo | ||
| 267 | + TaskExceptionHost->>hccp component: PrintUbDfxInfo→PrintUbRegisters→RaGetAuxInfo | ||
| 268 | + Note right of hccp component: RaCtxGetAuxInfo reads UB registers | ||
| 269 | + TaskExceptionHost->>error_manager component: ReportErrorMsg→RPT_INPUT_ERR | ||
| 270 | + Note right of error_manager component: TASK_NOTIFY_WAIT→EI0002, UB/Write→EI0018 | ||
| 271 | + TaskExceptionHost->>TaskExceptionHost: GetAicpuCqeErrInfo | ||
| 272 | + Note right of TaskExceptionHost: Report to cluster monitoring, triggered when ubCqeStatus is non-zero | ||
| 273 | + else AICPU has no error reported | ||
| 274 | + TaskExceptionHost->>TaskExceptionHost: HandleHostErrorReport | ||
| 275 | + Note right of TaskExceptionHost: Host side prints error info independently, prints preceding tasks for TASK_NOTIFY_WAIT | ||
| 276 | + end | ||
| 277 | + end | ||
| 278 | + end | ||
| 279 | + end | ||
| 280 | +``` | ||
| 281 | + | ||
| 282 | +#### Aicpu Mode Device-Side Exception Handling Flow | ||
| 283 | + | ||
| 284 | +```mermaid | ||
| 285 | +sequenceDiagram | ||
| 286 | + participant AICPU background thread | ||
| 287 | + participant HcclCommTaskExceptionLite | ||
| 288 | + participant CollCommAicpu | ||
| 289 | + participant StreamLite | ||
| 290 | + participant TaskExceptionHost | ||
| 291 | + participant hal component | ||
| 292 | + | ||
| 293 | + AICPU background thread->>HcclCommTaskExceptionLite: Call | ||
| 294 | + | ||
| 295 | + alt stopCall_ is true | ||
| 296 | + Note right of HcclCommTaskExceptionLite: Skip to prevent flooding | ||
| 297 | + else stopCall_ is false | ||
| 298 | + HcclCommTaskExceptionLite->>HcclCommTaskExceptionLite: HandleExceptionCqe | ||
| 299 | + Note right of HcclCommTaskExceptionLite: Detect CQE exceptions in all communication domains | ||
| 300 | + | ||
| 301 | + loop Traverse all communication domains | ||
| 302 | + HcclCommTaskExceptionLite->>CollCommAicpu: GetAllThread | ||
| 303 | + loop Traverse all threads | ||
| 304 | + HcclCommTaskExceptionLite->>StreamLite: GetThreadCqe | ||
| 305 | + Note right of StreamLite: Get CQE through CqReportRecv | ||
| 306 | + StreamLite-->>HcclCommTaskExceptionLite: cqeException, cqeStatus | ||
| 307 | + | ||
| 308 | + alt Exception CQE exists | ||
| 309 | + HcclCommTaskExceptionLite->>HcclCommTaskExceptionLite: PrintTaskExceptionBySqeId | ||
| 310 | + Note right of HcclCommTaskExceptionLite: Print Task/EID/Group/OpData | ||
| 311 | + HcclCommTaskExceptionLite->>HcclCommTaskExceptionLite: ReportErrMsg | ||
| 312 | + | ||
| 313 | + alt IsErrorReported is false | ||
| 314 | + HcclCommTaskExceptionLite->>HcclCommTaskExceptionLite: GenerateErrorMessageReport | ||
| 315 | + HcclCommTaskExceptionLite->>TaskExceptionHost: SendErrorMessageReportToHost | ||
| 316 | + Note right of TaskExceptionHost: Report through HDC channel | ||
| 317 | + HcclCommTaskExceptionLite->>hal component: SendTaskExceptionByMBox→dlHalEschedSubmitEvent | ||
| 318 | + Note right of hal component: Mailbox reporting, <br/>See table below for error code conversion | ||
| 319 | + HcclCommTaskExceptionLite->>CollCommAicpu: SetErrorReported(true) | ||
| 320 | + end | ||
| 321 | + | ||
| 322 | + alt NotifyWait timeout (sqeType==PLACE_HOLDER) | ||
| 323 | + HcclCommTaskExceptionLite->>CollCommAicpu: PrintCommTaskException | ||
| 324 | + Note right of CollCommAicpu: Print all streams of the current communication domain | ||
| 325 | + loop Traverse other communication domains | ||
| 326 | + HcclCommTaskExceptionLite->>HcclCommTaskExceptionLite: PrintAllCommTaskException | ||
| 327 | + end | ||
| 328 | + end | ||
| 329 | + end | ||
| 330 | + end | ||
| 331 | + end | ||
| 332 | + | ||
| 333 | + alt HandleExceptionCqe fails | ||
| 334 | + Note right of HcclCommTaskExceptionLite: stopCall_=true, stop subsequent calls | ||
| 335 | + end | ||
| 336 | + end | ||
| 337 | +``` | ||
| 338 | + | ||
| 339 | +**SendTaskExceptionByMBox Error Code Conversion Table** (Source: `hcclCommTaskExceptionLite.cc:429-435`, constants defined in: `hcomm_task_scheduler_error.h`) | ||
| 340 | + | ||
| 341 | +| CQE sqeType | CQE errorCode | TS Error Code | Value | Description | | ||
| 342 | +|-------------|--------------|-----------|-----|------| | ||
| 343 | +| UB (sqeType=9) | 0x02 | TS_ERROR_HCCL_OP_UB_DDRC_FAILED | 0x3ea | UB local side returns ERROR | | ||
| 344 | +| UB (sqeType=9) | 0x03 | TS_ERROR_HCCL_OP_UB_POISON_FAILED | 0x3eb | UB remote side returns ERROR | | ||
| 345 | +| UB (sqeType=9) | 0x05 | TS_ERROR_HCCL_OP_UB_LINK_FAILED | 0x3ec | UB network exception, taack timeout | | ||
| 346 | +| UB (sqeType=9) | Other | TS_ERROR_HCCL_OTHER_ERROR | 0x223 | UB other error | | ||
| 347 | +| SDMA (sqeType=11) | 0x09 | TS_ERROR_SDMA_LINK_ERROR | 0x222 | SDMA write copy timeout acknowledgment or address decoding error | | ||
| 348 | +| SDMA (sqeType=11) | 0x0a | TS_ERROR_SDMA_POISON_ERROR | 0x221 | SDMA read copy timeout acknowledgment or read HBM returns ERROR | | ||
| 349 | +| SDMA (sqeType=11) | 0x08 | TS_ERROR_SDMA_DDRC_ERROR | 0x220 | SDMA read HBM returns ERROR | | ||
| 350 | +| SDMA (sqeType=11) | Other | TS_ERROR_HCCL_OTHER_ERROR | 0x223 | SDMA other error | | ||
| 351 | +| Other sqeType | - | TS_ERROR_HCCL_OTHER_ERROR | 0x223 | Non-UB or SDMA error | | ||
| 352 | + | ||
| 353 | +#### CCU Mode Exception Handling Flow | ||
| 354 | + | ||
| 355 | +```mermaid | ||
| 356 | +sequenceDiagram | ||
| 357 | + participant Runtime | ||
| 358 | + participant TaskExceptionHost | ||
| 359 | + participant MirrorTaskManager | ||
| 360 | + participant CcuTaskException | ||
| 361 | + participant CcuKernelMgr | ||
| 362 | + participant CcuComponent | ||
| 363 | + participant hccp component | ||
| 364 | + | ||
| 365 | + Runtime->>TaskExceptionHost: Process | ||
| 366 | + TaskExceptionHost->>MirrorTaskManager: FindTaskInfo | ||
| 367 | + MirrorTaskManager-->>TaskExceptionHost: curTask | ||
| 368 | + Note right of TaskExceptionHost: Dispatched when taskType==TASK_CCU | ||
| 369 | + | ||
| 370 | + TaskExceptionHost->>CcuTaskException: ProcessCcuException | ||
| 371 | + Note right of CcuTaskException: Print baseInfo/groupRank/opData | ||
| 372 | + | ||
| 373 | + CcuTaskException->>CcuKernelMgr: InitChannelMap | ||
| 374 | + Note right of CcuKernelMgr: GetInstance→GetKernel→GetChannels, build channelId→handle mapping | ||
| 375 | + | ||
| 376 | + loop Traverse ccuExDetailInfo.missionInfo | ||
| 377 | + CcuTaskException->>CcuTaskException: PrintCcuErrorInfo | ||
| 378 | + CcuTaskException->>CcuTaskException: PrintPanicLogInfo | ||
| 379 | + Note right of CcuTaskException: Print CCUM DFX registers | ||
| 380 | + end | ||
| 381 | + | ||
| 382 | + CcuTaskException->>CcuComponent: CleanTaskKillState | ||
| 383 | + CcuTaskException->>CcuComponent: CleanDieCkes | ||
| 384 | + | ||
| 385 | + Note over CcuTaskException,hccp component: PrintCcuErrorInfo internal flow | ||
| 386 | + | ||
| 387 | + CcuTaskException->>CcuTaskException: GetCcuErrorMsg | ||
| 388 | + Note right of CcuTaskException: Generate CcuErrorInfo list | ||
| 389 | + | ||
| 390 | + CcuTaskException->>hccp component: HccpRaCustomChannel | ||
| 391 | + Note right of hccp component: GetCcuMissionContext, read MissionContext registers | ||
| 392 | + hccp component-->>CcuTaskException: missionContext | ||
| 393 | + | ||
| 394 | + CcuTaskException->>CcuKernelMgr: GetKernel | ||
| 395 | + CcuKernelMgr-->>CcuTaskException: kernel | ||
| 396 | + | ||
| 397 | + CcuTaskException->>CcuKernelMgr: GetRepByInstrId | ||
| 398 | + Note right of CcuKernelMgr: kernel is cast to CcuRepContext, find the Rep corresponding to the exception instruction | ||
| 399 | + CcuKernelMgr-->>CcuTaskException: rep | ||
| 400 | + | ||
| 401 | + CcuTaskException->>CcuTaskException: GenStatusInfo | ||
| 402 | + Note right of CcuTaskException: Generate Mission status info, report EI0002 | ||
| 403 | + | ||
| 404 | + alt LOOPGROUP | ||
| 405 | + CcuTaskException->>CcuTaskException: GenErrorInfoLoopGroup | ||
| 406 | + Note right of CcuTaskException: Parse LoopGroup→traverse Loop→GenErrorInfoLoop | ||
| 407 | + else LOC_WAIT_EVENT/LOC_WAIT_NOTIFY | ||
| 408 | + CcuTaskException->>CcuTaskException: GenErrorInfoByRepType | ||
| 409 | + Note right of CcuTaskException: Check CKE expected value vs actual value | ||
| 410 | + else Other Rep | ||
| 411 | + CcuTaskException->>CcuTaskException: GenErrorInfoByRepType | ||
| 412 | + Note right of CcuTaskException: HANDLER_MAP dispatches to Read/Write and other handlers | ||
| 413 | + end | ||
| 414 | + | ||
| 415 | + CcuTaskException->>CcuTaskException: PrintCcuErrorLog | ||
| 416 | + Note right of CcuTaskException: Print error info by CcuErrorType | ||
| 417 | + | ||
| 418 | + alt UB error (missionStatus 0x01~0x05) | ||
| 419 | + CcuTaskException->>hccp component: PrintCcuUbRegisters→RaBatchQueryJettyStatus | ||
| 420 | + Note right of hccp component: Query Jetty status | ||
| 421 | + CcuTaskException->>hccp component: PrintUbRegisters→RaGetAuxInfo | ||
| 422 | + Note right of hccp component: RaCtxGetAuxInfo reads UB registers | ||
| 423 | + alt repType is READ/WRITE/BUF_READ/BUF_WRITE | ||
| 424 | + CcuTaskException->>CcuTaskException: GetCcuCqeErrorInfo | ||
| 425 | + Note right of CcuTaskException: ClusterMoniterGetCcuCqeErrInfo, report to cluster monitoring | ||
| 426 | + end | ||
| 427 | + end | ||
| 428 | +``` | ||
| 429 | + | ||
| 430 | +--- | ||
| 431 | + | ||
| 432 | +## Interface Description (Class Diagram) | ||
| 433 | + | ||
| 434 | +```mermaid | ||
| 435 | +classDiagram | ||
| 436 | + class TaskExceptionHost { | ||
| 437 | + -mutex taskExceptionMutex_ | ||
| 438 | + -unordered_set~u64~ CommRegisterMap_ | ||
| 439 | + -bool hasAicpuReport_ | ||
| 440 | + +GetInstance(s32) TaskExceptionHost*$ | ||
| 441 | + +Register(u64) HcclResult | ||
| 442 | + +UnRegister(u64) HcclResult | ||
| 443 | + +ProcessCallback(rtExceptionInfo_t*) void$ | ||
| 444 | + -Process(rtExceptionInfo_t*) void | ||
| 445 | + -HandleAicpuErrorReport(rtExceptionInfo_t*, ErrorMessageReport, TaskInfo) void | ||
| 446 | + -HandleHostErrorReport(rtExceptionInfo_t*, TaskInfo) void | ||
| 447 | + -ReportErrorMsg(TaskInfo, string, ErrorMessageReport, rtExceptionInfo_t*) void | ||
| 448 | + -GetGroupRankInfo(TaskInfo) string | ||
| 449 | + -ProcessException(rtExceptionInfo_t*, TaskInfo) void | ||
| 450 | + -PrintTaskContextInfo(uint32_t, uint32_t, uint32_t) void | ||
| 451 | + -PrintUbDfxInfo(rtExceptionInfo_t*, ErrorMessageReport) void | ||
| 452 | + -PrintGroupErrorMessage(...) void | ||
| 453 | + -PrintOpDataErrorMessage(...) void | ||
| 454 | + -PrintUbRegisters(s32, RdmaHandle) HcclResult | ||
| 455 | + -GetAicpuCqeErrInfo(...) void | ||
| 456 | + -ClusterMoniterGetAicpuCqeErrInfo(...) void | ||
| 457 | + -GetAicpuCqeErrRemoteLocalIdByRankId(CollComm, uint32_t, u32&) void | ||
| 458 | + -GetAicpuCqeErrNetInstanceByRankId(CollComm, uint32_t, string&) void | ||
| 459 | + } | ||
| 460 | + | ||
| 461 | + class CcuTaskException { | ||
| 462 | + +ProcessCcuException(rtExceptionInfo_t*, TaskInfo) void$ | ||
| 463 | + -InitChannelMap(s32, u64) HcclResult$ | ||
| 464 | + -GetCcuErrorMsg(deviceId, missionStatus, ParaCcu, vector~CcuErrorInfo~&) HcclResult$ | ||
| 465 | + -GenErrorInfoByRepType(ErrorInfoBase, CcuRepBase*, vector~CcuErrorInfo~&) void$ | ||
| 466 | + -GenErrorInfoLoop(ErrorInfoBase, CcuRepContext&, vector~CcuErrorInfo~&) HcclResult$ | ||
| 467 | + -GenErrorInfoLoopGroup(ErrorInfoBase, CcuRepBase*, CcuRepContext&, vector~CcuErrorInfo~&) HcclResult$ | ||
| 468 | + -GenStatusInfo(ErrorInfoBase, vector~CcuErrorInfo~&) void$ | ||
| 469 | + -PrintCcuErrorInfo(...) void$ | ||
| 470 | + -PrintCcuErrorLog(...) void$ | ||
| 471 | + -PrintCcuUbRegisters(...) HcclResult$ | ||
| 472 | + -GetCcuJettys(...) HcclResult$ | ||
| 473 | + -PrintUbRegisters(s32, RdmaHandle) HcclResult$ | ||
| 474 | + -GetCcuErrorMsgByType(...) string$ | ||
| 475 | + -GetCcuCqeErrorInfo(...) void$ | ||
| 476 | + -GetRankIdByChannelId(...) RankId$ | ||
| 477 | + -GetAddrPairByChannelId(...) pair$ | ||
| 478 | + } | ||
| 479 | + | ||
| 480 | + class HcclCommTaskExceptionLite { | ||
| 481 | + -bool stopCall_ | ||
| 482 | + -u32 devId_ | ||
| 483 | + -MirrorTaskManager* mirrorTaskManager_ | ||
| 484 | + -unordered_map threadsPrinted_ | ||
| 485 | + +GetInstance() HcclCommTaskExceptionLite&$ | ||
| 486 | + +Init(u32) void | ||
| 487 | + +Call() void | ||
| 488 | + +PrintAllCommTaskException() HcclResult | ||
| 489 | + +PrintCommTaskException(CollCommAicpu) HcclResult | ||
| 490 | + -HandleExceptionCqe() HcclResult | ||
| 491 | + -GetThreadCqe(thread, cqeException, cqeStatus) HcclResult | ||
| 492 | + -ProcessCqe(aicpuComm, exceptionInfo, cqeStatus, aicpuCommInfo) HcclResult | ||
| 493 | + -ReportErrMsg(aicpuComm, exceptionInfo) HcclResult | ||
| 494 | + -GenerateErrorMessageReport(...) HcclResult | ||
| 495 | + -GenerateTaskErrMsg(...) void | ||
| 496 | + -FillNotifyErrMsg(...) void | ||
| 497 | + -FillReduceErrMsg(...) void | ||
| 498 | + -FillDmaErrMsg(...) void | ||
| 499 | + -FillSdmaErrMsg(...) void | ||
| 500 | + -FillUbErrMsg(...) void | ||
| 501 | + -FillReduceInlineErrMsg(...) void | ||
| 502 | + -SendTaskExceptionByMBox(notifyId, tsId, exceptionInfo) HcclResult | ||
| 503 | + -SwitchUBCqeErrCodeToTsErrCode(u32) uint16_t | ||
| 504 | + -SwitchSdmaCqeErrCodeToTsErrCode(u32) uint16_t | ||
| 505 | + -PrintTaskExceptionBySqeId(...) HcclResult | ||
| 506 | + -PrintTaskContextInfo(...) HcclResult | ||
| 507 | + -CollectTaskContext(...) HcclResult | ||
| 508 | + -PrintEid(TaskInfo) void | ||
| 509 | + -GetGroupInfo(CollCommAicpu) string | ||
| 510 | + -GetSqeId(uint16_t, uint16_t) u32 | ||
| 511 | + } | ||
| 512 | + | ||
| 513 | + class CcuErrorInfo { | ||
| 514 | + +CcuErrorType type | ||
| 515 | + +CcuRepType repType | ||
| 516 | + +uint8_t dieId | ||
| 517 | + +uint8_t missionId | ||
| 518 | + +uint16_t instrId | ||
| 519 | + +union msg | ||
| 520 | + +SetBaseInfo(...) void | ||
| 521 | + } | ||
| 522 | + | ||
| 523 | + class ErrorInfoBase { | ||
| 524 | + +int32_t deviceId | ||
| 525 | + +uint8_t dieId | ||
| 526 | + +uint8_t missionId | ||
| 527 | + +uint16_t currentInsId | ||
| 528 | + +uint16_t status | ||
| 529 | + } | ||
| 530 | + | ||
| 531 | + class CcuLoopContext { | ||
| 532 | + +GetCurrentIns() uint16_t | ||
| 533 | + +GetCurrentCnt() uint16_t | ||
| 534 | + +GetAddrStride() uint32_t | ||
| 535 | + } | ||
| 536 | + | ||
| 537 | + class CcuMissionContext { | ||
| 538 | + +GetStatus() uint16_t | ||
| 539 | + +GetCurrentIns() uint16_t | ||
| 540 | + +GetStartIns() uint16_t | ||
| 541 | + +GetEndIns() uint16_t | ||
| 542 | + } | ||
| 543 | + | ||
| 544 | + class CcuComponent { | ||
| 545 | + +GetInstance(devLogicId) CcuComponent$ | ||
| 546 | + +CleanTaskKillState() HcclResult | ||
| 547 | + +CleanDieCkes(dieId) HcclResult | ||
| 548 | + } | ||
| 549 | + | ||
| 550 | + TaskExceptionHost ..> CcuTaskException : dispatched when TASK_CCU | ||
| 551 | + HcclCommTaskExceptionLite ..> TaskExceptionHost : HDC channel reporting | ||
| 552 | + HcclCommTaskExceptionLite ..> hal component : dlHalEschedSubmitEvent | ||
| 553 | + TaskExceptionHost ..> hccp component : RaGetAuxInfo/HccpRaCustomChannel | ||
| 554 | + TaskExceptionHost ..> error_manager component : RPT_INPUT_ERR | ||
| 555 | + CcuTaskException ..> CcuComponent : cleanup status | ||
| 556 | + CcuTaskException ..> hccp component : HccpRaCustomChannel/RaGetAuxInfo | ||
| 557 | + CcuTaskException ..> CcuErrorInfo : generates | ||
| 558 | + CcuTaskException ..> ErrorInfoBase : uses | ||
| 559 | + CcuTaskException ..> CcuLoopContext : reads registers | ||
| 560 | + CcuTaskException ..> CcuMissionContext : reads registers | ||
| 561 | +``` | ||
| 562 | + | ||
| 563 | +--- | ||
| 564 | + | ||
| 565 | +## Interface Description | ||
| 566 | + | ||
| 567 | +### TaskExceptionHost | ||
| 568 | + | ||
| 569 | +| Interface | Type | Parameters | Return Value | Description | | ||
| 570 | +|------|------|------|--------|----------| | ||
| 571 | +| `GetInstance(s32)` | Public static | [in] deviceLogicID | `TaskExceptionHost*` | Gets the exception handler for the specified device. Supports up to 65 devices. | | ||
| 572 | +| `Register(u64)` | Public | [in] commHandle | `HcclResult` | Writes commHandle to CommRegisterMap_. The first comm registration calls `aclrtSetExceptionInfoCallback(ProcessCallback)` to register the exception callback with RTS. | | ||
| 573 | +| `UnRegister(u64)` | Public | [in] commHandle | `HcclResult` | Removes commHandle from CommRegisterMap_. When the last comm is unregistered, sets the callback to nullptr. | | ||
| 574 | +| `ProcessCallback(rtExceptionInfo_t*)` | Public static | [in] exceptionInfo | void | Runtime exception callback entry point. Obtains the handler through GetInstance and forwards to Process. | | ||
| 575 | +| `HandleAicpuErrorReport(rtExceptionInfo_t*, const ErrorMessageReport&, const TaskInfo&)` | Private | [in] exceptionInfo, [in] errorMessage, [in] taskInfo | void | Handles errors already reported by the AICPU side: prints BaseInfo/ParaInfo/GroupInfo/OpDataInfo, calls PrintUbDfxInfo, ReportErrorMsg. When ubCqeStatus is non-zero, calls GetAicpuCqeErrInfo. | | ||
| 576 | +| `HandleHostErrorReport(rtExceptionInfo_t*, const TaskInfo&)` | Private | [in] exceptionInfo, [in] taskInfo | void | Host-side independent exception handling: prints preceding task context for TASK_NOTIFY_WAIT and reports EI0002, prints cluster monitoring error info. | | ||
| 577 | +| `ReportErrorMsg(const TaskInfo&, const string&, const ErrorMessageReport&, rtExceptionInfo_t*)` | Private | [in] exceptionTaskInfo, [in] groupRankContent, [in] errorMessage, [in] exceptionInfo | void | Reports different error codes based on task type: TASK_NOTIFY_WAIT→EI0002, UB/Write type→EI0018. | | ||
| 578 | +| `ProcessException(rtExceptionInfo_t*, const TaskInfo&)` | Private | [in] exceptionInfo, [in] taskInfo | void | General exception handling entry point: checks hasAicpuReport_, obtains ErrorMessageReport through CollComm::GetAicpuTaskException if not reported, dispatches to HandleAicpuErrorReport or HandleHostErrorReport. | | ||
| 579 | +| `PrintTaskContextInfo(uint32_t, uint32_t, uint32_t)` | Private | [in] deviceId, [in] streamId, [in] taskId | void | Prints up to 50 preceding task context entries before the exception task. | | ||
| 580 | +| `PrintUbDfxInfo(rtExceptionInfo_t*, const ErrorMessageReport&)` | Private | [in] exceptionInfo, [in] errorMessage | void | For UB type tasks (TASK_WRITE_WITH_NOTIFY/TASK_UB, and so on), prints UB CQE status and EID information, and reads UB DFX registers. | | ||
| 581 | +| `PrintUbRegisters(s32, RdmaHandle)` | Private | [in] devLogicId, [in] rdmaHandle | `HcclResult` | Reads and prints UB DFX register information through RaGetAuxInfo. | | ||
| 582 | +| `GetGroupRankInfo(const TaskInfo&)` | Private | [in] taskInfo | string | Gets group/rankSize/rankId information from TaskInfo. | | ||
| 583 | +| `GetAicpuCqeErrRemoteLocalIdByRankId(CollComm*, uint32_t, u32&)` | Private | [in] collComm, [in] rankid, [out] remoteLocalId | void | Gets the LocalId corresponding to the remote rank through RankGraph. | | ||
| 584 | +| `GetAicpuCqeErrNetInstanceByRankId(CollComm*, uint32_t, string&)` | Private | [in] collComm, [in] rankid, [out] netInstanceId | void | Gets the NetInstanceId corresponding to the remote rank through RankGraph. | | ||
| 585 | + | ||
| 586 | +### CcuTaskException | ||
| 587 | + | ||
| 588 | +| Interface | Type | Parameters | Return Value | Description | | ||
| 589 | +|------|------|------|--------|----------| | ||
| 590 | +| `ProcessCcuException(const rtExceptionInfo_t*, const TaskInfo&)` | Public static | [in] exceptionInfo, [in] taskInfo | void | Main entry point for CCU task exception handling: initializes channel mapping, traverses Missions, prints errors and registers, cleans up TaskKill status. | | ||
| 591 | +| `InitChannelMap(s32, u64)` | Private static | [in] deviceId, [in] ccuKernelHandle | `HcclResult` | Initializes the global channelId→channelHandle mapping `g_channelIdToHandle`. | | ||
| 592 | +| `GetCcuErrorMsg(int32_t, uint16_t, const ParaCcu&, vector<CcuErrorInfo>&)` | Private static | [in] deviceId, [in] missionStatus, [in] ccuTaskParam, [out] errorInfo | `HcclResult` | Core method for obtaining CCU error information: reads MissionContext, finds the exception Rep, dispatches to different GenErrorInfo methods. | | ||
| 593 | +| `GenErrorInfoByRepType(const ErrorInfoBase&, shared_ptr<CcuRepBase>, vector<CcuErrorInfo>&)` | Private static | [in] baseInfo, [in] repBase, [out] errorInfo | void | Dispatches to the corresponding GenErrorInfo method based on CcuRepType (uses HANDLER_MAP function table). | | ||
| 594 | +| `GenErrorInfoLoop(const ErrorInfoBase&, CcuRepContext&, vector<CcuErrorInfo>&)` | Private static | [in] baseInfo, [in] ctx, [out] errorInfo | `HcclResult` | Parses Loop type exceptions: reads LoopContext registers, recursively parses Reps within the Loop. | | ||
| 595 | +| `GenErrorInfoLoopGroup(const ErrorInfoBase&, shared_ptr<CcuRepBase>, CcuRepContext&, vector<CcuErrorInfo>&)` | Private static | [in] baseInfo, [in] repBase, [in] ctx, [out] errorInfo | `HcclResult` | Parses LoopGroup type exceptions: expands all Loops and parses them individually. | | ||
| 596 | +| `PrintCcuUbRegisters(const vector<CcuErrorInfo>&, s32, const TaskInfo&)` | Private static | [in] errorInfos, [in] devLogicId, [in] taskInfo | `HcclResult` | Gets CCU Jetty status and prints UB registers for error Jetties. | | ||
| 597 | +| `GetCcuJettys(const CcuErrorInfo&, pair<CcuChannelInfo, vector<CcuJetty*>>&)` | Private static | [in] errorInfo, [out] ctx | `HcclResult` | Gets channelId from CcuErrorInfo and obtains Jetties through the channel→endpoint→ctxPool chain. | | ||
| 598 | +| `GetCcuCqeErrorInfo(const CcuErrorInfo&, const TaskInfo&, u32, uint8_t)` | Private static | [in] ccuErrorInfo, [in] taskInfo, [in] locDeviceId, [in] missionStatus | void | Gets remoteRankId and NetInstanceId through channelId, calls ClusterMoniterGetCcuCqeErrInfo to report to cluster monitoring. | | ||
| 599 | + | ||
| 600 | +### HcclCommTaskExceptionLite | ||
| 601 | + | ||
| 602 | +| Interface | Type | Parameters | Return Value | Description | | ||
| 603 | +|------|------|------|--------|----------| | ||
| 604 | +| `GetInstance()` | Public static | None | `HcclCommTaskExceptionLite&` | Gets the singleton instance. | | ||
| 605 | +| `Init(u32)` | Public | [in] devId | void | Initializes the device ID. | | ||
| 606 | +| `Call()` | Public | None | void | Daemon thread callback entry point. Calls HandleExceptionCqe. Sets stopCall_ on failure to prevent flooding. | | ||
| 607 | +| `HandleExceptionCqe()` | Private | None | `HcclResult` | Traverses all threads in all communication domains, detects and handles CQE exceptions. | | ||
| 608 | +| `GetThreadCqe(Thread*, rtLogicCqReport_t&, CqeStatus&)` | Private | [in] thread, [out] cqeException, [out] cqeStatus | `HcclResult` | Gets CQE exception information for the specified thread through CqReportRecv. | | ||
| 609 | +| `ProcessCqe(CollCommAicpu*, const rtLogicCqReport_t&, const CqeStatus&, const vector<pair<string,CollCommAicpuMgr*>>&)` | Private | [in] aicpuComm, [in] exceptionInfo, [in] cqeStatus, [in] aicpuCommInfo | `HcclResult` | Processes CQE exceptions: prints TaskException, reports to Host, prints all communication domain information on NotifyWait timeout. | | ||
| 610 | +| `ReportErrMsg(CollCommAicpu*, const rtLogicCqReport_t&)` | Private | [in] aicpuComm, [in] exceptionInfo | `HcclResult` | Checks IsErrorReported. If not reported, generates ErrorMessageReport, reports to Host through HDC, notifies TSFW through Mailbox, and sets ErrorReported(true). | | ||
| 611 | +| `GenerateErrorMessageReport(CollCommAicpu*, const TaskInfo&, const rtLogicCqReport_t&, ErrorMessageReport&)` | Private | [in] aicpuComm, [in] taskInfo, [in] exceptionInfo, [out] errMsgInfo | `HcclResult` | Fills common fields of ErrorMessageReport based on task information, then calls GenerateTaskErrMsg to fill task-type-specific fields. | | ||
| 612 | +| `GenerateTaskErrMsg(const TaskInfo&, ErrorMessageReport&, const rtLogicCqReport_t&)` | Private | [in] taskInfo, [out] errMsgInfo, [in] exceptionInfo | void | Dispatches to FillNotifyErrMsg/FillReduceErrMsg/FillDmaErrMsg/FillUbErrMsg/FillSdmaErrMsg/FillReduceInlineErrMsg based on taskType. | | ||
| 613 | +| `FillNotifyErrMsg(const TaskInfo&, ErrorMessageReport&)` | Private | [in] taskInfo, [out] errMsgInfo | void | Fills notifyId and notifyValue for NOTIFY_WAIT/NOTIFY_RECORD types. | | ||
| 614 | +| `FillReduceErrMsg(const TaskInfo&, ErrorMessageReport&, const rtLogicCqReport_t&)` | Private | [in] taskInfo, [out] errMsgInfo, [in] exceptionInfo | void | Fills reduceOp, notifyId, locEid, rmtEid, ubCqeStatus, and so on for UB_REDUCE_INLINE/WRITE_REDUCE_WITH_NOTIFY types. | | ||
| 615 | +| `FillDmaErrMsg(const TaskInfo&, ErrorMessageReport&, const rtLogicCqReport_t&)` | Private | [in] taskInfo, [out] errMsgInfo, [in] exceptionInfo | void | Fills for UB_INLINE_WRITE/WRITE_WITH_NOTIFY types. Internally calls FillUbErrMsg. | | ||
| 616 | +| `FillUbErrMsg(const TaskInfo&, ErrorMessageReport&, const rtLogicCqReport_t&)` | Private | [in] taskInfo, [out] errMsgInfo, [in] exceptionInfo | void | Fills locEid, rmtEid, ubCqeStatus, linkType, size, and so on for UB types. | | ||
| 617 | +| `FillSdmaErrMsg(const TaskInfo&, ErrorMessageReport&)` | Private | [in] taskInfo, [out] errMsgInfo | void | Fills linkType, size, srcAddr, dstAddr for SDMA types. | | ||
| 618 | +| `FillReduceInlineErrMsg(const TaskInfo&, ErrorMessageReport&)` | Private | [in] taskInfo, [out] errMsgInfo | void | Fills reduceOp for REDUCE_INLINE types. | | ||
| 619 | +| `SendTaskExceptionByMBox(u32, u32, const rtLogicCqReport_t&)` | Private | [in] notifyId, [in] tsId, [in] exceptionInfo | `HcclResult` | Reports task exception events to TSFW through Mailbox, including error code conversion (UB/SDMA error codes to TS error codes). | | ||
| 620 | +| `SwitchUBCqeErrCodeToTsErrCode(u32)` | Private | [in] cqeErrCode | `uint16_t` | Converts UB CQE error codes to TS error codes. | | ||
| 621 | +| `SwitchSdmaCqeErrCodeToTsErrCode(u32)` | Private | [in] cqeErrCode | `uint16_t` | Converts SDMA CQE error codes to TS error codes. | | ||
| 622 | +| `PrintAllCommTaskException()` | Public | None | `HcclResult` | Prints all Task exception information for all communication domains. | | ||
| 623 | +| `PrintCommTaskException(CollCommAicpu*)` | Public | [in] aicpuComm | `HcclResult` | Prints Task exception information for all threads in the specified communication domain. | | ||
| 624 | +| `PrintTaskExceptionBySqeId(CollCommAicpu*, u32, u32)` | Private | [in] aicpuComm, [in] sqId, [in] sqeId | `HcclResult` | Prints Task exception information for the specified sqId/sqeId: BaseInfo/ParaInfo, EID, GroupInfo, OpData/TaskContext. | | ||
| 625 | +| `PrintTaskContextInfo(CollCommAicpu*, u32, u32)` | Private | [in] aicpuComm, [in] sqId, [in] taskId | `HcclResult` | Prints up to 50 preceding task context entries before the exception task (printed in segments by opIndex). | | ||
| 626 | +| `CollectTaskContext(CollCommAicpu*, u32, u32, vector<shared_ptr<TaskInfo>>&)` | Private | [in] aicpuComm, [in] sqId, [in] taskId, [out] taskContext | `HcclResult` | Collects up to 50 task entries before the exception task from the MirrorTaskManagerLite queue. | | ||
| 627 | +| `PrintEid(const TaskInfo&)` | Private | [in] taskInfo | void | Prints localEid and remoteEid for UB type tasks. | | ||
| 628 | +| `GetGroupInfo(CollCommAicpu*)` | Private | [in] aicpuComm | string | Gets the group/rankSize/localRank information of the communication domain. | | ||
| 629 | +| `GetSqeId(uint16_t, uint16_t)` | Private | [in] taskId, [in] streamId | `u32` | Combines taskId and streamId into sqeId. | | ||
| 630 | + | ||
| 631 | +### Global Callback Registration Interfaces | ||
| 632 | + | ||
| 633 | +| Interface | Parameters | Description | | ||
| 634 | +|------|------|----------| | ||
| 635 | +| `RegisterGetAicpuCqeErrInfoCallBackHcomm(callback)` | AICPU CQE error info callback | Registers the callback for AICPU CQE error information reporting to cluster monitoring. | | ||
| 636 | +| `RegisterAicpuGetErrStatusVecCallBack(callback)` | AICPU error status vector callback | Registers the callback for getting the AICPU-side abnormal device status list. | | ||
| 637 | +| `RegisterGetCcuCqeErrInfoCallBackHcomm(callback)` | CCU CQE error info callback | Registers the callback for CCU CQE error information reporting to cluster monitoring. | | ||
| 638 | +| `RegisterCcuGetErrStatusVecCallBack(callback)` | CCU error status vector callback | Registers the callback for getting the CCU-side abnormal device status list. | | ||
| 639 | + | ||
| 640 | +--- | ||
| 641 | + | ||
| 642 | +## Usage Limitations | ||
| 643 | + | ||
| 644 | +### Supported Scenarios | ||
| 645 | + | ||
| 646 | +| Chip | Mode | Host Side | AICPU Side | Description | | ||
| 647 | +|------|------|---------|----------|------| | ||
| 648 | +| Ascend 950PR/Ascend 950DT | AICPU | Supported | Supported | Indop tasks use the new flow; non-Indop tasks fall back to legacy TaskExceptionHandler. | | ||
| 649 | +| Ascend 950PR/Ascend 950DT | CCU | Supported | Not applicable | TASK_CCU type is handled by CcuTaskException. | | ||
| 650 | +| Ascend 950PR/Ascend 950DT | MC2 | Fall back to legacy | Not applicable | Falls back to TaskExceptionHandler under the legacy directory. | | ||
| 651 | + | ||
| 652 | +### Constraint Specifications | ||
| 653 | + | ||
| 654 | +1. **Device count limit**: Supports up to 65 devices (`MAX_MODULE_DEVICE_NUM_V2 = 65`). `TaskExceptionHostManager::GetHandler` returns nullptr when devId >= 65. | ||
| 655 | +2. **CCU message length**: CCU transfer length must not exceed 256MB (`CCU_MSG_256MB_LEN`). A warning is printed when exceeded. | ||
| 656 | +3. **AICPU duplicate report prevention**: The Host side uses the `hasAicpuReport_` flag to prevent the same TaskExceptionHost from repeatedly handling AICPU-reported errors. The AICPU side uses the `CollCommAicpu::IsErrorReported()` flag to prevent duplicate reporting from the same communication domain. | ||
| 657 | +4. **Host-side comm registration management**: Uses `CommRegisterMap_` to manage commHandle registration. The Process callback validates whether the commHandle is registered. The RTS callback is cleared when the last comm is unregistered. | ||
| 658 | +5. **AICPU-side flooding prevention**: `HcclCommTaskExceptionLite` sets `stopCall_=true` after `HandleExceptionCqe` fails, stopping subsequent calls. | ||
| 659 | +6. **CQE error information retrieval**: The CCU side uses the `isGetCqeErrInfo` atomic flag to avoid redundant CQE error information retrieval. | ||
| 660 | +7. **Cluster monitoring report limit**: The abnormal device information list is limited to up to 3 entries (`maxListSize = 3`). | ||
| 661 | +8. **Task context print limit**: Prints up to 50 task entries before the exception task (`TASK_CONTEXT_SIZE = 50`). The single print length does not exceed `TASK_CONTEXT_INFO_SIZE`. | ||
| 662 | +9. **CCU Mission count**: Currently `ccuMissionNum` is 1, only handling a single Mission exception. | ||
| 663 | +10. **CCU instruction backtracking limit**: Prints up to 10 instructions before the error instruction (`loopUpInstrNum = 10`). | ||
| 664 | +11. **CCU Loop expansion**: Instructions within a Loop are recursively expanded through `GenErrorInfoLoop`. LoopGroup expands all Loops through `GenErrorInfoLoopGroup`. | ||
| 665 | +12. **Error code conversion**: AICPU-side UB and SDMA error codes must be converted to TS error codes before reporting through Mailbox. Unrecognized error codes are uniformly converted to `TS_ERROR_HCCL_OTHER_ERROR`. | ||
| 666 | +13. **Thread safety**: `g_communicatorCallbackMapV2` and `g_channelIdToHandle` are both protected by mutexes for concurrent access. | ||
| @@ -0,0 +1,37 @@ | |||
| 1 | +# Topo Address Info | ||
| 2 | + | ||
| 3 | +## Introduction | ||
| 4 | + | ||
| 5 | +In the Ascend 950 chip generation, the hyperplane uses the Unified Bus for networking. Multiple different topology networking methods are used across different product forms. This module is used to discover the endpoint addresses of each edge under different topologies. | ||
| 6 | + | ||
| 7 | +## Networking Introduction | ||
| 8 | + | ||
| 9 | +Two main types of networking are used: MESH and CLOS. For related information, refer to the paper: https://arxiv.org/abs/2503.20377 | ||
| 10 | + | ||
| 11 | +### MESH Networking | ||
| 12 | + | ||
| 13 | +Each NPU has a direct physical link to every other NPU, so there is a pair of independent communication addresses. For example, if there are 8 NPUs on the same NPU board, there are 8 * 7 / 2 = 28 physical paths, that is, 28 pairs of communication addresses. | ||
| 14 | + | ||
| 15 | +### CLOS Networking | ||
| 16 | + | ||
| 17 | +Any two NPUs communicate through a switch chip. Therefore, one NPU requires one address. In common networking scenarios, due to reliability and other reasons, CLOS networking can usually be divided into multiple planes, with each plane corresponding to one address. For example, in a liquid-cooled POD, NPUs use two independent logical ports to connect to two separate network planes. | ||
| 18 | + | ||
| 19 | +## Networking Planning | ||
| 20 | + | ||
| 21 | +### Network Layer Description | ||
| 22 | + | ||
| 23 | +In the 950 chip generation, the network is divided into multiple layers based on communication quality and range. | ||
| 24 | + | ||
| 25 | +| Network Layer | Description | | ||
| 26 | +|:-------| :-----------| | ||
| 27 | +| 0 | Highest communication quality and lowest latency. Mostly MESH networking, mainly the fullmesh network within the same NPU board and the box-level network in POD form. | | ||
| 28 | +| 1 | Second-highest communication quality and medium latency. CLOS networking with a larger communication range, at the super node level, still within the scale-up range. | | ||
| 29 | +| 2 | Lowest communication quality. CLOS networking with the entire cluster as the communication range. Mainly scale-out networks such as ROCE or UBOE. | | ||
| 30 | + | ||
| 31 | +### Network Address Description | ||
| 32 | + | ||
| 33 | +| Network Layer | Description | | ||
| 34 | +|:-------| :-----------| | ||
| 35 | +| 0 | Because MESH networking is the primary method, there are multiple pairs of communication addresses. In topo_addr_info, this is expressed as the address of each port on each NPU. The address type is EID. | | ||
| 36 | +| 1 | Fill in the address based on the networking plane. In multi-plane networking, the number of addresses is the same as the number of planes. Traffic is distributed among different planes during collective communication. The address type is EID. | | ||
| 37 | +| 2 | The address planning is the same as layer 1. The address type is IP address. | | ||
| @@ -0,0 +1,101 @@ | |||
| 1 | +# HCCL LLT | ||
| 2 | + | ||
| 3 | +## Overview | ||
| 4 | + | ||
| 5 | +HCCL LLT (Low Level Test) is the test framework for HCCL, designed to systematically verify the functional completeness and performance stability of HCCL components at all levels. LLT covers multiple parts of HCCL including the algorithm layer, framework layer, platform layer, and external interface layer. Through comprehensive test cases, it ensures the reliability and efficiency of HCCL in various business scenarios. | ||
| 6 | + | ||
| 7 | +## Directory Structure | ||
| 8 | + | ||
| 9 | +```text | ||
| 10 | +test/ | ||
| 11 | +├── legacy # Historical version compatibility test framework | ||
| 12 | +│ ├── common # Common utilities | ||
| 13 | +│ ├── depends # Test dependencies on other component headers | ||
| 14 | +│ ├── st # ST integration test cases | ||
| 15 | +│ │ ├── algorithm # Communication algorithm test cases | ||
| 16 | +│ │ ├── fwk # Communication framework test cases | ||
| 17 | +│ │ ├── service # Service layer test cases | ||
| 18 | +│ │ └── test_case # Test cases | ||
| 19 | +│ └── ut # UT integration test cases | ||
| 20 | +│ ├── aicpu # AICPU-specific test cases | ||
| 21 | +│ ├── all_source_code # Source code file paths | ||
| 22 | +│ ├── common # Common utilities | ||
| 23 | +│ ├── framework # Communication framework test cases | ||
| 24 | +│ ├── service # Service layer test cases | ||
| 25 | +│ └── unified_platform # Unified platform layer test cases | ||
| 26 | +├── st/algorithm # ST integration test cases (algorithm analyzer) | ||
| 27 | +│ ├── testcase # Test cases | ||
| 28 | +│ └── utils # Common utilities | ||
| 29 | +└── ut # UT unit test cases | ||
| 30 | + ├── aicpu_kfc # MC2-related tests | ||
| 31 | + ├── common # Common utilities | ||
| 32 | + ├── depends # Test dependencies on other component headers | ||
| 33 | + ├── device # Device tests | ||
| 34 | + ├── framework # Communication framework test cases | ||
| 35 | + ├── impl # Communication algorithm implementation test cases | ||
| 36 | + ├── inter # Interface adaptation layer test cases | ||
| 37 | + ├── misc # Miscellaneous test cases | ||
| 38 | + ├── platform # Communication platform implementation test cases | ||
| 39 | + └── stub # Test stub functions | ||
| 40 | +``` | ||
| 41 | + | ||
| 42 | +## Build and Run | ||
| 43 | + | ||
| 44 | +Execute the following commands from the repository root directory: | ||
| 45 | + | ||
| 46 | +```bash | ||
| 47 | +# Build and run all unit test cases | ||
| 48 | +bash build.sh --ut | ||
| 49 | + | ||
| 50 | +# Build and run all integration test cases | ||
| 51 | +bash build.sh --st | ||
| 52 | +# Build and run individual test suite cases | ||
| 53 | +bash build.sh --open_hccl_test | ||
| 54 | +bash build.sh --executor_hccl_test | ||
| 55 | +bash build.sh --executor_reduce_hccl_test | ||
| 56 | +bash build.sh --executor_pipeline_hccl_test | ||
| 57 | +# Manually execute test cases | ||
| 58 | +./build/test/st/algorithm/testcase/testcase/open_hccl_test | ||
| 59 | +./build/test/st/algorithm/testcase/testcase/executor_hccl_test | ||
| 60 | +./build/test/st/algorithm/testcase/testcase/executor_reduce_hccl_test | ||
| 61 | +./build/test/st/algorithm/testcase/testcase/executor_pipeline_hccl_test | ||
| 62 | +``` | ||
| 63 | + | ||
| 64 | +## Executable Output Location | ||
| 65 | + | ||
| 66 | +All executables are output to the `build/test` directory by default. The path can be adjusted by modifying `CMakeLists.txt`: | ||
| 67 | + | ||
| 68 | +```cmake | ||
| 69 | +set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${HCCL_OPEN_CODE_ROOT}/build/test) | ||
| 70 | +set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${HCCL_OPEN_CODE_ROOT}/build/test) | ||
| 71 | +``` | ||
| 72 | + | ||
| 73 | +## How to Write Test Cases | ||
| 74 | + | ||
| 75 | +> HCCL LLT test cases are implemented using the Google Test framework. For detailed writing guidelines, refer to the [Google Test User Guide](https://google.github.io/googletest/). | ||
| 76 | + | ||
| 77 | +1. Select the appropriate directory based on the test target, e.g., `test/algorithm` or `test/framework` | ||
| 78 | +2. Create a new test class based on Google Test | ||
| 79 | +3. Update the corresponding directory's `CMakeLists.txt` to add the test entry | ||
| 80 | + | ||
| 81 | +Example test code: | ||
| 82 | + | ||
| 83 | +```c++ | ||
| 84 | +#include "gtest/gtest.h" | ||
| 85 | + | ||
| 86 | +TEST(MyTestClass, MyTestCase) { | ||
| 87 | + // Implement assertion logic | ||
| 88 | + EXPECT_EQ(actual_value, expected_value); | ||
| 89 | +} | ||
| 90 | +``` | ||
| 91 | + | ||
| 92 | +## How to Run Specific Test Cases | ||
| 93 | + | ||
| 94 | +To run specific test cases individually, refer to the [Google Test User Guide](https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests) and add the `--gtest_filter` parameter when executing the test. | ||
| 95 | + | ||
| 96 | +Using `hccl_utest_framework_op_base_api` as an example: | ||
| 97 | + | ||
| 98 | +```bash | ||
| 99 | +# Run only test cases of the HcclCommInitRootInfoTest test class | ||
| 100 | +./build/test/hccl_utest_framework_op_base_api --gtest_filter=HcclCommInitRootInfoTest.* | ||
| 101 | +``` | ||
| @@ -26,7 +26,6 @@ chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run | |||
| 26 | ./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend | 26 | ./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend |
| 27 | ``` | 27 | ``` |
| 28 | 28 | ||
| 29 | - | ||
| 30 | ### 2.2 hccl_test编译 | 29 | ### 2.2 hccl_test编译 |
| 31 | 30 | ||
| 32 | hccl_test是昇腾官方提供的HCCL性能测试工具，详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html)，HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。 | 31 | hccl_test是昇腾官方提供的HCCL性能测试工具，详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html)，HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。 |
| @@ -110,7 +109,6 @@ export HCCL_OP_EXPANSION_MODE="CCU_SCHED" | |||
| 110 | 109 | ||
| 111 | 2. 执行 | 110 | 2. 执行 |
| 112 | 111 | ||
| 113 | - | ||
| 114 | ```bash | 112 | ```bash |
| 115 | # 需要进入到新的bin文件目录下执行hccl-vm | 113 | # 需要进入到新的bin文件目录下执行hccl-vm |
| 116 | cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | 114 | cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin |
| @@ -133,6 +131,7 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | |||
| 133 | ``` | 131 | ``` |
| 134 | 132 | ||
| 135 | 3. 验证hccl_test用例运行结果 | 133 | 3. 验证hccl_test用例运行结果 |
| 134 | + | ||
| 136 | [Runner结果查看](#491-runner插件结果) | 135 | [Runner结果查看](#491-runner插件结果) |
| 137 | [Checker结果查看](#492-checker插件结果) | 136 | [Checker结果查看](#492-checker插件结果) |
| 138 | 137 | ||
| @@ -361,7 +360,6 @@ links: | |||
| 361 | device_to_switch_links: | 360 | device_to_switch_links: |
| 362 | # 如下示例表示：device0到device15都通过die0的port连接到交换机。结合portGroup可知，device0到device15都通过portGroup[0/4, 0/5, 0/6, 0/7]连接到交换机。 | 361 | # 如下示例表示：device0到device15都通过die0的port连接到交换机。结合portGroup可知，device0到device15都通过portGroup[0/4, 0/5, 0/6, 0/7]连接到交换机。 |
| 363 | - {die_id: 0, devices_range: [0, 15]} | 362 | - {die_id: 0, devices_range: [0, 15]} |
| 364 | - | ||
| 365 | ``` | 363 | ``` |
| 366 | 364 | ||
| 367 | **字段说明**： | 365 | **字段说明**： |
| @@ -398,7 +396,6 @@ server_list: | |||
| 398 | - {super_pod_id: 1, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | 396 | - {super_pod_id: 1, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} |
| 399 | - {super_pod_id: 2, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | 397 | - {super_pod_id: 2, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} |
| 400 | - {super_pod_id: 3, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | 398 | - {super_pod_id: 3, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} |
| 401 | - | ||
| 402 | ``` | 399 | ``` |
| 403 | 400 | ||
| 404 | 上述配置文件描述了一个包含4个超节点，32个Server，共128个NPU卡的集群拓扑。其中，每个Server/Pod采用ascend950_server_topo_normal拓扑类型。 | 401 | 上述配置文件描述了一个包含4个超节点，32个Server，共128个NPU卡的集群拓扑。其中，每个Server/Pod采用ascend950_server_topo_normal拓扑类型。 |
| @@ -0,0 +1,814 @@ | |||
| 1 | +# HCCL-VM User Guide | ||
| 2 | + | ||
| 3 | +## 1. Overview | ||
| 4 | + | ||
| 5 | +HCCL-VM is a virtual execution environment for high-performance collective communication targeting Huawei Ascend NPU cards. This tool enables the development and functional verification of HCCL collective communication operators without real Ascend hardware. | ||
| 6 | + | ||
| 7 | + | ||
| 8 | + | ||
| 9 | +## 2. Prerequisites | ||
| 10 | + | ||
| 11 | +| Dependency | Version Requirement | | ||
| 12 | +|------------|---------------------| | ||
| 13 | +| System Architecture | x86_64 Ubuntu 22.04 or later | | ||
| 14 | +| Specification | Ascend950, for others refer to [Tool Specification Constraints](#45-tool-specification-constraints) | | ||
| 15 | + | ||
| 16 | +### 2.1 CANN Package Installation | ||
| 17 | + | ||
| 18 | +Install the latest CANN Toolkit development package and CANN ops operator package [Download Link](https://ascend.devcloud.huaweicloud.com/artifactory/cann-run-mirror/software/master/) | ||
| 19 | + | ||
| 20 | +```bash | ||
| 21 | +# Ensure the installation packages have executable permissions | ||
| 22 | +chmod +x Ascend-cann-toolkit_9.1.0_linux-x86_64.run | ||
| 23 | +chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run | ||
| 24 | +# Installation commands | ||
| 25 | +./Ascend-cann-toolkit_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend | ||
| 26 | +./Ascend-cann-950-ops_9.1.0_linux-x86_64.run --install --install-path=/home/workspace/Ascend | ||
| 27 | +``` | ||
| 28 | + | ||
| 29 | +### 2.2 hccl_test Compilation | ||
| 30 | + | ||
| 31 | +hccl_test is the official HCCL performance testing tool provided by Ascend. See [HCCL Performance Testing Tool](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html). HCCL-VM supports running hccl_test cases in a virtual environment. Please first refer to the [hccl_test Case Build](#42-hccl-test-case-build) section to compile the test case binary. | ||
| 32 | + | ||
| 33 | +Note: Optional. PyTorch test cases will be supported in the future. | ||
| 34 | + | ||
| 35 | +--- | ||
| 36 | + | ||
| 37 | +## One-Click Installation | ||
| 38 | + | ||
| 39 | +Complete dependency installation, source code retrieval, CANN detection, and compilation in one line (default `campus-2026` profile): | ||
| 40 | + | ||
| 41 | +```bash | ||
| 42 | +curl -fsSL https://raw.gitcode.com/cann/hcomm/raw/competition%2Fcampus-2026/test/hccl_vm/hccl_vm_installer | bash | ||
| 43 | +``` | ||
| 44 | + | ||
| 45 | +You can also download and run locally (for review or offline distribution): `bash hccl_vm_installer`. Append parameters with `... | bash -s -- --workspace /root/hvm`. | ||
| 46 | + | ||
| 47 | +**Prerequisites**: x86_64 Linux; the toolchain must meet hcomm build.md requirements — gcc/g++ 7.3.0–13.3.x, cmake ≥ 3.16.0 (applies to both host and aarch64 cross-compilers). Ubuntu 22.04 / 24.04 work out of the box. Later versions with default gcc (14/15) exceeding the range will trigger a warning; the script continues, but a compliant environment is recommended. | ||
| 48 | + | ||
| 49 | +**CANN**: The script only probes for CANN in the working directory `<workspace>/Ascend` (or the path specified by `--ascend-path`). If not found, it automatically downloads and installs the matching version to that location, with consistent behavior for both root and regular users. `--offline` only checks, never downloads. In offline environments without public internet, it falls back to printing instructions for self-provided CANN. | ||
| 50 | + | ||
| 51 | +**hccl_test**: By default, OpenMPI and the hccl_test performance testing tool are also compiled. Use `--skip-hccl-test` to disable. | ||
| 52 | + | ||
| 53 | +**Common Parameters**: | ||
| 54 | +- `--profile <name>`: Profile (default `campus-2026`, use `--list-profiles` to list all) | ||
| 55 | +- `--workspace <path>`: Working directory for source code, compilation, and artifacts (default: current directory) | ||
| 56 | +- `--ascend-path <path>`: Specify the CANN directory; reuse if exists, install if not | ||
| 57 | +- `--reinstall-cann`: Re-download and overwrite existing CANN (use when version mismatch; kept by default) | ||
| 58 | +- `--offline`: Use existing CANN only, never download | ||
| 59 | +- `--skip-hccl-test`: Skip hccl_test compilation | ||
| 60 | +- `-h`: Full help | ||
| 61 | + | ||
| 62 | +After completion, the tool is located at `<workspace>/hcomm/test/hccl_vm/hccl_vm_install/bin/hccl-vm`. Delete the working directory to clean up all tool artifacts (system dependencies installed via apt require manual `apt remove`). This tool does not modify CANN. | ||
| 63 | + | ||
| 64 | +--- | ||
| 65 | + | ||
| 66 | +## 3. Quick Start | ||
| 67 | + | ||
| 68 | +### 3.1 Tool Build & Installation | ||
| 69 | + | ||
| 70 | +```bash | ||
| 71 | +# 1. Create working directory | ||
| 72 | +mkdir -p /home/workspace | ||
| 73 | +cd /home/workspace | ||
| 74 | + | ||
| 75 | +# 2. Download dependency source code | ||
| 76 | +git clone https://gitcode.com/cann/hccl.git | ||
| 77 | +git clone https://gitcode.com/cann/hcomm.git | ||
| 78 | + | ||
| 79 | +# 3. Install third-party dependencies | ||
| 80 | +sudo apt-get update | ||
| 81 | +sudo apt install build-essential cmake libsqlite3-dev rdma-core libibverbs-dev pkg-config gcc-aarch64-linux-gnu g++-aarch64-linux-gnu qemu-user-static binfmt-support | ||
| 82 | + | ||
| 83 | +# 4. Compile the HCCL-VM tool. After downloading the hcomm code, the tool source is at: /home/workspace/hcomm/test/hccl_vm | ||
| 84 | +cd /home/workspace/hcomm/test/hccl_vm | ||
| 85 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 86 | +export HCCL_CODE_HOME=/home/workspace/hccl | ||
| 87 | +export HCOMM_CODE_HOME=/home/workspace/hcomm | ||
| 88 | +bash ./build.sh --full | ||
| 89 | +``` | ||
| 90 | + | ||
| 91 | +### 3.2 Usage Examples | ||
| 92 | + | ||
| 93 | +#### 3.2.1 Environment Configuration | ||
| 94 | + | ||
| 95 | +Refer to [hccl_rootinfo File Contents](#47-hccl_rootinfojson-file) to create and configure the `hccl_rootinfo.json` file. | ||
| 96 | + | ||
| 97 | +#### 3.2.2 CCU Mode | ||
| 98 | + | ||
| 99 | +1. Environment variable configuration. | ||
| 100 | + | ||
| 101 | +```bash | ||
| 102 | +# Enter the tool installation directory | ||
| 103 | +cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install | ||
| 104 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 105 | +export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | ||
| 106 | +export RANK_TABLE_FILE=$(pwd)/data/ranktable.json | ||
| 107 | +export HCCL_OP_EXPANSION_MODE="CCU_SCHED" | ||
| 108 | +``` | ||
| 109 | + | ||
| 110 | +2. Execution | ||
| 111 | + | ||
| 112 | +```bash | ||
| 113 | +# Navigate to the new bin directory to execute hccl-vm | ||
| 114 | +cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | ||
| 115 | + | ||
| 116 | +# Select the Ascend cluster topology configuration file, start the tool, initialize the cluster environment, and enter the tool command line | ||
| 117 | +./hccl-vm start ascend950_cluster_32_server_normal.yaml | ||
| 118 | + | ||
| 119 | +# To enable the runner plugin (optional) | ||
| 120 | +(hvm)$> hccl-vm plugin install @runner | ||
| 121 | + | ||
| 122 | +# Select the communication domain configuration file for this operator execution (run hccl_test in a cluster with 1 super node, 1 Server, 1 NPU) | ||
| 123 | +(hvm)$> hccl-vm mock-comm 112 | ||
| 124 | +(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | ||
| 125 | + | ||
| 126 | +# Execute checker validation | ||
| 127 | +(hvm)$> hccl-vm plugin run @checker | ||
| 128 | + | ||
| 129 | +# Exit the tool terminal | ||
| 130 | +(hvm)$> exit | ||
| 131 | +``` | ||
| 132 | + | ||
| 133 | +3. Verify hccl_test case execution results | ||
| 134 | + | ||
| 135 | +[Runner Result Viewing](#491-runner-plugin-results) | ||
| 136 | +[Checker Result Viewing](#492-checker-plugin-results) | ||
| 137 | + | ||
| 138 | +#### 3.2.3 AICPU Mode | ||
| 139 | + | ||
| 140 | +AICPU expansion mode requires executing algorithm expansion steps on the device side. Therefore, the hccl-vm tool needs to compile and simulate execution of HCCL's device-side symbols. Since device-side symbols use the ARM architecture, a cross-compiler is needed on x86 environments for compilation, and QEMU is required for runtime simulation of AICPU mode execution. | ||
| 141 | + | ||
| 142 | +Device-side symbols are compiled using hccl and hcomm source code. To ensure correct Host-Device communication protocol, the Host-side installation package must also be compiled and replaced. | ||
| 143 | + | ||
| 144 | +1. HCCL device-side symbol compilation, installation, copying, etc. | ||
| 145 | + | ||
| 146 | +```bash | ||
| 147 | +cd /home/workspace/hcomm/test/hccl_vm/ | ||
| 148 | +bash ./build_pkg.sh | ||
| 149 | +``` | ||
| 150 | + | ||
| 151 | +2. Environment variable configuration. | ||
| 152 | + | ||
| 153 | +```bash | ||
| 154 | +# Enter the tool installation directory | ||
| 155 | +cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install | ||
| 156 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 157 | +export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | ||
| 158 | +export RANK_TABLE_FILE=$(pwd)/data/ranktable.json | ||
| 159 | +export HCCL_OP_EXPANSION_MODE="AI_CPU" | ||
| 160 | +``` | ||
| 161 | + | ||
| 162 | +3. Execution | ||
| 163 | + | ||
| 164 | +```bash | ||
| 165 | +# Navigate to the new bin directory to execute hccl-vm | ||
| 166 | +cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | ||
| 167 | + | ||
| 168 | +# Select the Ascend cluster topology configuration file, start the tool, initialize the cluster environment, and enter the tool command line | ||
| 169 | +./hccl-vm start ascend950_cluster_32_server_normal.yaml | ||
| 170 | + | ||
| 171 | +# To enable the runner plugin (optional) | ||
| 172 | +(hvm)$> hccl-vm plugin install @runner | ||
| 173 | + | ||
| 174 | +# Select the communication domain configuration file for this operator execution (run hccl_test in a cluster with 1 super node, 1 Server, 1 NPU) | ||
| 175 | +(hvm)$> hccl-vm mock-comm 112 | ||
| 176 | +(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | ||
| 177 | + | ||
| 178 | +# Execute checker validation | ||
| 179 | +(hvm)$> hccl-vm plugin run @checker | ||
| 180 | + | ||
| 181 | +# Exit the tool terminal | ||
| 182 | +(hvm)$> exit | ||
| 183 | +``` | ||
| 184 | + | ||
| 185 | +4. Verify hccl_test case execution results [Runner Result Viewing](#491-runner-plugin-results) [Checker Result Viewing](#492-checker-plugin-results) | ||
| 186 | + | ||
| 187 | +#### 3.2.4 AIV Mode | ||
| 188 | + | ||
| 189 | +1. Environment variable configuration. | ||
| 190 | + | ||
| 191 | +```bash | ||
| 192 | +# Enter the tool installation directory | ||
| 193 | +cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install | ||
| 194 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 195 | +export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | ||
| 196 | +export RANK_TABLE_FILE=$(pwd)/data/ranktable.json | ||
| 197 | +export HCCL_OP_EXPANSION_MODE="AIV" | ||
| 198 | +``` | ||
| 199 | + | ||
| 200 | +2. Execution | ||
| 201 | + | ||
| 202 | +```bash | ||
| 203 | +# Navigate to the new bin directory to execute hccl-vm | ||
| 204 | +cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | ||
| 205 | + | ||
| 206 | +# Select the Ascend cluster topology configuration file, start the tool, initialize the cluster environment, and enter the tool command line | ||
| 207 | +./hccl-vm start ascend950_cluster_32_server_normal.yaml | ||
| 208 | + | ||
| 209 | +# To enable the runner plugin (optional) | ||
| 210 | +(hvm)$> hccl-vm plugin install @runner | ||
| 211 | + | ||
| 212 | +# Select the communication domain configuration file for this operator execution (run hccl_test in a cluster with 1 super node, 1 Server, 1 NPU) | ||
| 213 | +(hvm)$> hccl-vm mock-comm 112 | ||
| 214 | +(hvm)$> mpirun --allow-run-as-root --oversubscribe -np 2 ${ASCEND_HOME_PATH}/tools/hccl_test/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 > log.txt | ||
| 215 | + | ||
| 216 | +# Execute checker validation | ||
| 217 | +(hvm)$> hccl-vm plugin run @checker | ||
| 218 | + | ||
| 219 | +# Exit the tool terminal | ||
| 220 | +(hvm)$> exit | ||
| 221 | +``` | ||
| 222 | + | ||
| 223 | +3. Verify hccl_test case execution results [Runner Result Viewing](#491-runner-plugin-results) [Checker Result Viewing](#492-checker-plugin-results) | ||
| 224 | + | ||
| 225 | +### 3.3 PyTorch Example | ||
| 226 | + | ||
| 227 | +Not yet supported. | ||
| 228 | + | ||
| 229 | +### 3.4 hccl Code Modification Verification Example | ||
| 230 | + | ||
| 231 | +If you modify CANN operator package code, such as adding a new algorithm type, follow these steps to apply your changes. The `build_pkg.sh` script helps you build, install, and copy device-side dependency symbols. Set the environment variables before execution: | ||
| 232 | + | ||
| 233 | +```bash | ||
| 234 | +# Assume your CANN installation directory is: /home/workspace/Ascend | ||
| 235 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 236 | +# Configure hccl code repository path | ||
| 237 | +export HCCL_CODE_HOME=/home/workspace/hccl | ||
| 238 | +# Configure hcomm code repository path | ||
| 239 | +export HCOMM_CODE_HOME=/home/workspace/hcomm | ||
| 240 | +``` | ||
| 241 | + | ||
| 242 | +1. If you modified the CANN hccl repository code, run `bash build_pkg.sh --install hccl`. | ||
| 243 | +2. If you modified the CANN hcomm repository code, run `bash build_pkg.sh --install hcomm`. | ||
| 244 | +3. Refer to the [Usage Examples](#32-usage-examples) section and re-run the test cases. | ||
| 245 | + | ||
| 246 | +--- | ||
| 247 | + | ||
| 248 | +## 4. Detailed Guide | ||
| 249 | + | ||
| 250 | +### 4.1 Tool Environment Variable Configuration | ||
| 251 | + | ||
| 252 | +**HCCL-VM Environment Variables**: | ||
| 253 | + | ||
| 254 | +| Environment Variable | Purpose | Example | | ||
| 255 | +|----------------------|---------|---------| | ||
| 256 | +| `HCCL_CODE_HOME` | Specifies the HCCL source code path for HCCL-VM compilation. Not configured by default. | `export HCCL_CODE_HOME=/home/workspace/hccl` | | ||
| 257 | +| `HCOMM_CODE_HOME` | Specifies the HCOMM source code path for HCCL-VM compilation. Not configured by default. | `export HCOMM_CODE_HOME=/home/workspace/hcomm` | | ||
| 258 | +| `HCCLVM_ENABLE_DUMP_DATA` | Enables Runner plugin to dump input & output data. When enabled, each operator's input & output data is dumped to the `all_rank_input_output.txt` file during test execution. | `export HCCLVM_ENABLE_DUMP_DATA=1` to enable, `export HCCLVM_ENABLE_DUMP_DATA=0` to disable | | ||
| 259 | + | ||
| 260 | +### 4.2 HCCL-Test Case Build | ||
| 261 | + | ||
| 262 | +The hccl_test source code is located in the CANN package installation directory. It supports compilation and execution in both OpenMPI and MPICH environments. See [OpenMPI and MPICH Environment Case Execution Differences](#48-differences-in-running-cases-between-openmpi-and-mpich-environments) for runtime differences. This guide uses OpenMPI as an example. | ||
| 263 | + | ||
| 264 | +#### 4.2.1 OpenMPI Environment Compilation | ||
| 265 | + | ||
| 266 | +1. Install OpenMPI | ||
| 267 | + | ||
| 268 | +```bash | ||
| 269 | +sudo apt-get update | ||
| 270 | +sudo apt install openmpi-bin libopenmpi-dev | ||
| 271 | +``` | ||
| 272 | + | ||
| 273 | +2. Compile hccl_test | ||
| 274 | + | ||
| 275 | +```bash | ||
| 276 | +# Change CANN installation directory permissions | ||
| 277 | +chmod -R 755 /home/workspace/Ascend | ||
| 278 | + | ||
| 279 | +# Enter the hccl_test source code directory | ||
| 280 | +cd /home/workspace/Ascend/cann/tools/hccl_test | ||
| 281 | + | ||
| 282 | +# Set CANN environment variables | ||
| 283 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 284 | + | ||
| 285 | +# Temporarily modify the Makefile script | ||
| 286 | +if ! grep -q '\-lmpi_cxx' Makefile; then | ||
| 287 | + sed -i 's/-lmpi/-lmpi -lmpi_cxx/g' Makefile | ||
| 288 | +fi | ||
| 289 | + | ||
| 290 | +# Compile hccl_test cases | ||
| 291 | +MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH} | ||
| 292 | +``` | ||
| 293 | + | ||
| 294 | +#### 4.2.2 MPICH Environment Compilation | ||
| 295 | + | ||
| 296 | +Assume the mpich path is: `/usr/lib/mpich`. | ||
| 297 | + | ||
| 298 | +```bash | ||
| 299 | +# Enter the hccl_test source code directory | ||
| 300 | +cd /home/workspace/Ascend/cann/tools/hccl_test | ||
| 301 | + | ||
| 302 | +# Set CANN environment variables | ||
| 303 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 304 | + | ||
| 305 | +# Configure environment variables | ||
| 306 | +export LD_LIBRARY_PATH=/usr/lib/mpich/lib/:${ASCEND_HOME_PATH}/lib64/:${ASCEND_HOME_PATH}/x86_64-linux/devlib:$LD_LIBRARY_PATH | ||
| 307 | + | ||
| 308 | +# Compile hccl_test cases | ||
| 309 | +make MPI_HOME=/usr/lib/mpich/ ASCEND_DIR=${ASCEND_HOME_PATH} | ||
| 310 | +``` | ||
| 311 | + | ||
| 312 | +### 4.3 Ascend Cluster Topology Configuration File Description | ||
| 313 | + | ||
| 314 | +#### 4.3.1 Server/Pod Topology Configuration File Description | ||
| 315 | + | ||
| 316 | +An Ascend cluster topology is composed of one or more Server/Pod sub-topologies combined according to CLOS hierarchical network rules. Therefore, before generating a cluster topology, users need to confirm the topology type of each Server/Pod. | ||
| 317 | +Users can either use predefined topology types provided by the HCCL-VM tool or define custom Server/Pod topology types according to the configuration file format requirements. | ||
| 318 | + | ||
| 319 | +Describing the topological network relationship of a Server/Pod mainly includes the following aspects: | ||
| 320 | + | ||
| 321 | +- **Port Allocation Table**: Describes the physical port configuration of an NPU card, such as NPU-to-NPU direct ports (P2P), NPU out-of-chassis ports (P2NET), etc. | ||
| 322 | +- **Link Configuration Table**: Describes the connection relationships among all NPU cards within a Server/Pod, such as full mesh. | ||
| 323 | +- **PortBound**: Describes the binding relationship of certain ports on an NPU card, where multiple ports are bound into a PortGroup. | ||
| 324 | + | ||
| 325 | +```yaml | ||
| 326 | +type: "server_intra_links" | ||
| 327 | +name: "ascend950_links_topo_demo" | ||
| 328 | +description: "ascend950 chip standard topology connection relationship description file" | ||
| 329 | + | ||
| 330 | +soc_version: "Ascend950" | ||
| 331 | +device_num: 16 | ||
| 332 | + | ||
| 333 | +device_ports_allocate_map: | ||
| 334 | + # port allocation table: 0: unused, 1: device direct connect, 2: device to switch, 3: d2h port | ||
| 335 | + # portId: 0 1 2 3 4 5 6 7 8 | ||
| 336 | + - {die_id: 0, pin_map: [1, 1, 1, 0, 2, 2, 2, 2, 3]} # die0 | ||
| 337 | + - {die_id: 1, pin_map: [0, 0, 0, 0, 0, 0, 0, 0, 0]} # die1 | ||
| 338 | + | ||
| 339 | +# port_group: describes which ports are merged into a portGroup. Ports in the same portGroup share the same IP address. | ||
| 340 | +port_group: | ||
| 341 | + - {layer: 0, ports: ["0/4", "0/5", "0/6", "0/7"]} | ||
| 342 | + | ||
| 343 | +links: | ||
| 344 | + # ── Method: every 8 devices form full interconnect ── | ||
| 345 | + - link_mode: "fullmesh" | ||
| 346 | + connections: | ||
| 347 | + # The following example shows that die0 devices 0, 1, 2, 3 are all fully connected via die0 ports | ||
| 348 | + - {die_id: 0, devices_range: [0, 3]} | ||
| 349 | + - {die_id: 0, devices_range: [4, 7]} | ||
| 350 | + - {die_id: 0, devices_range: [8, 11]} | ||
| 351 | + - {die_id: 0, devices_range: [12, 15]} | ||
| 352 | + | ||
| 353 | + #- link_mode: "enum" | ||
| 354 | + # device_to_device_links: | ||
| 355 | + # The following example shows that device 0 and 1 both connect via die0 ports to devices 1, 3, 5, 7 on die1 ports respectively. | ||
| 356 | + # i.e.: device0 connects to device1, device3, device5, device7; device1 connects to device3, device5, device7 | ||
| 357 | + # - {src_die_id: 1, src_local_id_range: [0, 2], dst_die_id: 1, dst_local_id_range: [1, 3, 5, 7]} | ||
| 358 | + | ||
| 359 | + - link_mode: "enum" | ||
| 360 | + device_to_switch_links: | ||
| 361 | + # The following example shows that devices 0 to 15 all connect to the switch via die0 ports. Combined with portGroup, device0 to device15 all connect to the switch via portGroup[0/4, 0/5, 0/6, 0/7]. | ||
| 362 | + - {die_id: 0, devices_range: [0, 15]} | ||
| 363 | +``` | ||
| 364 | + | ||
| 365 | +**Field Descriptions**: | ||
| 366 | + | ||
| 367 | +- **soc_version**: Chip model, e.g., `Ascend950`. | ||
| 368 | +- **device_num**: Total number of devices, determined by chip model and topology type. | ||
| 369 | +- **device_ports_allocate_map**: Port allocation table describing each die's port configuration. 1 represents device direct connect ports, 2 represents device-to-switch ports, 3 represents d2h ports. | ||
| 370 | +- **port_group**: Describes which ports are merged into a portGroup. Ports in the same portGroup share the same IP address. Ports not configured default to one port per portGroup. | ||
| 371 | +- **links**: Link configuration table describing the connection relationships among all NPU cards within a Server/Pod, and between NPUs and switches. | ||
| 372 | + - **NPU direct connections**: The tool provides two methods for configuring NPU direct connections: | ||
| 373 | + - **link_mode == "fullmesh"**: Indicates all devices are fully connected based on one die's ports. New typical connection modes can be added as new link_mode types, such as "ring". | ||
| 374 | + - **link_mode == "enum"**: Enumeration method. When the NPU connection method within a Server/Pod is complex, all link relationships can be described through enumeration. | ||
| 375 | + - **NPU-to-switch connections**: Users can configure NPU-to-switch connection relationships using the enumeration method. | ||
| 376 | +- **device_to_device_links**: Describes NPU-to-NPU connection relationships. | ||
| 377 | +- **device_to_switch_links**: Describes NPU-to-switch connection relationships. | ||
| 378 | + | ||
| 379 | +#### 4.3.2 Cluster Topology Configuration File Description | ||
| 380 | + | ||
| 381 | +An Ascend cluster network is composed of one or more Server/Pod sub-topologies combined according to CLOS hierarchical network rules. Users can choose different Server/Pod topology types based on cluster size and requirements. | ||
| 382 | + | ||
| 383 | +Users can define custom cluster topology configuration files according to the following format: | ||
| 384 | + | ||
| 385 | +```yaml | ||
| 386 | +name: "ascend950_cluster_32_server_normal" | ||
| 387 | +description: "Ascend950 normal networking: 32 super nodes, 1 server per super node" | ||
| 388 | + | ||
| 389 | +# Total number of super nodes | ||
| 390 | +super_node_num: 4 | ||
| 391 | +# Total number of servers/pods | ||
| 392 | +server_num: 32 | ||
| 393 | +server_list: | ||
| 394 | + # 0-7 servers: all use ascend950_server_topo_normal topology type | ||
| 395 | + - {super_pod_id: 0, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | ||
| 396 | + - {super_pod_id: 1, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | ||
| 397 | + - {super_pod_id: 2, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | ||
| 398 | + - {super_pod_id: 3, id_range: [0, 7], soc_version: "Ascend950", server_topo: "ascend950_server_topo_normal.yaml"} | ||
| 399 | +``` | ||
| 400 | + | ||
| 401 | +The configuration file above describes a cluster topology with 4 super nodes, 32 Servers, and a total of 128 NPU cards. Each Server/Pod uses the `ascend950_server_topo_normal` topology type. | ||
| 402 | + | ||
| 403 | +**Field Descriptions**: | ||
| 404 | + | ||
| 405 | +- **super_node_num**: Total number of super nodes. | ||
| 406 | +- **server_num**: Total number of servers/pods. | ||
| 407 | +- **server_list**: Configuration information for each Server/Pod, including super node ID, device ID range, chip model, and Server/Pod topology configuration file path. | ||
| 408 | + | ||
| 409 | +#### 4.3.3 Communication Domain Configuration File Description | ||
| 410 | + | ||
| 411 | +In an Ascend cluster environment, users need to select different communication domain configuration files based on the communication domain required by the operator to be executed. | ||
| 412 | + | ||
| 413 | +The tool provides the `hccl-vm mock-comm` command to read and configure operator communication domain configuration files. The communication domain configuration file format is yaml, located at `hccl_vm_install/config/topo_meta`. If the corresponding configuration file does not exist in the directory, the user needs to create one first. | ||
| 414 | + | ||
| 415 | +The hccl-vm tool supports asymmetric topology communication domain configuration, as shown below: | ||
| 416 | + | ||
| 417 | +```yaml | ||
| 418 | +# 1. Global statistics: podNum, serNum, rankNum are all less than 1024 | ||
| 419 | +meta: | ||
| 420 | + podNum: 1 # Total number of super nodes | ||
| 421 | + serNum: 2 # Total number of servers | ||
| 422 | + rankNum: 6 # Total number of ranks | ||
| 423 | + | ||
| 424 | +# 2. Detailed topology structure | ||
| 425 | +topology: | ||
| 426 | + - podId: 0 | ||
| 427 | + servers: | ||
| 428 | + - serId: 0 | ||
| 429 | + # Local IDs of ranks running on this server | ||
| 430 | + ranks: [0, 2] | ||
| 431 | + - serId: 1 | ||
| 432 | + # Local IDs of ranks running on this server | ||
| 433 | + ranks: [1, 3, 5, 7] | ||
| 434 | +``` | ||
| 435 | + | ||
| 436 | +**Notes**: | ||
| 437 | + | ||
| 438 | +- When configuring the communication domain, the tool regenerates the `topo.json` and `ranktable.json` files based on the specified communication domain configuration ID. | ||
| 439 | +- In the communication domain configuration yaml file above, the `ranks` field represents the list of local IDs (i.e., device physical IDs) of ranks actually running on each server. | ||
| 440 | + | ||
| 441 | +#### 4.3.4 topo and ranktable.json File Description | ||
| 442 | + | ||
| 443 | +The `topo.json` and `ranktable.json` files do not need to be created manually. The tool automatically generates them based on the following information: | ||
| 444 | + | ||
| 445 | +- **Topology configuration ID**: The ID specified by the user at startup (e.g., 112, 113, etc.) | ||
| 446 | +- **Chip type**: The chip type automatically identified based on the runtime environment. | ||
| 447 | + | ||
| 448 | +Although the configuration files are automatically generated by the tool, understanding their structure helps in understanding topology configuration. | ||
| 449 | + | ||
| 450 | +**topo.json Structure**: | ||
| 451 | + | ||
| 452 | +`topo.json` describes the connection relationships among all devices within a server: | ||
| 453 | + | ||
| 454 | +```json | ||
| 455 | +{ | ||
| 456 | + "server": { | ||
| 457 | + "device_count": 8, | ||
| 458 | + "groups": [ | ||
| 459 | + { | ||
| 460 | + "group_id": 0, | ||
| 461 | + "device_start": 0, | ||
| 462 | + "device_count": 8, | ||
| 463 | + "topo_layout": "1D" | ||
| 464 | + } | ||
| 465 | + ] | ||
| 466 | + }, | ||
| 467 | + "ports": [ | ||
| 468 | + { | ||
| 469 | + "ccu": "die0", | ||
| 470 | + "port_pattern": "0/{0-6}", | ||
| 471 | + "protocol": "HCCS", | ||
| 472 | + "func_id": 2, | ||
| 473 | + "usage": "peer2peer", | ||
| 474 | + "ip_binding": "independent" | ||
| 475 | + }, | ||
| 476 | + { | ||
| 477 | + "ccu": "die0", | ||
| 478 | + "port_pattern": "0/7,0/8", | ||
| 479 | + "protocol": "ROCE", | ||
| 480 | + "func_id": 3, | ||
| 481 | + "usage": "peer2net", | ||
| 482 | + "ip_binding": "independent" | ||
| 483 | + } | ||
| 484 | + ], | ||
| 485 | + "links": [ | ||
| 486 | + { | ||
| 487 | + "net_layer": 0, | ||
| 488 | + "link_type": "PEER2PEER", | ||
| 489 | + "topo_type": "1DMESH", | ||
| 490 | + "ccu": "die0", | ||
| 491 | + "port_pattern": "0/{0-6}", | ||
| 492 | + "connect_pattern": "full_mesh", | ||
| 493 | + "group_id": 0 | ||
| 494 | + }, | ||
| 495 | + { | ||
| 496 | + "net_layer": 1, | ||
| 497 | + "link_type": "PEER2NET", | ||
| 498 | + "topo_type": "CLOS", | ||
| 499 | + "ccu": "die0", | ||
| 500 | + "port_pattern": "0/7,0/8", | ||
| 501 | + "connect_pattern": "all_to_net", | ||
| 502 | + "group_id": 0 | ||
| 503 | + } | ||
| 504 | + ] | ||
| 505 | +} | ||
| 506 | +``` | ||
| 507 | + | ||
| 508 | +**Field Descriptions**: | ||
| 509 | + | ||
| 510 | +- `server.device_count`: Total number of devices. | ||
| 511 | +- `server.groups`: Device grouping information. | ||
| 512 | +- `ports`: Port configuration. | ||
| 513 | + - `usage`: Port purpose (`peer2peer` for inter-device connections, `peer2net` for external connections) | ||
| 514 | +- `links`: Link configuration. | ||
| 515 | + - `link_type`: Link type (`PEER2PEER` or `PEER2NET`) | ||
| 516 | + - `topo_type`: Topology type (`1DMESH`, `CLOS`, etc.) | ||
| 517 | + | ||
| 518 | +**ranktable.json Structure**: | ||
| 519 | + | ||
| 520 | +`ranktable.json` describes the device and IP mapping for the current run: | ||
| 521 | + | ||
| 522 | +```json | ||
| 523 | +{ | ||
| 524 | + "version": "1.0", | ||
| 525 | + "server_count": 1, | ||
| 526 | + "device_count": 8, | ||
| 527 | + "server_list": [ | ||
| 528 | + { | ||
| 529 | + "server_id": 0, | ||
| 530 | + "device_id": 0, | ||
| 531 | + "device_ip": "192.168.1.10", | ||
| 532 | + "port": "2222" | ||
| 533 | + } | ||
| 534 | + ] | ||
| 535 | +} | ||
| 536 | +``` | ||
| 537 | + | ||
| 538 | +**Field Descriptions**: | ||
| 539 | + | ||
| 540 | +- `server_count`: Number of servers. | ||
| 541 | +- `device_count`: Total number of devices. | ||
| 542 | +- `server_list`: Server and device list. | ||
| 543 | + - `device_ip`: Device IP address. | ||
| 544 | + - `port`: Device port number. | ||
| 545 | + | ||
| 546 | +### 4.4 hccl_config.sh File Description | ||
| 547 | + | ||
| 548 | +The `hccl_config.sh` file contains the environment variable configuration required for running HCCL_Test cases. The environment variables are consistent with those used for Hccl_Test cases on real hardware. | ||
| 549 | +Users need to modify the `hccl_config.sh` script according to their own use cases and requirements to configure the HCCL test case runtime environment variables. | ||
| 550 | + | ||
| 551 | +```bash | ||
| 552 | +#!/bin/bash | ||
| 553 | +# hccl_config.sh - HCCL environment variable configuration | ||
| 554 | + | ||
| 555 | +remove_files_by_prefix() { | ||
| 556 | + if [ "$#" -ne 1 ]; then | ||
| 557 | + echo "Usage: remove_files_by_prefix <prefix>" >&2 | ||
| 558 | + return 2 | ||
| 559 | + fi | ||
| 560 | + | ||
| 561 | + local prefix="$1" | ||
| 562 | + if [ -z "$prefix" ]; then | ||
| 563 | + return 0 | ||
| 564 | + fi | ||
| 565 | + | ||
| 566 | + shopt -s nullglob | ||
| 567 | + local any_deleted=0 | ||
| 568 | + for f in "${prefix}"*; do | ||
| 569 | + if [ -f "$f" ]; then | ||
| 570 | + rm -f -- "$f" && any_deleted=1 | ||
| 571 | + fi | ||
| 572 | + done | ||
| 573 | + shopt -u nullglob | ||
| 574 | + | ||
| 575 | + # Return 0 regardless of whether files were deleted, to ensure script continues | ||
| 576 | + return 0 | ||
| 577 | +} | ||
| 578 | + | ||
| 579 | +# Clean up redundant files in the data/ directory (temporary files generated in CCU mode) | ||
| 580 | +cd "${HCCL_VM_INSTALL_DIR}/data" 2>/dev/null && { | ||
| 581 | + remove_files_by_prefix "sqe_info_rank_" | ||
| 582 | + remove_files_by_prefix "mc_instr_info_rank_" | ||
| 583 | + rm -f "all_rank_input_output.txt" | ||
| 584 | + cd "${HCCL_VM_INSTALL_DIR}" | ||
| 585 | +} | ||
| 586 | + | ||
| 587 | +# Set CANN environment variables | ||
| 588 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 589 | + | ||
| 590 | +# Disable HCCL heartbeat function | ||
| 591 | +export HCCL_DFS_CONFIG=cluster_heartbeat:off | ||
| 592 | + | ||
| 593 | +# Set HCCL-VM installation path, inferred from the script itself (compatible with bin/ and script/ subdirectories) | ||
| 594 | +_INSTALL_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| 595 | +case "$(basename "${_INSTALL_SCRIPT_DIR}")" in | ||
| 596 | + bin|script) | ||
| 597 | + export HCCL_VM_INSTALL_DIR="$(dirname "${_INSTALL_SCRIPT_DIR}")" | ||
| 598 | + ;; | ||
| 599 | + *) | ||
| 600 | + export HCCL_VM_INSTALL_DIR="${_INSTALL_SCRIPT_DIR}" | ||
| 601 | + ;; | ||
| 602 | +esac | ||
| 603 | +unset _INSTALL_SCRIPT_DIR | ||
| 604 | + | ||
| 605 | +# Configure LD_LIBRARY_PATH | ||
| 606 | +export LD_LIBRARY_PATH=$ASCEND_HOME_PATH/lib64:$ASCEND_HOME_PATH/devlib:$LD_LIBRARY_PATH | ||
| 607 | + | ||
| 608 | +# Set ranktable.json file path (consistent with mock-comm generation path) | ||
| 609 | +export RANK_TABLE_FILE=${HCCL_VM_INSTALL_DIR}/data/ranktable.json | ||
| 610 | + | ||
| 611 | +# Set log level | ||
| 612 | +export ASCEND_GLOBAL_LOG_LEVEL=1 | ||
| 613 | + | ||
| 614 | +# Enable log output to stdout | ||
| 615 | +export ASCEND_SLOG_PRINT_TO_STDOUT=1 | ||
| 616 | + | ||
| 617 | +# Set HCCL operation mode (CCU, AI_CPU, AIV, etc.) | ||
| 618 | +export HCCL_OP_EXPANSION_MODE="CCU_SCHED" | ||
| 619 | +# Or set HCCL runtime parameters (AI_CPU expansion mode) AI_CPU mode environment variables cannot be set simultaneously with other modes | ||
| 620 | +# export HCCL_OP_EXPANSION_MODE="AI_CPU" | ||
| 621 | +# Or set HCCL runtime parameters (AIV expansion mode) AIV mode environment variables cannot be set simultaneously with other modes | ||
| 622 | +# export HCCL_OP_EXPANSION_MODE="AIV" | ||
| 623 | + | ||
| 624 | +echo "HCCL-VM environment configured successfully!" | ||
| 625 | +``` | ||
| 626 | + | ||
| 627 | +### 4.5 Tool Specification Constraints | ||
| 628 | + | ||
| 629 | +**Supported Operator Types**: | ||
| 630 | + | ||
| 631 | +The supported operator types include: allgather/allreduce/alltoall/reduce/reduce\_scatter/scatter/alltoallv. | ||
| 632 | + | ||
| 633 | +**Supported Data Types**: | ||
| 634 | + | ||
| 635 | +The HCCL-VM tool supports the following data types: int8/int16/int32/fp16/fp32/uint8/uint16/uint32/bfp16/hif8/fp8e4m3/fp8e5m2/fp8e8m0. | ||
| 636 | + | ||
| 637 | +The HCCL-VM Runner plugin supports the following data types: | ||
| 638 | + | ||
| 639 | +| ReduceOp | DataType | | ||
| 640 | +|----------|------------------------------------| | ||
| 641 | +| `ADD` | `int8/int16/int32/uint8` | | ||
| 642 | +| `MIN` | `int8/int16/int32/uint8` | | ||
| 643 | +| `MAX` | `int8/int16/int32/uint8` | | ||
| 644 | + | ||
| 645 | +**Hardware Specifications**: | ||
| 646 | + | ||
| 647 | +Currently, this tool only supports the Ascend950 chip. A single server supports a maximum of 8 cards. For more than 8 cards, cross-server execution is required. | ||
| 648 | + | ||
| 649 | +### 4.6 HCCL-VM Plugin Features | ||
| 650 | + | ||
| 651 | +#### 4.6.1 Runner Plugin | ||
| 652 | + | ||
| 653 | +The Runner plugin simulates the execution of task sequences generated by HCCL operator orchestration and outputs data. | ||
| 654 | +The simulation runner plugin is **disabled** by default during hccl_test case execution. After the hccl_test case calls the operator interface, it waits for the operator task to complete via the `aclrtSynchronizeStream` interface. The simulation runner tool waits until all ranks are in a waiting state, then starts simulating the execution of all rank tasks. After execution completes, it notifies each rank's test case to continue. | ||
| 655 | + | ||
| 656 | +After test case execution, users can view the input buffer and output buffer data for each rank in the `all_rank_input_output.txt` file in the execution directory. This feature is disabled by default and can be enabled via the corresponding command before test execution. | ||
| 657 | + | ||
| 658 | +**Installation and Uninstallation**: | ||
| 659 | + | ||
| 660 | +The Runner plugin supports installation and uninstallation via the `hccl-vm plugin install/uninstall` command. The runner plugin must be installed after entering the hccl-vm tool command line and before executing the test case. Subsequent executions will then run the runner. | ||
| 661 | + | ||
| 662 | +```bash | ||
| 663 | +# Install runner plugin | ||
| 664 | +(hvm)$> hccl-vm plugin install @runner | ||
| 665 | + | ||
| 666 | +# Uninstall runner plugin | ||
| 667 | +(hvm)$> hccl-vm plugin uninstall @runner | ||
| 668 | +``` | ||
| 669 | + | ||
| 670 | +#### 4.6.2 Checker Plugin | ||
| 671 | + | ||
| 672 | +The Checker plugin is an algorithm analyzer plugin. It forms a DAG from all tasks generated by HCCL and analyzes it to detect memory conflicts. By simulating execution of the DAG, it also detects semantic errors. | ||
| 673 | + | ||
| 674 | +The Checker plugin is started manually by the user via a command. | ||
| 675 | + | ||
| 676 | +The Checker plugin is currently in a transition period. Checker V3 is a refactored version of the original Checker, mainly improving validation performance. By default, the new Checker (Checker V3) is used. You can adjust this by modifying the configuration parameters in the Checker's `manifest.json` file. | ||
| 677 | + | ||
| 678 | +```bash | ||
| 679 | +# Configuration file located at /pathto/hccl_vm_install/plugin/checker/manifest.json | ||
| 680 | + | ||
| 681 | +{ | ||
| 682 | + "name": "checker", // Checker plugin name | ||
| 683 | + "version": "1.0.0", // Checker plugin version | ||
| 684 | + "entry": "./checker", // Checker plugin startup command | ||
| 685 | + "dependency": { | ||
| 686 | + "min_core_version": "1.0.0" | ||
| 687 | + }, | ||
| 688 | + "setting": { // Checker plugin configuration | ||
| 689 | + "enable_new_checker": true, // Whether to enable the new Checker (Checker V3, enabled by default) | ||
| 690 | + "enable_old_checker": false, // Whether to enable the old Checker (disabled by default) | ||
| 691 | + "enable_insight_dump": false, // Whether to enable visualization data output (disabled by default, only supported by old Checker) | ||
| 692 | + "enable_memory_snapshot_dump": false // Whether to enable visualization memory snapshot data output (disabled by default, only supported by old Checker, requires visualization data output "enable_insight_dump" to be enabled first) | ||
| 693 | + } | ||
| 694 | +} | ||
| 695 | +``` | ||
| 696 | + | ||
| 697 | +### 4.7 hccl_rootinfo.json File | ||
| 698 | + | ||
| 699 | +Currently, the tool uses the `ranktable.json` file for communication domain initialization. Therefore, the `hccl_rootinfo.json` file is only needed to provide the path to the `topo.json` file. | ||
| 700 | +If the `hccl_rootinfo.json` file does not exist under the `/etc` path, users need to create it with the following content: | ||
| 701 | + | ||
| 702 | +```json | ||
| 703 | +{ | ||
| 704 | + "version": "2.0", | ||
| 705 | + "topo_file_path": "/home/workspace/hcomm/test/hccl_vm/hccl_vm_install/data/topo.json" | ||
| 706 | +} | ||
| 707 | +``` | ||
| 708 | + | ||
| 709 | +### 4.8 Differences in Running Cases Between OpenMPI and MPICH Environments | ||
| 710 | + | ||
| 711 | +Before running hccl_test cases, users can use the `which` command to determine which mpirun is being used in the current environment. | ||
| 712 | + | ||
| 713 | +#### 4.8.1 Environment Variable Configuration Differences | ||
| 714 | + | ||
| 715 | +OpenMPI is generally the default configuration in most environments. If using OpenMPI to run cases, no additional environment variable configuration is typically needed. | ||
| 716 | +If using the MPICH environment to run cases, environment variables must be configured as follows: | ||
| 717 | + | ||
| 718 | +```bash | ||
| 719 | +# Configure mpich environment variables | ||
| 720 | +export LD_LIBRARY_PATH=/usr/lib/mpich/lib/:${ASCEND_HOME_PATH}/lib64/:${ASCEND_HOME_PATH}/x86_64-linux/devlib:$LD_LIBRARY_PATH | ||
| 721 | +export PATH=/usr/lib/mpich/bin:$PATH | ||
| 722 | +``` | ||
| 723 | + | ||
| 724 | +#### 4.8.2 mpirun Command Parameter Differences | ||
| 725 | + | ||
| 726 | +In the OpenMPI environment, run hccl_test cases as follows: | ||
| 727 | + | ||
| 728 | +```bash | ||
| 729 | +export HCCL_TEST_PATH=/home/workspace/Ascend/cann/tools/hccl_test | ||
| 730 | +mpirun --allow-run-as-root --oversubscribe -np 2 ${HCCL_TEST_PATH}/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 | ||
| 731 | +``` | ||
| 732 | + | ||
| 733 | +**Parameter Descriptions**: | ||
| 734 | + | ||
| 735 | +- --allow-run-as-root: OpenMPI-specific parameter that allows MPI processes to run as the root user, for use in environments without root privileges. | ||
| 736 | +- --oversubscribe: OpenMPI-specific parameter, removes CPU slot limits, allowing a single node to launch processes when the number of processes exceeds the number of logical CPU cores—i.e., running with oversubscription/excess allocation. | ||
| 737 | +- -np 2: Specifies 2 processes, consistent with the number of nodes. | ||
| 738 | + | ||
| 739 | +In the MPICH environment, run hccl_test cases as follows: | ||
| 740 | + | ||
| 741 | +```bash | ||
| 742 | +export HCCL_TEST_PATH=/home/workspace/Ascend/cann/tools/hccl_test | ||
| 743 | +mpirun -np 2 ${HCCL_TEST_PATH}/bin/reduce_scatter_test -b 64 -e 64 -d int32 -o sum -w 0 -n 1 -c 1 | ||
| 744 | +``` | ||
| 745 | + | ||
| 746 | +**Parameter Descriptions**: | ||
| 747 | + | ||
| 748 | +- -np 2: Specifies 2 processes, consistent with the number of nodes. | ||
| 749 | + | ||
| 750 | +### 4.9 Result Viewing | ||
| 751 | + | ||
| 752 | +#### 4.9.1 Runner Plugin Results | ||
| 753 | + | ||
| 754 | +If the runner plugin is installed via `hccl-vm plugin install @runner` in the hccl-vm terminal, the runner plugin is automatically triggered after operator execution completes. The final result depends on hccl_test verification. Users should check the redirected log file for `[error]` level logs and the final verification result: | ||
| 755 | + | ||
| 756 | +```bash | ||
| 757 | +data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result: | ||
| 758 | +64 | 1000.00 | 0.00006 | success | ||
| 759 | +``` | ||
| 760 | + | ||
| 761 | +#### 4.9.2 Checker Plugin Results | ||
| 762 | + | ||
| 763 | +After executing `hccl-vm plugin run @checker` in the hccl-vm terminal, the Checker validation process and results are printed to the terminal. Users should check for `[error]` level logs and the final validation result: | ||
| 764 | + | ||
| 765 | +```bash | ||
| 766 | +[info][PID:144373][TID:144880][main.cc][RunChecker] [RunChecker] op[0] Checker Success. | ||
| 767 | +``` | ||
| 768 | + | ||
| 769 | +--- | ||
| 770 | + | ||
| 771 | +### 4.10 Large Memory Reuse (Check-Only Mode) | ||
| 772 | + | ||
| 773 | +Check-only mode is used for scenarios where only Checker validation is needed in large-scale clusters. When enabled, a single large memory allocation of 200MB to 4GB reuses the same 4GB shared pool `HcclCommPool`, shared across all ranks and allowing mutual overwriting. This significantly reduces `/dev/shm` usage. In this mode, the content of large blocks is not guaranteed to be correct, making it suitable only for the Checker V3 validation pipeline that does not read buffer data. Do not enable this mode when numerically correct results are needed. | ||
| 774 | + | ||
| 775 | +Check-only mode is a session-level switch. Append `--check-only` after the `start` subcommand to explicitly enable it. Without this flag, the default normal mode is used, where large blocks use real independent allocation with no correctness impact. Allocations smaller than 200MB always use real allocation. A single block exceeding 4GB in check-only mode results in a direct error. Check-only mode does not conflict with Runner, but if Runner is installed while check-only mode is enabled, large block reuse still takes effect and may overwrite Runner data. The tool prints a warning in this case. | ||
| 776 | + | ||
| 777 | +```bash | ||
| 778 | +# Start the tool with check-only mode enabled | ||
| 779 | +./hccl-vm start ascend950_cluster_32_server_normal.yaml --check-only | ||
| 780 | +``` | ||
| 781 | + | ||
| 782 | +--- | ||
| 783 | + | ||
| 784 | +## 5 Appendix | ||
| 785 | + | ||
| 786 | +### Open Source Third-Party Software Dependencies | ||
| 787 | + | ||
| 788 | +When compiling this project, the following third-party open-source software is required. For offline compilation, download and rename the packages, then place them in the `third_party` directory under the project. | ||
| 789 | + | ||
| 790 | +| Open Source Software | Version | Download URL | | ||
| 791 | +|---------------------|---------|--------------| | ||
| 792 | +| CLI11 | 2.2.0 | [cli11-2.2.0.tar.gz](https://raw.gitcode.com/src-openeuler/cli11/blobs/58c912141164a5c0f0139bfa91343fefe151d525/cli11-2.2.0.tar.gz) | | ||
| 793 | +| json | 3.11.3 | [include.zip](https://gitcode.com/cann-src-third-party/json/releases/download/v3.11.3/include.zip) | | ||
| 794 | +| spdlog | 1.11.0 | [spdlog-v1.11.0.tar.gz](https://raw.gitcode.com/src-openeuler/spdlog/blobs/c2dfb1aca26c607393665c836155613ff283de66/v1.11.0.tar.gz) | | ||
| 795 | +| yaml-cpp | 0.8.0 | [yaml-cpp-0.8.0.tar.gz](https://raw.gitcode.com/src-openeuler/yaml-cpp/blobs/d1ead4fff417073b9cdbf98b8b55eb0efc00b0ba/yaml-cpp-0.8.0.tar.gz) | | ||
| 796 | +| sqlite | 3.51.0 | [sqlite-amalgamation-3510300.zip](https://www.sqlite.org/2026/sqlite-amalgamation-3510300.zip) | | ||
| 797 | +| googletest | 1.14.0 | [googletest-1.14.0.tar.gz](https://gitcode.com/cann-src-third-party/googletest/releases/download/v1.14.0/googletest-1.14.0.tar.gz) | | ||
| 798 | + | ||
| 799 | +### Glossary | ||
| 800 | + | ||
| 801 | +| Term | Description | | ||
| 802 | +|------------|------------------------------------------------------------------| | ||
| 803 | +| HCCL | Huawei Collective Communication Library | | ||
| 804 | +| NPU | Neural Processing Unit | | ||
| 805 | +| CANN | Compute Architecture for Neural Networks, Huawei Ascend AI processor software stack | | ||
| 806 | +| MPI | Message Passing Interface | | ||
| 807 | +| CCU | Collective Communication Unit | | ||
| 808 | +| Topology | Device connection relationship | | ||
| 809 | +| Rank | Process identifier in MPI | | ||
| 810 | + | ||
| 811 | +--- | ||
| 812 | + | ||
| 813 | +**Document Version**: v1.1. | ||
| 814 | +**Last Updated**: 2026-06-30. | ||
| @@ -0,0 +1,737 @@ | |||
| 1 | +# Checker Quick Start Guide | ||
| 2 | + | ||
| 3 | +This document introduces the basic concepts and processing flow of Checker. For error code specific scenarios, see [Error Code FAQ](faq/modules/checker_faq_en.md). | ||
| 4 | + | ||
| 5 | +[toc] | ||
| 6 | + | ||
| 7 | +## 1 What is Checker | ||
| 8 | + | ||
| 9 | +Checker is a static verification tool. It does not intervene in operator execution. Instead, it reads the records left after operator execution, reconstructs the execution graph, and performs static analysis to determine whether the operator execution is logically correct. | ||
| 10 | +- Checker input: operator information + task data of each rank + CCU instruction sequence | ||
| 11 | +- Checker output: verification conclusion (success/failure) + error log | ||
| 12 | + | ||
| 13 | +--- | ||
| 14 | + | ||
| 15 | +## 2 Checker Overall Flow | ||
| 16 | + | ||
| 17 | +### 2.1 Key Terms | ||
| 18 | + | ||
| 19 | +| Term | Description | | ||
| 20 | +|------|-------------| | ||
| 21 | +| Communication domain | A group of communication members, describing the communication scope | | ||
| 22 | +| Communication member | Usually referred to as a rank, the smallest logical entity participating in communication. Each rank is assigned a unique identifier called `rankId` | | ||
| 23 | +| Communication operator | A collective communication operation, such as `AllReduce` or `AllGather`. Different communication algorithms may be used depending on network topology, data volume, hardware resources, etc. | | ||
| 24 | +| Task | The core data structure of Checker, describing a record of an atomic operation for a rank, such as memory copy, Reduce, memory move, etc. | | ||
| 25 | +| Task graph | The core data structure of Checker, used to express the task nodes generated by an operator execution and their dependency relationships | | ||
| 26 | +| Node | Each node on the task graph is a task | | ||
| 27 | +| Stream | A queue within a rank that executes tasks sequentially. Each task uses `streamId` to record which queue it belongs to | | ||
| 28 | +| Task type | Different task types serve different purposes, such as memory copy, Reduce, or synchronization | | ||
| 29 | + | ||
| 30 | +### 2.2 Checker Processing Flow | ||
| 31 | + | ||
| 32 | +```mermaid | ||
| 33 | +flowchart TD | ||
| 34 | + INPUT["Input"] | ||
| 35 | + GG["Graph Generation\nGenGraph"] | ||
| 36 | + ST["Single Task Check\nSingleTaskCheck"] | ||
| 37 | + MC["Memory Conflict Detection\nMemConflict"] | ||
| 38 | + SC["Semantic Check\nSemanticCheck"] | ||
| 39 | + INPUT --> GG --> ST --> MC --> SC | ||
| 40 | +``` | ||
| 41 | + | ||
| 42 | +| Phase | Purpose | | ||
| 43 | +|-------|---------| | ||
| 44 | +| Graph Generation | Generate the task graph based on Checker input. In CCU mode, CCU instructions are converted into the task graph | | ||
| 45 | +| Single Task Check | Check whether the memory range of a single task is valid, and verify the slave stream head/tail structure | | ||
| 46 | +| Memory Conflict Detection | Check for unprotected memory overlaps between concurrent tasks | | ||
| 47 | +| Semantic Check | Simulate the operator execution process and verify whether the final output meets operator expectations | | ||
| 48 | + | ||
| 49 | +--- | ||
| 50 | + | ||
| 51 | +## 3 Task Graph | ||
| 52 | + | ||
| 53 | +The task graph is the core data structure of Checker, used to express the task nodes generated by an operator execution and their dependency relationships. It is generated during the graph generation phase based on Checker input. | ||
| 54 | + | ||
| 55 | +### 3.1 Nodes and Edges | ||
| 56 | + | ||
| 57 | +Each node on the task graph is a task with a specific task type indicating the operation it performs. Common types are: | ||
| 58 | + | ||
| 59 | +| Task Type | Description | Core Fields | | ||
| 60 | +|-----------|-------------|-------------| | ||
| 61 | +| `TRANS_MEM` | Memory data copy | `srcRankId`, `srcOffset`<br>`dstRankId`, `dstOffset`<br>`len`, `type` | | ||
| 62 | +| `BATCH_TRANS_MEM` | Batch memory data copy, containing multiple `(src -> dst)` relationships per node | `srcs[]`<br>`dsts[]` | | ||
| 63 | +| `REDUCE` | Data reduce | `srcRankId`, `srcOffset`<br>`dstRankId`, `dstOffset`<br>`type`, `dataCount`, `dataType`, `reduceOp` | | ||
| 64 | +| `BATCH_REDUCE` | Batch data reduce, containing multiple `(src -> dst)` reduce relationships per node | `srcs[][]`<br>`dsts[]`<br>`dataType`, `reduceOp` | | ||
| 65 | +| `RECORD` / `WAIT` | Synchronization tasks, representing sending and waiting for synchronization signals respectively | `srcRankId` (sender)<br>`dstRankId` (waiter)<br>`notifyId` | | ||
| 66 | + | ||
| 67 | +In addition to the real execution tasks above, the task graph also includes `START` / `END` virtual boundary nodes. They do not correspond to actual data copy or computation, but serve to mark the boundaries of the main graph, subgraphs, and Loop structures. | ||
| 68 | + | ||
| 69 | +| Virtual Node Type | Supported `boundaryType` | Description | | ||
| 70 | +|-------------------|--------------------------|-------------| | ||
| 71 | +| `START` | `MAIN_GRAPH`, `CCU_SUB_GRAPH`, `AIV_SUB_GRAPH`, `LOOP` | Start boundary node. Marks the entry of the entire task graph, or the start of a CCU/AIV subgraph or Loop fragment | | ||
| 72 | +| `END` | `CCU_SUB_GRAPH`, `AIV_SUB_GRAPH`, `LOOP` | End boundary node. Marks the end of a CCU/AIV subgraph or Loop fragment, and converges the tail nodes within the boundary | | ||
| 73 | + | ||
| 74 | +Edges represent the execution order relationships between nodes. A directed edge means the tail node executes after the head node. Edges can be categorized as follows: | ||
| 75 | + | ||
| 76 | +| Edge Type | Description | | ||
| 77 | +|-----------|-------------| | ||
| 78 | +| Sequential edge | Dependency edge connecting tasks in execution order within the same stream. Example: two sequential tasks on `rank0/stream0`, three sequential tasks on `rank1/stream0` | | ||
| 79 | +| Synchronization edge | Dependency edge between synchronization task nodes. Example: a `WAIT` node needs to wait for a signal from a `RECORD` node before it can proceed. An edge from `RECORD` to `WAIT` represents this dependency | | ||
| 80 | + | ||
| 81 | +```mermaid | ||
| 82 | +flowchart LR | ||
| 83 | + START(["start"]) | ||
| 84 | + | ||
| 85 | + subgraph SG0["rank0 / stream0"] | ||
| 86 | + direction LR | ||
| 87 | + R0T1["TRANS_MEM"] --> R0REC["RECORD\nnotifyId=42"] | ||
| 88 | + end | ||
| 89 | + | ||
| 90 | + subgraph SG1["rank1 / stream0"] | ||
| 91 | + direction LR | ||
| 92 | + R1T1["TRANS_MEM"] --> R1WAI["WAIT\nnotifyId=42"] --> R1T2["REDUCE"] | ||
| 93 | + end | ||
| 94 | + | ||
| 95 | + START --> R0T1 | ||
| 96 | + START --> R1T1 | ||
| 97 | + R0REC -.-> R1WAI | ||
| 98 | +``` | ||
| 99 | + | ||
| 100 | +### 3.2 Task Graph Examples | ||
| 101 | + | ||
| 102 | +#### 3.2.1 AICPU Mode | ||
| 103 | + | ||
| 104 | +In AICPU mode, the task graph typically consists of `RECORD`, `WAIT`, and `TRANS_MEM` nodes. Below is a typical 2-rank `AllGather` example with each rank containing two streams, showing sequential edges in actual execution order and dashed lines for `RECORD` to `WAIT` synchronization dependencies. | ||
| 105 | + | ||
| 106 | +```mermaid | ||
| 107 | +flowchart LR | ||
| 108 | + subgraph R0["rank0"] | ||
| 109 | + direction TB | ||
| 110 | + subgraph R0S0["stream0 main stream"] | ||
| 111 | + direction LR | ||
| 112 | + R0REC0["RECORD\nnotifyId=101"] --> R0WAI1S0["WAIT\nnotifyId=101"] | ||
| 113 | + end | ||
| 114 | + subgraph R0S1["stream1 slave stream"] | ||
| 115 | + direction LR | ||
| 116 | + R0WAI1S1["WAIT\nnotifyId=100"] --> R0TM0["TRANS_MEM\nsrc=Rank0-INPUT-[0x0,0x100)\ndst=Rank0-CCL-[0x0,0x100)"] --> R0REC2["RECORD\nnotifyId=102"] --> R0WAI3["WAIT\nnotifyId=103"] --> R0TM1["TRANS_MEM\nsrc=Rank0-CCL-[0x0,0x100)\ndst=Rank1-CCL-[0x0,0x100)"] --> R0REC4["RECORD\nnotifyId=104"] --> R0WAI5["WAIT\nnotifyId=105"] --> R0TM2["TRANS_MEM\nsrc=Rank0-CCL-[0x0,0x200)\ndst=Rank0-OUTPUT-[0x0,0x200)"] --> R0REC1["RECORD\nnotifyId=100"] | ||
| 117 | + end | ||
| 118 | + end | ||
| 119 | + | ||
| 120 | + subgraph R1["rank1"] | ||
| 121 | + direction TB | ||
| 122 | + subgraph R1S0["stream0 main stream"] | ||
| 123 | + direction LR | ||
| 124 | + R1REC6["RECORD\nnotifyId=106"] --> R1WAI7["WAIT\nnotifyId=107"] | ||
| 125 | + end | ||
| 126 | + subgraph R1S1["stream1 slave stream"] | ||
| 127 | + direction LR | ||
| 128 | + R1WAI6["WAIT\nnotifyId=106"] --> R1TM0["TRANS_MEM\nsrc=Rank1-INPUT-[0x0,0x100)\ndst=Rank1-CCL-[0x100,0x200)"] --> R1REC3["RECORD\nnotifyId=103"] --> R1WAI2["WAIT\nnotifyId=102"] --> R1TM1["TRANS_MEM\nsrc=Rank1-CCL-[0x100,0x200)\ndst=Rank0-CCL-[0x100,0x200)"] --> R1REC5["RECORD\nnotifyId=105"] --> R1WAI4["WAIT\nnotifyId=104"] --> R1TM2["TRANS_MEM\nsrc=Rank1-CCL-[0x0,0x200)\ndst=Rank1-OUTPUT-[0x0,0x200)"] --> R1REC7["RECORD\nnotifyId=107"] | ||
| 129 | + end | ||
| 130 | + end | ||
| 131 | + | ||
| 132 | + R0REC0 -.-> R0WAI1S1 | ||
| 133 | + R0REC1 -.-> R0WAI1S0 | ||
| 134 | + R1REC6 -.-> R1WAI6 | ||
| 135 | + R1REC7 -.-> R1WAI7 | ||
| 136 | + R0REC2 -.-> R1WAI2 | ||
| 137 | + R1REC3 -.-> R0WAI3 | ||
| 138 | + R0REC4 -.-> R1WAI4 | ||
| 139 | + R1REC5 -.-> R0WAI5 | ||
| 140 | +``` | ||
| 141 | + | ||
| 142 | +#### 3.2.2 CCU Mode | ||
| 143 | + | ||
| 144 | +In CCU mode, Checker expands CCU instructions into CCU subgraphs. Using the 2-rank `AllReduce` data flow, the synchronization operations outside the CCU subgraph are omitted, retaining only the internal task sequence of the CCU subgraph. CCU uses `cke` / `mask` instead of `notifyId` for synchronization, and uses the CCU `MS` type for the intermediate buffer. | ||
| 145 | + | ||
| 146 | +```mermaid | ||
| 147 | +flowchart LR | ||
| 148 | + subgraph R0["rank0"] | ||
| 149 | + direction TB | ||
| 150 | + subgraph R0Q0["Stream0 / CCU subgraph"] | ||
| 151 | + direction LR | ||
| 152 | + R0START["START\nboundaryType=CCU_SUB_GRAPH"] --> R0WAI0["WAIT\ncke=100\nmask=0x1"] --> R0TM0_CCU["TRANS_MEM\nsrc=Rank0-INPUT-[0x0,0x100)\ndst=Rank0-MS-[0x0,0x100)"] --> R0REC2_CCU["RECORD\ncke=102\nmask=0x1"] --> R0WAI3_CCU["WAIT\ncke=103\nmask=0x1"] --> R0TM1_CCU["TRANS_MEM\nsrc=Rank0-MS-[0x0,0x100)\ndst=Rank1-MS-[0x0,0x100)"] --> R0REC4_CCU["RECORD\ncke=104\nmask=0x1"] --> R0WAI5_CCU["WAIT\ncke=105\nmask=0x1"] --> R0TM2_CCU["TRANS_MEM\nsrc=Rank0-MS-[0x0,0x200)\ndst=Rank0-OUTPUT-[0x0,0x200)"] --> R0REC1_CCU["RECORD\ncke=100\nmask=0x1"] --> R0END["END\nboundaryType=CCU_SUB_GRAPH"] | ||
| 153 | + end | ||
| 154 | + end | ||
| 155 | + | ||
| 156 | + subgraph R1["rank1"] | ||
| 157 | + direction TB | ||
| 158 | + subgraph R1Q0["Stream0 / CCU subgraph"] | ||
| 159 | + direction LR | ||
| 160 | + R1START["START\nboundaryType=CCU_SUB_GRAPH"] --> R1WAI6_CCU["WAIT\ncke=106\nmask=0x1"] --> R1TM0_CCU["TRANS_MEM\nsrc=Rank1-INPUT-[0x0,0x100)\ndst=Rank1-MS-[0x100,0x200)"] --> R1REC3_CCU["RECORD\ncke=103\nmask=0x1"] --> R1WAI2_CCU["WAIT\ncke=102\nmask=0x1"] --> R1TM1_CCU["TRANS_MEM\nsrc=Rank1-MS-[0x100,0x200)\ndst=Rank0-MS-[0x100,0x200)"] --> R1REC5_CCU["RECORD\ncke=105\nmask=0x1"] --> R1WAI4_CCU["WAIT\ncke=104\nmask=0x1"] --> R1TM2_CCU["TRANS_MEM\nsrc=Rank1-MS-[0x0,0x200)\ndst=Rank1-OUTPUT-[0x0,0x200)"] --> R1REC7_CCU["RECORD\ncke=107\nmask=0x1"] --> R1END["END\nboundaryType=CCU_SUB_GRAPH"] | ||
| 161 | + end | ||
| 162 | + end | ||
| 163 | + | ||
| 164 | + R0REC2_CCU -.-> R1WAI2_CCU | ||
| 165 | + R1REC3_CCU -.-> R0WAI3_CCU | ||
| 166 | + R0REC4_CCU -.-> R1WAI4_CCU | ||
| 167 | + R1REC5_CCU -.-> R0WAI5_CCU | ||
| 168 | +``` | ||
| 169 | + | ||
| 170 | +#### 3.2.3 Graphviz Visualization | ||
| 171 | + | ||
| 172 | +Checker provides task graph export capability for outputting the task graph as a Graphviz `.dot` file. | ||
| 173 | + | ||
| 174 | +The exported content includes not only "which nodes exist" but also commonly used debugging information directly in the graph: | ||
| 175 | +- Nodes arranged by `rank / stream` for easy observation of sequential relationships on the same execution queue | ||
| 176 | +- Solid lines for normal dependency edges, dashed lines for `RECORD -> WAIT` synchronization dependencies | ||
| 177 | +- Node labels include task type, `nodeId`, location information, key fields such as memory slices, `notifyId`, or `cke/mask` | ||
| 178 | + | ||
| 179 | +Usage: | ||
| 180 | +- Checker automatically attempts to export the `.dot` file after graph generation without requiring an additional switch | ||
| 181 | +- After successful export, search the logs for `[GraphvizDot]` to find the output path, typically `hccl_vm_install/data/` | ||
| 182 | +- Output file name format: `TaskGraph_YYYYMMDDHHMMSS.dot` | ||
| 183 | + | ||
| 184 | +> After obtaining the `.dot` file, use `Microsoft VS Code` plugins such as `Graphviz Interactive Preview` for instant browsing. | ||
| 185 | + | ||
| 186 | +--- | ||
| 187 | + | ||
| 188 | +## 4. Single Task Check | ||
| 189 | + | ||
| 190 | +This phase checks whether the memory range of a single task is valid and verifies the slave stream head/tail structure. | ||
| 191 | + | ||
| 192 | +### 4.1 MemSlice | ||
| 193 | + | ||
| 194 | +The most important information in memory copy and reduce tasks is the memory slice (MemSlice). A memory slice consists of the following: | ||
| 195 | + | ||
| 196 | +``` | ||
| 197 | +MemSlice = { rankId, type, offset, len } | ||
| 198 | +``` | ||
| 199 | + | ||
| 200 | +- `rankId` indicates which rank the memory belongs to | ||
| 201 | +- `type` indicates the memory type | ||
| 202 | + | Memory Type | Usage | | ||
| 203 | + |-------------|-------| | ||
| 204 | + | INPUT | Operator input buffer | | ||
| 205 | + | OUTPUT | Operator output buffer | | ||
| 206 | + | CCL | CCL buffer | | ||
| 207 | + | MS_CCU | CCU MS | | ||
| 208 | +- `offset` and `len` together define the memory access range | ||
| 209 | + - `offset` is the starting address of this access on the memory slice | ||
| 210 | + - `len` is the length of this memory access | ||
| 211 | + - The access range uses half-open notation: `[offset, offset + length)` | ||
| 212 | + | ||
| 213 | +The check points for a single task's memory slice are: | ||
| 214 | +- `offset + length` must not overflow the `uint64` upper bound, otherwise an error is reported | ||
| 215 | +- Multiple MemSlices within the same task with the same `(rankId, memType)` must not overlap, otherwise an error is reported | ||
| 216 | + | ||
| 217 | + ```mermaid | ||
| 218 | + gantt | ||
| 219 | + title MemSlice Range Comparison | ||
| 220 | + dateFormat x | ||
| 221 | + axisFormat %L | ||
| 222 | + tickInterval 100millisecond | ||
| 223 | + | ||
| 224 | + section Valid (no overlap) | ||
| 225 | + Slice A 0x000-0x400 : 0, 400 | ||
| 226 | + Slice B 0x400-0x800 : 400, 800 | ||
| 227 | + | ||
| 228 | + section Invalid (overlap) | ||
| 229 | + Slice A 0x000-0x600 : crit, 0, 600 | ||
| 230 | + Slice B 0x400-0x800 : crit, 400, 800 | ||
| 231 | + ``` | ||
| 232 | + | ||
| 233 | +- Different `type` values represent independent address spaces. The same `offset` under different `type` values is not considered overlapping | ||
| 234 | +- `offset + length` must not exceed the boundary of the current `type` address space | ||
| 235 | + | ||
| 236 | + ```mermaid | ||
| 237 | + gantt | ||
| 238 | + title MemSlice Boundary Check | ||
| 239 | + dateFormat x | ||
| 240 | + axisFormat %L | ||
| 241 | + tickInterval 100millisecond | ||
| 242 | + | ||
| 243 | + section Valid (within bounds) | ||
| 244 | + Type Space [0x000,0x800) : 0, 800 | ||
| 245 | + MemSlice [0x200,0x500) : 200, 500 | ||
| 246 | + | ||
| 247 | + section Invalid (out of bounds) | ||
| 248 | + Type Space [0x000,0x800) : 0, 800 | ||
| 249 | + MemSlice [0x600,0x900) : crit, 600, 900 | ||
| 250 | + ``` | ||
| 251 | + | ||
| 252 | +### 4.2 Slave Stream Structure Check | ||
| 253 | + | ||
| 254 | +The slave stream executes auxiliary tasks for the operator, such as data pre-copy. In the HCCL programming model, the main stream triggers the slave stream via a synchronization task `RECORD -> WAIT`. After the slave stream completes, it notifies the main stream through another set of synchronization tasks. Therefore, the slave stream must satisfy a fixed head/tail structure: first task is `WAIT` && last task is `RECORD`. | ||
| 255 | + | ||
| 256 | +The following diagram shows an incorrect example, with the offending nodes highlighted in red: | ||
| 257 | + | ||
| 258 | +```mermaid | ||
| 259 | +flowchart LR | ||
| 260 | + START(["start"]) | ||
| 261 | + | ||
| 262 | + subgraph MAIN["rank0 / stream0 (main stream)"] | ||
| 263 | + M0["TRANS_MEM"] | ||
| 264 | + M1["RECORD\nnotifyId=10\ntriggers slave stream"] | ||
| 265 | + M2["WAIT\nnotifyId=11\nwaiting for slave stream"] | ||
| 266 | + M3["REDUCE"] | ||
| 267 | + M0 --> M1 --> M2 --> M3 | ||
| 268 | + end | ||
| 269 | + | ||
| 270 | + subgraph SLAVE["rank0 / stream1 (slave stream, invalid example)"] | ||
| 271 | + S0["TRANS_MEM\nerror: first task is not WAIT"] | ||
| 272 | + S1["REDUCE"] | ||
| 273 | + S2["WAIT\nerror: last task is not RECORD"] | ||
| 274 | + S0 --> S1 --> S2 | ||
| 275 | + end | ||
| 276 | + | ||
| 277 | + START --> M0 | ||
| 278 | + M1 -. sync edge .-> S0 | ||
| 279 | + S2 -. sync edge .-> M2 | ||
| 280 | + | ||
| 281 | + classDef invalid fill:#fde2e2,stroke:#c62828,stroke-width:2px,color:#7f1d1d; | ||
| 282 | + class S0,S2 invalid; | ||
| 283 | +``` | ||
| 284 | + | ||
| 285 | +--- | ||
| 286 | + | ||
| 287 | +## 5. Memory Conflict Check | ||
| 288 | + | ||
| 289 | +This phase checks whether there is a potential memory conflict in the task graph. A memory conflict occurs when multiple memory operations access the same memory segment at the same time, and at least one operation is a write. When a memory conflict occurs, the value of the conflicting memory segment is indeterminate, leading to accuracy issues in collective communication operators. | ||
| 290 | + | ||
| 291 | +### 5.1 Memory Conflict Criteria | ||
| 292 | + | ||
| 293 | +Two memory-accessing task nodes are judged as having a memory conflict when all three conditions below are met: | ||
| 294 | +1. The two nodes may execute concurrently (no path exists between the two nodes on the task graph) | ||
| 295 | +2. The accessed memory address ranges overlap | ||
| 296 | +3. At least one is a write operation | ||
| 297 | + | ||
| 298 | +Checker efficiently checks every pair of memory-accessing task nodes to ensure no false negatives. | ||
| 299 | + | ||
| 300 | +### 5.2 Memory Conflict Example | ||
| 301 | + | ||
| 302 | +The following diagram shows a task graph with a memory conflict: | ||
| 303 | + | ||
| 304 | +```mermaid | ||
| 305 | +flowchart LR | ||
| 306 | + START(["start"]) | ||
| 307 | + | ||
| 308 | + subgraph R0["rank0 / stream0"] | ||
| 309 | + R0A["R0A\n\nTRANS_MEM\nsrc={rankId=0, type=INPUT, offset=0x000, len=0x400}\ndst={rankId=0, type=CCL, offset=0x000, len=0x400}"] | ||
| 310 | + R0REC["RECORD\nnotifyId=7"] | ||
| 311 | + R0B["R0B\n\nTRANS_MEM\nsrc={rankId=0, type=CCL, offset=0x200, len=0x200}\ndst={rankId=1, type=OUTPUT, offset=0x000, len=0x200}"] | ||
| 312 | + R0A --> R0REC --> R0B | ||
| 313 | + end | ||
| 314 | + | ||
| 315 | + subgraph R1["rank1 / stream0"] | ||
| 316 | + R1WAIT["WAIT\nnotifyId=7"] | ||
| 317 | + R1B["R1B\n\nTRANS_MEM\nsrc={rankId=0, type=CCL, offset=0x000, len=0x200}\ndst={rankId=1, type=OUTPUT, offset=0x000, len=0x200}"] | ||
| 318 | + R1WAIT --> R1B | ||
| 319 | + end | ||
| 320 | + | ||
| 321 | + START --> R0A | ||
| 322 | + START --> R1WAIT | ||
| 323 | + R0REC -.-> R1WAIT | ||
| 324 | + | ||
| 325 | + classDef conflict fill:#fde2e2,stroke:#c62828,stroke-width:2px,color:#7f1d1d; | ||
| 326 | + class R0B,R1B conflict; | ||
| 327 | +``` | ||
| 328 | + | ||
| 329 | +- `R0A` and `R0B` execute sequentially on the same stream, so concurrent execution is not possible and no memory conflict occurs | ||
| 330 | +- `R0A` and `R1B` have their execution order constrained by synchronization nodes: `R0A -> R0RECORD -> R1WAIT -> R1B`, so no memory conflict occurs | ||
| 331 | +- `R0B` and `R1B` can execute concurrently, and their write memory `dst` completely overlaps, so a memory conflict exists | ||
| 332 | + | ||
| 333 | +### 5.3 Conflict Log Interpretation | ||
| 334 | + | ||
| 335 | +The error log format for memory conflict is as follows: | ||
| 336 | + | ||
| 337 | +```text | ||
| 338 | +[ErrorCode: 302] Two tasks may access the same memory range in parallel, and at least one access is a write. | ||
| 339 | + Conflict memory : rank 0 OUTPUT | ||
| 340 | + Overlap range : [0x0,0xc80) | ||
| 341 | + Conflict task 1: | ||
| 342 | + node 17, action=write | ||
| 343 | + access range : [0x0,0xc80) | ||
| 344 | + task : [TaskTransMem] node=17, rank=1, stream=0, queue=0, protocol=SDMA, src=rank 1 CCL [0x0,0xc80), dst=rank 0 OUTPUT [0x0,0xc80) | ||
| 345 | + Conflict task 2: | ||
| 346 | + node 23, action=write | ||
| 347 | + access range : [0x0,0xc80) | ||
| 348 | + task : [TaskTransMem] node=23, rank=2, stream=0, queue=0, protocol=SDMA, src=rank 2 CCL [0x0,0xc80), dst=rank 0 OUTPUT [0x0,0xc80) | ||
| 349 | +``` | ||
| 350 | + | ||
| 351 | +Log description: | ||
| 352 | + | ||
| 353 | +| Field | Meaning | | ||
| 354 | +|-------|---------| | ||
| 355 | +| `[ErrorCode: 302]` | Memory conflict error code, corresponding to `MEMCONFLICT_DETECTED` | | ||
| 356 | +| `Conflict memory : rank X TYPE` | Location of the conflicting memory | | ||
| 357 | +| `Overlap range : [start,end)` | The actual overlapping address range of the two accesses | | ||
| 358 | +| `Conflict task 1 / Conflict task 2` | The two accesses determined to be "concurrently executable with overlapping addresses" | | ||
| 359 | +| `node X, action=read/write` | Node ID on the task graph and the read/write type of this access. If at least one is `write`, a conflict may be reported | | ||
| 360 | +| `access range : [start,end)` | The complete address range covered by this access, may not be identical to the `Overlap range` | | ||
| 361 | +| `task :` | Specific task details (task type, node ID, location, src/dst memory ranges, etc.) | | ||
| 362 | + | ||
| 363 | +--- | ||
| 364 | + | ||
| 365 | +## 6. Semantic Check | ||
| 366 | + | ||
| 367 | +The semantic check phase traverses the task graph topologically, simulates the operator execution process, and verifies whether the final output meets operator expectations. | ||
| 368 | + | ||
| 369 | +### 6.1 BufferSemantic | ||
| 370 | + | ||
| 371 | +During the semantic check, Checker maintains data source records for each memory segment: | ||
| 372 | + | ||
| 373 | +``` | ||
| 374 | +BufferSemantic = { | ||
| 375 | + startAddr: Memory segment start address (offset) | ||
| 376 | + size: Memory segment length (len) | ||
| 377 | + srcBufs: Set of memory sources, each item is {rankId, bufferType, srcAddr} | ||
| 378 | + isReduce: Whether it is a Reduce operation | ||
| 379 | + reduceType: Reduce operation type SUM/MAX/MIN/... | ||
| 380 | +} | ||
| 381 | +``` | ||
| 382 | + | ||
| 383 | +`srcBufs` records the data source of the memory. `bufferType` indicates the buffer type the source belongs to, such as `INPUT`, `OUTPUT`, or `CCL`. | ||
| 384 | + | ||
| 385 | +When each node executes, each `(src -> dst)` relationship is translated into one of the following two operations, then written back to the target address space. `(src -> dst)` represents a set of copy or reduce relationships from source address to destination address: | ||
| 386 | + | ||
| 387 | +| TaskType | Operation | Behavior | | ||
| 388 | +|----------|-----------|----------| | ||
| 389 | +| `TRANS_MEM` / `BATCH_TRANS_MEM` | overwrite | First clears the existing semantics of the target range, then copies the source semantics over | | ||
| 390 | +| `REDUCE` / `BATCH_REDUCE` | reduce | Requires that the target range is pre-filled with semantics, otherwise an error is reported. Then appends the new source to `srcBufs` and sets `isReduce=true` | | ||
| 391 | + | ||
| 392 | +### 6.2 OUTPUT Expectations by Operator | ||
| 393 | + | ||
| 394 | +The goal of semantic check is to determine whether each rank's `OUTPUT` meets the current operator's expectations. The expectations for different operators are: | ||
| 395 | + | ||
| 396 | +| Operator | OUTPUT Semantic Expectation | | ||
| 397 | +|----------|-----------------------------| | ||
| 398 | +| AllReduce | Each rank's OUTPUT is the reduce result of all ranks' INPUT | | ||
| 399 | +| AllGather | Each rank's OUTPUT is the concatenation of all ranks' INPUT in order | | ||
| 400 | +| ReduceScatter | Each rank's OUTPUT is the fragment of the global reduce assigned to this rank | | ||
| 401 | +| AllGatherV | Same as AllGather, but each rank contributes a different size | | ||
| 402 | +| ReduceScatterV | Same as ReduceScatter, but each rank's fragment size differs | | ||
| 403 | +| Send/Recv | The target rank's OUTPUT equals the source rank's INPUT, single source without reduce | | ||
| 404 | +| BatchSendRecv | Multiple pairs of Send/Recv simultaneously | | ||
| 405 | +| Broadcast | All ranks' OUTPUT equal the root rank's INPUT | | ||
| 406 | +| Reduce | Only the root rank's OUTPUT is the reduce result of all ranks' INPUT | | ||
| 407 | +| All2All | Each rank's `OUTPUT[i]` equals `rank i`'s `INPUT[this rank's offset]` | | ||
| 408 | + | ||
| 409 | +The diagrams below show the OUTPUT expectations for each operator using a 2-rank collective communication operator example: | ||
| 410 | + | ||
| 411 | +**AllReduce** | ||
| 412 | + | ||
| 413 | +```mermaid | ||
| 414 | +gantt | ||
| 415 | + title AllReduce (2-rank) | ||
| 416 | + dateFormat x | ||
| 417 | + axisFormat %L | ||
| 418 | + tickInterval 100millisecond | ||
| 419 | + | ||
| 420 | + section rank0.INPUT | ||
| 421 | + rank0.INPUT : 0, 100 | ||
| 422 | + | ||
| 423 | + section rank1.INPUT | ||
| 424 | + rank1.INPUT : 0, 100 | ||
| 425 | + | ||
| 426 | + section rank0.OUTPUT | ||
| 427 | + rank0.INPUT + rank1.INPUT : 0, 100 | ||
| 428 | + | ||
| 429 | + section rank1.OUTPUT | ||
| 430 | + rank0.INPUT + rank1.INPUT : 0, 100 | ||
| 431 | +``` | ||
| 432 | + | ||
| 433 | +**AllGather / AllGatherV** | ||
| 434 | + | ||
| 435 | +```mermaid | ||
| 436 | +gantt | ||
| 437 | + title AllGather (2-rank) | ||
| 438 | + dateFormat x | ||
| 439 | + axisFormat %L | ||
| 440 | + tickInterval 100millisecond | ||
| 441 | + | ||
| 442 | + section rank0.INPUT | ||
| 443 | + rank0.INPUT : 0, 100 | ||
| 444 | + | ||
| 445 | + section rank1.INPUT | ||
| 446 | + rank1.INPUT : 0, 100 | ||
| 447 | + | ||
| 448 | + section rank0.OUTPUT | ||
| 449 | + rank0.INPUT : 0, 100 | ||
| 450 | + rank1.INPUT : 100, 200 | ||
| 451 | + | ||
| 452 | + section rank1.OUTPUT | ||
| 453 | + rank0.INPUT : 0, 100 | ||
| 454 | + rank1.INPUT : 100, 200 | ||
| 455 | +``` | ||
| 456 | + | ||
| 457 | +`AllGatherV` semantics are the same as above, except that each rank's contribution length can differ. | ||
| 458 | + | ||
| 459 | +**ReduceScatter / ReduceScatterV** | ||
| 460 | + | ||
| 461 | +```mermaid | ||
| 462 | +gantt | ||
| 463 | + title ReduceScatter (2-rank, each slice Len=100) | ||
| 464 | + dateFormat x | ||
| 465 | + axisFormat %L | ||
| 466 | + tickInterval 100millisecond | ||
| 467 | + | ||
| 468 | + section rank0.INPUT | ||
| 469 | + rank0.INPUT1 : 0, 50 | ||
| 470 | + rank0.INPUT2 : 50, 100 | ||
| 471 | + | ||
| 472 | + section rank1.INPUT | ||
| 473 | + rank1.INPUT1 : 0, 50 | ||
| 474 | + rank1.INPUT2 : 50, 100 | ||
| 475 | + | ||
| 476 | + section rank0.OUTPUT | ||
| 477 | + rank0.INPUT1 + rank1.INPUT1 : 0, 50 | ||
| 478 | + | ||
| 479 | + section rank1.OUTPUT | ||
| 480 | + rank0.INPUT2 + rank1.INPUT2 : 50, 100 | ||
| 481 | +``` | ||
| 482 | + | ||
| 483 | +`ReduceScatterV` semantics are the same as above, except that each rank's output fragment size can differ. | ||
| 484 | + | ||
| 485 | +**Send/Recv** | ||
| 486 | + | ||
| 487 | +```mermaid | ||
| 488 | +gantt | ||
| 489 | + title Send/Recv (2-rank, src=0, dst=1) | ||
| 490 | + dateFormat x | ||
| 491 | + axisFormat %L | ||
| 492 | + tickInterval 100millisecond | ||
| 493 | + | ||
| 494 | + section rank0.INPUT | ||
| 495 | + rank0.INPUT : 0, 100 | ||
| 496 | + | ||
| 497 | + section rank1.INPUT | ||
| 498 | + None: 0,0 | ||
| 499 | + | ||
| 500 | + section rank0.OUTPUT | ||
| 501 | + None: 0,0 | ||
| 502 | + | ||
| 503 | + section rank1.OUTPUT | ||
| 504 | + rank0.INPUT : 0, 100 | ||
| 505 | +``` | ||
| 506 | + | ||
| 507 | +**BatchSendRecv** | ||
| 508 | + | ||
| 509 | +```mermaid | ||
| 510 | +gantt | ||
| 511 | + title BatchSendRecv (2-rank) | ||
| 512 | + dateFormat x | ||
| 513 | + axisFormat %L | ||
| 514 | + tickInterval 100millisecond | ||
| 515 | + | ||
| 516 | + section rank0.INPUT | ||
| 517 | + rank0.INPUT : 0, 100 | ||
| 518 | + | ||
| 519 | + section rank1.INPUT | ||
| 520 | + rank1.INPUT : 0, 100 | ||
| 521 | + | ||
| 522 | + section rank0.OUTPUT | ||
| 523 | + rank1.INPUT : 0, 100 | ||
| 524 | + | ||
| 525 | + section rank1.OUTPUT | ||
| 526 | + rank0.INPUT : 0, 100 | ||
| 527 | +``` | ||
| 528 | + | ||
| 529 | +**Broadcast** | ||
| 530 | + | ||
| 531 | +```mermaid | ||
| 532 | +gantt | ||
| 533 | + title Broadcast (2-rank, root=0) | ||
| 534 | + dateFormat x | ||
| 535 | + axisFormat %L | ||
| 536 | + tickInterval 100millisecond | ||
| 537 | + | ||
| 538 | + section rank0.INPUT | ||
| 539 | + rank0.INPUT : 0, 100 | ||
| 540 | + | ||
| 541 | + section rank1.INPUT | ||
| 542 | + None: 0,0 | ||
| 543 | + | ||
| 544 | + section rank0.OUTPUT | ||
| 545 | + rank0.INPUT : 0, 100 | ||
| 546 | + | ||
| 547 | + section rank1.OUTPUT | ||
| 548 | + rank0.INPUT : 0, 100 | ||
| 549 | +``` | ||
| 550 | + | ||
| 551 | +**Reduce** | ||
| 552 | + | ||
| 553 | +```mermaid | ||
| 554 | +gantt | ||
| 555 | + title Reduce (2-rank, root=0, SUM) | ||
| 556 | + dateFormat x | ||
| 557 | + axisFormat %L | ||
| 558 | + tickInterval 100millisecond | ||
| 559 | + | ||
| 560 | + section rank0.INPUT | ||
| 561 | + rank0.INPUT : 0, 100 | ||
| 562 | + | ||
| 563 | + section rank1.INPUT | ||
| 564 | + rank1.INPUT : 0, 100 | ||
| 565 | + | ||
| 566 | + section rank0.OUTPUT | ||
| 567 | + rank0.INPUT + rank1.INPUT : 0, 100 | ||
| 568 | + | ||
| 569 | + section rank1.OUTPUT | ||
| 570 | + None: 0,0 | ||
| 571 | +``` | ||
| 572 | + | ||
| 573 | +**All2All** | ||
| 574 | + | ||
| 575 | +```mermaid | ||
| 576 | +gantt | ||
| 577 | + title All2All (2-rank) | ||
| 578 | + dateFormat x | ||
| 579 | + axisFormat %L | ||
| 580 | + tickInterval 100millisecond | ||
| 581 | + | ||
| 582 | + section rank0.INPUT | ||
| 583 | + rank0.INPUT1 : 0, 100 | ||
| 584 | + rank0.INPUT2 : 100, 200 | ||
| 585 | + | ||
| 586 | + section rank1.INPUT | ||
| 587 | + rank1.INPUT1 : 0, 100 | ||
| 588 | + rank1.INPUT2 : 100, 200 | ||
| 589 | + | ||
| 590 | + section rank0.OUTPUT | ||
| 591 | + rank0.INPUT1 : 0, 100 | ||
| 592 | + rank1.INPUT1 : 100, 200 | ||
| 593 | + | ||
| 594 | + section rank1.OUTPUT | ||
| 595 | + rank0.INPUT2 : 0, 100 | ||
| 596 | + rank1.INPUT2 : 100, 200 | ||
| 597 | +``` | ||
| 598 | + | ||
| 599 | +### 6.3 Final Verification Flow | ||
| 600 | + | ||
| 601 | +```mermaid | ||
| 602 | +flowchart LR | ||
| 603 | + SIM["Topologically traverse the task graph, simulate each task, record BufferSemantic for each memory segment"] | ||
| 604 | + CHK["Check whether each rank's OUTPUT meets operator expectations"] | ||
| 605 | + PASS["Pass"] | ||
| 606 | + FAIL["Fail"] | ||
| 607 | + | ||
| 608 | + SIM --> CHK --> PASS | ||
| 609 | + CHK --> FAIL | ||
| 610 | +``` | ||
| 611 | + | ||
| 612 | +Using a 4-rank `AllReduce` as an example: | ||
| 613 | + | ||
| 614 | +``` | ||
| 615 | +rank0.OUTPUT[0,L) expected: | ||
| 616 | + sources = { rank0.INPUT, rank1.INPUT, rank2.INPUT, rank3.INPUT } | ||
| 617 | + reduceType = SUM | ||
| 618 | + | ||
| 619 | +Assuming sources is missing rank3.INPUT: | ||
| 620 | + actualSourceRankCount=3, expectedRankSize=4 -> check fails | ||
| 621 | +``` | ||
| 622 | + | ||
| 623 | +### 6.4 Semantic Propagation Example | ||
| 624 | + | ||
| 625 | +Using a 2-rank `AllGather` (each rank INPUT size 100 bytes) as an example to illustrate the semantic propagation process. | ||
| 626 | + | ||
| 627 | +**Initial State** | ||
| 628 | + | ||
| 629 | +Each rank's INPUT already has its own initial semantics (source pointing to itself): | ||
| 630 | + | ||
| 631 | +``` | ||
| 632 | +rank0.INPUT[0, 100): srcBufs = { (rank0, INPUT, 0) } | ||
| 633 | +rank1.INPUT[0, 100): srcBufs = { (rank1, INPUT, 0) } | ||
| 634 | +rank0.OUTPUT: empty | ||
| 635 | +rank1.OUTPUT: empty | ||
| 636 | +``` | ||
| 637 | + | ||
| 638 | +**Propagation Process** | ||
| 639 | + | ||
| 640 | +The diagram below shows the semantic filling process of rank0.OUTPUT. Each arrow represents an overwrite operation: reading the source buffer's semantics and writing them to the corresponding range of the target buffer. | ||
| 641 | + | ||
| 642 | +```mermaid | ||
| 643 | +flowchart LR | ||
| 644 | + subgraph SRC["Initial Semantics"] | ||
| 645 | + I0["rank0.INPUT[0,100)\nsrc = rank0"] | ||
| 646 | + I1["rank1.INPUT[0,100)\nsrc = rank1"] | ||
| 647 | + end | ||
| 648 | + | ||
| 649 | + subgraph TASK["TRANS_MEM task (overwrite)"] | ||
| 650 | + T1["rank0.INPUT[0,100) → rank0.OUTPUT[0,100)"] | ||
| 651 | + T2["rank1.INPUT[0,100) → rank0.OUTPUT[100,200)"] | ||
| 652 | + end | ||
| 653 | + | ||
| 654 | + subgraph DST["Final Semantics (rank0.OUTPUT)"] | ||
| 655 | + F0A["[0,100)\nsrc = rank0.INPUT"] | ||
| 656 | + F0B["[100,200)\nsrc = rank1.INPUT"] | ||
| 657 | + end | ||
| 658 | + | ||
| 659 | + I0 --> T1 --> F0A | ||
| 660 | + I1 --> T2 --> F0B | ||
| 661 | +``` | ||
| 662 | + | ||
| 663 | +The propagation process for rank1.OUTPUT is similar. Finally, both ranks' OUTPUT are filled, sources are correct, and the check passes. | ||
| 664 | + | ||
| 665 | +**Range Splitting** | ||
| 666 | + | ||
| 667 | +If the write range does not align with the existing semantic boundary, Checker splits first, then overwrites. For example, rank0.OUTPUT[0,100) already has a full semantic segment, and then [35,65) is written: | ||
| 668 | + | ||
| 669 | +```mermaid | ||
| 670 | +gantt | ||
| 671 | + title Range Splitting Illustration | ||
| 672 | + dateFormat x | ||
| 673 | + axisFormat %L | ||
| 674 | + tickInterval 100millisecond | ||
| 675 | + | ||
| 676 | + section Before Write | ||
| 677 | + rank0.OUTPUT[0,100) src=rank0.INPUT : 0, 100 | ||
| 678 | + | ||
| 679 | + section After Write | ||
| 680 | + [0,35) retain original semantics : 0, 35 | ||
| 681 | + [35,65) replace with new source :crit, 35, 65 | ||
| 682 | + [65,100) retain original semantics : 65, 100 | ||
| 683 | +``` | ||
| 684 | + | ||
| 685 | +Before the write, the original semantic block is split at offsets 35 and 65, [35,65) is replaced with the new source, and the remaining parts are kept unchanged. | ||
| 686 | + | ||
| 687 | +### 6.5 Two Root Causes of Semantic Errors | ||
| 688 | + | ||
| 689 | +Semantic check failures essentially have only two types of problems: | ||
| 690 | + | ||
| 691 | +- **Missing data**: The OUTPUT range was not written to, or was written incompletely (missing head, fragmented, or missing tail). | ||
| 692 | +- **Incorrect data source**: The OUTPUT is fully written, but the source rank, offset, or reduce type does not match expectations. | ||
| 693 | + | ||
| 694 | +When debugging, first determine the problem type: if data is missing, focus on whether the task scheduling lacks a transmission; if the data source is incorrect, focus on whether the src/dst addresses and reduceOp are correct. | ||
| 695 | + | ||
| 696 | +### 6.6 Error Log Interpretation | ||
| 697 | + | ||
| 698 | +```text | ||
| 699 | +[ErrorCode: 407] AllGather output range [0x1000,0x1400) for rank 3 should come from rank 4, but it actually comes from rank 5. | ||
| 700 | +Current result range detail: | ||
| 701 | + range=[0x1000,0x1400), size=0x400, sourceCount=1 | ||
| 702 | + sources: | ||
| 703 | + - sourceRank=5, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 704 | +``` | ||
| 705 | + | ||
| 706 | +Log description: | ||
| 707 | + | ||
| 708 | +| Line | Meaning | | ||
| 709 | +|------|---------| | ||
| 710 | +| Line 1 | Error code and main error message. `407` indicates an output source attribute error; `output range [0x1000,0x1400) for rank 3` specifies the failing output rank and range; `should come from rank 4, but it actually comes from rank 5` indicates the expected source rank differs from the actual source rank | | ||
| 711 | +| `Current result range detail` | Full semantic expansion of the current output range for further debugging | | ||
| 712 | +| `range / size / sourceCount` | Address range, length, and number of sources for the current output semantic block | | ||
| 713 | +| `sources` | Source list for the current range, each item includes source rank, source buffer type, and source address | | ||
| 714 | + | ||
| 715 | +--- | ||
| 716 | + | ||
| 717 | +## 7. Quick Term Reference | ||
| 718 | + | ||
| 719 | +| Term | Description | | ||
| 720 | +|------|-------------| | ||
| 721 | +| Communication domain | A group of communication members, describing the communication scope | | ||
| 722 | +| Communication member | Usually referred to as a rank, the smallest logical entity participating in communication. Each rank is assigned a unique identifier called `rankId` | | ||
| 723 | +| Communication operator | A collective communication operation, such as `AllReduce` or `AllGather` | | ||
| 724 | +| Task | The core data structure of Checker, which describes a record of one atomic operation performed by a given rank, such as memory transfer, Reduce, memory copy, etc | | ||
| 725 | +| Task graph | The core data structure of Checker, used to express task nodes and their dependency relationships | | ||
| 726 | +| Node | Each node on the task graph is a task | | ||
| 727 | +| Stream | A queue within a rank that executes tasks in sequential order, where each task uses a streamId to record which queue it belongs to | | ||
| 728 | +| Task type | Different task types serve different purposes, such as memory copy, Reduce, or synchronization | | ||
| 729 | +| Queue | CCU internal serial instruction queue | | ||
| 730 | +| MemSlice | Memory access range `{rankId, type, offset, len}` | | ||
| 731 | +| Slave stream | An auxiliary stream for executing secondary tasks, requiring first `WAIT`, last `RECORD` | | ||
| 732 | +| `RECORD` / `WAIT` | A pair of synchronization tasks for sending and waiting for synchronization signals | | ||
| 733 | +| Synchronization edge | A cross-stream or cross-rank execution dependency established by `RECORD -> WAIT` | | ||
| 734 | +| Memory conflict | Multiple memory operations access the same memory segment at the same time with at least one write | | ||
| 735 | +| BufferSemantic | An important data structure in the semantic check phase, recording where a memory segment's data comes from | | ||
| 736 | +| `reduceType` | The reduce type in semantics, such as `SUM`, `MAX`, `MIN`, used to describe how multi-source data is merged | | ||
| 737 | +| OUTPUT expectation | The correct semantic definition that the final output of a communication operator should satisfy, used for final comparison with actual results | | ||
| @@ -0,0 +1,1372 @@ | |||
| 1 | +# HCCL-VM FAQ Test Document | ||
| 2 | + | ||
| 3 | +> This document is used to test the FAQ HTML generation framework. | ||
| 4 | + | ||
| 5 | +--- | ||
| 6 | + | ||
| 7 | +## Module: HCCL-VM | ||
| 8 | + | ||
| 9 | +### Submodule: Command Line | ||
| 10 | + | ||
| 11 | +--- | ||
| 12 | + | ||
| 13 | +#### FAQ-E001 | ||
| 14 | + | ||
| 15 | +**Title:** Communication domain not configured | ||
| 16 | + | ||
| 17 | +**Error code:** | ||
| 18 | +``` | ||
| 19 | +NA (4) | ||
| 20 | +``` | ||
| 21 | + | ||
| 22 | +**Error function:** | ||
| 23 | +``` | ||
| 24 | +db_sim_runner_common.cc::GetDeviceByRankId() | ||
| 25 | +``` | ||
| 26 | + | ||
| 27 | +**Key log:** | ||
| 28 | +``` | ||
| 29 | +[error][PID:173579][TID:173579][db_sim_runner_common.cc][GetDeviceByRankId] cannot find rank by rank id 0 | ||
| 30 | +[error][PID:173579][TID:173579][aclrt_device_stub.cc][aclrtSetDevice] [DEVICE_STUB]device not found by rankId:0 | ||
| 31 | +acl interface return err ./common/src/hccl_test_common.cc:861, retcode: 100000. | ||
| 32 | +This is an error in device_init. | ||
| 33 | +``` | ||
| 34 | + | ||
| 35 | +**Symptoms:** When executing a business case, the device with rank id 0 cannot be found. | ||
| 36 | + | ||
| 37 | +**Troubleshooting:** | ||
| 38 | +``` | ||
| 39 | +[Possible Causes] | ||
| 40 | +Before executing the business case, users need to determine the communication domain size used by the operator and configure the communication domain using the `hccl-vm mock-comm aa` command. The aa.yaml file is located at $HCCL_VM_INSTALL_DIR/config/topo_meta/aa.yaml. | ||
| 41 | +``` | ||
| 42 | +--- | ||
| 43 | + | ||
| 44 | +#### FAQ-E002 | ||
| 45 | + | ||
| 46 | +**Title:** RANK_TABLE_FILE not set | ||
| 47 | + | ||
| 48 | +**Error code:** | ||
| 49 | +``` | ||
| 50 | +HCCL_SIM_E_PARA (1) | ||
| 51 | +``` | ||
| 52 | + | ||
| 53 | +**Error function:** | ||
| 54 | +``` | ||
| 55 | +hccl_comm_stub.cc::HcclCommInitRootInfo() | ||
| 56 | +``` | ||
| 57 | + | ||
| 58 | +**Key log:** | ||
| 59 | +``` | ||
| 60 | +RANK_TABLE_FILE env not set, please check your config. | ||
| 61 | +``` | ||
| 62 | + | ||
| 63 | +**Symptoms:** The rank table configuration file cannot be found during communication domain initialization. | ||
| 64 | + | ||
| 65 | +**Troubleshooting:** | ||
| 66 | +``` | ||
| 67 | +[Possible Causes] | ||
| 68 | +1. Environment variable not set | ||
| 69 | +2. Incorrect file path | ||
| 70 | + | ||
| 71 | +[Solution] | ||
| 72 | +export RANK_TABLE_FILE=/path/to/rank_table.json | ||
| 73 | +``` | ||
| 74 | +--- | ||
| 75 | + | ||
| 76 | +#### FAQ-E003 | ||
| 77 | + | ||
| 78 | +**Title:** HCCL_VM_INSTALL_DIR not set | ||
| 79 | + | ||
| 80 | +**Error code:** | ||
| 81 | +``` | ||
| 82 | +HCCL_SIM_E_INTERNAL (4) | ||
| 83 | +``` | ||
| 84 | + | ||
| 85 | +**Error function:** | ||
| 86 | +``` | ||
| 87 | +hccl_op_stub.cc::VirtualExecuteAivKernel() | ||
| 88 | +``` | ||
| 89 | + | ||
| 90 | +**Key log:** | ||
| 91 | +``` | ||
| 92 | +[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set, can not locate <path> for kernel <name> | ||
| 93 | +``` | ||
| 94 | + | ||
| 95 | +**Symptoms:** AIV kernel virtual execution failed; the corresponding .so file cannot be found. | ||
| 96 | + | ||
| 97 | +**Troubleshooting:** | ||
| 98 | +``` | ||
| 99 | +[Solution] | ||
| 100 | +export HCCL_VM_INSTALL_DIR=/path/to/hccl_vm/install/dir | ||
| 101 | +``` | ||
| 102 | +--- | ||
| 103 | + | ||
| 104 | +#### FAQ-E004 | ||
| 105 | + | ||
| 106 | +**Title:** Repeated execution of start command in a subshell | ||
| 107 | + | ||
| 108 | +**Error code:** | ||
| 109 | +``` | ||
| 110 | +NA (no error code, only WARNING) | ||
| 111 | +``` | ||
| 112 | + | ||
| 113 | +**Error function:** | ||
| 114 | +``` | ||
| 115 | +subcmd_start.cc::StartCommand::Execute() | ||
| 116 | +``` | ||
| 117 | + | ||
| 118 | +**Key log:** | ||
| 119 | +``` | ||
| 120 | +[warning][PID:<PID>][TID:<TID>][subcmd_start.cc][Execute] hccl-vm has already started. Please do not start it again in a sub-bash. | ||
| 121 | +``` | ||
| 122 | + | ||
| 123 | +**Symptoms:** In the hvm subshell environment, executing the `hccl-vm start` command again causes the system to prompt that it has already started and ignore this operation. | ||
| 124 | + | ||
| 125 | +**Troubleshooting:** | ||
| 126 | +``` | ||
| 127 | +[Possible Causes] | ||
| 128 | +`hccl-vm start` forks a sub-bash process. When the user enters `hccl-vm start` again inside that sub-bash (prompt `(hvm)$>`), the system refuses to start again. | ||
| 129 | + | ||
| 130 | +[Solution] | ||
| 131 | +Do not execute `hccl-vm start` repeatedly inside the subshell. To restart the simulation environment, first exit the current subshell (type `exit`), then re-execute `hccl-vm start`. | ||
| 132 | +``` | ||
| 133 | +--- | ||
| 134 | + | ||
| 135 | +#### FAQ-E005 | ||
| 136 | + | ||
| 137 | +**Title:** Fork subprocess failed | ||
| 138 | + | ||
| 139 | +**Error code:** | ||
| 140 | +``` | ||
| 141 | +HCCL_SIM_HOST_ERROR_CMD (no standard error code) | ||
| 142 | +``` | ||
| 143 | + | ||
| 144 | +**Error function:** | ||
| 145 | +``` | ||
| 146 | +cmd_base_utils.cc::StartHvmCmd() | ||
| 147 | +``` | ||
| 148 | + | ||
| 149 | +**Key log:** | ||
| 150 | +``` | ||
| 151 | +fork failed: Resource temporarily unavailable | ||
| 152 | +``` | ||
| 153 | + | ||
| 154 | +**Symptoms:** After executing the `hccl-vm start` command, the system cannot create a subshell process, and the simulation environment fails to start. | ||
| 155 | + | ||
| 156 | +**Troubleshooting:** | ||
| 157 | +``` | ||
| 158 | +[Possible Causes] | ||
| 159 | +1. The system user process limit has been reached (ulimit -u) | ||
| 160 | +2. Insufficient system memory to allocate resources for the new process | ||
| 161 | +3. PID resources exhausted (/proc/sys/kernel/pid_max) | ||
| 162 | + | ||
| 163 | +[Steps] | ||
| 164 | +ulimit -u | ||
| 165 | +cat /proc/sys/kernel/pid_max | ||
| 166 | +free -m | ||
| 167 | +ps -eLf | wc -l | ||
| 168 | + | ||
| 169 | +[Solution] | ||
| 170 | +1. Increase the user process limit: `ulimit -u <larger value>` | ||
| 171 | +2. Clean up zombie processes remaining in the system | ||
| 172 | +3. Check if other programs are consuming excessive system resources | ||
| 173 | +``` | ||
| 174 | +--- | ||
| 175 | + | ||
| 176 | +#### FAQ-E006 | ||
| 177 | + | ||
| 178 | +**Title:** Plugin name format error | ||
| 179 | + | ||
| 180 | +**Error code:** | ||
| 181 | +``` | ||
| 182 | +NA (CLI parameter validation) | ||
| 183 | +``` | ||
| 184 | + | ||
| 185 | +**Error function:** | ||
| 186 | +``` | ||
| 187 | +subcmd_plugin.cc::PluginCommand::Setup() | ||
| 188 | +``` | ||
| 189 | + | ||
| 190 | +**Key log:** | ||
| 191 | +``` | ||
| 192 | +[HVM] [ERROR] Install plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 193 | +[HVM] [ERROR] Uninstall plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 194 | +[HVM] [ERROR] Run plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 195 | +``` | ||
| 196 | + | ||
| 197 | +**Symptoms:** When executing the `hccl-vm plugin install/run/uninstall` command, CLI parameter validation fails and the operation is rejected. | ||
| 198 | + | ||
| 199 | +**Troubleshooting:** | ||
| 200 | +``` | ||
| 201 | +[Possible Causes] | ||
| 202 | +The plugin name does not start with the `@` symbol. For example, entering `hccl-vm plugin install runner` instead of `hccl-vm plugin install @runner`. | ||
| 203 | + | ||
| 204 | +[Solution] | ||
| 205 | +Ensure the plugin name starts with `@`, for example: | ||
| 206 | +hccl-vm plugin install @runner | ||
| 207 | +hccl-vm plugin install @checker | ||
| 208 | +hccl-vm plugin uninstall @runner | ||
| 209 | +``` | ||
| 210 | +--- | ||
| 211 | + | ||
| 212 | +#### FAQ-E007 | ||
| 213 | + | ||
| 214 | +**Title:** Topology configuration file not found | ||
| 215 | + | ||
| 216 | +**Error code:** | ||
| 217 | +``` | ||
| 218 | +NA (CLI parameter validation) | ||
| 219 | +``` | ||
| 220 | + | ||
| 221 | +**Error function:** | ||
| 222 | +``` | ||
| 223 | +cmd_base_utils.cc::FileInModelDir() | ||
| 224 | +``` | ||
| 225 | + | ||
| 226 | +**Key log:** | ||
| 227 | +``` | ||
| 228 | +[HVM] model File not found: <install_path>/config/topo_meta/<name>.yaml | ||
| 229 | +``` | ||
| 230 | + | ||
| 231 | +**Symptoms:** When executing the `hccl-vm mock-comm <name>` command, the specified topology yaml configuration file does not exist, and CLI parameter validation directly rejects the operation. The communication domain configuration file describes the scale of the operator's communication domain (e.g., how many super nodes, how many servers, and which cards are selected within each server; see the file description for details). | ||
| 232 | + | ||
| 233 | +**Troubleshooting:** | ||
| 234 | +``` | ||
| 235 | +[Possible Causes] | ||
| 236 | +1. The specified topology name is misspelled | ||
| 237 | +2. The corresponding yaml file is not placed in the `$HCCL_VM_INSTALL_DIR/config/topo_meta/` directory | ||
| 238 | +3. Incorrect file extension (should be `.yaml`) | ||
| 239 | + | ||
| 240 | +[Steps] | ||
| 241 | +ls $HCCL_VM_INSTALL_DIR/config/topo_meta/ | ||
| 242 | + | ||
| 243 | +[Solution] | ||
| 244 | +Ensure the topology yaml file is placed in the correct directory and the filename matches the command parameter. For example, executing `hccl-vm mock-comm 121` requires the `config/topo_meta/121.yaml` file to exist. | ||
| 245 | +``` | ||
| 246 | +--- | ||
| 247 | + | ||
| 248 | +#### FAQ-E008 | ||
| 249 | + | ||
| 250 | +**Title:** YAML topology file format parsing error | ||
| 251 | + | ||
| 252 | +**Error code:** | ||
| 253 | +``` | ||
| 254 | +NA (runtime parsing error) | ||
| 255 | +``` | ||
| 256 | + | ||
| 257 | +**Error function:** | ||
| 258 | +``` | ||
| 259 | +cmd_cluster_model_utils.cc::ParseYamlTopoImpl() | ||
| 260 | +``` | ||
| 261 | + | ||
| 262 | +**Key log:** | ||
| 263 | +``` | ||
| 264 | +[error][PID:<PID>][TID:<TID>][cmd_cluster_model_utils.cc][ParseYamlTopoImpl] Exception when parsing YAML: <detail> | ||
| 265 | +``` | ||
| 266 | + | ||
| 267 | +**Symptoms:** When executing the `hccl-vm mock-comm <name>` command, the YAML topology configuration file parsing fails, and communication domain initialization is interrupted. | ||
| 268 | + | ||
| 269 | +**Troubleshooting:** | ||
| 270 | +``` | ||
| 271 | +[Possible Causes] | ||
| 272 | +1. Syntax errors in the YAML file (e.g., incorrect indentation, missing space after colon, illegal characters) | ||
| 273 | +2. Unsupported field types or formats in the YAML file | ||
| 274 | +3. YAML file encoding is not UTF-8 | ||
| 275 | + | ||
| 276 | +[Steps] | ||
| 277 | +# Use python to verify the yaml format | ||
| 278 | +python3 -c "import yaml; yaml.safe_load(open('$HCCL_VM_INSTALL_DIR/config/topo_meta/<name>.yaml'))" | ||
| 279 | + | ||
| 280 | +[Solution] | ||
| 281 | +Fix the YAML file syntax errors based on the `<detail>` information in the log. Common issues include: | ||
| 282 | +1. Indentation must use spaces, not Tab | ||
| 283 | +2. A space is required after the colon in key-value pairs | ||
| 284 | +3. Indentation of list items (`-`) must be consistent with their parent level | ||
| 285 | +``` | ||
| 286 | +--- | ||
| 287 | + | ||
| 288 | +### Submodule: Memory Management | ||
| 289 | + | ||
| 290 | +--- | ||
| 291 | + | ||
| 292 | +#### FAQ-M001 | ||
| 293 | + | ||
| 294 | +**Title:** Device memory allocation exceeded limit | ||
| 295 | + | ||
| 296 | +**Error code:** | ||
| 297 | +``` | ||
| 298 | +HCCL_SIM_E_MEMORY (3) | ||
| 299 | +``` | ||
| 300 | + | ||
| 301 | +**Error function:** | ||
| 302 | +``` | ||
| 303 | +store_sim_device_memory_manager.cc::AllocPhyMem() | ||
| 304 | +``` | ||
| 305 | + | ||
| 306 | +**Key log:** | ||
| 307 | +``` | ||
| 308 | +dev:<N> alloc phy mem:<ADDR> size:<SIZE> exceeds pool ceiling:<CEILING>, reject | ||
| 309 | +``` | ||
| 310 | + | ||
| 311 | +**Symptoms:** The device memory allocation request exceeds the simulated memory pool ceiling. | ||
| 312 | + | ||
| 313 | +**Diagram:** | ||
| 314 | +```mermaid | ||
| 315 | +graph LR | ||
| 316 | + A[Memory allocation request] --> B{Check pool ceiling} | ||
| 317 | + B -->|Within limit| C[Allocation successful] | ||
| 318 | + B -->|Exceeded| D[Allocation rejected] | ||
| 319 | + D --> E[Error: exceeds pool ceiling] | ||
| 320 | +``` | ||
| 321 | +--- | ||
| 322 | + | ||
| 323 | +#### FAQ-M002 | ||
| 324 | + | ||
| 325 | +**Title:** Shared memory creation failed | ||
| 326 | + | ||
| 327 | +**Error code:** | ||
| 328 | +``` | ||
| 329 | +HCCL_SIM_E_SYSCALL (8) | ||
| 330 | +``` | ||
| 331 | + | ||
| 332 | +**Error function:** | ||
| 333 | +``` | ||
| 334 | +store_sim_shm_ops.cc::ShmCreate() | ||
| 335 | +``` | ||
| 336 | + | ||
| 337 | +**Key log:** | ||
| 338 | +``` | ||
| 339 | +[SHM_OPS] create: shm_open failed, name: <name> | ||
| 340 | +[SHM_OPS] create: ftruncate failed, name: <name> | ||
| 341 | +[SHM_OPS] create: mmap failed, name: <name> | ||
| 342 | +``` | ||
| 343 | + | ||
| 344 | +**Symptoms:** Unable to create a shared memory segment. | ||
| 345 | + | ||
| 346 | +**Troubleshooting:** | ||
| 347 | +``` | ||
| 348 | +[Possible Causes] | ||
| 349 | +1. Insufficient `/dev/shm` space | ||
| 350 | +2. Insufficient permissions | ||
| 351 | +3. A shared memory with the same name already exists and conflicts | ||
| 352 | + | ||
| 353 | +[Steps] | ||
| 354 | +df -h /dev/shm | ||
| 355 | +ls /dev/shm/ | grep hccl | ||
| 356 | +``` | ||
| 357 | +--- | ||
| 358 | + | ||
| 359 | +#### FAQ-M003 | ||
| 360 | + | ||
| 361 | +**Title:** Communication memory allocation failed | ||
| 362 | + | ||
| 363 | +**Error code:** | ||
| 364 | +``` | ||
| 365 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 366 | +``` | ||
| 367 | + | ||
| 368 | +**Error function:** | ||
| 369 | +``` | ||
| 370 | +store_sim_comm_memory_manager.cc | ||
| 371 | +``` | ||
| 372 | + | ||
| 373 | +**Key log:** | ||
| 374 | +``` | ||
| 375 | +[COMM_MEM] alloc failed, name: <name> | ||
| 376 | +[COMM_MEM] acquire failed, name: <name> | ||
| 377 | +[COMM_MEM] write size too large, size: <N>, max: <MAX> | ||
| 378 | +``` | ||
| 379 | + | ||
| 380 | +**Symptoms:** Cross-process communication memory operation failed. | ||
| 381 | + | ||
| 382 | +--- | ||
| 383 | + | ||
| 384 | +### Submodule: Stub Proxy | ||
| 385 | + | ||
| 386 | +--- | ||
| 387 | + | ||
| 388 | +#### FAQ-PX001 | ||
| 389 | + | ||
| 390 | +**Title:** AIV Kernel virtual execution failed | ||
| 391 | + | ||
| 392 | +**Error code:** | ||
| 393 | +``` | ||
| 394 | +HCCL_SIM_E_INTERNAL (4) | ||
| 395 | +``` | ||
| 396 | + | ||
| 397 | +**Error function:** | ||
| 398 | +``` | ||
| 399 | +hccl_op_stub.cc::VirtualExecuteAivKernel() | ||
| 400 | +``` | ||
| 401 | + | ||
| 402 | +**Key log:** | ||
| 403 | +``` | ||
| 404 | +[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set | ||
| 405 | +[virtual-aiv] missing aiv stub shared library, kernel=<name> | ||
| 406 | +[virtual-aiv] dlopen <so> failed, err = <error> | ||
| 407 | +[virtual-aiv] dlsym <symbol> from <so> failed, err = <error> | ||
| 408 | +``` | ||
| 409 | + | ||
| 410 | +**Symptoms:** AIV kernel execution failed in the virtual environment. | ||
| 411 | + | ||
| 412 | +**Troubleshooting:** | ||
| 413 | +``` | ||
| 414 | +[Steps] | ||
| 415 | +echo $HCCL_VM_INSTALL_DIR | ||
| 416 | +ls -la $HCCL_VM_INSTALL_DIR/lib/aiv/ | ||
| 417 | +nm -D $HCCL_VM_INSTALL_DIR/lib/aiv/<kernel>.so | grep <symbol> | ||
| 418 | +``` | ||
| 419 | +--- | ||
| 420 | + | ||
| 421 | +#### FAQ-PX002 | ||
| 422 | + | ||
| 423 | +**Title:** Operator database recording failed | ||
| 424 | + | ||
| 425 | +**Error code:** | ||
| 426 | +``` | ||
| 427 | +HCCL_SIM_E_INTERNAL (4) | ||
| 428 | +``` | ||
| 429 | + | ||
| 430 | +**Error function:** | ||
| 431 | +``` | ||
| 432 | +hccl_op_stub.cc::RecordOpDbInfo() | ||
| 433 | +``` | ||
| 434 | + | ||
| 435 | +**Key log:** | ||
| 436 | +``` | ||
| 437 | +[RecordOpDbInfo] insert op detail+mem failed | ||
| 438 | +[HcclAllReduce] record op db info failed | ||
| 439 | +``` | ||
| 440 | + | ||
| 441 | +**Symptoms:** HCCL collective communication operator parameters cannot be written to the simulation database. | ||
| 442 | + | ||
| 443 | +**Affected operators:** AlltoAll, AlltoAllV, AllGather, Broadcast, AllReduce, Scatter, Reduce, ReduceScatter | ||
| 444 | + | ||
| 445 | +--- | ||
| 446 | + | ||
| 447 | +#### FAQ-PX003 | ||
| 448 | + | ||
| 449 | +**Title:** QP not found or status error | ||
| 450 | + | ||
| 451 | +**Error code:** | ||
| 452 | +``` | ||
| 453 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 454 | +``` | ||
| 455 | + | ||
| 456 | +**Error function:** | ||
| 457 | +``` | ||
| 458 | +hccp_stub.cc::RaSendWr() | ||
| 459 | +``` | ||
| 460 | + | ||
| 461 | +**Key log:** | ||
| 462 | +``` | ||
| 463 | +[HCCP] RaSendWr: QP <N> not found | ||
| 464 | +[HCCP] RaSendWr: QP <N> not in RTS state, current state:<N> | ||
| 465 | +``` | ||
| 466 | + | ||
| 467 | +**Symptoms:** RDMA QP operation failed—QP does not exist or has not reached the RTS state. | ||
| 468 | + | ||
| 469 | +**Diagram:** | ||
| 470 | +```mermaid | ||
| 471 | +stateDiagram-v2 | ||
| 472 | + [*] --> INIT | ||
| 473 | + INIT --> RTR: RaQpConnect | ||
| 474 | + RTR --> RTS: RaTypicalQpModify | ||
| 475 | + RTS --> [*]: Ready to send data | ||
| 476 | + RTS --> ERROR: Abnormal state | ||
| 477 | + INIT --> ERROR: Not properly initialized | ||
| 478 | +``` | ||
| 479 | +--- | ||
| 480 | + | ||
| 481 | +#### FAQ-PX004 | ||
| 482 | + | ||
| 483 | +**Title:** EndPoint lookup failed | ||
| 484 | + | ||
| 485 | +**Error code:** | ||
| 486 | +``` | ||
| 487 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 488 | +``` | ||
| 489 | + | ||
| 490 | +**Error function:** | ||
| 491 | +``` | ||
| 492 | +hccp_stub.cc::RaCtxQpImport() | ||
| 493 | +``` | ||
| 494 | + | ||
| 495 | +**Key log:** | ||
| 496 | +``` | ||
| 497 | +[HCCP] cannot find endpoint addr:<IP> | ||
| 498 | +Get remote endpoint failed. ip:<IP>, eid:<EID> | ||
| 499 | +``` | ||
| 500 | + | ||
| 501 | +**Symptoms:** Network endpoint lookup failed. | ||
| 502 | + | ||
| 503 | +**Troubleshooting:** | ||
| 504 | +``` | ||
| 505 | +[Possible Causes] | ||
| 506 | +The IP address is not in the endpoint list configured in the rank table. | ||
| 507 | +``` | ||
| 508 | +--- | ||
| 509 | + | ||
| 510 | +#### FAQ-PX005 | ||
| 511 | + | ||
| 512 | +**Title:** CCU microcode loading failed | ||
| 513 | + | ||
| 514 | +**Error code:** | ||
| 515 | +``` | ||
| 516 | +HCCL_SIM_E_INTERNAL (4) | ||
| 517 | +``` | ||
| 518 | + | ||
| 519 | +**Error function:** | ||
| 520 | +``` | ||
| 521 | +hccp_ccu_stub.cc::LoadMicrocodeInstruction() | ||
| 522 | +``` | ||
| 523 | + | ||
| 524 | +**Key log:** | ||
| 525 | +``` | ||
| 526 | +[LoadMicrocodeInstruction] get device by logic id <N> failed. | ||
| 527 | +[LoadMicrocodeInstruction] get ccu from device by die id <N> failed. | ||
| 528 | +[LoadMicrocodeInstruction] insert instr failed | ||
| 529 | +``` | ||
| 530 | + | ||
| 531 | +**Symptoms:** CCU microcode instruction failed to load into the simulator. | ||
| 532 | + | ||
| 533 | +--- | ||
| 534 | + | ||
| 535 | +#### FAQ-PX006 | ||
| 536 | + | ||
| 537 | +**Title:** Unable to get current Context | ||
| 538 | + | ||
| 539 | +**Error code:** | ||
| 540 | +``` | ||
| 541 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 542 | +``` | ||
| 543 | + | ||
| 544 | +**Error function:** | ||
| 545 | +``` | ||
| 546 | +hccp_stub.cc::RaRdevInit() | ||
| 547 | +``` | ||
| 548 | + | ||
| 549 | +**Key log:** | ||
| 550 | +``` | ||
| 551 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaRdevInit] can not get CurrContext: <N> | ||
| 552 | +``` | ||
| 553 | + | ||
| 554 | +**Symptoms:** During RDMA device initialization, the active Context cannot be obtained from the current Runner, causing RDMA device creation to fail. | ||
| 555 | + | ||
| 556 | +**Troubleshooting:** | ||
| 557 | +``` | ||
| 558 | +[Possible Causes] | ||
| 559 | +1. The application layer did not call `aclrtSetDevice`/`aclrtCreateContext` to initialize the device and context | ||
| 560 | +2. The Context was destroyed prematurely | ||
| 561 | +3. The current_ctx_id in the Runner's TLS (Thread Local Storage) is invalid | ||
| 562 | +4. The application layer called other runtime interfaces to obtain a context before calling `aclrtSetDevice` to initialize the device context | ||
| 563 | + | ||
| 564 | +[Steps] | ||
| 565 | +# Check the Context table | ||
| 566 | +hccl-vm table show Context | ||
| 567 | +# Check the current_ctx_id in the Runner table | ||
| 568 | +hccl-vm table show Runner | ||
| 569 | + | ||
| 570 | +[Solution] | ||
| 571 | +Ensure that the application layer has correctly called `aclrtSetDevice` and `aclrtCreateContext` before performing RDMA operations, and that the Context has not been destroyed prematurely. | ||
| 572 | +``` | ||
| 573 | +--- | ||
| 574 | + | ||
| 575 | +#### FAQ-PX007 | ||
| 576 | + | ||
| 577 | +**Title:** AICPU binary file not found | ||
| 578 | + | ||
| 579 | +**Error code:** | ||
| 580 | +``` | ||
| 581 | +ACL_ERROR_RT_FEATURE_NOT_SUPPORT | ||
| 582 | +``` | ||
| 583 | + | ||
| 584 | +**Error function:** | ||
| 585 | +``` | ||
| 586 | +aclrt_kernel_stub.cc::aclrtDestroyBinary() | ||
| 587 | +``` | ||
| 588 | + | ||
| 589 | +**Key log:** | ||
| 590 | +``` | ||
| 591 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][aclrtDestroyBinary] can not find this binary | ||
| 592 | +``` | ||
| 593 | + | ||
| 594 | +**Symptoms:** When destroying an AICPU binary object, the corresponding binary handle cannot be found in the global kernel binary registry. | ||
| 595 | + | ||
| 596 | +**Troubleshooting:** | ||
| 597 | +``` | ||
| 598 | +[Possible Causes] | ||
| 599 | +1. The binary file was not properly loaded (`aclrtLoadBinary` was not executed or failed) | ||
| 600 | +2. The binary handle was destroyed twice (double-free) | ||
| 601 | +3. The binary object was accessed concurrently in a multi-threaded environment, causing state inconsistency | ||
| 602 | + | ||
| 603 | +[Steps] | ||
| 604 | +# Check for duplicate destroy calls | ||
| 605 | +# Verify the return value of aclrtLoadBinary | ||
| 606 | + | ||
| 607 | +[Solution] | ||
| 608 | +Ensure that `aclrtLoadBinary` returns successfully before calling `aclrtDestroyBinary`, and do not destroy the same binary object twice. | ||
| 609 | +``` | ||
| 610 | +--- | ||
| 611 | + | ||
| 612 | +#### FAQ-PX008 | ||
| 613 | + | ||
| 614 | +**Title:** AICPU device process exited abnormally | ||
| 615 | + | ||
| 616 | +**Error code:** | ||
| 617 | +``` | ||
| 618 | +NA (process-level error) | ||
| 619 | +``` | ||
| 620 | + | ||
| 621 | +**Error function:** | ||
| 622 | +``` | ||
| 623 | +aclrt_kernel_stub.cc::WaitAicpuProcess() | ||
| 624 | +``` | ||
| 625 | + | ||
| 626 | +**Key log:** | ||
| 627 | +``` | ||
| 628 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] exited with status <N> | ||
| 629 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] killed by signal <N> | ||
| 630 | +``` | ||
| 631 | + | ||
| 632 | +**Symptoms:** The AICPU device subprocess exited abnormally or was killed by a signal, causing the main process to also exit (`exit(EXIT_FAILURE)`). | ||
| 633 | + | ||
| 634 | +**Troubleshooting:** | ||
| 635 | +``` | ||
| 636 | +[Possible Causes] | ||
| 637 | +1. Uncaught exception or segmentation fault inside the AICPU process | ||
| 638 | +2. Insufficient system resources (memory, file descriptors, etc.) causing the subprocess to be killed by OOM killer | ||
| 639 | +3. Bugs in the AICPU binary file itself | ||
| 640 | +4. Missing shared libraries required by the subprocess | ||
| 641 | + | ||
| 642 | +[Steps] | ||
| 643 | +# Check system logs for OOM records | ||
| 644 | +dmesg | grep -i "oom\|killed" | ||
| 645 | +# Verify the integrity of the AICPU binary file | ||
| 646 | +ls -la $HCCL_VM_INSTALL_DIR/bin/ | ||
| 647 | +# Check system resources | ||
| 648 | +ulimit -a | ||
| 649 | +free -m | ||
| 650 | + | ||
| 651 | +[Solution] | ||
| 652 | +1. Check whether the AICPU binary file is correctly compiled and deployed | ||
| 653 | +2. Ensure sufficient system resources (memory, file descriptor limits, etc.) | ||
| 654 | +3. If killed by a signal, further locate the cause based on the signal number (e.g., 11=SIGSEGV, 9=SIGKILL) | ||
| 655 | +``` | ||
| 656 | +--- | ||
| 657 | + | ||
| 658 | +#### FAQ-PX009 | ||
| 659 | + | ||
| 660 | +**Title:** No ranks found when CCU loads microcode | ||
| 661 | + | ||
| 662 | +**Error code:** | ||
| 663 | +``` | ||
| 664 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 665 | +``` | ||
| 666 | + | ||
| 667 | +**Error function:** | ||
| 668 | +``` | ||
| 669 | +hccp_ccu_stub.cc::LoadMicrocodeInstruction() | ||
| 670 | +``` | ||
| 671 | + | ||
| 672 | +**Key log:** | ||
| 673 | +``` | ||
| 674 | +[error][PID:<PID>][TID:<TID>][hccp_ccu_stub.cc][LoadMicrocodeInstruction] can not find any rank | ||
| 675 | +``` | ||
| 676 | + | ||
| 677 | +**Symptoms:** During CCU microcode instruction loading, no rank records can be found in the Rank table corresponding to the current device. | ||
| 678 | + | ||
| 679 | +**Troubleshooting:** | ||
| 680 | +``` | ||
| 681 | +[Possible Causes] | ||
| 682 | +1. The communication domain has not been initialized via the `mock-comm` command, so the Rank table is empty | ||
| 683 | +2. The current device ID does not exist in the communication domain configuration | ||
| 684 | + | ||
| 685 | +[Steps] | ||
| 686 | +# Check if the Rank table has data | ||
| 687 | +hccl-vm table show Rank | ||
| 688 | +# Check the Device table | ||
| 689 | +hccl-vm table show Device | ||
| 690 | + | ||
| 691 | +[Solution] | ||
| 692 | +Ensure that before performing CCU-related operations, the communication domain has been correctly initialized via the `hccl-vm mock-comm` command, and the communication domain configuration covers the current device. | ||
| 693 | +``` | ||
| 694 | +--- | ||
| 695 | + | ||
| 696 | +#### FAQ-PX010 | ||
| 697 | + | ||
| 698 | +**Title:** Failed to find device by rankId | ||
| 699 | + | ||
| 700 | +**Error code:** | ||
| 701 | +``` | ||
| 702 | +HCCL_E_NOT_FOUND | ||
| 703 | +``` | ||
| 704 | + | ||
| 705 | +**Error function:** | ||
| 706 | +``` | ||
| 707 | +aclrt_device_stub.cc::hrtSetDevice() | ||
| 708 | +``` | ||
| 709 | + | ||
| 710 | +**Key log:** | ||
| 711 | +``` | ||
| 712 | +[error][PID:<PID>][TID:<TID>][aclrt_device_stub.cc][hrtSetDevice] device not found by rankId:<N> | ||
| 713 | +``` | ||
| 714 | + | ||
| 715 | +**Symptoms:** When calling `aclrtSetDevice` to set the current device, looking up the device by rankId fails. | ||
| 716 | + | ||
| 717 | +**Troubleshooting:** | ||
| 718 | +``` | ||
| 719 | +[Possible Causes] | ||
| 720 | +1. The rankId exceeds the actual rank range in the communication domain — e.g., the communication domain is configured with 4 NPUs, but mpirun starts 6 NPU processes, causing rankIds 4 and 5 to report device not found. | ||
| 721 | +2. The communication domain has not been initialized (the `mock-comm` command was not executed) — [High probability] The tool initializes the Rank table only after the communication domain is initialized. | ||
| 722 | +3. The ranktable configuration does not match the actual number of ranks used — possibly `RANK_TABLE_FILE` points to the wrong file path. | ||
| 723 | + | ||
| 724 | +[Steps] | ||
| 725 | +# Check if the rankId is within the valid range | ||
| 726 | +hccl-vm table show Rank | ||
| 727 | + | ||
| 728 | +[Solution] | ||
| 729 | +Ensure that the rankId is within the legal range of the communication domain configuration (0 to rank_count-1), and that the `RANK_TABLE_FILE` environment variable points to the correct ranktable.json file. | ||
| 730 | +``` | ||
| 731 | +--- | ||
| 732 | + | ||
| 733 | +#### FAQ-PX011 | ||
| 734 | + | ||
| 735 | +**Title:** Stub interface not yet implemented | ||
| 736 | + | ||
| 737 | +**Error code:** | ||
| 738 | +``` | ||
| 739 | +HCCL_SIM_E_INTERNAL (4) or NA | ||
| 740 | +``` | ||
| 741 | + | ||
| 742 | +**Error function:** | ||
| 743 | +``` | ||
| 744 | +Multiple stub function files (hccp_stub.cc, ascend_hal_stub.cc, aclrt_kernel_stub.cc, etc.) | ||
| 745 | +``` | ||
| 746 | + | ||
| 747 | +**Key log:** | ||
| 748 | +``` | ||
| 749 | +[warning][PID:<PID>][TID:<TID>][ascend_hal_stub.cc][*] [STUB] is empty | ||
| 750 | +[warning][PID:<PID>][TID:<TID>][hccp_stub.cc][*] [STUB] is empty | ||
| 751 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetAuxInfo] Not support yet | ||
| 752 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetCrErrInfoList] Not support yet | ||
| 753 | +``` | ||
| 754 | + | ||
| 755 | +**Symptoms:** The application layer called a low-level driver or runtime interface that is not yet implemented by the simulator. The log shows `[STUB] is empty` or `Not support yet` warnings/errors. Such stub functions return default values directly (usually 0 or success) without performing any actual operations. | ||
| 756 | + | ||
| 757 | +**Troubleshooting:** | ||
| 758 | +``` | ||
| 759 | +[Possible Causes] | ||
| 760 | +The current version of the simulator only implements the core interface subset required for HCCL collective communication. Some low-level driver interfaces (such as drvGetDeviceCapability, RaCtxGetAuxInfo, drvMemPrefetch, etc.) are not on the core path of HCCL communication, so the stub function body is empty or marked as unsupported. | ||
| 761 | + Generally, flows supported by the HCCL-VM tool do not call these interfaces, so such warnings should not occur. If the user calls the wrong application layer interface or enters an incorrect HCCL business flow, such warnings may appear. | ||
| 762 | + | ||
| 763 | +[Solution] | ||
| 764 | +1. Such warnings usually do not affect the correctness simulation of HCCL operators and can be safely ignored. | ||
| 765 | +2. If the warning is accompanied by functional anomalies, it means the application depends on an unimplemented interface. Please report it to the simulator development team. | ||
| 766 | +3. If a stub implementation for a specific interface is needed, contact the development team for prioritized adaptation. | ||
| 767 | +``` | ||
| 768 | + | ||
| 769 | +**Main interface types involved:** | ||
| 770 | +1. **Driver layer interfaces** (`ascend_hal_stub.cc`): drvGetDeviceCapability, drvMemPrefetch, drvStreamQuery, etc., approximately 315 interfaces | ||
| 771 | +2. **RDMA interfaces** (`hccp_stub.cc`): RaRestoreSnapshot, RaRdevInitWithBackup, RaCtxGetAuxInfo, etc., approximately 44 interfaces | ||
| 772 | +3. **Runtime adaptation layer** (`adapter_rts_stub.cc`): some aclrt extension interfaces | ||
| 773 | +4. **TSD client** (`tsd_client_stub.cc`): TSD-related interfaces | ||
| 774 | + | ||
| 775 | +--- | ||
| 776 | + | ||
| 777 | +### Submodule: Networking | ||
| 778 | + | ||
| 779 | +--- | ||
| 780 | + | ||
| 781 | +#### FAQ-N001 | ||
| 782 | + | ||
| 783 | +**Title:** Ranktable environment variable configuration error | ||
| 784 | + | ||
| 785 | +**Error code:** | ||
| 786 | +``` | ||
| 787 | +NA (1) | ||
| 788 | +``` | ||
| 789 | + | ||
| 790 | +**Error function:** | ||
| 791 | +``` | ||
| 792 | +param_check_v2.cc::RanktableRealPath | ||
| 793 | +``` | ||
| 794 | + | ||
| 795 | +**Key log:** | ||
| 796 | +``` | ||
| 797 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172019]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path | ||
| 798 | + | ||
| 799 | +[info][PID:172021][TID:172021][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172021][hrtGetDeviceRefresh]deviceLogicId[3] | ||
| 800 | +[error][PID:172020][TID:172020][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172020]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path | ||
| 801 | + | ||
| 802 | +[info][PID:172018][TID:172018][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172018][hrtGetDeviceRefresh]deviceLogicId[0] | ||
| 803 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base_v2.cc:294][172019][HcclCommInitClusterInfoV2]call trace: hcclRet -> 1 | ||
| 804 | + | ||
| 805 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base.cc:811] [172019][operator()]call trace: hcclRet -> 1 | ||
| 806 | +``` | ||
| 807 | + | ||
| 808 | +**Symptoms:** Running a test case fails to initialize the communication domain. | ||
| 809 | + | ||
| 810 | +**Troubleshooting:** | ||
| 811 | +``` | ||
| 812 | +[Possible Causes] | ||
| 813 | +The ranktable.json file path is configured incorrectly. Check the RANK_TABLE_FILE environment variable. The ranktable.json is generated by the tool, and its path is $HCCL_VM_INSTALL_DIR/data/ranktable.json. | ||
| 814 | + | ||
| 815 | +[Steps] | ||
| 816 | +echo $RANK_TABLE_FILE | ||
| 817 | + | ||
| 818 | +[Solution] | ||
| 819 | +Ensure the RANK_TABLE_FILE environment variable is correctly set to point to the ranktable.json file path. | ||
| 820 | +``` | ||
| 821 | +--- | ||
| 822 | + | ||
| 823 | +#### FAQ-N002 | ||
| 824 | + | ||
| 825 | +**Title:** topo.json path configuration error | ||
| 826 | + | ||
| 827 | +**Error code:** | ||
| 828 | +``` | ||
| 829 | +NA (1) | ||
| 830 | +``` | ||
| 831 | + | ||
| 832 | +**Error function:** | ||
| 833 | +``` | ||
| 834 | +communicator_impl.cc::GetTopoFilePath | ||
| 835 | +``` | ||
| 836 | + | ||
| 837 | +**Key log:** | ||
| 838 | +``` | ||
| 839 | +[error][PID:172635][TID:172635][log_stub.cc][DlogPrintStub] [HCCL_LOG][communicator_impl.cc:1339][172635][GetTopoFilePath] topo_file_path[/home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/topo.json] is not a valid real path | ||
| 840 | +``` | ||
| 841 | + | ||
| 842 | +**Symptoms:** Running a test case fails to initialize the communication domain. | ||
| 843 | + | ||
| 844 | +**Troubleshooting:** | ||
| 845 | +``` | ||
| 846 | +[Possible Causes] | ||
| 847 | +The topo.json file path is incorrectly configured in the /etc/hccl_rootinfo.json file. Check the topo_file_path field. The topo.json is generated by the tool, and its path is $HCCL_VM_INSTALL_DIR/data/topo.json. | ||
| 848 | + | ||
| 849 | +[Steps] | ||
| 850 | +echo $TOPO_FILE_PATH | ||
| 851 | + | ||
| 852 | +[Solution] | ||
| 853 | +Ensure the TOPO_FILE_PATH environment variable is correctly set to point to the topo.json file path. | ||
| 854 | +``` | ||
| 855 | +--- | ||
| 856 | + | ||
| 857 | +#### FAQ-N003 | ||
| 858 | + | ||
| 859 | +**Title:** mock-comm command error | ||
| 860 | + | ||
| 861 | +**Error code:** | ||
| 862 | +``` | ||
| 863 | +NA | ||
| 864 | +``` | ||
| 865 | + | ||
| 866 | +**Error function:** | ||
| 867 | +``` | ||
| 868 | +db_sim_runner_ops.cc::GetServerKeyById | ||
| 869 | +``` | ||
| 870 | + | ||
| 871 | +**Key log:** | ||
| 872 | +``` | ||
| 873 | +(hvm)$> hccl-vm mock-comm 144 | ||
| 874 | +[error][PID:172799][TID:172875][db_sim_runner_ops.cc][GetServerKeyById] can not find server by id: 0, 2 | ||
| 875 | +[error][PID:172799][TID:172875][topo_ascend_cluster_parser.cc][InitDynamicModelData] cannot find device by physical id 0 | ||
| 876 | +[error][PID:172799][TID:172875][cmd_base_utils.cc][InitHvmCommEnv] [HVM] InitHvmCommEnv failed | ||
| 877 | +[error][PID:172799][TID:172875][subcmd_mock_comm.cc][Execute] [HVM] Failed to initialize mock communication environment. Cleaning up environment. | ||
| 878 | +``` | ||
| 879 | + | ||
| 880 | +**Symptoms:** Before running a test case, configuring the communication domain via the mock-comm command fails. | ||
| 881 | + | ||
| 882 | +**Troubleshooting:** | ||
| 883 | +``` | ||
| 884 | +[Possible Causes] | ||
| 885 | +The communication domain 144 configured by the mock-comm command exceeds the cluster configuration used when starting the tool. For example, the cluster started by the tool has only 2 servers per super node, but communication domain 144 indicates that the super node has 4 servers. | ||
| 886 | + | ||
| 887 | +[Steps] | ||
| 888 | +Check the cluster configuration file used when starting the tool and the communication domain configuration file for the mock-comm command. | ||
| 889 | + | ||
| 890 | +[Solution] | ||
| 891 | +Check the cluster configuration used when starting the tool to confirm the number of servers per super node. If communication domain 144 is indeed needed, ensure the tool is started with a larger cluster networking configuration. | ||
| 892 | +Ensure that the communication domain configured by the mock-comm command does not exceed the cluster configuration used when starting the tool. | ||
| 893 | +``` | ||
| 894 | +--- | ||
| 895 | + | ||
| 896 | +#### FAQ-N004 | ||
| 897 | + | ||
| 898 | +**Title:** EndPoint IP lookup failed | ||
| 899 | + | ||
| 900 | +**Error code:** | ||
| 901 | +``` | ||
| 902 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 903 | +``` | ||
| 904 | + | ||
| 905 | +**Error function:** | ||
| 906 | +``` | ||
| 907 | +topo_ascend_cluster_parser.cc::AddLinkInfo() | ||
| 908 | +``` | ||
| 909 | + | ||
| 910 | +**Key log:** | ||
| 911 | +``` | ||
| 912 | +cannot find endPoint by ip <IP_ADDR> | ||
| 913 | +``` | ||
| 914 | + | ||
| 915 | +**Symptoms:** The IP address referenced in the network link configuration does not exist in the topology. | ||
| 916 | + | ||
| 917 | +--- | ||
| 918 | + | ||
| 919 | +#### FAQ-N005 | ||
| 920 | + | ||
| 921 | +**Title:** Superpod index out of range | ||
| 922 | + | ||
| 923 | +**Error code:** | ||
| 924 | +``` | ||
| 925 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 926 | +``` | ||
| 927 | + | ||
| 928 | +**Error function:** | ||
| 929 | +``` | ||
| 930 | +topo_ascend_cluster_parser.cc::InitDynamicModelData() | ||
| 931 | +``` | ||
| 932 | + | ||
| 933 | +**Key log:** | ||
| 934 | +``` | ||
| 935 | +[InitDynamicModelData] superpod index <N> out of range | ||
| 936 | +``` | ||
| 937 | + | ||
| 938 | +**Symptoms:** When parsing the ranktable to generate ranktable.json, the referenced superpod index exceeds the actual number of superpods in the cluster, causing initialization to fail. | ||
| 939 | + | ||
| 940 | +**Troubleshooting:** | ||
| 941 | +``` | ||
| 942 | +[Possible Causes] | ||
| 943 | +The number of superpods to which the devices in the ranktable belong exceeds the cluster networking configuration used when starting the tool. For example, the cluster has only 1 superpod, but the ranktable references a second superpod. | ||
| 944 | + | ||
| 945 | +[Steps] | ||
| 946 | +1. Check the cluster networking configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of superpods. | ||
| 947 | +2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to verify that the referenced superpod index is within range. | ||
| 948 | + | ||
| 949 | +[Solution] | ||
| 950 | +Ensure that the number of superpods referenced by the communication domain configured via mock-comm does not exceed the cluster networking configuration. If more superpods are needed, start the tool with a larger cluster networking configuration. | ||
| 951 | +``` | ||
| 952 | +--- | ||
| 953 | + | ||
| 954 | +#### FAQ-N006 | ||
| 955 | + | ||
| 956 | +**Title:** Server index out of range | ||
| 957 | + | ||
| 958 | +**Error code:** | ||
| 959 | +``` | ||
| 960 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 961 | +``` | ||
| 962 | + | ||
| 963 | +**Error function:** | ||
| 964 | +``` | ||
| 965 | +topo_ascend_cluster_parser.cc::InitDynamicModelData() | ||
| 966 | +``` | ||
| 967 | + | ||
| 968 | +**Key log:** | ||
| 969 | +``` | ||
| 970 | +[InitDynamicModelData] server index <N> out of range in superpod <M> | ||
| 971 | +``` | ||
| 972 | + | ||
| 973 | +**Symptoms:** When parsing the ranktable to generate ranktable.json, the referenced server index exceeds the actual number of servers in the superpod, causing initialization to fail. | ||
| 974 | + | ||
| 975 | +**Troubleshooting:** | ||
| 976 | +``` | ||
| 977 | +[Possible Causes] | ||
| 978 | +The number of servers under a certain superpod in the ranktable exceeds the number of servers in that superpod in the cluster networking configuration used when starting the tool. For example, the cluster networking has 2 servers per superpod, but the ranktable references a third server. | ||
| 979 | + | ||
| 980 | +[Steps] | ||
| 981 | +1. Check the cluster networking configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of servers per superpod. | ||
| 982 | +2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to verify that the referenced server index is within range. | ||
| 983 | + | ||
| 984 | +[Solution] | ||
| 985 | +Ensure that the number of servers per superpod in the communication domain configured via mock-comm does not exceed the cluster networking configuration. If more servers are needed, start the tool with a larger cluster networking configuration. | ||
| 986 | +``` | ||
| 987 | +--- | ||
| 988 | + | ||
| 989 | +#### FAQ-N007 | ||
| 990 | + | ||
| 991 | +**Title:** Failed to find device by physical ID | ||
| 992 | + | ||
| 993 | +**Error code:** | ||
| 994 | +``` | ||
| 995 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 996 | +``` | ||
| 997 | + | ||
| 998 | +**Error function:** | ||
| 999 | +``` | ||
| 1000 | +topo_ascend_cluster_parser.cc::InitDynamicModelData() | ||
| 1001 | +``` | ||
| 1002 | + | ||
| 1003 | +**Key log:** | ||
| 1004 | +``` | ||
| 1005 | +[InitDynamicModelData] cannot find device by physical id <N> | ||
| 1006 | +``` | ||
| 1007 | + | ||
| 1008 | +**Symptoms:** When parsing the ranktable, finding a device by its physical device ID fails, typically occurring when configuring the communication domain via mock-comm. | ||
| 1009 | + | ||
| 1010 | +**Troubleshooting:** | ||
| 1011 | +``` | ||
| 1012 | +[Possible Causes] | ||
| 1013 | +The physical device ID referenced in the communication domain configured by the mock-comm command exceeds the actual device range in the cluster networking. For example, the cluster has only 2 devices (physical id 0 and 1), but the communication domain configuration references physical id 2. | ||
| 1014 | + | ||
| 1015 | +[Steps] | ||
| 1016 | +1. Check the cluster networking configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of devices per server. | ||
| 1017 | +2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to verify that the referenced device_id is within range. | ||
| 1018 | + | ||
| 1019 | +[Solution] | ||
| 1020 | +Ensure that the physical device IDs referenced in the communication domain configured via mock-comm do not exceed the device range in the cluster networking configuration. If more devices are needed, start the tool with a larger cluster networking configuration. | ||
| 1021 | +``` | ||
| 1022 | +--- | ||
| 1023 | + | ||
| 1024 | +### Submodule: Database | ||
| 1025 | + | ||
| 1026 | +--- | ||
| 1027 | + | ||
| 1028 | +#### FAQ-DB001 | ||
| 1029 | + | ||
| 1030 | +**Title:** SQLite database connection failed | ||
| 1031 | + | ||
| 1032 | +**Error code:** | ||
| 1033 | +``` | ||
| 1034 | +HCCL_SIM_E_OPEN_FILE_FAILURE (10) | ||
| 1035 | +``` | ||
| 1036 | + | ||
| 1037 | +**Error function:** | ||
| 1038 | +``` | ||
| 1039 | +db_hccl_db_sqlite.cc::Connect() | ||
| 1040 | +``` | ||
| 1041 | + | ||
| 1042 | +**Key log:** | ||
| 1043 | +``` | ||
| 1044 | +[dbInit] Connect database failed | ||
| 1045 | +Connect database:<path> failed | ||
| 1046 | +``` | ||
| 1047 | + | ||
| 1048 | +**Symptoms:** Unable to connect to the SQLite database file. | ||
| 1049 | + | ||
| 1050 | +**Troubleshooting:** | ||
| 1051 | +``` | ||
| 1052 | +[Possible Causes] | ||
| 1053 | +1. The database file does not exist | ||
| 1054 | +2. Insufficient file permissions | ||
| 1055 | +3. The file is locked by another process | ||
| 1056 | +``` | ||
| 1057 | +--- | ||
| 1058 | + | ||
| 1059 | +#### FAQ-DB002 | ||
| 1060 | + | ||
| 1061 | +**Title:** Database backup file not found | ||
| 1062 | + | ||
| 1063 | +**Error code:** | ||
| 1064 | +``` | ||
| 1065 | +HCCL_SIM_E_OPEN_FILE_FAILURE (10) | ||
| 1066 | +``` | ||
| 1067 | + | ||
| 1068 | +**Error function:** | ||
| 1069 | +``` | ||
| 1070 | +sim_loader.cc::BackupDatabase() | ||
| 1071 | +``` | ||
| 1072 | + | ||
| 1073 | +**Key log:** | ||
| 1074 | +``` | ||
| 1075 | +[Loader] Backup database file not found: <dbPath> | ||
| 1076 | +``` | ||
| 1077 | + | ||
| 1078 | +**Symptoms:** The Loader cannot find the simulation database file. | ||
| 1079 | + | ||
| 1080 | +**Troubleshooting:** | ||
| 1081 | +``` | ||
| 1082 | +[Possible Causes] | ||
| 1083 | +1. Incorrect simulation data file path configuration | ||
| 1084 | +2. Simulation data has not been generated yet | ||
| 1085 | +3. Insufficient file permissions | ||
| 1086 | + | ||
| 1087 | +[Steps] | ||
| 1088 | +ls -la <dbPath> | ||
| 1089 | +``` | ||
| 1090 | +--- | ||
| 1091 | + | ||
| 1092 | +#### FAQ-DB003 | ||
| 1093 | + | ||
| 1094 | +**Title:** SQLite query failed | ||
| 1095 | + | ||
| 1096 | +**Error code:** | ||
| 1097 | +``` | ||
| 1098 | +HCCL_SIM_E_INTERNAL (4) | ||
| 1099 | +``` | ||
| 1100 | + | ||
| 1101 | +**Error function:** | ||
| 1102 | +``` | ||
| 1103 | +db_hccl_db_sqlite.cc | ||
| 1104 | +``` | ||
| 1105 | + | ||
| 1106 | +**Key log:** | ||
| 1107 | +``` | ||
| 1108 | +Prepare failed: <error> sql:<SQL> | ||
| 1109 | +Step failed: <error>, sql:<SQL> | ||
| 1110 | +``` | ||
| 1111 | + | ||
| 1112 | +**Symptoms:** SQL query execution failed. | ||
| 1113 | + | ||
| 1114 | +**Troubleshooting:** | ||
| 1115 | +``` | ||
| 1116 | +[Possible Causes] | ||
| 1117 | +1. Database table structure mismatch (version incompatibility) | ||
| 1118 | +2. Database file corruption | ||
| 1119 | +3. Insufficient disk space | ||
| 1120 | +``` | ||
| 1121 | +--- | ||
| 1122 | + | ||
| 1123 | +## Module: Plugin | ||
| 1124 | + | ||
| 1125 | +### Submodule: checker | ||
| 1126 | + | ||
| 1127 | +--- | ||
| 1128 | + | ||
| 1129 | +##### HCCL_SIM_E_INTERNAL (4) | ||
| 1130 | + | ||
| 1131 | +--- | ||
| 1132 | + | ||
| 1133 | +#### FAQ-C001 | ||
| 1134 | + | ||
| 1135 | +**Title:** Memory slice overflow | ||
| 1136 | + | ||
| 1137 | +**Error function:** | ||
| 1138 | +``` | ||
| 1139 | +task_graph_single_task_check_v3.cc::CheckMemorySlice() | ||
| 1140 | +``` | ||
| 1141 | + | ||
| 1142 | +**Key log:** | ||
| 1143 | +``` | ||
| 1144 | +[TaskGraphSingleTaskCheckV3] Memory slice overflow while accumulating coverage, <detail> | ||
| 1145 | +[TaskGraphSingleTaskCheckV3] Memory slice overflow, node=<node>, slice=<slice> | ||
| 1146 | +``` | ||
| 1147 | + | ||
| 1148 | +**Symptoms:** The memory slice coverage of a single task exceeds the total size of the target buffer. | ||
| 1149 | + | ||
| 1150 | +**Troubleshooting:** | ||
| 1151 | +``` | ||
| 1152 | +[Possible Causes] | ||
| 1153 | +1. Incorrect memory offset calculation in the HCCL algorithm layer (HCCL business issue) | ||
| 1154 | +2. Mismatch between the memory layout information in the simulation data and the task parameters (tool data issue) | ||
| 1155 | +``` | ||
| 1156 | + | ||
| 1157 | +**Diagram:** | ||
| 1158 | +```mermaid | ||
| 1159 | +graph LR | ||
| 1160 | + A[Task memory slice] --> B[Target Buffer] | ||
| 1161 | + A -.->|Out of bounds| C[Overflow error] | ||
| 1162 | + style C fill:#f96,stroke:#333 | ||
| 1163 | +``` | ||
| 1164 | +--- | ||
| 1165 | + | ||
| 1166 | +#### FAQ-C002 | ||
| 1167 | + | ||
| 1168 | +**Title:** Buffer semantic incomplete | ||
| 1169 | + | ||
| 1170 | +**Error function:** | ||
| 1171 | +``` | ||
| 1172 | +task_graph_semantic_check_v3.cc::CheckBufferContinuity() | ||
| 1173 | +``` | ||
| 1174 | + | ||
| 1175 | +**Key log:** | ||
| 1176 | +``` | ||
| 1177 | +[TaskGraphSemanticCheckV3] Head gap, expect start=0x<ADDR>, actual start=0x<ADDR> | ||
| 1178 | +[TaskGraphSemanticCheckV3] Middle gap, prev end=0x<ADDR>, cur start=0x<ADDR> | ||
| 1179 | +[TaskGraphSemanticCheckV3] Tail gap, expect end=0x<ADDR>, actual end=0x<ADDR> | ||
| 1180 | +``` | ||
| 1181 | + | ||
| 1182 | +**Symptoms:** There are gaps in the data semantic coverage of the target buffer. | ||
| 1183 | + | ||
| 1184 | +**Troubleshooting:** | ||
| 1185 | +``` | ||
| 1186 | +[Possible Causes] | ||
| 1187 | +1. The HCCL algorithm missed some data regions | ||
| 1188 | +2. The Checker failed to correctly trace the transfer path | ||
| 1189 | +``` | ||
| 1190 | + | ||
| 1191 | +**Diagram:** | ||
| 1192 | +```mermaid | ||
| 1193 | +graph TB | ||
| 1194 | + subgraph Target Buffer | ||
| 1195 | + A[Address 0x0] --> B[Address 0x100] | ||
| 1196 | + B -.->|Gap| C[Address 0x200] | ||
| 1197 | + C --> D[Address 0x300] | ||
| 1198 | + end | ||
| 1199 | + style B fill:#ff9,stroke:#333 | ||
| 1200 | + style C fill:#ff9,stroke:#333 | ||
| 1201 | +``` | ||
| 1202 | +--- | ||
| 1203 | + | ||
| 1204 | +#### FAQ-C003 | ||
| 1205 | + | ||
| 1206 | +**Title:** Reduce semantic error | ||
| 1207 | + | ||
| 1208 | +**Error function:** | ||
| 1209 | +``` | ||
| 1210 | +task_graph_semantic_check_v3.cc::CheckReduceSemantics() | ||
| 1211 | +``` | ||
| 1212 | + | ||
| 1213 | +**Key log:** | ||
| 1214 | +``` | ||
| 1215 | +[TaskGraphSemanticCheckV3] Reduce type mismatch, pair=<pair> | ||
| 1216 | +[TaskGraphSemanticCheckV3] Duplicate reduce source, pair=<pair>, srcOffset=0x<ADDR> | ||
| 1217 | +[TaskGraphSemanticCheckV3] Destination reduce semantic incomplete, pair=<pair> | ||
| 1218 | +``` | ||
| 1219 | + | ||
| 1220 | +**Symptoms:** Data semantic validation of the Reduce operation failed. | ||
| 1221 | + | ||
| 1222 | +**Troubleshooting:** | ||
| 1223 | +``` | ||
| 1224 | +[Possible Causes] | ||
| 1225 | +1. Data type mismatch in the Reduce operation | ||
| 1226 | +2. Duplicate reduce source exists | ||
| 1227 | +3. The target buffer is not fully covered by all reduce sources | ||
| 1228 | +``` | ||
| 1229 | +--- | ||
| 1230 | + | ||
| 1231 | +##### HCCL_SIM_E_PARA (1) | ||
| 1232 | + | ||
| 1233 | +--- | ||
| 1234 | + | ||
| 1235 | +#### FAQ-C004 | ||
| 1236 | + | ||
| 1237 | +**Title:** rankSize is zero | ||
| 1238 | + | ||
| 1239 | +**Error function:** | ||
| 1240 | +``` | ||
| 1241 | +task_graph_semantic_check_v3.cc | ||
| 1242 | +``` | ||
| 1243 | + | ||
| 1244 | +**Key log:** | ||
| 1245 | +``` | ||
| 1246 | +[TaskGraphSemanticCheckV3] rankSize is zero | ||
| 1247 | +``` | ||
| 1248 | + | ||
| 1249 | +**Symptoms:** The number of ranks participating in communication is 0 during semantic check. | ||
| 1250 | + | ||
| 1251 | +**Troubleshooting:** | ||
| 1252 | +``` | ||
| 1253 | +[Possible Causes] | ||
| 1254 | +The communication domain has not been properly initialized, or the rank table parsing failed. | ||
| 1255 | +``` | ||
| 1256 | +--- | ||
| 1257 | + | ||
| 1258 | +#### FAQ-C005 | ||
| 1259 | + | ||
| 1260 | +**Title:** Batch Trans pair size mismatch | ||
| 1261 | + | ||
| 1262 | +**Error function:** | ||
| 1263 | +``` | ||
| 1264 | +task_graph_single_task_check_v3.cc::CheckBatchTrans() | ||
| 1265 | +``` | ||
| 1266 | + | ||
| 1267 | +**Key log:** | ||
| 1268 | +``` | ||
| 1269 | +[TaskGraphSingleTaskCheckV3] Batch trans slice length mismatch, node=<node>, label=<label>, index=<N> | ||
| 1270 | +[TaskGraphSingleTaskCheckV3] Batch trans pair size mismatch, node=<node>, label=<label> | ||
| 1271 | +``` | ||
| 1272 | + | ||
| 1273 | +**Symptoms:** Slice length or pair count mismatch in batch transfer operations. | ||
| 1274 | + | ||
| 1275 | +**Troubleshooting:** | ||
| 1276 | +``` | ||
| 1277 | +[Possible Causes] | ||
| 1278 | +Uneven data distribution among ranks for operators such as AlltoAll. | ||
| 1279 | +``` | ||
| 1280 | +--- | ||
| 1281 | + | ||
| 1282 | +##### HCCL_SIM_E_NOT_SUPPORT (5) | ||
| 1283 | + | ||
| 1284 | +--- | ||
| 1285 | + | ||
| 1286 | +#### FAQ-C006 | ||
| 1287 | + | ||
| 1288 | +**Title:** Unsupported memory type | ||
| 1289 | + | ||
| 1290 | +**Error function:** | ||
| 1291 | +``` | ||
| 1292 | +task_graph_single_task_check_v3.cc | ||
| 1293 | +``` | ||
| 1294 | + | ||
| 1295 | +**Key log:** | ||
| 1296 | +``` | ||
| 1297 | +[TaskGraphSingleTaskCheckV3] Unsupported memory type, node=<node>, slice=<slice> | ||
| 1298 | +[TaskGraphSingleTaskCheckV3] Invalid memory slice, node=<node>, slice=<slice> | ||
| 1299 | +``` | ||
| 1300 | + | ||
| 1301 | +**Symptoms:** The memory slice type is not within the range supported by Checker. | ||
| 1302 | + | ||
| 1303 | +--- | ||
| 1304 | + | ||
| 1305 | +##### HCCL_SIM_E_OPEN_FILE_FAILURE (10) | ||
| 1306 | + | ||
| 1307 | +--- | ||
| 1308 | + | ||
| 1309 | +#### FAQ-C007 | ||
| 1310 | + | ||
| 1311 | +**Title:** Dump file write failed | ||
| 1312 | + | ||
| 1313 | +**Error function:** | ||
| 1314 | +``` | ||
| 1315 | +dump_manager.cc, dump_v3_manager.cc | ||
| 1316 | +``` | ||
| 1317 | + | ||
| 1318 | +**Key log:** | ||
| 1319 | +``` | ||
| 1320 | +[DumpManager::WriteMsgpackFile] failed to open file: <path> | ||
| 1321 | +[DumpManager::WriteJsonFile] json serialize failed: <error>, file: <path> | ||
| 1322 | +[DumpV3Manager::WriteMsgpack] failed to open file: <path> | ||
| 1323 | +``` | ||
| 1324 | + | ||
| 1325 | +**Symptoms:** Failed to write Checker intermediate result dump file. | ||
| 1326 | + | ||
| 1327 | +**Troubleshooting:** | ||
| 1328 | +``` | ||
| 1329 | +[Possible Causes] | ||
| 1330 | +1. Insufficient disk space | ||
| 1331 | +2. Directory does not exist or lacks write permissions | ||
| 1332 | +``` | ||
| 1333 | +--- | ||
| 1334 | + | ||
| 1335 | +#### FAQ-C008 | ||
| 1336 | + | ||
| 1337 | +**Title:** Binary file magic number mismatch | ||
| 1338 | + | ||
| 1339 | +**Error function:** | ||
| 1340 | +``` | ||
| 1341 | +binary_data_operator.cc::FileHeaderRead() | ||
| 1342 | +``` | ||
| 1343 | + | ||
| 1344 | +**Key log:** | ||
| 1345 | +``` | ||
| 1346 | +[FileHeaderRead] Unmatched magic number:0x<N>≠0x<M> | ||
| 1347 | +``` | ||
| 1348 | + | ||
| 1349 | +**Symptoms:** When reading the simulation data file, the magic number in the file header does not match. | ||
| 1350 | + | ||
| 1351 | +**Troubleshooting:** | ||
| 1352 | +``` | ||
| 1353 | +[Possible Causes] | ||
| 1354 | +1. The data file version is incompatible with the tool version | ||
| 1355 | +2. The file is corrupted | ||
| 1356 | +``` | ||
| 1357 | +--- | ||
| 1358 | + | ||
| 1359 | +## Appendix: Error Code Quick Reference | ||
| 1360 | + | ||
| 1361 | +| Error Code | Enum Value | Description | | ||
| 1362 | +|--------|--------|------| | ||
| 1363 | +| 0 | HCCL_SIM_SUCCESS | Success | | ||
| 1364 | +| 1 | HCCL_SIM_E_PARA | Parameter error | | ||
| 1365 | +| 2 | HCCL_SIM_E_PTR | Null pointer | | ||
| 1366 | +| 3 | HCCL_SIM_E_MEMORY | Memory error | | ||
| 1367 | +| 4 | HCCL_SIM_E_INTERNAL | Internal error | | ||
| 1368 | +| 5 | HCCL_SIM_E_NOT_SUPPORT | Unsupported feature | | ||
| 1369 | +| 6 | HCCL_SIM_E_NOT_FOUND | Resource not found | | ||
| 1370 | +| 8 | HCCL_SIM_E_SYSCALL | System call error | | ||
| 1371 | +| 9 | HCCL_SIM_E_TIMEOUT | Timeout | | ||
| 1372 | +| 10 | HCCL_SIM_E_OPEN_FILE_FAILURE | File open failed | | ||
| @@ -0,0 +1,1119 @@ | |||
| 1 | +# Checker Error Code FAQ | ||
| 2 | + | ||
| 3 | +--- | ||
| 4 | + | ||
| 5 | +## Module: Checker | ||
| 6 | + | ||
| 7 | +### Submodule: Graph Generation Stage | ||
| 8 | + | ||
| 9 | +--- | ||
| 10 | + | ||
| 11 | +#### FAQ-CHK101 | ||
| 12 | + | ||
| 13 | +**Title:** Graph Translation Failed | ||
| 14 | + | ||
| 15 | +**Error Code:** | ||
| 16 | + | ||
| 17 | +``` | ||
| 18 | +GRAPH_TRANSLATE_FAILED (101) | ||
| 19 | +``` | ||
| 20 | + | ||
| 21 | +**Key Log:** | ||
| 22 | + | ||
| 23 | +``` | ||
| 24 | +[GenGraph] [ErrorCode: 101] Failed to convert one task into a graph node, taskIndex=128, ret=1, taskMeta=taskType=0, rankId=3, streamId=7, srcRankId=3, dstRankId=4, src=[0x0,0x400), dst=[0x1000,0x1400), protocol=1 | ||
| 25 | +``` | ||
| 26 | + | ||
| 27 | +**Symptom:** During graph generation, the input task meta cannot be translated into an internal graph node. This commonly occurs when the task type is unsupported or field combinations do not meet translation conditions. | ||
| 28 | + | ||
| 29 | +**Troubleshooting Guide:** | ||
| 30 | + | ||
| 31 | +``` | ||
| 32 | +[Possible Causes] | ||
| 33 | +1. No translation implementation exists for this `taskType`. | ||
| 34 | +2. Fields such as `rankId`, `streamId`, offset, or length are abnormal. | ||
| 35 | +3. The upstream-generated task meta itself is malformed. | ||
| 36 | + | ||
| 37 | +[Troubleshooting Steps] | ||
| 38 | +Typical error points: | ||
| 39 | +1. A normal task meta cannot be translated into a graph node: First locate the specific task in the original task list using `taskIndex`; then determine the root cause based on the task details. | ||
| 40 | +``` | ||
| 41 | +--- | ||
| 42 | + | ||
| 43 | +#### FAQ-CHK102 | ||
| 44 | + | ||
| 45 | +**Title:** Graph Generation Deadlock | ||
| 46 | + | ||
| 47 | +**Error Code:** | ||
| 48 | + | ||
| 49 | +``` | ||
| 50 | +GRAPH_DEADLOCK (102) | ||
| 51 | +``` | ||
| 52 | + | ||
| 53 | +**Key Log:** | ||
| 54 | + | ||
| 55 | +``` | ||
| 56 | +[GenGraph] [ErrorCode: 102] Local Record/Wait matching is stuck on this rank. Some Wait tasks are still blocked, but no new local Record task can unblock them, rankId=0, firstBlockedWaitNode=[TaskWaitAICPU] node=143, rank=0, stream=3, protocol=SDMA, notify={recordRank=0, waitRank=0, notifyId=17}, blockedWaitNodeCount=5 | ||
| 57 | +``` | ||
| 58 | + | ||
| 59 | +**Symptom:** During graph generation, synchronization pairing progression is stuck. Wait nodes are still waiting, but no new Record nodes can pair with them. | ||
| 60 | + | ||
| 61 | +**Troubleshooting Guide:** | ||
| 62 | + | ||
| 63 | +``` | ||
| 64 | +[Possible Causes] | ||
| 65 | +1. Number of Waits exceeds Records, or Records cannot participate in pairing due to other unmet dependencies. | ||
| 66 | +2. `notifyId` mismatch: Wait and Record have inconsistent `notifyIds`, preventing successful pairing. | ||
| 67 | +3. Record has already executed, but its subsequent Wait's predecessors are not yet satisfied, so the Wait cannot enter the pairable queue. | ||
| 68 | + | ||
| 69 | +[Troubleshooting Steps] | ||
| 70 | +1. Read the `notifyId`, `recordRank`, and `waitRank` of `firstBlockedWaitNode` from the log to determine which rank should theoretically unlock this Wait. | ||
| 71 | +2. Check if a matching Record exists on the same rank or the corresponding remote rank; if it exists, continue to examine whether they can pair correctly. | ||
| 72 | +``` | ||
| 73 | + | ||
| 74 | +--- | ||
| 75 | + | ||
| 76 | +#### FAQ-CHK103 | ||
| 77 | + | ||
| 78 | +**Title:** Unconsumed Synchronization Pairing Residue | ||
| 79 | + | ||
| 80 | +**Error Code:** | ||
| 81 | + | ||
| 82 | +``` | ||
| 83 | +GRAPH_UNMATCHED (103) | ||
| 84 | +``` | ||
| 85 | + | ||
| 86 | +**Key Log:** | ||
| 87 | + | ||
| 88 | +``` | ||
| 89 | +[GenGraph] [ErrorCode: 103] Found cross-rank Record tasks that were never consumed by any matching Wait task, recordRankId=0, waitRankId=3, notifyId=21, firstUnconsumedRecordNode=[TaskRecordAICPU] node=77, rank=0, stream=1, protocol=RDMA, notify={recordRank=0, waitRank=3, notifyId=21}, unconsumedRecordCount=2 | ||
| 90 | +``` | ||
| 91 | + | ||
| 92 | +**Symptom:** After synchronization pairing completes, there are still unconsumed synchronization nodes. Typically, Records exist without corresponding Waits. | ||
| 93 | + | ||
| 94 | +**Troubleshooting Guide:** | ||
| 95 | + | ||
| 96 | +``` | ||
| 97 | +[Possible Causes] | ||
| 98 | +1. Mismatch between the number of Records and Waits. | ||
| 99 | +2. Incorrect `notifyId` used. | ||
| 100 | + | ||
| 101 | +[Troubleshooting Steps] | ||
| 102 | +1. Verify that `recordRankId`, `waitRankId`, and `notifyId` match expectations. | ||
| 103 | +``` | ||
| 104 | +--- | ||
| 105 | + | ||
| 106 | +#### FAQ-CHK104 | ||
| 107 | + | ||
| 108 | +**Title:** AIV Group Member Missing | ||
| 109 | + | ||
| 110 | +**Error Code:** | ||
| 111 | + | ||
| 112 | +``` | ||
| 113 | +GRAPH_MEMBER_MISSING (104) | ||
| 114 | +``` | ||
| 115 | + | ||
| 116 | +**Symptom:** This error indicates that during graph generation in AIV mode, a group member is incomplete. | ||
| 117 | + | ||
| 118 | +--- | ||
| 119 | + | ||
| 120 | +#### FAQ-CHK105 | ||
| 121 | + | ||
| 122 | +**Title:** Invalid Graph Structure | ||
| 123 | + | ||
| 124 | +**Error Code:** | ||
| 125 | + | ||
| 126 | +``` | ||
| 127 | +GRAPH_STRUCTURE_INVALID (105) | ||
| 128 | +``` | ||
| 129 | + | ||
| 130 | +**Key Log:** | ||
| 131 | + | ||
| 132 | +``` | ||
| 133 | +[GenGraph] [ErrorCode: 105] Failed to remove one graph edge because the parent or child node does not exist, parentNodeId=91, childNodeId=123, parentNode=[TaskTransMem] node=91, rank=2, stream=0, protocol=SDMA, src=rank 2 INPUT [0x0,0x400), dst=rank 2 CCL [0x1000,0x1400), childNode=null | ||
| 134 | +``` | ||
| 135 | + | ||
| 136 | +**Symptom:** Graph edge relationships violate graph construction prerequisites, such as parent or child nodes not existing during edge removal or reconnection. | ||
| 137 | + | ||
| 138 | +**Troubleshooting Guide:** | ||
| 139 | + | ||
| 140 | +``` | ||
| 141 | +[Possible Causes] | ||
| 142 | +1. This issue is typically not an algorithm orchestration problem. | ||
| 143 | + | ||
| 144 | +[Troubleshooting Steps] | ||
| 145 | +1. First confirm whether the task node has been generated in `Checker`. | ||
| 146 | +2. Contact tool support personnel for assistance. | ||
| 147 | +``` | ||
| 148 | + | ||
| 149 | +--- | ||
| 150 | + | ||
| 151 | +#### FAQ-CHK106 | ||
| 152 | + | ||
| 153 | +**Title:** AIV Snapshot Inconsistency | ||
| 154 | + | ||
| 155 | +**Error Code:** | ||
| 156 | + | ||
| 157 | +``` | ||
| 158 | +GRAPH_SNAPSHOT_MISMATCH (106) | ||
| 159 | +``` | ||
| 160 | + | ||
| 161 | +**Symptom:** This error indicates that the snapshot or environment information loaded during graph generation in AIV mode is inconsistent. | ||
| 162 | + | ||
| 163 | +--- | ||
| 164 | + | ||
| 165 | +#### FAQ-CHK107 | ||
| 166 | + | ||
| 167 | +**Title:** Missing Graph Generation Resources | ||
| 168 | + | ||
| 169 | +**Error Code:** | ||
| 170 | + | ||
| 171 | +``` | ||
| 172 | +GRAPH_RESOURCE_NOT_FOUND (107) | ||
| 173 | +``` | ||
| 174 | + | ||
| 175 | +**Symptom:** This error indicates that resources, mappings, or data files required during graph generation in AIV mode are missing. | ||
| 176 | + | ||
| 177 | +--- | ||
| 178 | + | ||
| 179 | +#### FAQ-CHK108 | ||
| 180 | + | ||
| 181 | +**Title:** Register or HBM Uninitialized | ||
| 182 | + | ||
| 183 | +**Error Code:** | ||
| 184 | + | ||
| 185 | +``` | ||
| 186 | +GRAPH_REGISTER_UNINITIALIZED (108) | ||
| 187 | +``` | ||
| 188 | + | ||
| 189 | +**Key Log:** | ||
| 190 | + | ||
| 191 | +``` | ||
| 192 | +[GenGraphCCU] [ErrorCode: 108] Failed to read XN register before it was initialized, rankId=2, dieId=0, instrId=73, xnId=11 | ||
| 193 | + | ||
| 194 | +[GenGraphCCU] [ErrorCode: 108] Failed to read HBM content before it was initialized, rankId=2, dieId=0, instrId=73, hbmAddr=0x1000 | ||
| 195 | +``` | ||
| 196 | + | ||
| 197 | +**Symptom:** The current instruction cannot find initialized data when reading a register or HBM content. This typically indicates that the preceding write chain was not correctly established. | ||
| 198 | + | ||
| 199 | +**Troubleshooting Guide:** | ||
| 200 | + | ||
| 201 | +``` | ||
| 202 | +[Possible Causes] | ||
| 203 | +1. Preceding Load/Set/Store instructions have not yet executed. | ||
| 204 | +2. The queue responsible for initialization is blocked by dependencies (e.g., Wait) and cannot proceed. | ||
| 205 | +3. Address or register parsing is misaligned, accessing unintended locations. | ||
| 206 | + | ||
| 207 | +[Troubleshooting Steps] | ||
| 208 | +1. If the log provides `xnId`, trace back to the most recent valid write to that register in the queue. | ||
| 209 | +2. If the log provides `hbmAddr`, check whether a valid write exists for that address range in earlier tasks. | ||
| 210 | +``` | ||
| 211 | + | ||
| 212 | +--- | ||
| 213 | + | ||
| 214 | +#### FAQ-CHK109 | ||
| 215 | + | ||
| 216 | +**Title:** ID or Index Out of Range | ||
| 217 | + | ||
| 218 | +**Error Code:** | ||
| 219 | + | ||
| 220 | +``` | ||
| 221 | +GRAPH_OUT_OF_RANGE (109) | ||
| 222 | +``` | ||
| 223 | + | ||
| 224 | +**Key Log:** | ||
| 225 | + | ||
| 226 | +``` | ||
| 227 | +[GenGraphCCU] [ErrorCode: 109] dieId is out of range when converting address to MS id, dieId=4, maxDieId=1 | ||
| 228 | + | ||
| 229 | +[GenGraphCCU] [ErrorCode: 109] Xn register id is out of the valid range, xnId=37, validMin=0, validMax=31 | ||
| 230 | +``` | ||
| 231 | + | ||
| 232 | +**Symptom:** An ID, index, or address attribution field exceeds the current resource or instruction constraint range. This commonly occurs with `dieId`, register numbers, or address parsing intermediate results. | ||
| 233 | + | ||
| 234 | +**Troubleshooting Guide:** | ||
| 235 | + | ||
| 236 | +``` | ||
| 237 | +[Possible Causes] | ||
| 238 | +1. Resource pool size does not match task data (e.g., only a subset of dies is loaded). | ||
| 239 | +2. Address attribution parsing error maps a local address to a nonexistent resource ID. | ||
| 240 | +3. Instruction field parsing is misaligned, causing abnormal register numbers or index values. | ||
| 241 | +4. Current data comes from instruction sets of different versions or branches. | ||
| 242 | + | ||
| 243 | +[Troubleshooting Steps] | ||
| 244 | +1. If the log provides `dieId/maxDieId`, first verify that the die count in the current resource file matches the task data. | ||
| 245 | +2. If the log provides `xnId/validMin/validMax`, review the original instruction fields to confirm whether the register number was incorrectly parsed or calculated. | ||
| 246 | +``` | ||
| 247 | + | ||
| 248 | +--- | ||
| 249 | + | ||
| 250 | +#### FAQ-CHK110 | ||
| 251 | + | ||
| 252 | +**Title:** Invalid Address or Unmet Alignment Constraints | ||
| 253 | + | ||
| 254 | +**Error Code:** | ||
| 255 | + | ||
| 256 | +``` | ||
| 257 | +GRAPH_ADDRESS_INVALID (110) | ||
| 258 | +``` | ||
| 259 | + | ||
| 260 | +**Key Log:** | ||
| 261 | + | ||
| 262 | +``` | ||
| 263 | +[GenGraphCCU] [ErrorCode: 110] Address does not fall into any known MS address range, localMsAddr=0x27f0000, rawAddr=0x82ff000 | ||
| 264 | + | ||
| 265 | +[GenGraphCCU] [ErrorCode: 110] Load source address must be 8-byte aligned, sourceAddress=0x1003 | ||
| 266 | +``` | ||
| 267 | + | ||
| 268 | +**Symptom:** The address cannot be mapped to a known resource range in Checker, or Load/Store-related addresses or lengths do not meet the alignment constraints of the current instruction. | ||
| 269 | + | ||
| 270 | +**Troubleshooting Guide:** | ||
| 271 | + | ||
| 272 | +``` | ||
| 273 | +[Possible Causes] | ||
| 274 | +1. Base address table mismatch. | ||
| 275 | +2. The original address was incorrectly written upstream, causing abnormal values. | ||
| 276 | +3. Upstream address calculation is off. | ||
| 277 | +4. An instruction's address or length is not properly aligned. | ||
| 278 | + | ||
| 279 | +[Troubleshooting Steps] | ||
| 280 | +1. If the log provides `rawAddr/localMsAddr` or `addr/dieBaseAddr`, first determine which resource range the address should theoretically fall into. | ||
| 281 | +2. If the log provides `sourceAddress`, `hbmAddr`, or `dataLengthBytes`, verify whether it meets 8-byte or 64-byte granularity constraints. | ||
| 282 | +``` | ||
| 283 | + | ||
| 284 | +--- | ||
| 285 | + | ||
| 286 | +#### FAQ-CHK111 | ||
| 287 | + | ||
| 288 | +**Title:** Current Task or Instruction Not Supported | ||
| 289 | + | ||
| 290 | +**Error Code:** | ||
| 291 | + | ||
| 292 | +``` | ||
| 293 | +GRAPH_UNSUPPORTED (111) | ||
| 294 | +``` | ||
| 295 | + | ||
| 296 | +**Key Log:** | ||
| 297 | + | ||
| 298 | +``` | ||
| 299 | +[GenGraph] [ErrorCode: 111] This task type is not supported for CheckerV3 graph generation, taskIndex=128, taskMeta=taskType=9, rankId=3, streamId=7 | ||
| 300 | + | ||
| 301 | +[GenGraphCCU] [ErrorCode: 111] This CCU instruction type is not supported by CheckerV3 graph expansion, rankId=2, queueId=1, instructionHeader=0xf431 | ||
| 302 | +``` | ||
| 303 | + | ||
| 304 | +**Symptom:** A feature not yet supported by Checker is being used. | ||
| 305 | + | ||
| 306 | +**Troubleshooting Guide:** | ||
| 307 | + | ||
| 308 | +``` | ||
| 309 | +[Possible Causes] | ||
| 310 | +1. Checker does not yet support this feature. | ||
| 311 | + | ||
| 312 | +[Troubleshooting Steps] | ||
| 313 | +1. Check the corresponding fields in the log to confirm whether they match expectations. | ||
| 314 | +2. Contact tool support personnel for assistance. | ||
| 315 | +``` | ||
| 316 | + | ||
| 317 | +--- | ||
| 318 | + | ||
| 319 | +#### FAQ-CHK112 | ||
| 320 | + | ||
| 321 | +**Title:** Remote Rank Derivation Inconsistency | ||
| 322 | + | ||
| 323 | +**Error Code:** | ||
| 324 | + | ||
| 325 | +``` | ||
| 326 | +GRAPH_REMOTE_RANK_MISMATCH (112) | ||
| 327 | +``` | ||
| 328 | + | ||
| 329 | +**Key Log:** | ||
| 330 | + | ||
| 331 | +``` | ||
| 332 | +[GenGraphCCU] [ErrorCode: 112] Remote address resolves to a different rank than the selected channel, instruction=TransLocMemToRmtMem, rankId=2, dieId=0, queueId=1, instrId=73, channelId=7, expectedRemoteRankId=5, actualRemoteRankId=6, remoteAddr=140737488363520 | ||
| 333 | +``` | ||
| 334 | + | ||
| 335 | +**Symptom:** The remote rank derived from the channel or remote address is inconsistent. | ||
| 336 | + | ||
| 337 | +**Troubleshooting Guide:** | ||
| 338 | + | ||
| 339 | +``` | ||
| 340 | +[Possible Causes] | ||
| 341 | +1. Channel table error. | ||
| 342 | +2. Remote address incorrectly encoded. | ||
| 343 | + | ||
| 344 | +[Troubleshooting Steps] | ||
| 345 | +1. Verify the ranks to which `channelId` and `remoteAddr` belong to confirm they match expectations. | ||
| 346 | +``` | ||
| 347 | + | ||
| 348 | +--- | ||
| 349 | + | ||
| 350 | +#### FAQ-CHK113 | ||
| 351 | + | ||
| 352 | +**Title:** Merged Loop Emission Failed | ||
| 353 | + | ||
| 354 | +**Error Code:** | ||
| 355 | + | ||
| 356 | +``` | ||
| 357 | +GRAPH_LOOP_MERGE_ERROR (113) | ||
| 358 | +``` | ||
| 359 | + | ||
| 360 | +**Key Log:** | ||
| 361 | + | ||
| 362 | +``` | ||
| 363 | +[GenGraphCCU] [ErrorCode: 113] Failed to emit one merged loop instruction because the merged instruction entry is null, rankId=2, queueId=1 | ||
| 364 | + | ||
| 365 | +[GenGraphCCU] [ErrorCode: 113] Failed to emit one merged loop transfer task, rankId=2, queueId=1, mergedLoopInstr={rankId=2, dieId=0, instrId=73, srcs=4, dsts=4, waitOps=1, setOps=1} | ||
| 366 | +``` | ||
| 367 | + | ||
| 368 | +**Symptom:** Loop merging fails in CCU mode. | ||
| 369 | + | ||
| 370 | +**Troubleshooting Guide:** | ||
| 371 | + | ||
| 372 | +``` | ||
| 373 | +[Possible Causes] | ||
| 374 | +1. Resource conflicts (memory addresses, CKE, etc.) occur during loop serial or parallel expansion. | ||
| 375 | +Note: After loop merge failure, normal expansion will be attempted, which may impact performance. | ||
| 376 | + | ||
| 377 | +[Troubleshooting Steps] | ||
| 378 | +1. Confirm that the loop body instruction template design meets expectations. | ||
| 379 | + | ||
| 380 | +``` | ||
| 381 | + | ||
| 382 | +--- | ||
| 383 | + | ||
| 384 | +### Submodule: Single Task and Slave Stream Validation | ||
| 385 | + | ||
| 386 | +--- | ||
| 387 | + | ||
| 388 | +#### FAQ-CHK201 | ||
| 389 | + | ||
| 390 | +**Title:** Invalid Memory Slice | ||
| 391 | + | ||
| 392 | +**Error Code:** | ||
| 393 | + | ||
| 394 | +``` | ||
| 395 | +SINGLETASK_SLICE_INVALID (201) | ||
| 396 | +``` | ||
| 397 | + | ||
| 398 | +**Error Functions:** | ||
| 399 | + | ||
| 400 | +``` | ||
| 401 | +task_graph_single_task_check_v3.cc::CheckMemorySlice() | ||
| 402 | +task_graph_single_task_check_v3.cc::CheckBatchTrans() | ||
| 403 | +task_graph_mem_conflict_v3.cc | ||
| 404 | +task_graph_semantic_check_v3.cc | ||
| 405 | +``` | ||
| 406 | + | ||
| 407 | +**Key Log:** | ||
| 408 | + | ||
| 409 | +``` | ||
| 410 | +[MemConflict] [ErrorCode: 201] One memory slice is missing a valid rank or memory type, task=[TaskTransMem] node=42, rank=0, stream=2, protocol=SDMA, src=rank 0 INPUT [0x0,0x400), dst=rank 0 CCL [0x1000,0x1400), rankId=invalid, memType=invalid, offset=0x0, length=0x400 | ||
| 411 | + | ||
| 412 | + | ||
| 413 | +[SingleTaskCheck] [ErrorCode: 201] One memory slice is invalid because its end address overflows while total coverage is being calculated, task=[TaskBatchTransMem] node=108, rank=1, stream=5, protocol=CCU, pairCount=2, mergedPairCount=2, src0=rank 1 CCL [0xfffffffffffffff0,0xffffffffffffff30), dst0=rank 1 OUTPUT [0x0,0x40), memorySlice={rankId=1, memType=CCL, offset=0xfffffffffffffff0, length=0x40} | ||
| 414 | + | ||
| 415 | + | ||
| 416 | +[SingleTaskCheck] [ErrorCode: 201] Batch trans slice length mismatch, node=[TaskBatchTransMem] node=108, rank=1, stream=5, protocol=CCU, label=src, index=2, expectedLen=0x400, actualLen=0x200 | ||
| 417 | + | ||
| 418 | +[SingleTaskCheck] [ErrorCode: 201] Batch trans pair size mismatch, node=[TaskBatchTransMem] node=108, rank=1, stream=5, protocol=CCU, label=src, srcCount=4, dstCount=3 | ||
| 419 | + | ||
| 420 | +[SingleTaskCheck] [ErrorCode: 201] Batch reduce has different counts of source groups and target memory slices, task=[TaskBatchReduce] node=176, rank=2, stream=4, protocol=CCU, group=src, sourceGroupCount=3, targetMemorySliceCount=2 | ||
| 421 | + | ||
| 422 | +[SingleTaskCheck] [ErrorCode: 201] Source data size is not an integer multiple of target data size, task=[TaskBatchReduce] node=176, rank=2, stream=4, protocol=CCU, srcDataSize=0xc00, dstDataSize=0x800, group=src | ||
| 423 | +``` | ||
| 424 | + | ||
| 425 | +**Symptom:** The memory slice itself is invalid, or slices in the same group overlap. The issue may occur during single task validation, memory conflict checking, or semantic simulation. | ||
| 426 | + | ||
| 427 | +**Troubleshooting Guide:** | ||
| 428 | + | ||
| 429 | +``` | ||
| 430 | +[Possible Causes] | ||
| 431 | +1. Incomplete slice fields. | ||
| 432 | +2. Memory type conversion failed. | ||
| 433 | +3. Length or offset does not meet expectations. | ||
| 434 | +4. Slice overlap detected between loops during CCU loop merging. | ||
| 435 | + | ||
| 436 | +[Troubleshooting Steps] | ||
| 437 | +1. Observe whether the `rankId/memType/offset/length` fields of the slice in the log match expectations. | ||
| 438 | +``` | ||
| 439 | + | ||
| 440 | +--- | ||
| 441 | + | ||
| 442 | +#### FAQ-CHK202 | ||
| 443 | + | ||
| 444 | +**Title:** Slice Conflict Within a Single Task | ||
| 445 | + | ||
| 446 | +**Error Code:** | ||
| 447 | + | ||
| 448 | +``` | ||
| 449 | +SINGLETASK_SLICE_CONFLICT (202) | ||
| 450 | +``` | ||
| 451 | + | ||
| 452 | +**Key Log:** | ||
| 453 | + | ||
| 454 | +``` | ||
| 455 | +[SingleTaskCheck] [ErrorCode: 202] Two memory slices overlap inside the same task, task=[TaskReduce] node=57, rank=0, stream=4, protocol=CCU, dataType=0, reduceOp=0, srcs=[rank 0 CCL [0x1000,0x1400), rank 0 CCL [0x1200,0x1600)], dst=rank 0 OUTPUT [0x0,0x400), memorySlice1={rankId=0, memType=CCL, offset=0x1000, length=0x400}, memorySlice2={rankId=0, memType=CCL, offset=0x1200, length=0x400}, position=rankId=0, streamId=4 | ||
| 456 | +``` | ||
| 457 | + | ||
| 458 | +**Symptom:** Overlapping memory slices exist within a single task, causing address range intersections on the same buffer. | ||
| 459 | + | ||
| 460 | +**Troubleshooting Guide:** | ||
| 461 | + | ||
| 462 | +``` | ||
| 463 | +[Possible Causes] | ||
| 464 | +1. Source and destination addresses in Transmem overlap. | ||
| 465 | +2. Source segment partitioning in Reduce is incorrect. | ||
| 466 | +3. Overlap remains after Batch merging. | ||
| 467 | +Note: In CCU mode, source and destination addresses being identical is allowed. | ||
| 468 | + | ||
| 469 | +[Troubleshooting Steps] | ||
| 470 | +1. Observe whether the `rankId/memType/offset/length` fields of the slices in the log match expectations. | ||
| 471 | +``` | ||
| 472 | + | ||
| 473 | +--- | ||
| 474 | + | ||
| 475 | +#### FAQ-CHK203 | ||
| 476 | + | ||
| 477 | +**Title:** Invalid Slave Stream Structure | ||
| 478 | + | ||
| 479 | +**Error Code:** | ||
| 480 | + | ||
| 481 | +``` | ||
| 482 | +SINGLETASK_SLAVE_STREAM_INVALID (203) | ||
| 483 | +``` | ||
| 484 | + | ||
| 485 | +**Key Log:** | ||
| 486 | + | ||
| 487 | +``` | ||
| 488 | +[StreamCheck] [ErrorCode: 203] This slave stream is missing its start node or end node, rankId=0, streamId=6, taskCount=4, startNode=null, endNode=[TaskRecordAICPU] node=241, rank=0, stream=6, protocol=SDMA, notify={recordRank=0, waitRank=0, notifyId=32} | ||
| 489 | + | ||
| 490 | + | ||
| 491 | +[StreamCheck] [ErrorCode: 203] The first task in this slave stream is not a local WAIT task, rankId=0, streamId=6, actualFirstTaskType=TRANS_MEM, firstTask=[TaskTransMem] node=214, rank=0, stream=6, protocol=SDMA, src=rank 0 INPUT [0x0,0x400), dst=rank 0 CCL [0x4000,0x4400) | ||
| 492 | + | ||
| 493 | + | ||
| 494 | +[StreamCheck] [ErrorCode: 203] The last task in this slave stream is not a local RECORD task, rankId=0, streamId=6, actualLastTaskType=WAIT, lastTask=[TaskWaitAICPU] node=245, rank=0, stream=6, protocol=SDMA, notify={recordRank=0, waitRank=0, notifyId=32} | ||
| 495 | + | ||
| 496 | + | ||
| 497 | +[StreamCheck] [ErrorCode: 203] This slave stream still has no valid end node after empty local-copy tasks are skipped, rankId=0, streamId=6, skippedEmptyLocalCopyCount=3, currentTailNode=null | ||
| 498 | +``` | ||
| 499 | + | ||
| 500 | +**Symptom:** The slave stream structure violates Checker constraints. Common manifestations include missing valid head/tail nodes, the first task not being a local `WAIT`, or the last task not being a local `RECORD`. | ||
| 501 | + | ||
| 502 | +**Troubleshooting Guide:** | ||
| 503 | + | ||
| 504 | +``` | ||
| 505 | +[Possible Causes] | ||
| 506 | +1. Slave stream is missing head/tail synchronization nodes. | ||
| 507 | + | ||
| 508 | +[Troubleshooting Steps] | ||
| 509 | +1. Slave stream missing start or end node: First verify whether the stream node list itself is complete, then confirm whether the current head/tail nodes match expectations. | ||
| 510 | +``` | ||
| 511 | + | ||
| 512 | +--- | ||
| 513 | + | ||
| 514 | +### Submodule: Memory Conflict Validation | ||
| 515 | + | ||
| 516 | +--- | ||
| 517 | + | ||
| 518 | +#### FAQ-CHK301 | ||
W 🔴 [严重] [翻译正确性/漏译] checker_faq_en.md:438 从 FAQ-CHK301 起的 11 个 FAQ 条目标题与错误码与中文版 checker_faq.md 全面不一致（疑似基于不同版本源文档翻译），CHK409 内容截断，且完全缺失“Dump 输出”(CHK501)、“主流程与配置”(CHK901/902) 两个子模块。英文版 673 行 vs 中文版 1025 行，约 35% 内容缺失。 对比示例：
建议：以当前中文版 checker_faq.md 为准完整重译 CHK301-902，并补全 CHK501/901/902 两个子模块。   | |||
| 519 | + | ||
| 520 | +**Title:** Invalid Memory Conflict DAG | ||
| 521 | + | ||
| 522 | +**Error Code:** | ||
| 523 | + | ||
| 524 | +``` | ||
| 525 | +MEMCONFLICT_DAG_INVALID (301) | ||
| 526 | +``` | ||
| 527 | + | ||
| 528 | +**Key Log:** | ||
| 529 | + | ||
| 530 | +``` | ||
| 531 | +[MemConflict] [ErrorCode: 301] Reachability analysis cannot start because the main start node is invalid, mainStartNode=[TaskTransMem] node=42, rank=0, stream=2, protocol=SDMA, src=rank 0 INPUT [0x0,0x400), dst=rank 0 CCL [0x1000,0x1400) | ||
| 532 | + | ||
| 533 | + | ||
| 534 | +[MemConflict] [ErrorCode: 301] This data-move node is missing its reachability index, node=[TaskBatchTransMem] node=318, rank=3, stream=2, protocol=CCU, pairCount=4, mergedPairCount=2, src0=rank 3 CCL [0x8000,0x8400), dst0=rank 3 OUTPUT [0x0,0x400) | ||
| 535 | + | ||
| 536 | + | ||
| 537 | +[MemConflict] [ErrorCode: 301] This V3 graph is not a complete DAG from the main start node, topoSize=412, expectedTopoSize=415, reachableTaskCount=411, taskNodeCount=414, mainStartNodeId=-1 | ||
| 538 | +``` | ||
| 539 | + | ||
| 540 | +**Symptom:** The main graph structure required by memory conflict checking is abnormal. Common causes include invalid main graph start point, missing reachability index nodes, or the task graph not being a complete DAG. | ||
| 541 | + | ||
| 542 | +**Troubleshooting Guide:** | ||
| 543 | + | ||
| 544 | +``` | ||
| 545 | +[Possible Causes] | ||
| 546 | +1. The main graph generated during graph generation is incomplete. | ||
| 547 | +2. Some task nodes are not on the reachable path from the `main_start` head node. | ||
| 548 | + | ||
| 549 | +[Troubleshooting Steps] | ||
| 550 | +1. First confirm whether task graph generation executed correctly. | ||
| 551 | +2. Contact tool support personnel for assistance. | ||
| 552 | +``` | ||
| 553 | + | ||
| 554 | +--- | ||
| 555 | + | ||
| 556 | +#### FAQ-CHK302 | ||
| 557 | + | ||
| 558 | +**Title:** Real Memory Conflict Detected | ||
| 559 | + | ||
| 560 | +**Error Code:** | ||
| 561 | + | ||
| 562 | +``` | ||
| 563 | +MEMCONFLICT_DETECTED (302) | ||
| 564 | +``` | ||
| 565 | + | ||
| 566 | +**Key Log:** | ||
| 567 | + | ||
| 568 | +``` | ||
| 569 | +[MemConflict] [ErrorCode: 302] Two tasks may access the same memory range in parallel, and at least one access is a write. | ||
| 570 | + Conflict memory : rank 0 OUTPUT | ||
| 571 | + Overlap range : [0x1000,0x1400) | ||
| 572 | + Conflict task 1: | ||
| 573 | + node 214, action=write | ||
| 574 | + access range : [0x1000,0x1800) | ||
| 575 | + task : [TaskTransMem] node=214, rank=0, stream=6, protocol=SDMA, src=rank 0 CCL [0x4000,0x4800), dst=rank 0 OUTPUT [0x1000,0x1800) | ||
| 576 | + Conflict task 2: | ||
| 577 | + node 233, action=write | ||
| 578 | + access range : [0x1000,0x1400) | ||
| 579 | + task : [TaskReduce] node=233, rank=0, stream=8, protocol=CCU, dataType=0, reduceOp=0, srcs=[rank 0 CCL [0x5000,0x5400), rank 3 CCL [0x5000,0x5400)], dst=rank 0 OUTPUT [0x1000,0x1400) | ||
| 580 | +``` | ||
| 581 | + | ||
| 582 | +**Symptom:** A real concurrent memory conflict is detected. Two tasks access the same memory range, and at least one access is a write. | ||
| 583 | + | ||
| 584 | +**Troubleshooting Guide:** | ||
| 585 | + | ||
| 586 | +``` | ||
| 587 | +[Possible Causes] | ||
| 588 | +1. Synchronization constraints between two streams are missing. | ||
| 589 | +2. Tasks that should be serialized are incorrectly modeled as concurrent. | ||
| 590 | +3. Read/write range partitioning or address calculation is incorrect. | ||
| 591 | +Note: Only `read-write` / `write-write` conflicts are validated; `read-read` is not considered a conflict. | ||
| 592 | + | ||
| 593 | +[Troubleshooting Steps] | ||
| 594 | +1. Review the log to identify which two tasks conflict at which address range, and check whether task scheduling and synchronization signal design match expectations. | ||
| 595 | +``` | ||
| 596 | + | ||
| 597 | +--- | ||
| 598 | + | ||
| 599 | +### Submodule: Semantic Validation | ||
| 600 | + | ||
| 601 | +--- | ||
| 602 | + | ||
| 603 | +#### FAQ-CHK401 | ||
| 604 | + | ||
| 605 | +**Title:** Target Range Has No Semantic Source | ||
| 606 | + | ||
| 607 | +**Error Code:** | ||
| 608 | + | ||
| 609 | +``` | ||
| 610 | +SEMANTIC_BUFFER_EMPTY (401) | ||
| 611 | +``` | ||
| 612 | + | ||
| 613 | +**Key Log:** | ||
| 614 | + | ||
| 615 | +``` | ||
| 616 | +[SemanticCheck] [ErrorCode: 401] No source/output information was found for the target memory range, startAddr=0x0, size=0x1000 | ||
| 617 | +``` | ||
| 618 | + | ||
| 619 | +**Symptom:** The semantic check cannot find any available source data or output semantics for the target range. | ||
| 620 | + | ||
| 621 | +**Troubleshooting Guide:** | ||
| 622 | + | ||
| 623 | +``` | ||
| 624 | +[Possible Causes] | ||
| 625 | +1. The relevant write task has not yet executed. | ||
| 626 | +2. Earlier semantic construction failed prematurely due to other errors. | ||
| 627 | +Note: Checker only initializes INPUT memory semantics by default; subsequent propagation is based on memory operation tasks. | ||
| 628 | + | ||
| 629 | +[Troubleshooting Steps] | ||
| 630 | +1. First confirm which task should theoretically write to this result, and verify whether the src source address semantics it uses were correctly set. | ||
| 631 | +``` | ||
| 632 | + | ||
| 633 | +**Diagram:** | ||
| 634 | + | ||
| 635 | +```mermaid | ||
| 636 | +flowchart TB | ||
| 637 | + subgraph NORMAL["Normal Semantic Propagation"] | ||
| 638 | + N1["Task1\nINPUT -> CCL1"] --> N2["Task2\nCCL1 -> CCL2"] | ||
| 639 | + N2 --> N3["Task3\nCCL2 -> OUTPUT"] | ||
| 640 | + N3 --> N4["OUTPUT semantics complete"] | ||
| 641 | + end | ||
| 642 | + | ||
| 643 | + subgraph ABNORMAL["Missing Intermediate Task"] | ||
| 644 | + A1["Task1\nINPUT -> CCL1"] -.->|"Missing Task2\nCCL1 -> CCL2"| A2["CCL2 has no semantics"] | ||
| 645 | + A2 --> A3["Task3\nCCL2 -> OUTPUT"] | ||
| 646 | + A3 --> A4["Error: Task3 CCL2 source semantics missing"] | ||
| 647 | + end | ||
| 648 | + | ||
| 649 | + style A2 fill:#fdd,stroke:#c33 | ||
| 650 | + style A4 fill:#fdd,stroke:#c33 | ||
| 651 | +``` | ||
| 652 | + | ||
| 653 | +--- | ||
| 654 | + | ||
| 655 | +#### FAQ-CHK402 | ||
| 656 | + | ||
| 657 | +**Title:** Semantic Result Range Gap | ||
| 658 | + | ||
| 659 | +**Error Code:** | ||
| 660 | + | ||
| 661 | +``` | ||
| 662 | +SEMANTIC_GAP (402) | ||
| 663 | +``` | ||
| 664 | + | ||
| 665 | +**Key Log:** | ||
| 666 | + | ||
| 667 | +``` | ||
| 668 | +[SemanticCheck] [ErrorCode: 402] Output data does not start from the expected address; the beginning is missing, expectedStart=0x0, actualStart=0x400 | ||
| 669 | + | ||
| 670 | + | ||
| 671 | +[SemanticCheck] [ErrorCode: 402] Output data is broken in the middle; one piece ends at 0x800 but the next starts at 0xc00 | ||
| 672 | + | ||
| 673 | + | ||
| 674 | +[SemanticCheck] [ErrorCode: 402] Output data ends too early; the tail is missing, expectedEnd=0x2000, actualEnd=0x1c00 | ||
| 675 | +``` | ||
| 676 | + | ||
| 677 | +**Symptom:** The semantic result range is discontinuous. Common manifestations include missing beginning, middle gap, or incomplete tail coverage. | ||
| 678 | + | ||
| 679 | +**Troubleshooting Guide:** | ||
| 680 | + | ||
| 681 | +``` | ||
| 682 | +[Possible Causes] | ||
| 683 | +1. Relevant write task did not execute completely. | ||
| 684 | +2. Offset or length calculation does not meet expectations. | ||
| 685 | + | ||
| 686 | +[Troubleshooting Steps] | ||
| 687 | +1. First determine whether it is missing beginning, middle gap, or missing tail based on `expectedStart/actualStart`, breakpoint address, or `expectedEnd/actualEnd`. | ||
| 688 | +2. Then check the corresponding write task to identify which data segment was not written or has incorrect write length. | ||
| 689 | +``` | ||
| 690 | + | ||
| 691 | +**Diagram:** | ||
| 692 | + | ||
| 693 | +```mermaid | ||
| 694 | +--- | ||
| 695 | +config: | ||
| 696 | + gantt: | ||
| 697 | + displayMode: compact | ||
| 698 | +--- | ||
| 699 | +gantt | ||
| 700 | + title Target Buffer | ||
| 701 | + dateFormat x | ||
| 702 | + axisFormat 0x%L | ||
| 703 | + tickInterval 100millisecond | ||
| 704 | + todayMarker off | ||
| 705 | + | ||
| 706 | + section Address Layout | ||
| 707 | + 0x000-0x100 Existing data :done, blk0, 0, 100ms | ||
| 708 | + 0x100-0x200 Gap :crit, hole, 100, 100ms | ||
| 709 | + 0x200-0x300 Existing data :done, blk1, 200, 100ms | ||
| 710 | +``` | ||
| 711 | + | ||
| 712 | +--- | ||
| 713 | + | ||
| 714 | +#### FAQ-CHK403 | ||
| 715 | + | ||
| 716 | +**Title:** Incorrect Reduce Semantics | ||
| 717 | + | ||
| 718 | +**Error Code:** | ||
| 719 | + | ||
| 720 | +``` | ||
| 721 | +SEMANTIC_REDUCE_ERROR (403) | ||
| 722 | +``` | ||
| 723 | + | ||
| 724 | +**Key Log:** | ||
| 725 | + | ||
| 726 | +``` | ||
| 727 | +[SemanticCheck] [ErrorCode: 403] Target output range is only partially filled before reduce continues, dataMapping={operation=reduce, sourceMemorySlice={rankId=3, memoryType=CCL, offset=0x800, length=0x400}, targetMemorySlice={rankId=1, memoryType=OUTPUT, offset=0x0, length=0x400}, launchIdx=18446744073709551615, blockId=4294967295, pipeId=4294967295, taskId=4294967295, reduceType=HCCL_REDUCE_SUM}, outputRange=[0x0,0x400), pieceCount=1 | ||
| 728 | + | ||
| 729 | + | ||
| 730 | +[SemanticCheck] [ErrorCode: 403] Reduce result type is inconsistent while merging one source data range, dataMapping={operation=reduce, sourceMemorySlice={rankId=2, memoryType=CCL, offset=0x0, length=0x400}, targetMemorySlice={rankId=0, memoryType=OUTPUT, offset=0x0, length=0x400}, launchIdx=18446744073709551615, blockId=4294967295, pipeId=4294967295, taskId=4294967295, reduceType=HCCL_REDUCE_MAX} | ||
| 731 | + | ||
| 732 | + | ||
| 733 | +[SemanticCheck] [ErrorCode: 403] Source data needed by this reduce is missing, dataMapping={operation=reduce, sourceMemorySlice={rankId=5, memoryType=INPUT, offset=0x400, length=0x400}, targetMemorySlice={rankId=0, memoryType=OUTPUT, offset=0x400, length=0x400}, launchIdx=18446744073709551615, blockId=4294967295, pipeId=4294967295, taskId=4294967295, reduceType=HCCL_REDUCE_SUM} | ||
| 734 | +``` | ||
| 735 | + | ||
| 736 | +**Symptom:** The reduce semantic chain is incomplete or inconsistent. Common manifestations include continuing reduce before the target range is fully covered, inconsistent reduce types, or missing reduce source data. | ||
| 737 | + | ||
| 738 | +**Troubleshooting Guide:** | ||
| 739 | + | ||
| 740 | +``` | ||
| 741 | +[Possible Causes] | ||
| 742 | +1. Preceding overwrite or transfer did not fully cover the target or source range. | ||
| 743 | +2. Reduce execution order is abnormal, or different `reduceOp` values are written to the same target range. | ||
| 744 | +3. Some ranks did not correctly participate in reduce. | ||
| 745 | + | ||
| 746 | +[Troubleshooting Steps] | ||
| 747 | +1. Review the log information to determine whether it is range incompleteness, type inconsistency, or source data deficiency. | ||
| 748 | +2. Then check the corresponding preceding memory operation tasks to identify which semantic link is missing. | ||
| 749 | +``` | ||
| 750 | + | ||
| 751 | +--- | ||
| 752 | + | ||
| 753 | +#### FAQ-CHK404 | ||
| 754 | + | ||
| 755 | +**Title:** Overwrite Source Semantics Missing | ||
| 756 | + | ||
| 757 | +**Error Code:** | ||
| 758 | + | ||
| 759 | +``` | ||
| 760 | +SEMANTIC_SIMULATE_FAILED (404) | ||
| 761 | +``` | ||
| 762 | + | ||
| 763 | +**Key Log:** | ||
| 764 | + | ||
| 765 | +``` | ||
| 766 | +[SemanticCheck] [ErrorCode: 404] Source data needed by this overwrite is missing, dataMapping={operation=overwrite, sourceMemorySlice={rankId=1, memoryType=INPUT, offset=0x0, length=0x800}, targetMemorySlice={rankId=1, memoryType=OUTPUT, offset=0x0, length=0x800}, launchIdx=18446744073709551615, blockId=4294967295, pipeId=4294967295, taskId=4294967295} | ||
| 767 | +``` | ||
| 768 | + | ||
| 769 | +**Symptom:** The source range semantics required by overwrite are incomplete. The current semantic implementation will continue simulation but will warn that this overwrite is not a "complete memcpy semantics"; subsequent semantic analysis results may be affected. | ||
| 770 | + | ||
| 771 | +**Troubleshooting Guide:** | ||
| 772 | + | ||
| 773 | +``` | ||
| 774 | +[Possible Causes] | ||
| 775 | +1. The overwrite source range was not fully initialized earlier, or only partial source semantics were established. | ||
| 776 | +2. Preceding transfer/slice tasks have incorrect source address or length configuration, causing overwrite to read uninitialized semantic holes. | ||
| 777 | +3. Some dependency tasks are missing or have abnormal ordering, causing the corresponding source data to be unprepared when overwrite executes. | ||
| 778 | + | ||
| 779 | +[Troubleshooting Steps] | ||
| 780 | +1. Based on `sourceMemorySlice`, locate the overwrite source buffer range and verify whether complete semantics were established for that range in preceding tasks. | ||
| 781 | +2. Check preceding transfer, slice, reduce, and other tasks to confirm continuous address ranges, matched lengths, and no intermediate holes. | ||
| 782 | +3. If this is expected behavior, further confirm whether subsequent analysis allows "partial source semantics" to continue propagating; otherwise, complete the preceding data chain. | ||
| 783 | +``` | ||
| 784 | + | ||
| 785 | +--- | ||
| 786 | + | ||
| 787 | +#### FAQ-CHK405 | ||
| 788 | + | ||
| 789 | +**Title:** Final Output Validation Prerequisite Not Met | ||
| 790 | + | ||
| 791 | +**Error Code:** | ||
| 792 | + | ||
| 793 | +``` | ||
| 794 | +SEMANTIC_FINAL_CHECK_FAILED (405) | ||
| 795 | +``` | ||
| 796 | + | ||
| 797 | +**Key Log:** | ||
| 798 | + | ||
| 799 | +``` | ||
| 800 | +[SemanticCheck] [ErrorCode: 405] Send/Recv final output validation supports exactly 2 ranks, but got expectedRankSize=2, actualRankSize=3, sourceRank=1, targetRank=5 | ||
| 801 | +``` | ||
| 802 | + | ||
| 803 | +**Symptom:** The prerequisite for final output validation is not met (e.g., Send/Recv scenario uses a rank count other than 2). | ||
| 804 | + | ||
| 805 | +**Troubleshooting Guide:** | ||
| 806 | + | ||
| 807 | +``` | ||
| 808 | +[Possible Causes] | ||
| 809 | +1. Input rank count configuration is incorrect. | ||
| 810 | +2. Ranks not belonging to the same Send/Recv are mixed together. | ||
| 811 | + | ||
| 812 | +[Troubleshooting Steps] | ||
| 813 | +1. Confirm whether this round of Send/Recv validation should theoretically contain only two ranks. | ||
| 814 | +``` | ||
| 815 | + | ||
| 816 | +--- | ||
| 817 | + | ||
| 818 | +#### FAQ-CHK406 | ||
| 819 | + | ||
| 820 | +**Title:** Final Output Missing Data | ||
| 821 | + | ||
| 822 | +**Error Code:** | ||
| 823 | + | ||
| 824 | +``` | ||
| 825 | +SEMANTIC_FINAL_MISSING (406) | ||
| 826 | +``` | ||
| 827 | + | ||
| 828 | +**Key Log:** | ||
| 829 | + | ||
| 830 | +``` | ||
| 831 | +[SemanticCheck] [ErrorCode: 406] AllGatherV produced no result data for rank 3, but this rank is expected to receive data from all 8 participating ranks with an expected total result size of 0x1c00 bytes. | ||
| 832 | + | ||
| 833 | + | ||
| 834 | +[SemanticCheck] [ErrorCode: 406] Send/Recv output for rank 5 should continue at 0x0, but the next actual range starts at 0x400 (actual range: [0x400,0x800)). | ||
| 835 | + Current result range detail: | ||
| 836 | + range=[0x400,0x800), size=0x400, sourceCount=1 | ||
| 837 | + sources: | ||
| 838 | + - sourceRank=1, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 839 | + | ||
| 840 | + | ||
| 841 | +[SemanticCheck] [ErrorCode: 406] ReduceScatter output for rank 6 ends too early: the checker validated 0x1800 bytes in total, but the expected result size is 0x1c00. | ||
| 842 | +``` | ||
| 843 | + | ||
| 844 | +**Symptom:** The final output is missing data. Common manifestations include a rank having no results at all, incorrect result start address, or result tail not fully written. | ||
| 845 | + | ||
| 846 | +**Troubleshooting Guide:** | ||
| 847 | + | ||
| 848 | +``` | ||
| 849 | +[Possible Causes] | ||
| 850 | +1. Result write chain did not execute completely. | ||
| 851 | +2. Memory transfer process has missing data or abnormal offsets. | ||
| 852 | +3. Address offset or shard size does not meet expectations. | ||
| 853 | + | ||
| 854 | +[Troubleshooting Steps] | ||
| 855 | +1. First determine which result segment is missing based on `expectedStartAddr/actualStartAddr` or `expectedSize/checkedSize`. | ||
| 856 | +2. Then trace backwards from the corresponding rank's memory transfer tasks to confirm each memory transfer task matches expectations. | ||
| 857 | +``` | ||
| 858 | + | ||
| 859 | +--- | ||
| 860 | + | ||
| 861 | +#### FAQ-CHK407 | ||
| 862 | + | ||
| 863 | +**Title:** Final Output Source Attribute Error | ||
| 864 | + | ||
| 865 | +**Error Code:** | ||
| 866 | + | ||
| 867 | +``` | ||
| 868 | +SEMANTIC_FINAL_SRC_ERROR (407) | ||
| 869 | +``` | ||
| 870 | + | ||
| 871 | +**Key Log:** | ||
| 872 | + | ||
| 873 | +``` | ||
| 874 | +[SemanticCheck] [ErrorCode: 407] AllGatherV output range [0x1000,0x1400) for rank 3 should come from rank 4, but it actually comes from rank 5. | ||
| 875 | + Current result range detail: | ||
| 876 | + range=[0x1000,0x1400), size=0x400, sourceCount=1 | ||
| 877 | + sources: | ||
| 878 | + - sourceRank=5, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 879 | + | ||
| 880 | + | ||
| 881 | +[SemanticCheck] [ErrorCode: 407] AllReduce result range [0x0,0x400) for rank 0 should come from INPUT, but source rank 3 actually provides buffer type CCL. | ||
| 882 | + Current result range detail: | ||
| 883 | + range=[0x0,0x400), size=0x400, reduce=HCCL_REDUCE_SUM, sourceCount=8 | ||
| 884 | + sources: | ||
| 885 | + - sourceRank=0, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 886 | + - sourceRank=1, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 887 | + - sourceRank=2, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 888 | + - sourceRank=3, sourceBufferType=CCL, sourceAddr=0x0 | ||
| 889 | + - sourceRank=4, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 890 | + - sourceRank=5, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 891 | + - sourceRank=6, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 892 | + - sourceRank=7, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 893 | + | ||
| 894 | + | ||
| 895 | +[SemanticCheck] [ErrorCode: 407] Send/Recv output range [0x400,0x800) for rank 5 should take data from source rank 1 at input address 0x400, but it actually takes data from source rank 1 at input address 0x0. | ||
| 896 | + Current result range detail: | ||
| 897 | + range=[0x400,0x800), size=0x400, sourceCount=1 | ||
| 898 | + sources: | ||
| 899 | + - sourceRank=1, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 900 | +``` | ||
| 901 | + | ||
| 902 | +**Symptom:** The source attributes of the final output are incorrect. Possible manifestations include source rank, source buffer type, or source address not matching expectations. | ||
| 903 | + | ||
| 904 | +**Troubleshooting Guide:** | ||
| 905 | + | ||
| 906 | +``` | ||
| 907 | +[Possible Causes] | ||
| 908 | +1. Result concatenation order or rank semantic labeling is incorrect. | ||
| 909 | +2. Intermediate buffer is incorrectly treated as the final source. | ||
| 910 | +3. Address offset or shard order does not meet expectations. | ||
| 911 | + | ||
| 912 | +[Troubleshooting Steps] | ||
| 913 | +1. First observe `actualSourceRank`, `actualSourceBufferType`, `expectedAddr`, and `actualAddr` to determine the problem type. | ||
| 914 | +2. Then check the corresponding memory transfer tasks to confirm each memory transfer task matches expectations. | ||
| 915 | +``` | ||
| 916 | + | ||
| 917 | +--- | ||
| 918 | + | ||
| 919 | +#### FAQ-CHK408 | ||
| 920 | + | ||
| 921 | +**Title:** Single Source Data Size Too Large | ||
| 922 | + | ||
| 923 | +**Error Code:** | ||
| 924 | + | ||
| 925 | +``` | ||
| 926 | +SEMANTIC_FINAL_SIZE_ERROR (408) | ||
| 927 | +``` | ||
| 928 | + | ||
| 929 | +**Key Log:** | ||
| 930 | + | ||
| 931 | +``` | ||
| 932 | +[SemanticCheck] [ErrorCode: 408] AllGatherV data collected from rank 4 for rank 3 becomes larger than expected after outputRange [0x1000,0x1600). The accumulated size is 0x600, but the expected size from this source rank is 0x400. | ||
| 933 | + Current result range detail: | ||
| 934 | + range=[0x1000,0x1600), size=0x600, sourceCount=1 | ||
| 935 | + sources: | ||
| 936 | + - sourceRank=4, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 937 | +``` | ||
| 938 | + | ||
| 939 | +**Symptom:** The contribution data size from a single source rank in the final output exceeds the range permitted by operator semantics. | ||
| 940 | + | ||
| 941 | +**Troubleshooting Guide:** | ||
| 942 | + | ||
| 943 | +``` | ||
| 944 | +[Possible Causes] | ||
| 945 | +1. Length configuration for this source rank is incorrect. | ||
| 946 | +2. The same data segment is concatenated multiple times. | ||
| 947 | + | ||
| 948 | +[Troubleshooting Steps] | ||
| 949 | +1. First verify the counts/displs configuration corresponding to `expectedSize`, then confirm whether this source rank's result is being concatenated repeatedly. | ||
| 950 | +``` | ||
| 951 | + | ||
| 952 | +--- | ||
| 953 | + | ||
| 954 | +#### FAQ-CHK409 | ||
| 955 | + | ||
| 956 | +**Title:** Final Output Reduce Semantic Error | ||
| 957 | + | ||
| 958 | +**Error Code:** | ||
| 959 | + | ||
| 960 | +``` | ||
| 961 | +SEMANTIC_FINAL_REDUCE_ERROR (409) | ||
| 962 | +``` | ||
| 963 | + | ||
| 964 | +**Key Log:** | ||
| 965 | + | ||
| 966 | +``` | ||
| 967 | +[SemanticCheck] [ErrorCode: 409] Send/Recv output range [0x0,0x400) for rank 5 should come from exactly one source, but it actually comes from 2 sources. | ||
| 968 | + Current result range detail: | ||
| 969 | + range=[0x0,0x400), size=0x400, sourceCount=2 | ||
| 970 | + sources: | ||
| 971 | + - sourceRank=1, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 972 | + - sourceRank=2, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 973 | + | ||
| 974 | + | ||
| 975 | +[SemanticCheck] [ErrorCode: 409] AllReduce result range [0x0,0x400) for rank 0 was reduced with mode HCCL_REDUCE_MAX, but the operator expects reduce mode HCCL_REDUCE_SUM. | ||
| 976 | + Current result range detail: | ||
| 977 | + range=[0x0,0x400), size=0x400, reduce=HCCL_REDUCE_MAX, sourceCount=8 | ||
| 978 | + sources: | ||
| 979 | + - sourceRank=0, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 980 | + - sourceRank=1, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 981 | + - sourceRank=2, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 982 | + - sourceRank=3, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 983 | + - sourceRank=4, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 984 | + - sourceRank=5, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 985 | + - sourceRank=6, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 986 | + - sourceRank=7, sourceBufferType=INPUT, sourceAddr=0x0 | ||
| 987 | + | ||
| 988 | + | ||
| 989 | +[SemanticCheck] [ErrorCode: 409] ReduceScatter output range [0x0,0x400) for rank 6 expected 8 source ranks but got 6. | ||
| 990 | + Current result range detail: | ||
| 991 | + range=[0x0,0x400), size=0x400, reduce=HCCL_REDUCE_SUM, sourceCount=6 | ||
| 992 | + sources: | ||
| 993 | + - sourceRank=0, sourceBufferType=INPUT, sourceAddr=0x1800 | ||
| 994 | + - sourceRank=1, sourceBufferType=INPUT, sourceAddr=0x1800 | ||
| 995 | + - sourceRank=2, sourceBufferType=INPUT, sourceAddr=0x1800 | ||
| 996 | + - sourceRank=3, sourceBufferType=INPUT, sourceAddr=0x1800 | ||
| 997 | + - sourceRank=4, sourceBufferType=INPUT, sourceAddr=0x1800 | ||
| 998 | + - sourceRank=5, sourceBufferType=INPUT, sourceAddr=0x1800 | ||
| 999 | +``` | ||
| 1000 | + | ||
| 1001 | +**Symptom:** The reduce semantics in the final output are incorrect. Possible manifestations include a single-source operator having multiple sources, `reduceType` mismatch, or insufficient source rank count. | ||
| 1002 | + | ||
| 1003 | +**Troubleshooting Guide:** | ||
| 1004 | + | ||
| 1005 | +``` | ||
| 1006 | +[Possible Causes] | ||
| 1007 | +1. Overwrite/reduce merging logic does not meet expectations. | ||
| 1008 | +2. `reduceOp` is inconsistent, or intermediate semantics are corrupted. | ||
| 1009 | +3. Some rank contributions did not enter the result range. | ||
| 1010 | + | ||
| 1011 | +[Troubleshooting Steps] | ||
| 1012 | +1. First observe `sourceCount`, `expectedSourceCount`, `actualReduceType`, and `sources` list to determine whether it is multi-source, type inconsistency, or source deficiency. | ||
| 1013 | +2. Then check the corresponding memory transfer and Reduce tasks to confirm each memory operation matches expectations. | ||
| 1014 | +``` | ||
| 1015 | + | ||
| 1016 | +--- | ||
| 1017 | + | ||
| 1018 | +### Submodule: Dump Output | ||
| 1019 | + | ||
| 1020 | +--- | ||
| 1021 | + | ||
| 1022 | +#### FAQ-CHK501 | ||
| 1023 | + | ||
| 1024 | +**Title:** Dump Output Failed | ||
| 1025 | + | ||
| 1026 | +**Error Code:** | ||
| 1027 | + | ||
| 1028 | +``` | ||
| 1029 | +DUMP_FAILED (501) | ||
| 1030 | +``` | ||
| 1031 | + | ||
| 1032 | +**Symptom:** Dump manager initialization, file writing, or serialization fails, preventing validation results from being persisted to disk. | ||
| 1033 | + | ||
| 1034 | +**Troubleshooting Guide:** | ||
| 1035 | + | ||
| 1036 | +``` | ||
| 1037 | +[Possible Causes] | ||
| 1038 | +1. Output directory or target path is not writable. | ||
| 1039 | +2. Insufficient disk space, or dump path not ready. | ||
| 1040 | +3. File handle, flush, or serialization failure. | ||
| 1041 | + | ||
| 1042 | +[Troubleshooting Steps] | ||
| 1043 | +1. First check the dump output directory, permissions, and disk space. | ||
| 1044 | +2. Contact tool support personnel for assistance. | ||
| 1045 | +``` | ||
| 1046 | + | ||
| 1047 | +--- | ||
| 1048 | + | ||
| 1049 | +### Submodule: Main Flow and Configuration | ||
| 1050 | + | ||
| 1051 | +--- | ||
| 1052 | + | ||
| 1053 | +#### FAQ-CHK901 | ||
| 1054 | + | ||
| 1055 | +**Title:** Runtime General Error | ||
| 1056 | + | ||
| 1057 | +**Error Code:** | ||
| 1058 | + | ||
| 1059 | +``` | ||
| 1060 | +CHECKER_RUNTIME_ERROR (901) | ||
| 1061 | +``` | ||
| 1062 | + | ||
| 1063 | +**Key Log:** | ||
| 1064 | + | ||
| 1065 | +``` | ||
| 1066 | +[Main] [ErrorCode: 901] Failed to load instruction data for this rank, rankId=3 | ||
| 1067 | + | ||
| 1068 | + | ||
| 1069 | +[Main] [ErrorCode: 901] Unsupported collective type, collectiveTypeCode=37 | ||
| 1070 | + | ||
| 1071 | + | ||
| 1072 | +[SemanticCheck] [ErrorCode: 901] Semantic check initialization failed because the rank count is 0, collectiveType=AllReduce, dataType=FLOAT, elementCount=1024, reduceType=SUM | ||
| 1073 | + | ||
| 1074 | + | ||
| 1075 | +[SemanticCheck] [ErrorCode: 901] Output simulation stopped because some tasks still have unresolved dependencies, handledNodeCount=410, totalNodeCount=415, firstRemainingNode=[TaskReduce] node=233, rank=2, stream=5, protocol=CCU, dataType=0, reduceOp=0, srcs=[rank 2 CCL [0x2000,0x2400)], dst=rank 2 OUTPUT [0x0,0x400) | ||
| 1076 | +``` | ||
| 1077 | + | ||
| 1078 | +**Symptom:** A general exception occurs during main flow runtime. | ||
| 1079 | + | ||
| 1080 | +**Troubleshooting Guide:** | ||
| 1081 | + | ||
| 1082 | +``` | ||
| 1083 | +[Possible Causes] | ||
| 1084 | +1. This is typically an internal Checker issue. | ||
| 1085 | + | ||
| 1086 | +[Troubleshooting Steps] | ||
| 1087 | +1. First determine the error type. For `Unsupported` and similar errors, you may self-diagnose whether the data meets Checker requirements. | ||
| 1088 | +2. Contact tool support personnel for assistance. | ||
| 1089 | +``` | ||
| 1090 | + | ||
| 1091 | +--- | ||
| 1092 | + | ||
| 1093 | +#### FAQ-CHK902 | ||
| 1094 | + | ||
| 1095 | +**Title:** Configuration or Runtime Strategy Warning | ||
| 1096 | + | ||
| 1097 | +**Error Code:** | ||
| 1098 | + | ||
| 1099 | +``` | ||
| 1100 | +SETTING_WARNING (902) | ||
| 1101 | +``` | ||
| 1102 | + | ||
| 1103 | +**Key Log:** | ||
| 1104 | + | ||
| 1105 | +``` | ||
| 1106 | +[Main] [ErrorCode: 902] This op is skipped because both the new checker and the old checker are disabled, opIndex=47, newCheckerEnabled=0, oldCheckerEnabled=0 | ||
| 1107 | +``` | ||
| 1108 | + | ||
| 1109 | +**Symptom:** Configuration switches or runtime strategies do not meet the execution conditions for the current op (e.g., both new and old checkers are disabled simultaneously). | ||
| 1110 | + | ||
| 1111 | +**Troubleshooting Guide:** | ||
| 1112 | + | ||
| 1113 | +``` | ||
| 1114 | +[Possible Causes] | ||
| 1115 | +1. manifest.json or runtime parameters disabled the checker. | ||
| 1116 | + | ||
| 1117 | +[Troubleshooting Steps] | ||
| 1118 | +1. Prioritize checking the switch configuration to confirm at least one Checker is enabled. | ||
| 1119 | +``` | ||
| @@ -0,0 +1,1543 @@ | |||
| 1 | +# HCCL-VM FAQ Test Document | ||
| 2 | + | ||
| 3 | +> This document is used to test the FAQ HTML generation framework. | ||
| 4 | + | ||
| 5 | +--- | ||
| 6 | + | ||
| 7 | +## Module: CANN Package Installation and WSL Environment Configuration | ||
| 8 | + | ||
| 9 | +--- | ||
| 10 | + | ||
| 11 | +#### FAQ-C001 | ||
| 12 | + | ||
| 13 | +**Title:** WSL Environment Configuration | ||
| 14 | + | ||
| 15 | +**Error Code:** | ||
| 16 | +``` | ||
| 17 | +NA (4) | ||
| 18 | +``` | ||
| 19 | + | ||
| 20 | +**Error Function:** | ||
| 21 | +``` | ||
| 22 | +NA | ||
| 23 | +``` | ||
| 24 | + | ||
| 25 | +**Key Log:** | ||
| 26 | +``` | ||
| 27 | +[ 85%] Building CXX object src/legacy/ascend910/framework/CMakeFiles/hcomm.dir/common/src/config/env_config_host.cc.o | ||
| 28 | +{standard input}: Assembler messages: | ||
| 29 | +{standard input}:61985: Warning: end of file not at end of a line; newline inserted | ||
| 30 | +{standard input}:61986: Error: expecting operand after ','; got nothing | ||
| 31 | +{standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive | ||
| 32 | +c++: fatal error: Killed signal terminated program cc1plus | ||
| 33 | +compilation terminated. | ||
| 34 | +gmake[2]: *** [src/legacy/ascend910/framework/CMakeFiles/hcomm.dir/build.make:160: src/legacy/ascend910/framework/CMakeFiles/hcomm.dir/common/src/topo/topoinfo_ranktableParser.cc.o] Error 1 | ||
| 35 | +gmake[2]: *** Waiting for unfinished jobs.... | ||
| 36 | +gmake[1]: *** [CMakeFiles/Makefile2:6400: src/legacy/ascend910/framework/CMakeFiles/hcomm.dir/all] Error 2 | ||
| 37 | +gmake: *** [Makefile:156: all] Error 2 | ||
| 38 | + Full log: /home/zhf/workspace/.hccl_vm_install_logs/build-pkg-20260705-234844.log | ||
| 39 | + Or run manually: bash /home/zhf/workspace/hcomm/test/hccl_vm/build_pkg.sh --tool_path /home/zhf/workspace/hcomm/test/hccl_vm | ||
| 40 | +[ERROR] Sub-package compilation failed. Please check the log and retry. | ||
| 41 | +``` | ||
| 42 | + | ||
| 43 | +**Symptom:** When compiling the hcomm sub-package using the one-click command or manual compilation in the WSL environment, the above error occurs. | ||
| 44 | + | ||
| 45 | +**Troubleshooting Guide:** | ||
| 46 | +``` | ||
| 47 | +[Possible Causes] | ||
| 48 | +Compiling the hcomm sub-package has certain requirements for the WSL virtual Linux environment. | ||
| 49 | +1. Ensure the WSL system version is Ubuntu 22.04 or Ubuntu 24.04. | ||
| 50 | +2. Ensure the WSL settings meet the following conditions: Available memory >= 8GB, swap space >= 4GB. Users can configure these via WSL settings. | ||
| 51 | +``` | ||
| 52 | + | ||
| 53 | +--- | ||
| 54 | + | ||
| 55 | +## Module: HCCL-VM | ||
| 56 | + | ||
| 57 | +### Submodule: Command Line | ||
| 58 | + | ||
| 59 | +--- | ||
| 60 | + | ||
| 61 | +#### FAQ-E001 | ||
| 62 | + | ||
| 63 | +**Title:** Communication Domain Not Configured | ||
| 64 | + | ||
| 65 | +**Error Code:** | ||
| 66 | +``` | ||
| 67 | +NA (4) | ||
| 68 | +``` | ||
| 69 | + | ||
| 70 | +**Error Function:** | ||
| 71 | +``` | ||
| 72 | +db_sim_runner_common.cc::GetDeviceByRankId() | ||
| 73 | +``` | ||
| 74 | + | ||
| 75 | +**Key Log:** | ||
| 76 | +``` | ||
| 77 | +[error][PID:173579][TID:173579][db_sim_runner_common.cc][GetDeviceByRankId] cannot find rank by rank id 0 | ||
| 78 | +[error][PID:173579][TID:173579][aclrt_device_stub.cc][aclrtSetDevice] [DEVICE_STUB]device not found by rankId:0 | ||
| 79 | +acl interface return err ./common/src/hccl_test_common.cc:861, retcode: 100000. | ||
| 80 | +This is an error in device_init. | ||
| 81 | +``` | ||
| 82 | + | ||
| 83 | +**Symptom:** When executing a business case, an error occurs indicating that the device with rank id 0 cannot be found. | ||
| 84 | + | ||
| 85 | +**Troubleshooting Guide:** | ||
| 86 | +``` | ||
| 87 | +[Possible Causes] | ||
| 88 | +Before executing business cases, users need to determine the communication domain scale used by the current operator and configure the communication domain via the hccl-vm mock-comm aa command. The aa.yaml file path is $HCCL_VM_INSTALL_DIR/config/topo_meta/aa.yaml. | ||
| 89 | +``` | ||
| 90 | + | ||
| 91 | +--- | ||
| 92 | + | ||
| 93 | +#### FAQ-E002 | ||
| 94 | + | ||
| 95 | +**Title:** RANK_TABLE_FILE Not Set | ||
| 96 | + | ||
| 97 | +**Error Code:** | ||
| 98 | +``` | ||
| 99 | +HCCL_SIM_E_PARA (1) | ||
| 100 | +``` | ||
| 101 | + | ||
| 102 | +**Error Function:** | ||
| 103 | +``` | ||
| 104 | +hccl_comm_stub.cc::HcclCommInitRootInfo() | ||
| 105 | +``` | ||
| 106 | + | ||
| 107 | +**Key Log:** | ||
| 108 | +``` | ||
| 109 | +RANK_TABLE_FILE env not set, please check your config. | ||
| 110 | +``` | ||
| 111 | + | ||
| 112 | +**Symptom:** The rank table configuration file cannot be found during communication domain initialization. | ||
| 113 | + | ||
| 114 | +**Troubleshooting Guide:** | ||
| 115 | +``` | ||
| 116 | +[Possible Causes] | ||
| 117 | +1. Environment variable not set | ||
| 118 | +2. Incorrect file path | ||
| 119 | + | ||
| 120 | +[Solution] | ||
| 121 | +export RANK_TABLE_FILE=/path/to/rank_table.json | ||
| 122 | +``` | ||
| 123 | + | ||
| 124 | +--- | ||
| 125 | + | ||
| 126 | +#### FAQ-E003 | ||
| 127 | + | ||
| 128 | +**Title:** HCCL_VM_INSTALL_DIR Not Set | ||
| 129 | + | ||
| 130 | +**Error Code:** | ||
| 131 | +``` | ||
| 132 | +HCCL_SIM_E_INTERNAL (4) | ||
| 133 | +``` | ||
| 134 | + | ||
| 135 | +**Error Function:** | ||
| 136 | +``` | ||
| 137 | +hccl_op_stub.cc::VirtualExecuteAivKernel() | ||
| 138 | +``` | ||
| 139 | + | ||
| 140 | +**Key Log:** | ||
| 141 | +``` | ||
| 142 | +[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set, can not locate <path> for kernel <name> | ||
| 143 | +``` | ||
| 144 | + | ||
| 145 | +**Symptom:** AIV kernel virtual execution fails because the corresponding .so file cannot be found. | ||
| 146 | + | ||
| 147 | +**Troubleshooting Guide:** | ||
| 148 | +``` | ||
| 149 | +[Solution] | ||
| 150 | +export HCCL_VM_INSTALL_DIR=/path/to/hccl_vm/install/dir | ||
| 151 | +``` | ||
| 152 | + | ||
| 153 | +--- | ||
| 154 | + | ||
| 155 | +#### FAQ-E004 | ||
| 156 | + | ||
| 157 | +**Title:** Repeatedly Executing the start Command in a Sub-shell | ||
| 158 | + | ||
| 159 | +**Error Code:** | ||
| 160 | +``` | ||
| 161 | +NA (No error code, WARNING only) | ||
| 162 | +``` | ||
| 163 | + | ||
| 164 | +**Error Function:** | ||
| 165 | +``` | ||
| 166 | +subcmd_start.cc::StartCommand::Execute() | ||
| 167 | +``` | ||
| 168 | + | ||
| 169 | +**Key Log:** | ||
| 170 | +``` | ||
| 171 | +[warning][PID:<PID>][TID:<TID>][subcmd_start.cc][Execute] hccl-vm has already started. Please do not start it again in a sub-bash. | ||
| 172 | +``` | ||
| 173 | + | ||
| 174 | +**Symptom:** When executing the `hccl-vm start` command again in the hvm sub-shell environment, the system indicates that it has already been started and ignores the operation. | ||
| 175 | + | ||
| 176 | +**Troubleshooting Guide:** | ||
| 177 | +``` | ||
| 178 | +[Possible Causes] | ||
| 179 | +`hccl-vm start` forks a child bash process. When the user enters `hccl-vm start` again in this sub-bash (prompt is `(hvm)$>`), the system rejects the duplicate start. | ||
| 180 | + | ||
| 181 | +[Solution] | ||
| 182 | +Do not repeatedly execute `hccl-vm start` in a sub-shell. To restart the simulation environment, first exit the current sub-shell (type `exit`), then re-execute `hccl-vm start`. | ||
| 183 | +``` | ||
| 184 | + | ||
| 185 | +--- | ||
| 186 | + | ||
| 187 | +#### FAQ-E005 | ||
| 188 | + | ||
| 189 | +**Title:** Fork Child Process Failed | ||
| 190 | + | ||
| 191 | +**Error Code:** | ||
| 192 | +``` | ||
| 193 | +HCCL_SIM_HOST_ERROR_CMD (No standard error code) | ||
| 194 | +``` | ||
| 195 | + | ||
| 196 | +**Error Function:** | ||
| 197 | +``` | ||
| 198 | +cmd_base_utils.cc::StartHvmCmd() | ||
| 199 | +``` | ||
| 200 | + | ||
| 201 | +**Key Log:** | ||
| 202 | +``` | ||
| 203 | +fork failed: Resource temporarily unavailable | ||
| 204 | +``` | ||
| 205 | + | ||
| 206 | +**Symptom:** After executing the `hccl-vm start` command, the system cannot create a sub-shell process, and the simulation environment fails to start. | ||
| 207 | + | ||
| 208 | +**Troubleshooting Guide:** | ||
| 209 | +``` | ||
| 210 | +[Possible Causes] | ||
| 211 | +1. System user process limit reached (ulimit -u) | ||
| 212 | +2. Insufficient system memory to allocate resources for new processes | ||
| 213 | +3. PID resources exhausted (/proc/sys/kernel/pid_max) | ||
| 214 | + | ||
| 215 | +[Troubleshooting Steps] | ||
| 216 | +ulimit -u | ||
| 217 | +cat /proc/sys/kernel/pid_max | ||
| 218 | +free -m | ||
| 219 | +ps -eLf | wc -l | ||
| 220 | + | ||
| 221 | +[Solution] | ||
| 222 | +1. Increase user process limit: `ulimit -u <larger value>` | ||
| 223 | +2. Clean up zombie processes remaining in the system | ||
| 224 | +3. Check if other programs are consuming excessive system resources | ||
| 225 | +``` | ||
| 226 | + | ||
| 227 | +--- | ||
| 228 | + | ||
| 229 | +#### FAQ-E006 | ||
| 230 | + | ||
| 231 | +**Title:** Invalid Plugin Name Format | ||
| 232 | + | ||
| 233 | +**Error Code:** | ||
| 234 | +``` | ||
| 235 | +NA (CLI parameter validation) | ||
| 236 | +``` | ||
| 237 | + | ||
| 238 | +**Error Function:** | ||
| 239 | +``` | ||
| 240 | +subcmd_plugin.cc::PluginCommand::Setup() | ||
| 241 | +``` | ||
| 242 | + | ||
| 243 | +**Key Log:** | ||
| 244 | +``` | ||
| 245 | +[HVM] [ERROR] Install plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 246 | +[HVM] [ERROR] Uninstall plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 247 | +[HVM] [ERROR] Run plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 248 | +``` | ||
| 249 | + | ||
| 250 | +**Symptom:** When executing the `hccl-vm plugin install/run/uninstall` command, CLI parameter validation fails and refuses to execute. | ||
| 251 | + | ||
| 252 | +**Troubleshooting Guide:** | ||
| 253 | +``` | ||
| 254 | +[Possible Causes] | ||
| 255 | +The plugin name does not start with the '@' symbol. For example, inputting `hccl-vm plugin install runner` instead of `hccl-vm plugin install @runner`. | ||
| 256 | + | ||
| 257 | +[Solution] | ||
| 258 | +Ensure the plugin name starts with '@', for example: | ||
| 259 | +hccl-vm plugin install @runner | ||
| 260 | +hccl-vm plugin install @checker | ||
| 261 | +hccl-vm plugin uninstall @runner | ||
| 262 | +``` | ||
| 263 | + | ||
| 264 | +--- | ||
| 265 | + | ||
| 266 | +#### FAQ-E007 | ||
| 267 | + | ||
| 268 | +**Title:** Topology Configuration File Does Not Exist | ||
| 269 | + | ||
| 270 | +**Error Code:** | ||
| 271 | +``` | ||
| 272 | +NA (CLI parameter validation) | ||
| 273 | +``` | ||
| 274 | + | ||
| 275 | +**Error Function:** | ||
| 276 | +``` | ||
| 277 | +cmd_base_utils.cc::FileInModelDir() | ||
| 278 | +``` | ||
| 279 | + | ||
| 280 | +**Key Log:** | ||
| 281 | +``` | ||
| 282 | +[HVM] model File not found: <install_path>/config/topo_meta/<name>.yaml | ||
| 283 | +``` | ||
| 284 | + | ||
| 285 | +**Symptom:** When executing the `hccl-vm mock-comm <name>` command, the specified topology yaml configuration file does not exist, and CLI parameter validation directly rejects it. The communication domain configuration file is used to describe the scale of the communication domain for operator execution (e.g., how many super nodes the domain contains, how many servers, and which cards from each server to use — see file description for details). | ||
| 286 | + | ||
| 287 | +**Troubleshooting Guide:** | ||
| 288 | +``` | ||
| 289 | +[Possible Causes] | ||
| 290 | +1. Typo in the specified topology name | ||
| 291 | +2. The corresponding yaml file is not placed in the `$HCCL_VM_INSTALL_DIR/config/topo_meta/` directory | ||
| 292 | +3. Incorrect file extension (should be `.yaml`) | ||
| 293 | + | ||
| 294 | +[Troubleshooting Steps] | ||
| 295 | +ls $HCCL_VM_INSTALL_DIR/config/topo_meta/ | ||
| 296 | + | ||
| 297 | +[Solution] | ||
| 298 | +Confirm that the topology yaml file is placed in the correct directory and the filename matches the command parameter. For example, executing `hccl-vm mock-comm 121` requires the `config/topo_meta/121.yaml` file to exist. | ||
| 299 | +``` | ||
| 300 | + | ||
| 301 | +--- | ||
| 302 | + | ||
| 303 | +#### FAQ-E008 | ||
| 304 | + | ||
| 305 | +**Title:** YAML Topology File Format Parsing Exception | ||
| 306 | + | ||
| 307 | +**Error Code:** | ||
| 308 | +``` | ||
| 309 | +NA (Runtime parsing error) | ||
| 310 | +``` | ||
| 311 | + | ||
| 312 | +**Error Function:** | ||
| 313 | +``` | ||
| 314 | +cmd_cluster_model_utils.cc::ParseYamlTopoImpl() | ||
| 315 | +``` | ||
| 316 | + | ||
| 317 | +**Key Log:** | ||
| 318 | +``` | ||
| 319 | +[error][PID:<PID>][TID:<TID>][cmd_cluster_model_utils.cc][ParseYamlTopoImpl] Exception when parsing YAML: <detail> | ||
| 320 | +``` | ||
| 321 | + | ||
| 322 | +**Symptom:** When executing the `hccl-vm mock-comm <name>` command, the YAML topology configuration file fails to parse, and communication domain initialization is interrupted. | ||
| 323 | + | ||
| 324 | +**Troubleshooting Guide:** | ||
| 325 | +``` | ||
| 326 | +[Possible Causes] | ||
| 327 | +1. YAML file contains syntax errors (e.g., incorrect indentation, missing space after colon, illegal characters, etc.) | ||
| 328 | +2. YAML file contains unsupported field types or formats | ||
| 329 | +3. YAML file encoding is not UTF-8 | ||
| 330 | + | ||
| 331 | +[Troubleshooting Steps] | ||
| 332 | +# Use python to validate yaml format | ||
| 333 | +python3 -c "import yaml; yaml.safe_load(open('$HCCL_VM_INSTALL_DIR/config/topo_meta/<name>.yaml'))" | ||
| 334 | + | ||
| 335 | +[Solution] | ||
| 336 | +Correct the YAML file syntax errors based on the `<detail>` information in the log. Common issues include: | ||
| 337 | +1. Indentation must use spaces, not tabs | ||
| 338 | +2. A space is required after the colon in key-value pairs | ||
| 339 | +3. List item (`-`) indentation must be consistent with its level | ||
| 340 | +``` | ||
| 341 | + | ||
| 342 | +--- | ||
| 343 | + | ||
| 344 | +### Submodule: Memory Management | ||
| 345 | + | ||
| 346 | +--- | ||
| 347 | + | ||
| 348 | +#### FAQ-M001 | ||
| 349 | + | ||
| 350 | +**Title:** Device Memory Allocation Exceeded | ||
| 351 | + | ||
| 352 | +**Error Code:** | ||
| 353 | +``` | ||
| 354 | +HCCL_SIM_E_MEMORY (3) | ||
| 355 | +``` | ||
| 356 | + | ||
| 357 | +**Error Function:** | ||
| 358 | +``` | ||
| 359 | +store_sim_device_memory_manager.cc::AllocPhyMem() | ||
| 360 | +``` | ||
| 361 | + | ||
| 362 | +**Key Log:** | ||
| 363 | +``` | ||
| 364 | +dev:<N> alloc phy mem:<ADDR> size:<SIZE> exceeds pool ceiling:<CEILING>, reject | ||
| 365 | +``` | ||
| 366 | + | ||
| 367 | +**Symptom:** The device memory allocation request exceeds the simulated memory pool ceiling. | ||
| 368 | + | ||
| 369 | +**Diagram:** | ||
| 370 | +```mermaid | ||
| 371 | +graph LR | ||
| 372 | + A[Memory Allocation Request] --> B{Check Pool Ceiling} | ||
| 373 | + B -->|Not Exceeded| C[Allocation Successful] | ||
| 374 | + B -->|Exceeded| D[Reject Allocation] | ||
| 375 | + D --> E[Error: exceeds pool ceiling] | ||
| 376 | +``` | ||
| 377 | + | ||
| 378 | +--- | ||
| 379 | + | ||
| 380 | +#### FAQ-M002 | ||
| 381 | + | ||
| 382 | +**Title:** Shared Memory Creation Failed | ||
| 383 | + | ||
| 384 | +**Error Code:** | ||
| 385 | +``` | ||
| 386 | +HCCL_SIM_E_SYSCALL (8) | ||
| 387 | +``` | ||
| 388 | + | ||
| 389 | +**Error Function:** | ||
| 390 | +``` | ||
| 391 | +store_sim_shm_ops.cc::ShmCreate() | ||
| 392 | +``` | ||
| 393 | + | ||
| 394 | +**Key Log:** | ||
| 395 | +``` | ||
| 396 | +[SHM_OPS] create: shm_open failed, name: <name> | ||
| 397 | +[SHM_OPS] create: ftruncate failed, name: <name> | ||
| 398 | +[SHM_OPS] create: mmap failed, name: <name> | ||
| 399 | +``` | ||
| 400 | + | ||
| 401 | +**Symptom:** Unable to create a shared memory segment. | ||
| 402 | + | ||
| 403 | +**Troubleshooting Guide:** | ||
| 404 | +``` | ||
| 405 | +[Possible Causes] | ||
| 406 | +1. Insufficient `/dev/shm` space | ||
| 407 | +2. Insufficient permissions | ||
| 408 | +3. Shared memory with the same name already exists and conflicts | ||
| 409 | + | ||
| 410 | +[Troubleshooting Steps] | ||
| 411 | +df -h /dev/shm | ||
| 412 | +ls /dev/shm/ | grep hccl | ||
| 413 | +``` | ||
| 414 | + | ||
| 415 | +--- | ||
| 416 | + | ||
| 417 | +#### FAQ-M003 | ||
| 418 | + | ||
| 419 | +**Title:** Communication Memory Allocation Failed | ||
| 420 | + | ||
| 421 | +**Error Code:** | ||
| 422 | +``` | ||
| 423 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 424 | +``` | ||
| 425 | + | ||
| 426 | +**Error Function:** | ||
| 427 | +``` | ||
| 428 | +store_sim_comm_memory_manager.cc | ||
| 429 | +``` | ||
| 430 | + | ||
| 431 | +**Key Log:** | ||
| 432 | +``` | ||
| 433 | +[COMM_MEM] alloc failed, name: <name> | ||
| 434 | +[COMM_MEM] acquire failed, name: <name> | ||
| 435 | +[COMM_MEM] write size too large, size: <N>, max: <MAX> | ||
| 436 | +``` | ||
| 437 | + | ||
| 438 | +**Symptom:** Cross-process communication memory operation failed. | ||
| 439 | + | ||
| 440 | +--- | ||
| 441 | + | ||
| 442 | +#### FAQ-M004 | ||
| 443 | + | ||
| 444 | +**Title:** BUS Error When Operating Device Memory | ||
| 445 | + | ||
| 446 | +**Error Code:** `NA` | ||
| 447 | + | ||
| 448 | +**Error Function:** `CommunicationMemoryManager::WriteCommMem` | ||
| 449 | + | ||
| 450 | +**Key Log:** | ||
| 451 | +``` | ||
| 452 | +Bus error | ||
| 453 | +``` | ||
| 454 | + | ||
| 455 | +**Symptom:** The business process crashes directly with a bus error. | ||
| 456 | + | ||
| 457 | +**Possible Causes:** `/dev/shm` has no available space. Check with `df -h /dev/shm`. | ||
| 458 | + | ||
| 459 | +--- | ||
| 460 | + | ||
| 461 | +### Submodule: Stub Proxy (proxy) | ||
| 462 | + | ||
| 463 | +--- | ||
| 464 | + | ||
| 465 | +#### FAQ-PX001 | ||
| 466 | + | ||
| 467 | +**Title:** AIV Kernel Virtual Execution Failed | ||
| 468 | + | ||
| 469 | +**Error Code:** | ||
| 470 | +``` | ||
| 471 | +HCCL_SIM_E_INTERNAL (4) | ||
| 472 | +``` | ||
| 473 | + | ||
| 474 | +**Error Function:** | ||
| 475 | +``` | ||
| 476 | +hccl_op_stub.cc::VirtualExecuteAivKernel() | ||
| 477 | +``` | ||
| 478 | + | ||
| 479 | +**Key Log:** | ||
| 480 | +``` | ||
| 481 | +[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set | ||
| 482 | +[virtual-aiv] missing aiv stub shared library, kernel=<name> | ||
| 483 | +[virtual-aiv] dlopen <so> failed, err = <error> | ||
| 484 | +[virtual-aiv] dlsym <symbol> from <so> failed, err = <error> | ||
| 485 | +``` | ||
| 486 | + | ||
| 487 | +**Symptom:** AIV kernel execution fails in the virtual environment. | ||
| 488 | + | ||
| 489 | +**Troubleshooting Guide:** | ||
| 490 | +``` | ||
| 491 | +[Troubleshooting Steps] | ||
| 492 | +echo $HCCL_VM_INSTALL_DIR | ||
| 493 | +ls -la $HCCL_VM_INSTALL_DIR/lib/aiv/ | ||
| 494 | +nm -D $HCCL_VM_INSTALL_DIR/lib/aiv/<kernel>.so | grep <symbol> | ||
| 495 | +``` | ||
| 496 | + | ||
| 497 | +--- | ||
| 498 | + | ||
| 499 | +#### FAQ-PX002 | ||
| 500 | + | ||
| 501 | +**Title:** Operator Database Record Failed | ||
| 502 | + | ||
| 503 | +**Error Code:** | ||
| 504 | +``` | ||
| 505 | +HCCL_SIM_E_INTERNAL (4) | ||
| 506 | +``` | ||
| 507 | + | ||
| 508 | +**Error Function:** | ||
| 509 | +``` | ||
| 510 | +hccl_op_stub.cc::RecordOpDbInfo() | ||
| 511 | +``` | ||
| 512 | + | ||
| 513 | +**Key Log:** | ||
| 514 | +``` | ||
| 515 | +[RecordOpDbInfo] insert op detail+mem failed | ||
| 516 | +[HcclAllReduce] record op db info failed | ||
| 517 | +``` | ||
| 518 | + | ||
| 519 | +**Symptom:** HCCL collective communication operator parameters cannot be written to the simulation database. | ||
| 520 | + | ||
| 521 | +**Affected Operators:** AlltoAll, AlltoAllV, AllGather, Broadcast, AllReduce, Scatter, Reduce, ReduceScatter | ||
| 522 | + | ||
| 523 | +--- | ||
| 524 | + | ||
| 525 | +#### FAQ-PX003 | ||
| 526 | + | ||
| 527 | +**Title:** QP Not Found or State Error | ||
| 528 | + | ||
| 529 | +**Error Code:** | ||
| 530 | +``` | ||
| 531 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 532 | +``` | ||
| 533 | + | ||
| 534 | +**Error Function:** | ||
| 535 | +``` | ||
| 536 | +hccp_stub.cc::RaSendWr() | ||
| 537 | +``` | ||
| 538 | + | ||
| 539 | +**Key Log:** | ||
| 540 | +``` | ||
| 541 | +[HCCP] RaSendWr: QP <N> not found | ||
| 542 | +[HCCP] RaSendWr: QP <N> not in RTS state, current state:<N> | ||
| 543 | +``` | ||
| 544 | + | ||
| 545 | +**Symptom:** RDMA QP operation failed — QP does not exist or has not reached RTS state. | ||
| 546 | + | ||
| 547 | +**Diagram:** | ||
| 548 | +```mermaid | ||
| 549 | +stateDiagram-v2 | ||
| 550 | + [*] --> INIT | ||
| 551 | + INIT --> RTR: RaQpConnect | ||
| 552 | + RTR --> RTS: RaTypicalQpModify | ||
| 553 | + RTS --> [*]: Ready to send data | ||
| 554 | + RTS --> ERROR: State abnormal | ||
| 555 | + INIT --> ERROR: Not properly initialized | ||
| 556 | +``` | ||
| 557 | + | ||
| 558 | +--- | ||
| 559 | + | ||
| 560 | +#### FAQ-PX004 | ||
| 561 | + | ||
| 562 | +**Title:** EndPoint Lookup Failed | ||
| 563 | + | ||
| 564 | +**Error Code:** | ||
| 565 | +``` | ||
| 566 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 567 | +``` | ||
| 568 | + | ||
| 569 | +**Error Function:** | ||
| 570 | +``` | ||
| 571 | +hccp_stub.cc::RaCtxQpImport() | ||
| 572 | +``` | ||
| 573 | + | ||
| 574 | +**Key Log:** | ||
| 575 | +``` | ||
| 576 | +[HCCP] cannot find endpoint addr:<IP> | ||
| 577 | +Get remote endpoint failed. ip:<IP>, eid:<EID> | ||
| 578 | +``` | ||
| 579 | + | ||
| 580 | +**Symptom:** Network endpoint lookup failed. | ||
| 581 | + | ||
| 582 | +**Troubleshooting Guide:** | ||
| 583 | +``` | ||
| 584 | +[Possible Causes] | ||
| 585 | +The IP address is not in the endpoint list configured in the rank table. | ||
| 586 | +``` | ||
| 587 | + | ||
| 588 | +--- | ||
| 589 | + | ||
| 590 | +#### FAQ-PX005 | ||
| 591 | + | ||
| 592 | +**Title:** CCU Microcode Loading Failed | ||
| 593 | + | ||
| 594 | +**Error Code:** | ||
| 595 | +``` | ||
| 596 | +HCCL_SIM_E_INTERNAL (4) | ||
| 597 | +``` | ||
| 598 | + | ||
| 599 | +**Error Function:** | ||
| 600 | +``` | ||
| 601 | +hccp_ccu_stub.cc::LoadMicrocodeInstruction() | ||
| 602 | +``` | ||
| 603 | + | ||
| 604 | +**Key Log:** | ||
| 605 | +``` | ||
| 606 | +[LoadMicrocodeInstruction] get device by logic id <N> failed. | ||
| 607 | +[LoadMicrocodeInstruction] get ccu from device by die id <N> failed. | ||
| 608 | +[LoadMicrocodeInstruction] insert instr failed | ||
| 609 | +``` | ||
| 610 | + | ||
| 611 | +**Symptom:** CCU microcode instruction failed to load into the simulator. | ||
| 612 | + | ||
| 613 | +--- | ||
| 614 | + | ||
| 615 | +#### FAQ-PX006 | ||
| 616 | + | ||
| 617 | +**Title:** Unable to Obtain Current Context | ||
| 618 | + | ||
| 619 | +**Error Code:** | ||
| 620 | +``` | ||
| 621 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 622 | +``` | ||
| 623 | + | ||
| 624 | +**Error Function:** | ||
| 625 | +``` | ||
| 626 | +hccp_stub.cc::RaRdevInit() | ||
| 627 | +``` | ||
| 628 | + | ||
| 629 | +**Key Log:** | ||
| 630 | +``` | ||
| 631 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaRdevInit] can not get CurrContext: <N> | ||
| 632 | +``` | ||
| 633 | + | ||
| 634 | +**Symptom:** During RDMA device initialization, the active Context cannot be obtained through the current Runner, causing RDMA device creation to fail. | ||
| 635 | + | ||
| 636 | +**Troubleshooting Guide:** | ||
| 637 | +``` | ||
| 638 | +[Possible Causes] | ||
| 639 | +1. Application layer did not call `aclrtSetDevice`/`aclrtCreateContext` to initialize device and context | ||
| 640 | +2. Context has been destroyed prematurely | ||
| 641 | +3. current_ctx_id in Runner's TLS (Thread-Local Storage) is invalid | ||
| 642 | +4. Application layer called other runtime interfaces to obtain context before calling `aclrtSetDevice` to initialize device context | ||
| 643 | + | ||
| 644 | +[Troubleshooting Steps] | ||
| 645 | +# Check Context table | ||
| 646 | +hccl-vm table show Context | ||
| 647 | +# Check current_ctx_id in Runner table | ||
| 648 | +hccl-vm table show Runner | ||
| 649 | + | ||
| 650 | +[Solution] | ||
| 651 | +Confirm that the application layer has correctly called `aclrtSetDevice` and `aclrtCreateContext` before performing RDMA operations, and the Context has not been destroyed prematurely. | ||
| 652 | +``` | ||
| 653 | + | ||
| 654 | +--- | ||
| 655 | + | ||
| 656 | +#### FAQ-PX007 | ||
| 657 | + | ||
| 658 | +**Title:** AICPU Binary File Not Found | ||
| 659 | + | ||
| 660 | +**Error Code:** | ||
| 661 | +``` | ||
| 662 | +ACL_ERROR_RT_FEATURE_NOT_SUPPORT | ||
| 663 | +``` | ||
| 664 | + | ||
| 665 | +**Error Function:** | ||
| 666 | +``` | ||
| 667 | +aclrt_kernel_stub.cc::aclrtDestroyBinary() | ||
| 668 | +``` | ||
| 669 | + | ||
| 670 | +**Key Log:** | ||
| 671 | +``` | ||
| 672 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][aclrtDestroyBinary] can not find this binary | ||
| 673 | +``` | ||
| 674 | + | ||
| 675 | +**Symptom:** When destroying an AICPU binary object, the corresponding binary handle cannot be found in the global kernel binary registry. | ||
| 676 | + | ||
| 677 | +**Troubleshooting Guide:** | ||
| 678 | +``` | ||
| 679 | +[Possible Causes] | ||
| 680 | +1. The binary file was not correctly loaded (`aclrtLoadBinary` was not executed or failed) | ||
| 681 | +2. The binary handle has been destroyed repeatedly (double-free) | ||
| 682 | +3. The binary object was concurrently operated on in a multi-threaded environment, causing inconsistent state | ||
| 683 | + | ||
| 684 | +[Troubleshooting Steps] | ||
| 685 | +# Check for duplicate destroy calls | ||
| 686 | +# Verify the return value of aclrtLoadBinary | ||
| 687 | + | ||
| 688 | +[Solution] | ||
| 689 | +Ensure `aclrtLoadBinary` returns successfully before calling `aclrtDestroyBinary`, and do not repeatedly destroy the same binary object. | ||
| 690 | +``` | ||
| 691 | + | ||
| 692 | +--- | ||
| 693 | + | ||
| 694 | +#### FAQ-PX008 | ||
| 695 | + | ||
| 696 | +**Title:** AICPU Device Process Abnormal Exit | ||
| 697 | + | ||
| 698 | +**Error Code:** | ||
| 699 | +``` | ||
| 700 | +NA (Process-level error) | ||
| 701 | +``` | ||
| 702 | + | ||
| 703 | +**Error Function:** | ||
| 704 | +``` | ||
| 705 | +aclrt_kernel_stub.cc::WaitAicpuProcess() | ||
| 706 | +``` | ||
| 707 | + | ||
| 708 | +**Key Log:** | ||
| 709 | +``` | ||
| 710 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] exited with status <N> | ||
| 711 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] killed by signal <N> | ||
| 712 | +``` | ||
| 713 | + | ||
| 714 | +**Symptom:** The AICPU device child process exits abnormally or is killed by a signal, causing the main process to subsequently exit as well (`exit(EXIT_FAILURE)`). | ||
| 715 | + | ||
| 716 | +**Troubleshooting Guide:** | ||
| 717 | +``` | ||
| 718 | +[Possible Causes] | ||
| 719 | +1. Uncaught exception or segmentation fault inside the AICPU process | ||
| 720 | +2. Insufficient system resources (memory, file descriptors, etc.) causing the child process to be killed by the OOM killer | ||
| 721 | +3. The AICPU binary file itself contains bugs | ||
| 722 | +4. Missing shared libraries required by the child process | ||
| 723 | + | ||
| 724 | +[Troubleshooting Steps] | ||
| 725 | +# Check system logs for OOM records | ||
| 726 | +dmesg | grep -i "oom\|killed" | ||
| 727 | +# Verify AICPU binary file integrity | ||
| 728 | +ls -la $HCCL_VM_INSTALL_DIR/bin/ | ||
| 729 | +# Check system resources | ||
| 730 | +ulimit -a | ||
| 731 | +free -m | ||
| 732 | + | ||
| 733 | +[Solution] | ||
| 734 | +1. Check if the AICPU binary file is correctly compiled and deployed | ||
| 735 | +2. Confirm sufficient system resources (memory, file descriptor limits, etc.) | ||
| 736 | +3. If killed by a signal, further diagnose based on the signal number (e.g., 11=SIGSEGV, 9=SIGKILL) | ||
| 737 | +``` | ||
| 738 | + | ||
| 739 | +--- | ||
| 740 | + | ||
| 741 | +#### FAQ-PX009 | ||
| 742 | + | ||
| 743 | +**Title:** CCU Cannot Find Any Rank When Loading Microcode | ||
| 744 | + | ||
| 745 | +**Error Code:** | ||
| 746 | +``` | ||
| 747 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 748 | +``` | ||
| 749 | + | ||
| 750 | +**Error Function:** | ||
| 751 | +``` | ||
| 752 | +hccp_ccu_stub.cc::LoadMicrocodeInstruction() | ||
| 753 | +``` | ||
| 754 | + | ||
| 755 | +**Key Log:** | ||
| 756 | +``` | ||
| 757 | +[error][PID:<PID>][TID:<TID>][hccp_ccu_stub.cc][LoadMicrocodeInstruction] can not find any rank | ||
| 758 | +``` | ||
| 759 | + | ||
| 760 | +**Symptom:** During CCU microcode instruction loading, no rank records can be found in the Rank table corresponding to the current device. | ||
| 761 | + | ||
| 762 | +**Troubleshooting Guide:** | ||
| 763 | +``` | ||
| 764 | +[Possible Causes] | ||
| 765 | +1. Communication domain not initialized via the `mock-comm` command; Rank table is empty | ||
| 766 | +2. The current device ID does not exist in the communication domain configuration | ||
| 767 | + | ||
| 768 | +[Troubleshooting Steps] | ||
| 769 | +# Check if Rank table has data | ||
| 770 | +hccl-vm table show Rank | ||
| 771 | +# Check device table | ||
| 772 | +hccl-vm table show Device | ||
| 773 | + | ||
| 774 | +[Solution] | ||
| 775 | +Ensure that before performing CCU-related operations, the communication domain has been correctly initialized via the `hccl-vm mock-comm` command, and the communication domain configuration covers the current device. | ||
| 776 | +``` | ||
| 777 | + | ||
| 778 | +--- | ||
| 779 | + | ||
| 780 | +#### FAQ-PX010 | ||
| 781 | + | ||
| 782 | +**Title:** Device Lookup by rankId Failed | ||
| 783 | + | ||
| 784 | +**Error Code:** | ||
| 785 | +``` | ||
| 786 | +HCCL_E_NOT_FOUND | ||
| 787 | +``` | ||
| 788 | + | ||
| 789 | +cc | ||
| 790 | +``` | ||
| 791 | +aclrt_device_stub.cc::hrtSetDevice() | ||
| 792 | +``` | ||
| 793 | + | ||
| 794 | +**Key Log:** | ||
| 795 | +``` | ||
| 796 | +[error][PID:<PID>][TID:<TID>][aclrt_device_stub.cc][hrtSetDevice] device not found by rankId:<N> | ||
| 797 | +``` | ||
| 798 | + | ||
| 799 | +**Symptom:** When calling `aclrtSetDevice` to set the current device, device lookup by rankId fails. | ||
| 800 | + | ||
| 801 | +**Troubleshooting Guide:** | ||
| 802 | +``` | ||
| 803 | +[Possible Causes] | ||
| 804 | +1. rankId exceeds the actual rank range in the communication domain — e.g., the domain is configured with 4 NPUs, but mpirun started 6 NPU processes, causing rankid 4, 5 to report device not found. | ||
| 805 | +2. Communication domain not initialized (did not execute the `mock-comm` command) — [most likely] The Rank table is only initialized after the communication domain is initialized. | ||
| 806 | +3. Rank table configuration does not match the actual number of ranks used — possibly `RANK_TABLE_FILE` is configured with an incorrect file path. | ||
| 807 | + | ||
| 808 | +[Troubleshooting Steps] | ||
| 809 | +# Check if rankId is within valid range | ||
| 810 | +hccl-vm table show Rank | ||
| 811 | + | ||
| 812 | +[Solution] | ||
| 813 | +Confirm that rankId is within the valid range of the communication domain configuration (0 to rank_count-1), and the `RANK_TABLE_FILE` environment variable points to the correct ranktable.json file. | ||
| 814 | +``` | ||
| 815 | + | ||
| 816 | +--- | ||
| 817 | + | ||
| 818 | +#### FAQ-PX011 | ||
| 819 | + | ||
| 820 | +**Title:** Stub Interface Not Yet Implemented | ||
| 821 | + | ||
| 822 | +**Error Code:** | ||
| 823 | +``` | ||
| 824 | +HCCL_SIM_E_INTERNAL (4) or NA | ||
| 825 | +``` | ||
| 826 | + | ||
| 827 | +**Error Function:** | ||
| 828 | +``` | ||
| 829 | +Multiple stub function files (hccp_stub.cc, ascend_hal_stub.cc, aclrt_kernel_stub.cc, etc.) | ||
| 830 | +``` | ||
| 831 | + | ||
| 832 | +**Key Log:** | ||
| 833 | +``` | ||
| 834 | +[warning][PID:<PID>][TID:<TID>][ascend_hal_stub.cc][*] [STUB] is empty | ||
| 835 | +[warning][PID:<PID>][TID:<TID>][hccp_stub.cc][*] [STUB] is empty | ||
| 836 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetAuxInfo] Not support yet | ||
| 837 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetCrErrInfoList] Not support yet | ||
| 838 | +``` | ||
| 839 | + | ||
| 840 | +**Symptom:** The application layer calls underlying driver or runtime interfaces that the simulator has not yet implemented. Warnings/errors with `[STUB] is empty` or `Not support yet` appear in the log. These stub functions directly return default values (typically 0 or success) without performing any actual operations. | ||
| 841 | + | ||
| 842 | +**Troubleshooting Guide:** | ||
| 843 | +``` | ||
| 844 | +[Possible Causes] | ||
| 845 | +The current version of the simulator only implements the core interface subset required for HCCL collective communication. Some underlying driver interfaces (such as drvGetDeviceCapability, RaCtxGetAuxInfo, drvMemPrefetch, etc.) are not on the core path of HCCL communication, so the stub function body is empty or marked as unsupported. | ||
| 846 | + In general, for flows supported by the HCCL-VM tool, these interfaces will not be called, so such warnings will not appear. If the user calls an incorrect application-layer interface or enters an incorrect HCCL business flow, such warnings may appear. | ||
| 847 | + | ||
| 848 | +[Solution] | ||
| 849 | +1. Such warnings typically do not affect the correctness simulation of HCCL operators and can be safely ignored | ||
| 850 | +2. If the warning is accompanied by functional anomalies, it indicates the application depends on unimplemented interfaces. Please report to the simulator development team | ||
| 851 | +3. If stub implementation of specific interfaces is needed, contact the development team for prioritized adaptation | ||
| 852 | +``` | ||
| 853 | + | ||
| 854 | +**Main Interface Types Involved:** | ||
| 855 | +1. **Driver layer interfaces** (`ascend_hal_stub.cc`): Approximately 315 interfaces including drvGetDeviceCapability, drvMemPrefetch, drvStreamQuery, etc. | ||
| 856 | +2. **RDMA interfaces** (`hccp_stub.cc`): Approximately 44 interfaces including RaRestoreSnapshot, RaRdevInitWithBackup, RaCtxGetAuxInfo, etc. | ||
| 857 | +3. **Runtime adaptation layer** (`adapter_rts_stub.cc`): Some aclrt extension interfaces | ||
| 858 | +4. **TSD client** (`tsd_client_stub.cc`): TSD-related interfaces | ||
| 859 | + | ||
| 860 | +--- | ||
| 861 | + | ||
| 862 | +#### FAQ-PX012 | ||
| 863 | + | ||
| 864 | +**Title:** Socket Acquisition Failed | ||
| 865 | + | ||
| 866 | +**Error Code:** `NA` | ||
| 867 | + | ||
| 868 | +**Error Function:** `hccp_ra_socket_stub.cc::RaGetSockets()` | ||
| 869 | + | ||
| 870 | +**Key Log:** | ||
| 871 | +``` | ||
| 872 | +[RASOCKET_STUB]get socket failed local:socketFd peerAddr:ipaddr role:0 | ||
| 873 | +``` | ||
| 874 | + | ||
| 875 | +**Symptom:** Failed to obtain the desired link establishment socket handle. | ||
| 876 | + | ||
| 877 | +**Possible Causes:** | ||
| 878 | +1. The remote socket did not call RaSocketInit | ||
| 879 | +2. The remote socket did not call RaSocketBatchConnect | ||
| 880 | +3. The remote socket has already called RaSocketBatchClose | ||
| 881 | + | ||
| 882 | +**Diagram:** | ||
| 883 | +```mermaid | ||
| 884 | +graph TD | ||
| 885 | + A[1. RaSocketInit] --> B[2. RaSocketListenStart] | ||
| 886 | + B --> C[3. RaSocketBatchConnect] | ||
| 887 | + C --> D[4. RaGetSockets] | ||
| 888 | + D --> E[4. RaSend/Recv] | ||
| 889 | + E --> F[5. RaSocketBatchClose] | ||
| 890 | +``` | ||
| 891 | + | ||
| 892 | +--- | ||
| 893 | + | ||
| 894 | +#### FAQ-PX013 | ||
| 895 | + | ||
| 896 | +**Title:** Socket Buffer Allocation Failed | ||
| 897 | + | ||
| 898 | +**Error Code:** `NA` | ||
| 899 | + | ||
| 900 | +**Error Function:** `hccp_ra_socket_stub.cc::RaSocketBatchConnect()` | ||
| 901 | + | ||
| 902 | +**Key Log:** | ||
| 903 | +``` | ||
| 904 | +[RASOCKET_STUB] alloc sock name ra_sock_1_c2s mem failed | ||
| 905 | +``` | ||
| 906 | + | ||
| 907 | +**Symptom:** Socket link buffer memory allocation failed. | ||
| 908 | + | ||
| 909 | +**Possible Causes:** There may be leftover socket buffers not cleared. Check with `ls /dev/shm/`. | ||
| 910 | + | ||
| 911 | +--- | ||
| 912 | + | ||
| 913 | +#### FAQ-PX014 | ||
| 914 | + | ||
| 915 | +**Title:** waitpid Waiting for AICPU Device Process Failed | ||
| 916 | + | ||
| 917 | +**Error Code:** | ||
| 918 | +``` | ||
| 919 | +NA (Process-level error) | ||
| 920 | +``` | ||
| 921 | + | ||
| 922 | +**Error Function:** | ||
| 923 | +``` | ||
| 924 | +aclrt_kernel_stub.cc::WaitAicpuProcess() | ||
| 925 | +``` | ||
| 926 | + | ||
| 927 | +**Key Log:** | ||
| 928 | +``` | ||
| 929 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] waitpid failed for pid <PID>, errno: <N> (<description>) | ||
| 930 | +``` | ||
| 931 | + | ||
| 932 | +**Symptom:** When the main process is waiting for the AICPU device child process to exit, the `waitpid` system call does not return the target device process PID. The main process then calls `exit(EXIT_FAILURE)` to exit, interrupting the simulation. Unlike FAQ-PX008, in this scenario the exit status of the device child process was not successfully collected (waitpid itself failed), rather than the child process actively crashing. | ||
| 933 | + | ||
| 934 | +**Troubleshooting Guide:** | ||
| 935 | +``` | ||
| 936 | +[Possible Causes] | ||
| 937 | +waitpid returned an incorrect value. Common errno causes: | ||
| 938 | +1. ECHILD (10): The current process has no waitable child processes — the device child process has already been reaped by another thread/process, or fork parent-child relationship is disordered (e.g., the child process was preemptively reaped by wait4 in a signal handler) | ||
| 939 | +2. EINVAL (22): The options or signal parameters passed are invalid (typically a code logic bug) | ||
| 940 | +3. EINTR (4): waitpid was interrupted by a signal and did not automatically restart (rare; waitpid defaults to retrying on EINTR, but some call paths do not handle this) | ||
| 941 | + | ||
| 942 | +[Troubleshooting Steps] | ||
| 943 | +# Verify if the target PID exists and its parent process is the current process | ||
| 944 | +ps -eo pid,ppid,stat,cmd | grep <PID> | ||
| 945 | +# Check system logs for abnormal process reaping or signal handling | ||
| 946 | +dmesg | grep -i "process\|killed" | ||
| 947 | +# Check for duplicate WaitAicpuProcess calls or multi-threaded race conditions in child process reaping | ||
| 948 | +# (Code side) Check if SIGCHLD handler is registered and internally calls wait/waitpid | ||
| 949 | + | ||
| 950 | +``` | ||
| 951 | + | ||
| 952 | +--- | ||
| 953 | + | ||
| 954 | +#### FAQ-PX015 | ||
| 955 | + | ||
| 956 | +**Title:** Socket Link Establishment Cannot Find Remote IP Endpoint | ||
| 957 | + | ||
| 958 | +**Error Code:** | ||
| 959 | +``` | ||
| 960 | +NA (Returns -1, no standard error code) | ||
| 961 | +``` | ||
| 962 | + | ||
| 963 | +**Error Function:** | ||
| 964 | +``` | ||
| 965 | +hccp_ra_socket_stub.cc::RaSocketInit() | ||
| 966 | +hccp_ra_socket_stub.cc::RaSocketBatchConnect() | ||
| 967 | +``` | ||
| 968 | + | ||
| 969 | +**Key Log:** | ||
| 970 | +``` | ||
| 971 | +[RASOCKET_STUB] get device by phy id <N> failed | ||
| 972 | +[RASOCKET_STUB] cannot find remote ip <IP> | ||
| 973 | +``` | ||
| 974 | + | ||
| 975 | +**Symptom:** After `RaSocketInit` resolves the IP based on `rdevInfo.localIp` or `RaSocketBatchConnect` resolves the IP based on `conn[i].remoteIp`, the corresponding endpoint cannot be found in the EndPoint table, and socket initialization/link establishment returns -1 directly, interrupting the operation. The `<IP>` in the log is typically an IPv6 string with the prefix removed. | ||
| 976 | + | ||
| 977 | +**Troubleshooting Guide:** | ||
| 978 | +``` | ||
| 979 | +[Possible Causes] | ||
| 980 | +1. The endpoint for this IP is not configured in ranktable.json (EndPoint table is missing this entry) | ||
| 981 | +2. Communication domain not initialized (did not execute `mock-comm`); EndPoint table is empty | ||
| 982 | +3. IP address mismatch — the IP configured in the ranktable differs from the IP actually dispatched at runtime (e.g., IPv6 address abbreviation/prefix mapping differences) | ||
| 983 | +4. The ranktable's server_count or device_list count is insufficient; some ranks' IPs are not included in the endpoint table | ||
| 984 | + | ||
| 985 | +[Troubleshooting Steps] | ||
| 986 | +# Check if the endpoint table contains this IP | ||
| 987 | +hccl-vm table show EndPoint | ||
| 988 | +# Verify ranktable configuration | ||
| 989 | +echo $RANK_TABLE_FILE | ||
| 990 | +cat $RANK_TABLE_FILE | python3 -m json.tool | ||
| 991 | +# Check if communication domain has been initialized | ||
| 992 | +hccl-vm table show Rank | ||
| 993 | + | ||
| 994 | +``` | ||
| 995 | + | ||
| 996 | +--- | ||
| 997 | + | ||
| 998 | +#### FAQ-PX016 | ||
| 999 | + | ||
| 1000 | +**Title:** Socket Link Establishment Remote End Timeout Not Ready | ||
| 1001 | + | ||
| 1002 | +**Error Code:** | ||
| 1003 | +``` | ||
| 1004 | +NA (Function returns 0, but this connection did not establish a pair; subsequent RaGetSockets will trigger FAQ-PX012) | ||
| 1005 | +``` | ||
| 1006 | + | ||
| 1007 | +**Error Function:** | ||
| 1008 | +``` | ||
| 1009 | +hccp_ra_socket_stub.cc::RaSocketBatchConnect() | ||
| 1010 | +``` | ||
| 1011 | + | ||
| 1012 | +**Key Log:** | ||
| 1013 | +``` | ||
| 1014 | +[RASOCKET_STUB] can not find remote dev:<N>, endpoint:<N> | ||
| 1015 | +[RASOCKET_STUB] can not get break dev:<N> sock:<N> connect dev:<N> ip addr:<IP> tag:<tag> | ||
| 1016 | +``` | ||
| 1017 | + | ||
| 1018 | +**Symptom:** `RaSocketBatchConnect` polls and waits for the remote end to appear in the RaSocket table. After 600 iterations × 100ms = 60 seconds, the remote end is still not found, and times out with `break`. This connection did not establish a RaSocketPair, so subsequent `RaGetSockets` will not be able to obtain the socket and will report FAQ-PX012. | ||
| 1019 | + | ||
| 1020 | +**Troubleshooting Guide:** | ||
| 1021 | +``` | ||
| 1022 | +[Possible Causes] | ||
| 1023 | +1. The remote rank process has not started, or has not yet called `RaSocketInit`/`RaSocketListenStart` | ||
| 1024 | +2. Multi-rank startup order is disordered — when BatchConnect is initiated, the remote end has not yet completed socket initialization | ||
| 1025 | +3. The remote end's device_id/endpoint_id does not match the local end's query criteria (ranktable endpoint configuration inconsistent) | ||
| 1026 | +4. The remote end process exited abnormally, and the RaSocket record was not written to DB | ||
| 1027 | + | ||
| 1028 | +[Troubleshooting Steps] | ||
| 1029 | +# Check RaSocket table to confirm if the remote end has created a socket record | ||
| 1030 | +hccl-vm table show RaSocket | ||
| 1031 | +# Confirm all rank processes have started | ||
| 1032 | +ps -ef | grep <business process name> | ||
| 1033 | +# Confirm rank count in communication domain matches process count | ||
| 1034 | +hccl-vm table show Rank | ||
| 1035 | + | ||
| 1036 | +``` | ||
| 1037 | + | ||
| 1038 | +**Diagram:** | ||
| 1039 | +```mermaid | ||
| 1040 | +sequenceDiagram | ||
| 1041 | + participant L as Local End | ||
| 1042 | + participant DB as RaSocket Table | ||
| 1043 | + participant R as Remote End | ||
| 1044 | + L->>DB: RaSocketInit(Write local socket) | ||
| 1045 | + L->>DB: Poll remote socket(600 iterations x 100ms) | ||
| 1046 | + R-->>DB: RaSocketInit(Write remote socket) | ||
| 1047 | + DB-->>L: Remote end matched → Establish Pair | ||
| 1048 | + Note over L,DB: If remote end does not write within 60s → Timeout break | ||
| 1049 | +``` | ||
| 1050 | + | ||
| 1051 | +--- | ||
| 1052 | + | ||
| 1053 | +#### FAQ-PX017 | ||
| 1054 | + | ||
| 1055 | +**Title:** socketHandle Does Not Exist in RaSocket Table | ||
| 1056 | + | ||
| 1057 | +**Error Code:** | ||
| 1058 | +``` | ||
| 1059 | +NA (Returns -1) | ||
| 1060 | +``` | ||
| 1061 | + | ||
| 1062 | +**Error Function:** | ||
| 1063 | +``` | ||
| 1064 | +hccp_ra_socket_stub.cc::RaSocketListenStart() | ||
| 1065 | +hccp_ra_socket_stub.cc::RaSocketListenStop() | ||
| 1066 | +hccp_ra_socket_stub.cc::RaSocketBatchConnect() | ||
| 1067 | +hccp_ra_socket_stub.cc::RaGetSockets() | ||
| 1068 | +``` | ||
| 1069 | + | ||
| 1070 | +**Key Log:** | ||
| 1071 | +``` | ||
| 1072 | +[RASOCKET_STUB] can not get Socket:<N> | ||
| 1073 | +[RASOCKET_STUB] can not get Local Ra Socket:<N> | ||
| 1074 | +[RASOCKET_STUB] can not get local socket fd:<N> | ||
| 1075 | +``` | ||
| 1076 | + | ||
| 1077 | +**Symptom:** When calling `RaSocketListenStart`/`RaSocketListenStop`/`RaSocketBatchConnect`/`RaGetSockets`, the passed `socketHandle` cannot be found by `GetById` in the RaSocket table, and the corresponding operation fails with return -1. | ||
| 1078 | + | ||
| 1079 | +**Troubleshooting Guide:** | ||
| 1080 | +``` | ||
| 1081 | +[Possible Causes] | ||
| 1082 | +1. The socketHandle has been deleted by `RaSocketDeinit` (Deinit called before use) | ||
| 1083 | +2. The socketHandle value is invalid — comes from uninitialized memory or a handle from another process | ||
| 1084 | +3. Handle was passed across processes — the simulator's RaSocket table is an in-process DB; handles are not shared across processes. If the business layer passes fd between processes, the remote process will not be able to find it. | ||
| 1085 | +4. RaSocketInit returned failure (e.g., FAQ-PX015), but the caller did not check the return value and still uses the empty handle | ||
| 1086 | + | ||
| 1087 | +[Troubleshooting Steps] | ||
| 1088 | +# Confirm if the handle exists in RaSocket table | ||
| 1089 | +hccl-vm table show RaSocket | ||
| 1090 | +# Confirm call order — whether the handle is still used after Deinit | ||
| 1091 | +# Confirm handle source — whether it was successfully returned by RaSocketInit | ||
| 1092 | + | ||
| 1093 | +``` | ||
| 1094 | + | ||
| 1095 | +--- | ||
| 1096 | + | ||
| 1097 | +#### FAQ-PX018 | ||
| 1098 | + | ||
| 1099 | +**Title:** Socket Link Send/Receive Read/Write Failed | ||
| 1100 | + | ||
| 1101 | +**Error Code:** | ||
| 1102 | +``` | ||
| 1103 | +NA (Returns -1) | ||
| 1104 | +``` | ||
| 1105 | + | ||
| 1106 | +**Error Function:** | ||
| 1107 | +``` | ||
| 1108 | +hccp_ra_socket_stub.cc::RaSocketSend() | ||
| 1109 | +hccp_ra_socket_stub.cc::RaSocketRecv() | ||
| 1110 | +hccp_ra_socket_stub.cc::RaSocketRecvAsync() | ||
| 1111 | +``` | ||
| 1112 | + | ||
| 1113 | +**Key Log:** | ||
| 1114 | +``` | ||
| 1115 | +[RASOCKET_STUB] cannot pair socket:<N> role:<N>, key=<key> | ||
| 1116 | +[RASOCKET_STUB] socket pair:<N> role:<N>, key=<key> recv failed | ||
| 1117 | +[RASOCKET_STUB] socket pair:<N> role:<N>, key=<key> read try again | ||
| 1118 | +``` | ||
| 1119 | + | ||
| 1120 | +**Symptom:** `RaSocketSend` calls `WriteCommMem` which fails to write to c2s/s2c shared memory and returns -1; or `RaSocketRecv`/`RaSocketRecvAsync` calls `ReadCommMem` which returns -1. Link data send/receive is interrupted. Note that `read try again` is a WARN indicating no data available to read (remote end has not yet sent); it will sleep and retry, returning 0, which is outside the scope of this FAQ. | ||
| 1121 | + | ||
| 1122 | +**Troubleshooting Guide:** | ||
| 1123 | +``` | ||
| 1124 | +[Possible Causes] | ||
| 1125 | +1. The pair's c2s/s2c shared memory has been released by `RaSocketBatchClose` — after ref_cnt drops to zero, `DestoryRaSocketBufKeyByPairId` deleted the buffer, but some thread is still in Send/Recv | ||
| 1126 | +2. Insufficient `/dev/shm` space or leftover ra_sock_* old buffers causing memory segment conflicts | ||
| 1127 | +3. The shared memory segment corresponding to the key does not exist (pair was created but `GenRaSocketBufKeyByPairId` failed; however, the pair record already exists) | ||
| 1128 | +4. Caller passed an incorrect fdHandle — pairId parsing error; the corresponding key was never created | ||
| 1129 | + | ||
| 1130 | +[Troubleshooting Steps] | ||
| 1131 | +# Check if /dev/shm has leftover ra_sock_* buffers | ||
| 1132 | +ls /dev/shm/ | grep ra_sock | ||
| 1133 | +# Check /dev/shm space | ||
| 1134 | +df -h /dev/shm | ||
| 1135 | +# Check RaSocketPair status and buf_status | ||
| 1136 | +hccl-vm table show RaSocketPair | ||
| 1137 | + | ||
| 1138 | +``` | ||
| 1139 | + | ||
| 1140 | +--- | ||
| 1141 | + | ||
| 1142 | +### Submodule: Networking | ||
| 1143 | + | ||
| 1144 | +--- | ||
| 1145 | + | ||
| 1146 | +#### FAQ-N001 | ||
| 1147 | + | ||
| 1148 | +**Title:** Ranktable Environment Variable Configuration Error | ||
| 1149 | + | ||
| 1150 | +**Error Code:** | ||
| 1151 | +``` | ||
| 1152 | +NA (1) | ||
| 1153 | +``` | ||
| 1154 | + | ||
| 1155 | +**Error Function:** | ||
| 1156 | +``` | ||
| 1157 | +param_check_v2.cc::RanktableRealPath | ||
| 1158 | +``` | ||
| 1159 | + | ||
| 1160 | +**Key Log:** | ||
| 1161 | +``` | ||
| 1162 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172019]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path | ||
| 1163 | + | ||
| 1164 | +[info][PID:172021][TID:172021][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172021][hrtGetDeviceRefresh]deviceLogicId[3] | ||
| 1165 | +[error][PID:172020][TID:172020][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172020]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path | ||
| 1166 | + | ||
| 1167 | +[info][PID:172018][TID:172018][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172018][hrtGetDeviceRefresh]deviceLogicId[0] | ||
| 1168 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base_v2.cc:294][172019][HcclCommInitClusterInfoV2]call trace: hcclRet -> 1 | ||
| 1169 | + | ||
| 1170 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base.cc:811] [172019][operator()]call trace: hcclRet -> 1 | ||
| 1171 | +``` | ||
| 1172 | + | ||
| 1173 | +**Symptom:** Running test cases fails to initialize the communication domain. | ||
| 1174 | + | ||
| 1175 | +**Troubleshooting Guide:** | ||
| 1176 | +``` | ||
| 1177 | +[Possible Causes] | ||
| 1178 | +The ranktable.json file path is configured incorrectly. Check the RANK_TABLE_FILE environment variable configuration. ranktable.json is generated by the tool at $HCCL_VM_INSTALL_DIR/data/ranktable.json. | ||
| 1179 | + | ||
| 1180 | +[Troubleshooting Steps] | ||
| 1181 | +echo $RANK_TABLE_FILE | ||
| 1182 | + | ||
| 1183 | +[Solution] | ||
| 1184 | +Confirm that the RANK_TABLE_FILE environment variable is correctly configured, pointing to the ranktable.json file path. | ||
| 1185 | +``` | ||
| 1186 | + | ||
| 1187 | +--- | ||
| 1188 | + | ||
| 1189 | +#### FAQ-N002 | ||
| 1190 | + | ||
| 1191 | +**Title:** topo.json Path Configuration Error | ||
| 1192 | + | ||
| 1193 | +**Error Code:** | ||
| 1194 | +``` | ||
| 1195 | +NA (1) | ||
| 1196 | +``` | ||
| 1197 | + | ||
| 1198 | +**Error Function:** | ||
| 1199 | +``` | ||
| 1200 | +communicator_impl.cc::GetTopoFilePath | ||
| 1201 | +``` | ||
| 1202 | + | ||
| 1203 | +**Key Log:** | ||
| 1204 | +``` | ||
| 1205 | +[error][PID:172635][TID:172635][log_stub.cc][DlogPrintStub] [HCCL_LOG][communicator_impl.cc:1339][172635][GetTopoFilePath] topo_file_path[/home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/topo.json] is not a valid real path | ||
| 1206 | +``` | ||
| 1207 | + | ||
| 1208 | +**Symptom:** Running test cases fails to initialize the communication domain. | ||
| 1209 | + | ||
| 1210 | +**Troubleshooting Guide:** | ||
| 1211 | +``` | ||
| 1212 | +[Possible Causes] | ||
| 1213 | +The topo.json file path is configured incorrectly in the /etc/hccl_rootinfo.json file. Check the topo_file_path field. topo.json is generated by the tool at $HCCL_VM_INSTALL_DIR/data/topo.json. | ||
| 1214 | + | ||
| 1215 | +[Troubleshooting Steps] | ||
| 1216 | +echo $TOPO_FILE_PATH | ||
| 1217 | + | ||
| 1218 | +[Solution] | ||
| 1219 | +Confirm that the TOPO_FILE_PATH environment variable is correctly configured, pointing to the topo.json file path. | ||
| 1220 | +``` | ||
| 1221 | + | ||
| 1222 | +--- | ||
| 1223 | + | ||
| 1224 | +#### FAQ-N003 | ||
| 1225 | + | ||
| 1226 | +**Title:** mock-comm Command Error | ||
| 1227 | + | ||
| 1228 | +**Error Code:** | ||
| 1229 | +``` | ||
| 1230 | +NA | ||
| 1231 | +``` | ||
| 1232 | + | ||
| 1233 | +**Error Function:** | ||
| 1234 | +``` | ||
| 1235 | +db_sim_runner_ops.cc::GetServerKeyById | ||
| 1236 | +``` | ||
| 1237 | + | ||
| 1238 | +**Key Log:** | ||
| 1239 | +``` | ||
| 1240 | +(hvm)$> hccl-vm mock-comm 144 | ||
| 1241 | +[error][PID:172799][TID:172875][db_sim_runner_ops.cc][GetServerKeyById] can not find server by id: 0, 2 | ||
| 1242 | +[error][PID:172799][TID:172875][topo_ascend_cluster_parser.cc][InitDynamicModelData] cannot find device by physical id 0 | ||
| 1243 | +[error][PID:172799][TID:172875][cmd_base_utils.cc][InitHvmCommEnv] [HVM] InitHvmCommEnv failed | ||
| 1244 | +[error][PID:172799][TID:172875][subcmd_mock_comm.cc][Execute] [HVM] Failed to initialize mock communication environment. Cleaning up environment. | ||
| 1245 | +``` | ||
| 1246 | + | ||
| 1247 | +**Symptom:** Before running test cases, configuring the communication domain via the mock-comm command fails. | ||
| 1248 | + | ||
| 1249 | +**Troubleshooting Guide:** | ||
| 1250 | +``` | ||
| 1251 | +[Possible Causes] | ||
| 1252 | +The communication domain 144 configured by the mock-comm command exceeds the cluster configuration started by the tool. For example, if the cluster started by the tool has only 2 servers per super node, but communication domain 144 indicates 4 servers under that super node. | ||
| 1253 | + | ||
| 1254 | +[Troubleshooting Steps] | ||
| 1255 | +Confirm the cluster configuration file used when starting the tool and the communication domain configuration file for the mock-comm command. | ||
| 1256 | + | ||
| 1257 | +[Solution] | ||
| 1258 | +Check the cluster configuration started by the tool to confirm the number of servers per super node. If you truly need to configure communication domain 144, ensure the tool is started with a larger cluster network configuration. | ||
| 1259 | +Ensure the communication domain configured by the mock-comm command does not exceed the cluster configuration started by the tool. | ||
| 1260 | +``` | ||
| 1261 | + | ||
| 1262 | +--- | ||
| 1263 | + | ||
| 1264 | +#### FAQ-N004 | ||
| 1265 | + | ||
| 1266 | +**Title:** EndPoint IP Lookup Failed | ||
| 1267 | + | ||
| 1268 | +**Error Code:** | ||
| 1269 | +``` | ||
| 1270 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 1271 | +``` | ||
| 1272 | + | ||
| 1273 | +**Error Function:** | ||
| 1274 | +``` | ||
| 1275 | +topo_ascend_cluster_parser.cc::AddLinkInfo() | ||
| 1276 | +``` | ||
| 1277 | + | ||
| 1278 | +**Key Log:** | ||
| 1279 | +``` | ||
| 1280 | +cannot find endPoint by ip <IP_ADDR> | ||
| 1281 | +``` | ||
| 1282 | + | ||
| 1283 | +**Symptom:** The IP address referenced in the network link configuration does not exist in the topology. | ||
| 1284 | + | ||
| 1285 | +--- | ||
| 1286 | + | ||
| 1287 | +#### FAQ-N005 | ||
| 1288 | + | ||
| 1289 | +**Title:** Superpod Index Out of Range | ||
| 1290 | + | ||
| 1291 | +**Error Code:** | ||
| 1292 | +``` | ||
| 1293 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 1294 | +``` | ||
| 1295 | + | ||
| 1296 | +**Error Function:** | ||
| 1297 | +``` | ||
| 1298 | +topo_ascend_cluster_parser.cc::InitDynamicModelData() | ||
| 1299 | +``` | ||
| 1300 | + | ||
| 1301 | +**Key Log:** | ||
| 1302 | +``` | ||
| 1303 | +[InitDynamicModelData] superpod index <N> out of range | ||
| 1304 | +``` | ||
| 1305 | + | ||
| 1306 | +**Symptom:** When parsing the ranktable to generate ranktable.json, the referenced superpod index exceeds the actual number of superpods in the cluster, causing initialization to fail. | ||
| 1307 | + | ||
| 1308 | +**Troubleshooting Guide:** | ||
| 1309 | +``` | ||
| 1310 | +[Possible Causes] | ||
| 1311 | +The number of superpods to which the devices in the ranktable belong exceeds the cluster network configuration started by the tool. For example, the cluster has only 1 superpod, but the ranktable references a 2nd superpod. | ||
| 1312 | + | ||
| 1313 | +[Troubleshooting Steps] | ||
| 1314 | +1. Check the cluster network configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of superpods. | ||
| 1315 | +2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to confirm whether the referenced superpod indices exceed the range. | ||
| 1316 | + | ||
| 1317 | +[Solution] | ||
| 1318 | +Ensure the number of superpods referenced by the communication domain configured via the mock-comm command does not exceed the cluster network configuration. If more superpods are needed, start the tool with a larger cluster network configuration. | ||
| 1319 | +``` | ||
| 1320 | + | ||
| 1321 | +--- | ||
| 1322 | + | ||
| 1323 | +#### FAQ-N006 | ||
| 1324 | + | ||
| 1325 | +**Title:** Server Index Out of Range | ||
| 1326 | + | ||
| 1327 | +**Error Code:** | ||
| 1328 | +``` | ||
| 1329 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 1330 | +``` | ||
| 1331 | + | ||
| 1332 | +**Error Function:** | ||
| 1333 | +``` | ||
| 1334 | +topo_ascend_cluster_parser.cc::InitDynamicModelData() | ||
| 1335 | +``` | ||
| 1336 | + | ||
| 1337 | +**Key Log:** | ||
| 1338 | +``` | ||
| 1339 | +[InitDynamicModelData] server index <N> out of range in superpod <M> | ||
| 1340 | +``` | ||
| 1341 | + | ||
| 1342 | +**Symptom:** When parsing the ranktable to generate ranktable.json, the referenced server index exceeds the actual number of servers within the superpod, causing initialization to fail. | ||
| 1343 | + | ||
| 1344 | +**Troubleshooting Guide:** | ||
| 1345 | +``` | ||
| 1346 | +[Possible Causes] | ||
| 1347 | +The number of servers under a superpod in the ranktable exceeds the number of servers for that superpod in the cluster network configuration. For example, the cluster network has 2 servers per superpod, but the ranktable references a 3rd server. | ||
| 1348 | + | ||
| 1349 | +[Troubleshooting Steps] | ||
| 1350 | +1. Check the cluster network configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of servers per superpod. | ||
| 1351 | +2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to confirm whether the referenced server indices exceed the range. | ||
| 1352 | + | ||
| 1353 | +[Solution] | ||
| 1354 | +Ensure the number of servers per superpod in the communication domain configured via the mock-comm command does not exceed the cluster network configuration. If more servers are needed, start the tool with a larger cluster network configuration. | ||
| 1355 | +``` | ||
| 1356 | + | ||
| 1357 | +--- | ||
| 1358 | + | ||
| 1359 | +#### FAQ-N007 | ||
| 1360 | + | ||
| 1361 | +**Title:** Device Lookup by Physical ID Failed | ||
| 1362 | + | ||
| 1363 | +**Error Code:** | ||
| 1364 | +``` | ||
| 1365 | +HCCL_SIM_E_NOT_FOUND (6) | ||
| 1366 | +``` | ||
| 1367 | + | ||
| 1368 | +**Error Function:** | ||
| 1369 | +``` | ||
| 1370 | +topo_ascend_cluster_parser.cc::InitDynamicModelData() | ||
| 1371 | +``` | ||
| 1372 | + | ||
| 1373 | +**Key Log:** | ||
| 1374 | +``` | ||
| 1375 | +[InitDynamicModelData] cannot find device by physical id <N> | ||
| 1376 | +``` | ||
| 1377 | + | ||
| 1378 | +**Symptom:** When parsing the ranktable, device lookup by physical device ID fails, typically occurring when configuring the communication domain via mock-comm. | ||
| 1379 | + | ||
| 1380 | +**Troubleshooting Guide:** | ||
| 1381 | +``` | ||
| 1382 | +[Possible Causes] | ||
| 1383 | +The physical device ID referenced in the communication domain configured by the mock-comm command exceeds the actual device range in the cluster network. For example, the cluster has only 2 devices (physical id 0 and 1), but the communication domain configuration references physical id 2. | ||
| 1384 | + | ||
| 1385 | +[Troubleshooting Steps] | ||
| 1386 | +1. Check the cluster network configuration (topo_meta/*.yaml) used when starting the tool to confirm the number of devices per server. | ||
| 1387 | +2. Check the ranktable configuration ($HCCL_VM_INSTALL_DIR/data/ranktable.json) to confirm whether the referenced device_id exceeds the range. | ||
| 1388 | + | ||
| 1389 | +[Solution] | ||
| 1390 | +Ensure the physical device IDs referenced in the communication domain configured via the mock-comm command do not exceed the device range in the cluster network configuration. If more devices are needed, start the tool with a larger cluster network configuration. | ||
| 1391 | +``` | ||
| 1392 | + | ||
| 1393 | +--- | ||
| 1394 | + | ||
| 1395 | +### Submodule: Database | ||
| 1396 | + | ||
| 1397 | +--- | ||
| 1398 | + | ||
| 1399 | +#### FAQ-DB001 | ||
| 1400 | + | ||
| 1401 | +**Title:** SQLite Database Connection Failed | ||
| 1402 | + | ||
| 1403 | +**Error Code:** | ||
| 1404 | +``` | ||
| 1405 | +HCCL_SIM_E_OPEN_FILE_FAILURE (10) | ||
| 1406 | +``` | ||
| 1407 | + | ||
| 1408 | +**Error Function:** | ||
| 1409 | +``` | ||
| 1410 | +db_hccl_db_sqlite.cc::Connect() | ||
| 1411 | +``` | ||
| 1412 | + | ||
| 1413 | +**Key Log:** | ||
| 1414 | +``` | ||
| 1415 | +[dbInit] Connect database failed | ||
| 1416 | +Connect database:<path> failed | ||
| 1417 | +``` | ||
| 1418 | + | ||
| 1419 | +**Symptom:** Unable to connect to the SQLite database file. | ||
| 1420 | + | ||
| 1421 | +**Troubleshooting Guide:** | ||
| 1422 | +``` | ||
| 1423 | +[Possible Causes] | ||
| 1424 | +1. Database file does not exist | ||
| 1425 | +2. Insufficient file permissions | ||
| 1426 | +3. File is locked by another process | ||
| 1427 | +``` | ||
| 1428 | + | ||
| 1429 | +--- | ||
| 1430 | + | ||
| 1431 | +#### FAQ-DB002 | ||
| 1432 | + | ||
| 1433 | +**Title:** Database Backup File Not Found | ||
| 1434 | + | ||
| 1435 | +**Error Code:** | ||
| 1436 | +``` | ||
| 1437 | +HCCL_SIM_E_OPEN_FILE_FAILURE (10) | ||
| 1438 | +``` | ||
| 1439 | + | ||
| 1440 | +**Error Function:** | ||
| 1441 | +``` | ||
| 1442 | +sim_loader.cc::BackupDatabase() | ||
| 1443 | +``` | ||
| 1444 | + | ||
| 1445 | +**Key Log:** | ||
| 1446 | +``` | ||
| 1447 | +[Loader] Backup database file not found: <dbPath> | ||
| 1448 | +``` | ||
| 1449 | + | ||
| 1450 | +**Symptom:** Loader cannot find the simulation database file. | ||
| 1451 | + | ||
| 1452 | +**Troubleshooting Guide:** | ||
| 1453 | +``` | ||
| 1454 | +[Possible Causes] | ||
| 1455 | +1. Simulation data file path configured incorrectly | ||
| 1456 | +2. Simulation data not yet generated | ||
| 1457 | +3. Insufficient file permissions | ||
| 1458 | + | ||
| 1459 | +[Troubleshooting Steps] | ||
| 1460 | +ls -la <dbPath> | ||
| 1461 | +``` | ||
| 1462 | + | ||
| 1463 | +--- | ||
| 1464 | + | ||
| 1465 | +#### FAQ-DB003 | ||
| 1466 | + | ||
| 1467 | +**Title:** SQLite Query Failed | ||
| 1468 | + | ||
| 1469 | +**Error Code:** | ||
| 1470 | +``` | ||
| 1471 | +HCCL_SIM_E_INTERNAL (4) | ||
| 1472 | +``` | ||
| 1473 | + | ||
| 1474 | +**Error Function:** | ||
| 1475 | +``` | ||
| 1476 | +db_hccl_db_sqlite.cc | ||
| 1477 | +``` | ||
| 1478 | + | ||
| 1479 | +**Key Log:** | ||
| 1480 | +``` | ||
| 1481 | +Prepare failed: <error> sql:<SQL> | ||
| 1482 | +Step failed: <error>, sql:<SQL> | ||
| 1483 | +``` | ||
| 1484 | + | ||
| 1485 | +**Symptom:** SQL query execution failed. | ||
| 1486 | + | ||
| 1487 | +**Troubleshooting Guide:** | ||
| 1488 | +``` | ||
| 1489 | +[Possible Causes] | ||
| 1490 | +1. Database table structure mismatch (version incompatibility) | ||
| 1491 | +2. Database file corrupted | ||
| 1492 | +3. Insufficient disk space | ||
| 1493 | +``` | ||
| 1494 | + | ||
| 1495 | +--- | ||
| 1496 | + | ||
| 1497 | +## Module: Plugins | ||
| 1498 | + | ||
| 1499 | +### Submodule: checker | ||
| 1500 | + | ||
| 1501 | +> The error codes (101-902) for the Checker plugin have been migrated to the [Checker Error Code FAQ](checker_faq_en.md). This document no longer repeats them. Only entries that do not belong to the V3 error code system are retained below. | ||
| 1502 | + | ||
| 1503 | +--- | ||
| 1504 | + | ||
| 1505 | +#### FAQ-C008 | ||
| 1506 | + | ||
| 1507 | +**Title:** Binary File Magic Number Mismatch | ||
| 1508 | + | ||
| 1509 | +**Error Function:** | ||
| 1510 | +``` | ||
| 1511 | +binary_data_operator.cc::FileHeaderRead() | ||
| 1512 | +``` | ||
| 1513 | + | ||
| 1514 | +**Key Log:** | ||
| 1515 | +``` | ||
| 1516 | +[FileHeaderRead] Unmatched magic number:0x<N>≠0x<M> | ||
| 1517 | +``` | ||
| 1518 | + | ||
| 1519 | +**Symptom:** When reading the simulation data file, the magic number in the file header does not match. | ||
| 1520 | + | ||
| 1521 | +**Troubleshooting Guide:** | ||
| 1522 | +``` | ||
| 1523 | +[Possible Causes] | ||
| 1524 | +1. Data file version is incompatible with the tool version | ||
| 1525 | +2. File is corrupted | ||
| 1526 | +``` | ||
| 1527 | + | ||
| 1528 | +--- | ||
| 1529 | + | ||
| 1530 | +## Appendix: Error Code Quick Reference Table | ||
| 1531 | + | ||
| 1532 | +| Error Code | Enum Value | Description | | ||
| 1533 | +|--------|--------|------| | ||
| 1534 | +| 0 | HCCL_SIM_SUCCESS | Success | | ||
| 1535 | +| 1 | HCCL_SIM_E_PARA | Parameter error | | ||
| 1536 | +| 2 | HCCL_SIM_E_PTR | Null pointer | | ||
| 1537 | +| 3 | HCCL_SIM_E_MEMORY | Memory error | | ||
| 1538 | +| 4 | HCCL_SIM_E_INTERNAL | Internal error | | ||
| 1539 | +| 5 | HCCL_SIM_E_NOT_SUPPORT | Unsupported feature | | ||
| 1540 | +| 6 | HCCL_SIM_E_NOT_FOUND | Resource not found | | ||
| 1541 | +| 8 | HCCL_SIM_E_SYSCALL | System call error | | ||
| 1542 | +| 9 | HCCL_SIM_E_TIMEOUT | Timeout | | ||
| 1543 | +| 10 | HCCL_SIM_E_OPEN_FILE_FAILURE | File open failure | | ||
| @@ -0,0 +1,767 @@ | |||
| 1 | +# HCCL Simulator Requirements Analysis | ||
| 2 | + | ||
| 3 | +## 1 Background | ||
| 4 | + | ||
| 5 | +### 1.1 Purpose | ||
| 6 | + | ||
| 7 | +Provide an **offline, extensible, high-determinism** testing framework for the collective communication module: | ||
| 8 | + | ||
| 9 | +- **New algorithm development**. | ||
| 10 | + Support design verification of custom communication operators (e.g., scatter variants) with algorithm visualization analysis. | ||
| 11 | +- **Base package quality assurance**. | ||
| 12 | + Enable offline fast regression of HCCL test cases through HAL interfaces such as Runtime/Driver/Net. | ||
| 13 | +- **Architecture verification (long-term)**. | ||
| 14 | + Verify physical compatibility between algorithms and hardware topologies (HCCS/RoCE). | ||
| 15 | + | ||
| 16 | +### 1.2 Scope | ||
| 17 | + | ||
| 18 | +Simulation depth by layer: | ||
| 19 | + | ||
| 20 | +| Layer | Capability | Phase | | ||
| 21 | +|-------|------------|-------| | ||
| 22 | +| L1 | Communication semantics verification | ✅ Current | | ||
| 23 | +| L2 | Logical function reproduction | ⚠️ Challenge | | ||
| 24 | +| L3 | Network performance modeling | ⏳ Long-term | | ||
| 25 | +| L4 | Fault injection diagnostics | ⏳ Long-term | | ||
| 26 | + | ||
| 27 | +#### 1.2.1 Usage Scenarios | ||
| 28 | + | ||
| 29 | +| Role | Purpose | | ||
| 30 | +|------|---------| | ||
| 31 | +| Open source contributor | Verify new operator/algorithm semantics | | ||
| 32 | +| Tester | Run HCCL API test cases | | ||
| 33 | +| Developer | Execute Runtime/Driver-level test cases | | ||
| 34 | + | ||
| 35 | +#### 1.2.2 Input Forms | ||
| 36 | + | ||
| 37 | +- **Configuration-driven** | ||
| 38 | + YAML defines hardware topology (Appendix B) | ||
| 39 | +- **API hijacking** | ||
| 40 | + Redirect calls such as `hcclInitComm`/`aclrtMalloc` to the simulation backend. | ||
| 41 | + | ||
| 42 | +#### 1.2.3 Output Forms | ||
| 43 | + | ||
| 44 | +| Type | Purpose | | ||
| 45 | +|------|---------| | ||
| 46 | +| Program logs | User standard output | | ||
| 47 | +| Verification results | External checker report | | ||
| 48 | +| State snapshots | Binary buffer files | | ||
| 49 | +| Visualization | hccl-insight rendering analysis | | ||
| 50 | + | ||
| 51 | +--- | ||
| 52 | + | ||
| 53 | +## 2 System Context (Omitted) | ||
| 54 | + | ||
| 55 | +--- | ||
| 56 | + | ||
| 57 | +## 3 Requirements Overview | ||
| 58 | + | ||
| 59 | +### 3.1 Core Architecture | ||
| 60 | + | ||
| 61 | +#### 3.1.1 Design Philosophy | ||
| 62 | + | ||
| 63 | +**Non-invasive runtime hijacking**: | ||
| 64 | + | ||
| 65 | +```mermaid | ||
| 66 | +graph LR | ||
| 67 | +A[User binary] -->|LD_PRELOAD| B(libhccl-proxy.so) | ||
| 68 | +B --> C[Simulation backend] | ||
| 69 | +C --> D[CPU execution] | ||
| 70 | +``` | ||
| 71 | + | ||
| 72 | +**Core advantages**: | ||
| 73 | + | ||
| 74 | +1. **High fidelity**: Real binary execution path. | ||
| 75 | +2. **Zero intrusion**: No modification to user code required. | ||
| 76 | +3. **Strong extensibility**: Supports any HCCL/ACL program. | ||
| 77 | + | ||
| 78 | +**Determinism guarantee**: | ||
| 79 | + | ||
| 80 | +- L1: Communication semantics check (no concurrent communication domains/operators) | ||
| 81 | +- L2: Global event serialization. | ||
| 82 | +- L3/L4: Fixed seed random source or unified clock. | ||
| 83 | + | ||
| 84 | +### 3.2 Typical Workflows | ||
| 85 | + | ||
| 86 | +#### 3.2.1 Open Source Contributor Custom Operator Verification Flow (LLT) | ||
| 87 | + | ||
| 88 | +1. **Environment setup** | ||
| 89 | + | ||
| 90 | + - Clone the `hccl_ops` operator repository. | ||
| 91 | + - Build and run the preset **scatter operator test case** according to the contribution guide. | ||
| 92 | + | ||
| 93 | +2. **Operator development** | ||
| 94 | + | ||
| 95 | + - Implement the **CustomAllreduce operator** by referencing the scatter operator. | ||
| 96 | + - Build the operator and verify successful execution. | ||
| 97 | + | ||
| 98 | +3. **Execute verification test case** | ||
| 99 | + Run the Checker test case in the code project (example logic): | ||
| 100 | + | ||
| 101 | + ```python | ||
| 102 | + model = SimWorld("./topologies/cloud_matrix.yaml") # Initialize simulation model | ||
| 103 | + | ||
| 104 | + # Single-thread loop to simulate each rank execution | ||
| 105 | + foreach rank in rankGraph: | ||
| 106 | + HcclInitComm(...) | ||
| 107 | + ret = HcclScatter(rankGraph, scatter, array, 'root:0') # Execute original operator | ||
| 108 | + | ||
| 109 | + # Task graph generation | ||
| 110 | + taskGraph = model.GetStatus('taskGraph') | ||
| 111 | + | ||
| 112 | + # Operator semantic verification | ||
| 113 | + checker = Validator<Scatter>().Build(taskGraph) | ||
| 114 | + EXPECT(checker.CheckSemantic(), 'success') # Verify result | ||
| 115 | + ``` | ||
| 116 | + | ||
| 117 | +4. **Verify the custom operator**. | ||
| 118 | + Replace `HcclScatter` with `CustomAllreduce` in the code above and re-execute the test case. | ||
| 119 | + | ||
| 120 | +#### 3.2.2 Developer Local Iteration Before Code Submission | ||
| 121 | + | ||
| 122 | +1. **Start the simulator**: Run the `hccl-vm` command as root with the topology specified. | ||
| 123 | + | ||
| 124 | + ```bash | ||
| 125 | + root%> hccl-vm --topology=atlas900 | ||
| 126 | + ``` | ||
| 127 | + | ||
| 128 | + - **System feedback**: `info: entered hccl-vm` _(simulator: created simulation model)_. | ||
| 129 | + - **System state**: Prompt changes to `hccl-vm%>` _(simulator: interactive shell started)_. | ||
| 130 | + | ||
| 131 | +2. **Execute communication program**: Run the `scatter.bin` program in the simulator shell (can use MPI/slurm, etc.). | ||
| 132 | + | ||
| 133 | + ```bash | ||
| 134 | + hccl-vm%> ./scatter.bin | ||
| 135 | + ``` | ||
| 136 | + | ||
| 137 | +3. **Repeat operations**: Execute individual communication domain initialization test cases in the simulator shell as needed. | ||
| 138 | + | ||
| 139 | + ```bash | ||
| 140 | + hccl-vm%> ./test_init_comm | ||
| 141 | + ``` | ||
| 142 | + | ||
| 143 | +4. **Exit the simulator**: Enter the `exit` command in the simulator shell. | ||
| 144 | + | ||
| 145 | + ```bash | ||
| 146 | + hccl-vm%> exit | ||
| 147 | + ``` | ||
| 148 | + | ||
| 149 | + - **System feedback**: `info: exit hccl-vm` _(simulator: cleanup and exit)_. | ||
| 150 | + - **System state**: Prompt returns to `root%>`. | ||
| 151 | + | ||
| 152 | +#### 3.2.3 Multi-Server Testing | ||
| 153 | + | ||
| 154 | +1. **Log in to Server1**: Start the simulator as root. | ||
| 155 | + | ||
| 156 | + - **System state**: Prompt changes to `hccl-vm%>` _(simulator: interactive shell started)_. | ||
| 157 | + | ||
| 158 | +2. **Execute communication program**: Run the `scatter.bin` program in the simulator shell. | ||
| 159 | + | ||
| 160 | + ```bash | ||
| 161 | + hccl-vm%> py3 allreduce.py --host 90.91.103.38 --data-sample large-b16 --serverid 0 --deviceid 8 --op sum | ||
| 162 | + hccl-vm%> py3 allreduce.py --host 90.91.103.38 --data-sample small-b16 --serverid 1 --deviceid 1 --op sum | ||
| 163 | + ``` | ||
| 164 | + | ||
| 165 | +3. **Log in to other servers**: Repeat steps 1-2 _(simulator: interactive shell started)_. | ||
| 166 | + | ||
| 167 | +4. **View program logs on any server command line**: | ||
| 168 | + | ||
| 169 | + ```bash | ||
| 170 | + hccl-vm%> ... | ||
| 171 | + hccl-vm%> ... | ||
| 172 | + ``` | ||
| 173 | + | ||
| 174 | +### 3.3 Conventions | ||
| 175 | + | ||
| 176 | +#### 3.3.1 Terminology | ||
| 177 | + | ||
| 178 | +- hccl-vm: The controller program delivered by the simulator (interactive shell + backend process) | ||
| 179 | +- libhccl-vm.so: The core library delivered by the simulator | ||
| 180 | +- libhccl-proxy.so: The hijacking library (LD_PRELOAD) | ||
| 181 | +- IPC: Communication between libhccl-proxy.so and the hccl-vm backend via shared memory to execute the payload (tasks) constructed after hijacking ACL/HCCL calls | ||
| 182 | + | ||
| 183 | +#### 3.3.2 Key Mechanisms | ||
| 184 | + | ||
| 185 | +- CLI: | ||
| 186 | + 1. hccl-vm --topology=`describe-file-path` | ||
| 187 | + 2. Enter the interactive shell and execute user commands. | ||
| 188 | +- Environment variable: `LD_PRELOAD` points to `libhccl-proxy.so`, set by hccl-vm and effective in child processes. | ||
| 189 | +- Process model: hccl-vm uses `fork+exec` to execute user commands; the OS loader preferentially loads `libhccl-proxy.so`. | ||
| 190 | + | ||
| 191 | +--- | ||
| 192 | + | ||
| 193 | +## 4 System Functional Requirements | ||
| 194 | + | ||
| 195 | +```mermaid | ||
| 196 | +graph LR | ||
| 197 | + | ||
| 198 | + FR1["Modeler<br>═══════════<br> | ||
| 199 | + Simple → Complex<br> | ||
| 200 | + Virtual Network/Device<br>"]:::filter | ||
| 201 | + FR2.1["Proxy<br>═══════════<br> | ||
| 202 | + Hijack: HCCL/ACL API<br> | ||
| 203 | + Forward: Task Submission"]:::filter | ||
| 204 | + FR2.2["Simulator<br>═══════════<br> | ||
| 205 | + Schedule: Generate Task Graph<br> | ||
| 206 | + Execute: Hardware Operation Simulation"]:::filter | ||
| 207 | + FR3["Validator<br>═══════════<br> | ||
| 208 | + Operator Evaluation<br> | ||
| 209 | + Visualization Dashboard"]:::filter | ||
| 210 | + FR4[("Persistence<br>")]:::filter | ||
| 211 | + FR5>"Controller<br>Interactive Shell"]:::filter | ||
| 212 | + | ||
| 213 | + %% ===== Payload ===== | ||
| 214 | + input((payload)):::data -->|HCCL/ACL API| FR2.1 | ||
| 215 | + | ||
| 216 | + %% ===== Data Flow ===== | ||
| 217 | + A((System Description)):::data --> FR1 | ||
| 218 | + FR1 --> B((Initial Model)):::data | ||
| 219 | + B --> FR2.1 | ||
| 220 | + FR2.1 --> C((Task Model)):::data | ||
| 221 | + C --> FR2.2 | ||
| 222 | + FR2.2 --> D((Intermediate Model)):::data | ||
| 223 | + D --> FR3 | ||
| 224 | + | ||
| 225 | + %% ===== Storage ===== | ||
| 226 | + FR4:::storageNode --> B | ||
| 227 | + FR4 --> C | ||
| 228 | + FR4 --> D | ||
| 229 | + | ||
| 230 | + %% ===== Control Flow ===== | ||
| 231 | + FR5:::controlNode -.-> FR1 | ||
| 232 | + FR5 -.-> FR2.1 | ||
| 233 | + FR5 -.-> FR2.2 | ||
| 234 | + FR5 -.-> FR3 | ||
| 235 | + FR5 -.-> FR4 | ||
| 236 | + | ||
| 237 | + %% ===== Styles ===== | ||
| 238 | + classDef data fill:#E1F5FE,stroke:#039BE5,stroke-width:2px,color:#01579B; | ||
| 239 | + classDef filter fill:#E8F5E9,stroke:#4CAF50,stroke-width:2px,color:#1B5E20; | ||
| 240 | + classDef storageNode fill:#FFF3E0,stroke:#FF9800,stroke-width:2px,color:#E65100; | ||
| 241 | + classDef controlNode fill:#F3E5F5,stroke:#9C27B0,stroke-width:2px,color:#4A148C; | ||
| 242 | +``` | ||
| 243 | + | ||
| 244 | +Delivery matrix by project phase: | ||
| 245 | + | ||
| 246 | +| Phase | Modeler | Proxy | Simulator | Validator | Persistence | Controller | | ||
| 247 | +|-------|---------|-------|-----------|-----------|-------------|------------| | ||
| 248 | +| L1 | A3 simplified | HCCL-AICPU | Task graph generation | Checker porting | - | - | | ||
| 249 | +| L2 | A2/5 full line | HCCL/ACL | Sequential execution | A5 plugin-based porting | ✅ | ✅ | | ||
| 250 | +| L2 Challenge | Distributed | Batch forwarding | Parallel computing | Visualization | High-concurrency IO library | Cluster management | | ||
| 251 | + | ||
| 252 | +- **L1 phase**: Deliver simulator core library. | ||
| 253 | + `libhccl-vm.so` (A3/AICPU modeling simulation + validator) + `libhccl-proxy.so` (simplified proxy) | ||
| 254 | + Users implement communication semantic verification through LLT (FR2.1→FR3) | ||
| 255 | +- **L2 phase**: Deliver standalone controller. | ||
| 256 | + `hccl-vm` (FR5) + full modeler (FR1) + persistence (FR4) | ||
| 257 | + Supports command-line startup of non-intrusive environment for logical function reproduction. | ||
| 258 | +- **L2 Challenge**: Run the test team's existing distributed real-hardware test cases directly. | ||
| 259 | + | ||
| 260 | +### 4.1 FR1 Construct Collective Communication Simulation Environment | ||
| 261 | + | ||
| 262 | +**Constraints**: | ||
| 263 | + | ||
| 264 | +- ✅ Only supports dynamic linking (acl/aclRt/hccl-base) | ||
| 265 | +- ❌ Does not support setuid/setgid programs. | ||
| 266 | + | ||
| 267 | +**Core capability**: | ||
| 268 | + | ||
| 269 | +> Create/query: Collective communication simulation model. | ||
| 270 | + | ||
| 271 | +```python | ||
| 272 | +# Example | ||
| 273 | +model = SimWorld("./topologies/cloud_matrix.yaml") | ||
| 274 | +device = model.GetStatus("device0") | ||
| 275 | +``` | ||
| 276 | + | ||
| 277 | +**Interaction flow**: | ||
| 278 | + | ||
| 279 | +| Step | Action | Result | | ||
| 280 | +|------|--------|--------| | ||
| 281 | +| 1 | Load YAML configuration | Generate initial model | | ||
| 282 | +| 2 | Call `SimWorld()` | Return model handle | | ||
| 283 | +| 3 | Query `GetStatus()` | Get specified state data | | ||
| 284 | + | ||
| 285 | +**Failure scenario**: Configuration file error, print parsing details. | ||
| 286 | + | ||
| 287 | +### 4.2 FR2 Simulate Execution of Collective Communication Operators | ||
| 288 | + | ||
| 289 | +```mermaid | ||
| 290 | +graph LR | ||
| 291 | + | ||
| 292 | + B(API Hijack) -->|"Produce Tasks<br>(Memory Operations)"| C[Simulator Backend] | ||
| 293 | + C -->|Subgraph Assembly| D[(Task DAG)] | ||
| 294 | + D -.->|Distributed Computation| E[(Operator Output)] | ||
| 295 | + C -->|Simulated Compute<br>L2| E | ||
| 296 | + D -->|<br>L1| G(Logs/State Snapshots) | ||
| 297 | + E -->G | ||
| 298 | +``` | ||
| 299 | + | ||
| 300 | +#### 4.2.1 Core Capabilities (Automatic in Background) | ||
| 301 | + | ||
| 302 | +| Phase | Function | User (Developer) Visible Effect | | ||
| 303 | +|-------|----------|---------------------------------| | ||
| 304 | +| **Hijack** | Dynamically redirect HCCL/ACL API | Modify link options for original LLT test cases | | ||
| 305 | +| **Forward** | Package communication task metadata | ~~Invisible~~ | | ||
| 306 | +| **Simulate** | Generate global DAG task graph | Obtainable from the modeling interface | | ||
| 307 | +| **Execute** | Simulate memory copy and state migration | Obtainable from the modeling interface | | ||
| 308 | + | ||
| 309 | +**Prerequisites**: | ||
| 310 | + | ||
| 311 | +1. Developer has already developed a scatter operator variant program `./my_scatter.bin`. | ||
| 312 | +2. `./my_scatter.bin` uses the `lib-vm.so` interface for simulation modeling. | ||
| 313 | + | ||
| 314 | +**Success scenario 1**: Link the proxy library with `-lhccl-proxy` and run. | ||
| 315 | + | ||
| 316 | +```bash | ||
| 317 | +# Link at compile time (illustrative) | ||
| 318 | +ld ./scatter.bin -lhccl-proxy -lhccl-vm | ||
| 319 | +# Command-line execution | ||
| 320 | +./scatter.bin | ||
| 321 | +``` | ||
| 322 | + | ||
| 323 | +**Success scenario 2**: Hijack the proxy library via LD_PRELOAD. | ||
| 324 | + | ||
| 325 | +```bash | ||
| 326 | +# Single-node execution (automatic hijack → simulation) | ||
| 327 | +LD_PRELOAD=libhccl-proxy.so ./scatter.bin | ||
| 328 | +``` | ||
| 329 | + | ||
| 330 | +**Feature evolution, same scenario**: | ||
| 331 | + | ||
| 332 | +| Phase | Capability | User Operation Change | | ||
| 333 | +|-------|------------|----------------------| | ||
| 334 | +| L1 | Single-process multi-rank simulation | ~~No change~~ | | ||
| 335 | +| L2 | Multi-process single-server simulation | ~~No change~~ | | ||
| 336 | + | ||
| 337 | +### 4.3 FR3 Semantic Verification of Operator Results in Simulation Environment | ||
| 338 | + | ||
| 339 | +#### 4.3.1 Core Capabilities | ||
| 340 | + | ||
| 341 | +1. **Operator semantic verification**. | ||
| 342 | + - Supports communication semantic verification for preset operators (AllReduce/Scatter, etc.). | ||
| 343 | + - Provides a plugin-based extension interface. | ||
| 344 | +2. **Visualization analysis** | ||
| 345 | + - Generate interactive communication topology diagrams. | ||
| 346 | + - Mark semantic violation points. | ||
| 347 | +3. **Verification report generation**. | ||
| 348 | + - Structured error diagnostics (data consistency/timing conflicts) | ||
| 349 | + | ||
| 350 | +### 4.4 Success Scenarios | ||
| 351 | + | ||
| 352 | +#### 4.4.1 Scenario 1: Basic Operator Semantic Verification | ||
| 353 | + | ||
| 354 | +```mermaid | ||
| 355 | +sequenceDiagram | ||
| 356 | + participant User | ||
| 357 | + participant Validator | ||
| 358 | + participant SimWorld | ||
| 359 | + | ||
| 360 | + User->>Validator: Build(taskGraph, "Scatter") | ||
| 361 | + Validator->>SimWorld: GetStatus("dag") | ||
| 362 | + SimWorld-->>Validator: Return global task graph | ||
| 363 | + Validator->>Validator: Execute Scatter semantic rule check | ||
| 364 | + alt Check passes | ||
| 365 | + Validator-->>User: Return true | ||
| 366 | + else Check fails | ||
| 367 | + Validator-->>User: Return false + error code SEM_VIOLATION | ||
| 368 | + end | ||
| 369 | +``` | ||
| 370 | + | ||
| 371 | +**User operation**: | ||
| 372 | + | ||
| 373 | +```python | ||
| 374 | +# Load task graph | ||
| 375 | +task_graph = model.GetStatus('taskGraph') | ||
| 376 | + | ||
| 377 | +# Create validator | ||
| 378 | +scatter_validator = Validator<Scatter>() | ||
| 379 | +scatter_validator.Build(task_graph) | ||
| 380 | + | ||
| 381 | +# Execute verification | ||
| 382 | +if scatter_validator.CheckSemantic(): | ||
| 383 | + print("Scatter semantic verification passed") | ||
| 384 | +``` | ||
| 385 | + | ||
| 386 | +#### 4.4.2 Scenario 2: Custom Operator Hot Plug | ||
| 387 | + | ||
| 388 | +```bash | ||
| 389 | +# In hccl-vm interactive environment | ||
| 390 | +hccl-vm%> validator install ./custom_allreduce_validator.so | ||
| 391 | +[System] Success: Validator 'CustomAllReduce' registered | ||
| 392 | + | ||
| 393 | +# Code call | ||
| 394 | +validator = Validator<CustomAllReduce>() | ||
| 395 | +validator.Build(task_graph) | ||
| 396 | +validator.CheckSemantic() | ||
| 397 | +``` | ||
| 398 | + | ||
| 399 | +#### 4.4.3 Scenario 3: Visualization Diagnostics | ||
| 400 | + | ||
| 401 | +```mermaid | ||
| 402 | +flowchart TB | ||
| 403 | + A[Check failed] --> B{Error type?} | ||
| 404 | + B -->|Data consistency| C[Generate data flow comparison chart] | ||
| 405 | + B -->|Timing conflict| D[Generate timeline Gantt chart] | ||
| 406 | + C & D --> E[hccl-insight rendering] | ||
| 407 | + E --> F[Interactive HTML report] | ||
| 408 | +``` | ||
| 409 | + | ||
| 410 | +**Output example**: | ||
| 411 | + | ||
| 412 | +| Error type | Node | Expected | Actual | | ||
| 413 | +|-----------|------|----------|--------| | ||
| 414 | +| Data inconsistency | Rank1 | 0x7f8e (32782) | 0x0000 (0) | | ||
| 415 | +| Deadlock risk | Rank2 | Waiting for Rank3 | Timeout (>200ms) | | ||
| 416 | + | ||
| 417 | +--- | ||
| 418 | + | ||
| 419 | +### 4.4 Failure Scenarios | ||
| 420 | + | ||
| 421 | +#### 4.4.1 Scenario 1: Validator Plugin Load Failure | ||
| 422 | + | ||
| 423 | +**Trigger conditions**: | ||
| 424 | + | ||
| 425 | +- Plugin ABI version incompatibility. | ||
| 426 | +- Plugin does not export the `Validator_CreateInstance` symbol. | ||
| 427 | + | ||
| 428 | +**System response**: | ||
| 429 | + | ||
| 430 | +```bash | ||
| 431 | +hccl-vm%> validator install ./broken_validator.so | ||
| 432 | +[ERROR] Plugin load failed: | ||
| 433 | + - ABI version mismatch (expected v3, got v2) | ||
| 434 | + - Symbol 'Validator_CreateInstance' not found | ||
| 435 | +[Suggestion] Use validator check-abi ./broken_validator.so to check compatibility | ||
| 436 | +``` | ||
| 437 | + | ||
| 438 | +#### 4.4.2 Scenario 2: Invalid Task Graph Input | ||
| 439 | + | ||
| 440 | +**Trigger condition**: | ||
| 441 | + | ||
| 442 | +```python | ||
| 443 | +# Pass non-DAG structure | ||
| 444 | +validator.Build("invalid_data") | ||
| 445 | +``` | ||
| 446 | + | ||
| 447 | +**System response**: | ||
| 448 | + | ||
| 449 | +```python | ||
| 450 | +Traceback (most recent call last): | ||
| 451 | + File "test.py", line 12, in <module> | ||
| 452 | + validator.Build("invalid_data") | ||
| 453 | +hccl.error.InvalidGraphError: | ||
| 454 | + Expected TaskGraph object, got <class 'str'> | ||
| 455 | +``` | ||
| 456 | + | ||
| 457 | +#### 4.4.3 Scenario 3: Runtime State Conflict | ||
| 458 | + | ||
| 459 | +**Trigger condition**: | ||
| 460 | + | ||
| 461 | +```python | ||
| 462 | +# Attempt verification during communication execution | ||
| 463 | +while HcclAllReduceInner(is_running=True): | ||
| 464 | + validator.CheckSemantic() # Illegal call! | ||
| 465 | +``` | ||
| 466 | + | ||
| 467 | +**System response**: | ||
| 468 | + | ||
| 469 | +```text | ||
| 470 | +[FATAL] Validator state conflict: | ||
| 471 | + - Operation not allowed during HcclAllReduceInner execution | ||
| 472 | + - Call GetStatus('idle') before validation | ||
| 473 | +``` | ||
| 474 | + | ||
| 475 | +--- | ||
| 476 | + | ||
| 477 | +### 4.5 Constraints and Evolution | ||
| 478 | + | ||
| 479 | +| Capability | L1 Phase | L2 Phase | | ||
| 480 | +|------------|----------|----------| | ||
| 481 | +| **Preset validators** | Scatter | Full HCCL operators | | ||
| 482 | +| **Plugin mechanism** | Static linking | Dynamic loading (.so) | | ||
| 483 | +| **Visualization** | Text report | Interactive topology + timeline | | ||
| 484 | +| **Error location precision** | Node level | Buffer byte offset | | ||
| 485 | + | ||
| 486 | +**Key evolution path**: | ||
| 487 | + | ||
| 488 | +1. Provide `validator-template` code generator (L2); | ||
| 489 | +2. Support distributed verification coordination (L2 challenge phase); | ||
| 490 | +3. Integrate memory access trace tracking (L3). | ||
| 491 | + | ||
| 492 | +### 4.6 FR4 Persistence | ||
| 493 | + | ||
| 494 | +#### 4.6.1 `L2` Story: Simulator provides a persistence interface so that developers can quickly locate issues after testing. | ||
| 495 | + | ||
| 496 | +### 4.7 FR5 Controller | ||
| 497 | + | ||
| 498 | +Considering ease of use, users typically use HCCL test programs in a terminal. Therefore, the controller is delivered as a command-line program. | ||
| 499 | + | ||
| 500 | +#### 4.7.1 Controller Conventions | ||
| 501 | + | ||
| 502 | +- Topology switching is not supported at runtime. | ||
| 503 | +- The following description uses `shell` to represent the command line. | ||
| 504 | + | ||
| 505 | +#### 4.7.2 `L2` Story: Users start the simulator via an interactive shell, which automatically creates a virtual communication cluster in the background. Therefore, users intuitively expect to repeatedly test/verify their HCCL programs in this interactive shell environment. | ||
| 506 | + | ||
| 507 | +##### Shell Start Prerequisites | ||
| 508 | + | ||
| 509 | +- During installation, the simulator places multiple system configuration files in the same directory (e.g., `./topologies`), such as `cloud_matrix.yaml`, `atlas900.yaml`. | ||
| 510 | +- System configuration files are read-only and not corrupted. | ||
| 511 | + | ||
| 512 | +##### Success Scenario: System Model Discovery | ||
| 513 | + | ||
| 514 | +1. User runs `hccl-vm --list-topologies` to view supported systems. | ||
| 515 | +2. The command prints a list of all built-in system models such as cloud_matrix, atlas900, etc., without starting the simulator. | ||
| 516 | + | ||
| 517 | +##### Success Scenario: Successfully Start Simulator and Hijack Application | ||
| 518 | + | ||
| 519 | +1. User starts the `hccl-vm` controller, specifies the topology configuration file, and enters the interactive shell. | ||
| 520 | +2. `hccl-vm` parses and loads the system configuration file, calls the modeler, creates the initial system model, and indicates successful creation. | ||
| 521 | +3. User executes `./scatter_perf` in the shell, for example: | ||
| 522 | +4. `hccl-vm` sets `LD_PRELOAD=libhccl-proxy.so` for the child process and executes `fork/exec`. | ||
| 523 | +5. The OS loader loads `libhccl-proxy.so` first. Application calls to CUDA/NCCL APIs are hijacked. | ||
| 524 | +6. `libhccl-proxy.so` delegates to the `hccl-vm` backend via IPC for simulation. | ||
| 525 | +7. The application completes and returns standard output/error to the user; the return code is passed through to the hccl-vm interactive shell. | ||
| 526 | +8. User can `exit` the shell, and sim_run exits with code 0. Example interaction: | ||
| 527 | + | ||
| 528 | + ```bash | ||
| 529 | + root%>hccl-vm --topology=atlas900 | ||
| 530 | + info: entered hccl-vm | ||
| 531 | + hccl-vm%> | ||
| 532 | + hccl-vm%>./scatter_perf | ||
| 533 | + hccl-vm%>exit | ||
| 534 | + info: exit hccl-vm | ||
| 535 | + root%> | ||
| 536 | + ``` | ||
| 537 | + | ||
| 538 | +##### Success Scenario: Start with Default System Model | ||
| 539 | + | ||
| 540 | +1. User starts the simulator with `hccl-vm` without any parameters. | ||
| 541 | +2. `hccl-vm` finds and locates the `cloud_matrix.yaml` file in its preset directory. | ||
| 542 | +3. Subsequent flow is identical to `Success Scenario: Successfully Start Simulator and Hijack Application`. | ||
| 543 | + | ||
| 544 | +##### Failure Scenario: Specified System Configuration Does Not Exist | ||
| 545 | + | ||
| 546 | +1. User executes `hccl-vm --topology=non_existent_topo`. | ||
| 547 | +2. `hccl-vm` outputs an error message and terminates normally, e.g., "Error: System 'non_existent_topo' not found. Available systems: cloud_matrix, atlas900". | ||
| 548 | + | ||
| 549 | +##### Failure Scenario: Topology Configuration File Format Error | ||
| 550 | + | ||
| 551 | +1. The user-provided topology configuration file does not conform to YAML specifications or lacks required fields, causing `hccl-vm` to fail during initialization parsing. | ||
| 552 | +2. The simulator outputs a clear error message to stderr, such as "Error: Failed to parse topology file a.yaml, reason: ...", and immediately terminates the program. | ||
| 553 | + | ||
| 554 | +##### Failure Scenario: Topology Configuration File Does Not Exist or Is Unreadable | ||
| 555 | + | ||
| 556 | +#### 4.7.3 `L2` Story: Users can quickly configure a system model description file and start the simulator. Format reference **Appendix B**. | ||
| 557 | + | ||
| 558 | +#### 4.7.4 `L2` Story: Users manage validator plugins through `shell` subcommands. | ||
| 559 | + | ||
| 560 | +##### Success Scenario: User installs custom validator plugin and successfully completes verification | ||
| 561 | + | ||
| 562 | +##### Success Scenario: User views available validator plugins | ||
| 563 | + | ||
| 564 | +##### Failure Scenario: Validator plugin and simulator version mismatch | ||
| 565 | + | ||
| 566 | +##### Failure Scenario: Multiple validator plugin conflicts during installation | ||
| 567 | + | ||
| 568 | +#### 4.7.5 `L2` Story: Developers can export the state of the simulation environment (e.g., specific device memory buffer contents) to a file via controller subcommands for subsequent analysis or visualization. | ||
| 569 | + | ||
| 570 | +##### Subcommand Constraints | ||
| 571 | + | ||
| 572 | +- The test program is running in the `hccl-vm` interactive shell. | ||
| 573 | +- The export operation is triggered synchronously at any time during the test program. | ||
| 574 | +- The export is an instantaneous snapshot of the `hccl-vm` current system model. | ||
| 575 | + | ||
| 576 | +##### Success Scenario: Successfully Export Simulation Device State Buffer | ||
| 577 | + | ||
| 578 | +1. The user executes the export subcommand in the interactive shell. | ||
| 579 | + Example: `hccl-vm%> snapshot --path=/tmp/hccl_vm_state` | ||
| 580 | +2. `hccl-vm` calls the persistence interface and writes the system model state to the `hccl_vm_state` file. | ||
| 581 | +3. After writing, `hccl-vm` prints a success message to stdout, e.g., "Snapshot exported successfully: /tmp/global_sim_state.bin". | ||
| 582 | +4. The user can find the `hccl_vm_state` file under `/tmp/` and view the simulation device's memory at that moment. | ||
| 583 | + | ||
| 584 | +##### Success Scenario: Overwrite Existing File | ||
| 585 | + | ||
| 586 | +1. User executes the snapshot command specifying an existing output file path. | ||
| 587 | +2. `hccl-vm` prompts that the file exists and asks whether to overwrite. | ||
| 588 | +3. User confirms overwrite. | ||
| 589 | +4. `hccl-vm` overwrites the file and prints a success message. | ||
| 590 | + | ||
| 591 | +##### Failure Scenario: Insufficient Disk Space | ||
| 592 | + | ||
| 593 | +1. `hccl-vm` outputs a clear error message to stderr, such as "Error: Insufficient disk space to write snapshot file: ...", and terminates the write. | ||
| 594 | + | ||
| 595 | +##### Failure Scenario: No Write Permission or Directory Does Not Exist | ||
| 596 | + | ||
| 597 | +1. `hccl-vm` outputs a clear error message to stderr, such as "Error: No write permission or directory does not exist: ...", and terminates the write. | ||
| 598 | + | ||
| 599 | +#### 4.7.6 `L2` Story: In the interactive shell, users can visually view algorithm results via the visualization subcommand after activating the validator plugin. | ||
| 600 | + | ||
| 601 | +##### Prerequisites | ||
| 602 | + | ||
| 603 | +- `hccl-vm` has the `scatter` operator validator plugin installed. | ||
| 604 | +- `hccl-vm` integrates the `hccl-insight` command-line visualization tool. | ||
| 605 | +- After a simulator run, one or more binary algorithm verification result files (e.g., `rank0_output.bin`) have been successfully exported via the persistence API. | ||
| 606 | +- The snapshot file `rank0_output.bin` contains an 8x8 float32 matrix. | ||
| 607 | + | ||
| 608 | +##### Success Scenario: Render Single-Card Memory Data as a Heatmap | ||
| 609 | + | ||
| 610 | +1. The user runs the visualization tool in the interactive shell for the exported snapshot file. The user needs to provide metadata (data type and shape) for the tool to parse correctly, e.g.: | ||
| 611 | + | ||
| 612 | + ```bash | ||
| 613 | + hccl-insight --file=rank0_output.bin --dtype=float32 --shape=8x8 | ||
| 614 | + ``` | ||
| 615 | + | ||
| 616 | +2. `hccl-insight` (Go program) starts and parses the command-line arguments. | ||
| 617 | +3. The program reads the binary contents of `rank0_output.bin`. | ||
| 618 | +4. Based on the `--dtype=float32` and `--shape=8x8` parameters, the program parses the binary stream into a two-dimensional array. | ||
| 619 | +5. The program starts a built-in web server on a random available local port (e.g., `:9527`). | ||
| 620 | +6. The program prints a message to stdout: "Visualization service started, open <http://localhost:9527> in your browser". | ||
| 621 | +7. (Optional) The program automatically calls a system command to open the URL in the user's default browser. | ||
| 622 | +8. The user sees a page rendered by Vue/JS in the browser, displaying an 8x8 heatmap where each cell's color represents its corresponding value. The user can hover over cells to see precise values. | ||
| 623 | + | ||
| 624 | +##### Failure Scenario: Snapshot File Does Not Exist | ||
| 625 | + | ||
| 626 | +1. The `--file` parameter points to a non-existent file. | ||
| 627 | +2. `hccl-insight` fails when trying to open the file. | ||
| 628 | +3. The tool outputs a clear error message to stderr, such as "Error: File 'xxx.bin' not found", and exits with a non-zero status code. | ||
| 629 | + | ||
| 630 | +##### Failure Scenario: File Size Does Not Match Metadata | ||
| 631 | + | ||
| 632 | +1. User specifies `--shape=8x8` and `--dtype=float32` (requires 8 _8_ 4 = 256 bytes), but the actual file `rank0_output.bin` is only 100 bytes. | ||
| 633 | +2. After reading the file, `hccl-insight` finds that the file size does not match the expected size calculated from the metadata. | ||
| 634 | +3. The tool outputs a clear error message to stderr, such as "Error: Data format mismatch. Based on shape(8x8) and dtype(float32), 256 bytes are expected, but the file is only 100 bytes", and exits with a non-zero status code. | ||
| 635 | + | ||
| 636 | +##### Failure Scenario: Missing Required Metadata Parameters | ||
| 637 | + | ||
| 638 | +1. User runs `hccl-insight --file=rank0_output.bin` but does not provide `--dtype` or `--shape`. | ||
| 639 | +2. The command-line parsing library finds missing required parameters. | ||
| 640 | +3. The tool outputs usage help information to stderr, indicating that these parameters are required, and exits with a non-zero status code. | ||
| 641 | + | ||
| 642 | +#### 4.7.7 `L2` Story: Users can switch between L1/L2 proxies to select simulators with different speeds. | ||
| 643 | + | ||
| 644 | +- `hccl-vm` integrates multiple versions of the simulator core library, e.g.: | ||
| 645 | + - `libhccl-proxy-l1.so`: Only performs NCCL communication operator semantic layer simulation. | ||
| 646 | + - `libhccl-proxy-l2.so`: Runtime/Driver/Net layer logical function simulation (default) | ||
| 647 | +- `hccl-vm` integrates validator plugins. | ||
| 648 | +- Start `hccl-vm` and enter the interactive shell. | ||
| 649 | + | ||
| 650 | +##### Success Scenario (Default): Controller uses L2 proxy to hijack the user program at the HAL layer, attempts to simulate memory copy operations, and writes the algorithm result to the simulation device buffer. | ||
| 651 | + | ||
| 652 | +##### Success Scenario (Full Function): Controller uses L2 proxy, simulates algorithm low-level operations while calling validator plugins, outputting verification results and visualization files. | ||
| 653 | + | ||
| 654 | +##### Success Scenario (Lightest): Controller uses L1 proxy without memory copy. User manually views the algorithm's scheduled task graph via the persistence command. | ||
| 655 | + | ||
| 656 | +##### Success Scenario (Typical for Operator Research): Controller uses L1 proxy without memory copy, preset validator plugins, outputting verification results. | ||
| 657 | + | ||
| 658 | +1. Specify simulation depth through the proxy control subcommand. | ||
| 659 | +2. Based on the subcommand parameter `L1`, `hccl-vm` determines the core library to load is `libhccl-proxy-l1.so`. | ||
| 660 | +3. `hccl-vm` sets the `LD_PRELOAD` environment variable to the full path of `libhccl-proxy-l1.so`. | ||
| 661 | +4. After the test program starts, it is hijacked by `libhccl-proxy-l1.so`. When `ncclMemcpyWrite` is executed, only the task semantics are inferred through the validator, without simulating the full data movement operation. | ||
| 662 | +5. The program completes and outputs the semantic verification result. The total time is significantly faster than using `L2`. | ||
| 663 | + | ||
| 664 | + The complete command sequence may be: | ||
| 665 | + | ||
| 666 | + ```bash | ||
| 667 | + hccl-vm%>validator install scatter | ||
| 668 | + hccl-vm%>proxy -l1 | ||
| 669 | + hccl-vm%>./scatter_perf.bin | ||
| 670 | + info[validator]: scatter checking finished, result is ... | ||
| 671 | + ``` | ||
| 672 | + | ||
| 673 | +##### Failure Scenario: Specified Non-Existent Subcommand or Parameter | ||
| 674 | + | ||
| 675 | +### 4.8 Distributed Controller (NFR) | ||
| 676 | + | ||
| 677 | +For on-board test programs running on real hardware, the user needs to start the same test program on multiple servers manually or via scripts. To run the simulator, a copy of the simulator controller must be prepared on each server running the test program. Since the simulator needs to uniformly model the entire network topology and hardware environment, a distributed design for the controller is required. | ||
| 678 | + | ||
| 679 | +#### 4.8.1 `L2` Story: Users can use cluster management software such as k8s to collaboratively run real HCCL test cases on multiple servers. | ||
| 680 | + | ||
| 681 | +## 5 Appendices | ||
| 682 | + | ||
| 683 | +### 5.1 Appendix A: API Support Plan | ||
| 684 | + | ||
| 685 | +Convention: Implement the behavior of the following listed APIs offline on the host. For API calls not listed here, the simulator's default behavior is to print a warning message and return `0`, without performing any actual operation. | ||
| 686 | + | ||
| 687 | +#### 5.1.1 L1 Phase | ||
| 688 | + | ||
| 689 | +- HCCL communication domain management (25 APIs) | ||
| 690 | +- HCCL control plane programming (23 APIs) | ||
| 691 | +- HCCL AICPU programming (8 APIs) | ||
| 692 | + | ||
| 693 | +#### 5.1.2 L2 Phase | ||
| 694 | + | ||
| 695 | +### 5.2 Appendix B: Topology Configuration File Format (Schema) | ||
| 696 | + | ||
| 697 | +The topology configuration file uses YAML format to describe the simulated hardware environment. The file must contain the following fields: | ||
| 698 | + | ||
| 699 | +- `npus` (integer, required): Total number of NPUs. | ||
| 700 | +- `links` (list, required): A list describing point-to-point physical connections. | ||
| 701 | + | ||
| 702 | +Each `link` object contains: | ||
| 703 | + | ||
| 704 | +- `peer` (list of 2 integers, required): Describes the IDs of two interconnected NPUs. | ||
| 705 | +- `type` (string, optional): Connection type, such as "HCCS", "PCIe". In the current L1 phase, this field is for reference only and does not affect logic. | ||
| 706 | + | ||
| 707 | +**Example: `my_topo.yaml`** | ||
| 708 | + | ||
| 709 | +```yaml | ||
| 710 | +## Describes a 4-card ring topology | ||
| 711 | +npus: 4 | ||
| 712 | +links: | ||
| 713 | + - peer: [0, 1] | ||
| 714 | + type: "HCCS" | ||
| 715 | + - peer: [1, 2] | ||
| 716 | + type: "HCCS" | ||
| 717 | + - peer: [2, 3] | ||
| 718 | + type: "HCCS" | ||
| 719 | + - peer: [3, 0] | ||
| 720 | + type: "HCCS" | ||
| 721 | +``` | ||
| 722 | + | ||
| 723 | +### 5.3 Appendix C: Validator Plugin Interface (API) Definition | ||
| 724 | + | ||
| 725 | +To create a valid validator plugin, users need to implement and export one or more callback functions conforming to the following specification. The simulator uses `dlsym` to find these symbols. | ||
| 726 | + | ||
| 727 | +#### 5.3.1 Data Structures | ||
| 728 | + | ||
| 729 | +```c++ | ||
| 730 | +// sim_validator_api.h | ||
| 731 | + | ||
| 732 | +// Describes HCCL data types, consistent with real HCCL | ||
| 733 | +typedef enum { hcclInt8 = 0, ..., hcclFloat64 = 7 } SimHcclDataType_t; | ||
| 734 | + | ||
| 735 | +// Context information passed to the callback function | ||
| 736 | +struct SimCommContext { | ||
| 737 | + int world_size; // Number of ranks in the communication domain | ||
| 738 | + void** rank_output_buffers; // Pointer array, containing the address of each rank's output buffer in simulated memory | ||
| 739 | + size_t element_count; // Number of elements in the buffer | ||
| 740 | + SimHcclDataType_t datatype; // Data type | ||
| 741 | +}; | ||
| 742 | + | ||
| 743 | +// Return value of the plugin | ||
| 744 | +struct SimValidationResult { | ||
| 745 | + bool success; // Whether the check passed | ||
| 746 | + char error_message[256]; // Error message if failed | ||
| 747 | +}; | ||
| 748 | +``` | ||
| 749 | + | ||
| 750 | +##### 5.3.2 Callback Function Signature | ||
| 751 | + | ||
| 752 | +The plugin must implement C functions with the following naming format as needed: | ||
| 753 | +`SimValidationResult post_<hccl_function_name>_hook(const SimCommContext* context);` | ||
| 754 | + | ||
| 755 | +##### 5.3.3 Example: AllReduce Validation Function | ||
| 756 | + | ||
| 757 | +```c++ | ||
| 758 | +// Function to implement in allreduce_validator.so | ||
| 759 | +extern "C" SimValidationResult post_HcclAllReduceInner_hook(const SimCommContext* context) { | ||
| 760 | + // 1. Determine data type based on context->datatype | ||
| 761 | + // 2. Allocate memory on CPU and copy contents of all rank_output_buffers | ||
| 762 | + // 3. Implement AllReduce mathematical logic verification (e.g., check if all ranks' outputs are consistent) | ||
| 763 | + // 4. Return SimValidationResult | ||
| 764 | +} | ||
| 765 | +``` | ||
| 766 | + | ||
| 767 | +### 5.4 Appendix D: System Model | ||
| @@ -0,0 +1,746 @@ | |||
| 1 | +# HCCL VM Binary Stream File Format Specification | ||
| 2 | + | ||
| 3 | +## Document Information | ||
| 4 | + | ||
| 5 | +| Item | Content | | ||
| 6 | +|------|---------| | ||
| 7 | +| Version | v1.0 | | ||
| 8 | +| Update Date | 2025-01-16 | | ||
| 9 | +| Applicable Scenario | HCCL VM simulation data export and reading | | ||
| 10 | +| Author | HCCL VM Team | | ||
| 11 | + | ||
| 12 | +--- | ||
| 13 | + | ||
| 14 | +## 1. Overview | ||
| 15 | + | ||
| 16 | +The `DumpData` interface is used to export HCCL VM simulation runtime data as binary stream files, generating **3 types of files**: | ||
| 17 | + | ||
| 18 | +| No. | File Name Format | Magic Number | Purpose | | ||
| 19 | +|:---:|------------------|--------------|---------| | ||
| 20 | +| 1 | `{dataId}_hcclvm_syn_data.bin` | `0x48564D44` | Synthesis data (model, channel, memory layout) | | ||
| 21 | +| 2 | `{dataId}_hcclvm_instr_data.bin` | `0x434D4349` | Microcode instruction data | | ||
| 22 | +| 3 | `{dataId}_hcclvm_task_data.bin` | `0x48565444` | Task metadata | | ||
| 23 | + | ||
| 24 | +> **Note**: `{dataId}` is a unique identifier in timestamp format, e.g., `20250115_143052_3847`. | ||
| 25 | + | ||
| 26 | +--- | ||
| 27 | + | ||
| 28 | +## 2. Common File Header | ||
| 29 | + | ||
| 30 | +All binary files start with a **20-byte** file header. | ||
| 31 | + | ||
| 32 | +### 2.1 File Header Structure | ||
| 33 | + | ||
| 34 | +| Offset | Bytes | Type | Field | Description | | ||
| 35 | +|:------:|:-----:|------|-------|-------------| | ||
| 36 | +| 0 | 4 | uint32 | magic | Magic number, identifies file type | | ||
| 37 | +| 4 | 2 | uint16 | version | Version number (currently 1) | | ||
| 38 | +| 6 | 2 | uint16 | header_size | File header size (20 bytes) | | ||
| 39 | +| 8 | 4 | uint32 | flags | Flags (reserved) | | ||
| 40 | +| 12 | 4 | uint32 | count | Number of data entries | | ||
| 41 | +| 16 | 4 | uint32 | checksum | Checksum (reserved) | | ||
| 42 | + | ||
| 43 | +### 2.2 C/C++ Structure Definition | ||
| 44 | + | ||
| 45 | +```cpp | ||
| 46 | +#pragma pack(push, 1) | ||
| 47 | +struct FileHeader { | ||
| 48 | + uint32_t magic; // Magic number | ||
| 49 | + uint16_t version; // Version number | ||
| 50 | + uint16_t header_size; // File header size | ||
| 51 | + uint32_t flags; // Flags | ||
| 52 | + uint32_t count; // Number of data entries | ||
| 53 | + uint32_t checksum; // Checksum | ||
| 54 | +}; | ||
| 55 | +#pragma pack(pop) | ||
| 56 | +``` | ||
| 57 | + | ||
| 58 | +--- | ||
| 59 | + | ||
| 60 | +## 3. Synthesis Data File (`*_hcclvm_syn_data.bin`) | ||
| 61 | + | ||
| 62 | +### 3.1 Overall Structure | ||
| 63 | + | ||
| 64 | +| No. | Data Block | Size | Description | | ||
| 65 | +|:---:|------------|------|-------------| | ||
| 66 | +| 1 | FileHeader | 20 Bytes | File header | | ||
| 67 | +| 2 | ModelInfo | Variable | Model information (includes ModelInfoCommInner, VDataDesTag, All2AllDataDesTag) | | ||
| 68 | +| 3 | ChannelInfo or JettyInfo | Variable | One of two, determined by `op_expansion_mode` | | ||
| 69 | +| 4 | MemLayoutInfo | Variable | Memory layout information | | ||
| 70 | + | ||
| 71 | +**`op_expansion_mode` Values:** | ||
| 72 | + | ||
| 73 | +| Value | Mode | Data Structure Used | | ||
| 74 | +|:----:|------|--------------------| | ||
| 75 | +| 0 | CCU | ChannelInfo | | ||
| 76 | +| 1 | AICPU | JettyInfo | | ||
| 77 | + | ||
| 78 | +### 3.2 ModelInfo Structure | ||
| 79 | + | ||
| 80 | +#### 3.2.1 ModelInfoCommInner | ||
| 81 | + | ||
| 82 | +**Size**: 36 bytes. | ||
| 83 | + | ||
| 84 | +| Offset | Bytes | Type | Field | Description | | ||
| 85 | +|:------:|:-----:|------|-------|-------------| | ||
| 86 | +| 0 | 4 | uint32 | src_rank | Source Rank ID | | ||
| 87 | +| 4 | 4 | uint32 | dst_rank | Destination Rank ID | | ||
| 88 | +| 8 | 4 | uint32 | root | Root rank | | ||
| 89 | +| 12 | 4 | uint32 | rank_size | Total number of ranks | | ||
| 90 | +| 16 | 2 | uint16 | chip_type | Chip type | | ||
| 91 | +| 18 | 2 | uint16 | op_type | Operation type | | ||
| 92 | +| 20 | 2 | uint16 | reduce_op | Reduce operation type | | ||
| 93 | +| 22 | 2 | uint16 | data_type | Data type | | ||
| 94 | +| 24 | 8 | uint64 | data_count | Number of data elements | | ||
| 95 | +| 32 | 4 | uint32 | op_expansion_mode | Expansion mode (see enum) | | ||
| 96 | +| 36 | 8 | uint64 | ccu0_resource_base_addr | die0 CCU resource base address | | ||
| 97 | +| 44 | 8 | uint64 | ccu1_resource_base_addr | die1 CCU resource base address | | ||
| 98 | + | ||
| 99 | +#### 3.2.2 VDataDesTag | ||
| 100 | + | ||
| 101 | +**Purpose**: ReduceScatterV / AllGatherV operations. | ||
| 102 | + | ||
| 103 | +**Size**: Variable. | ||
| 104 | + | ||
| 105 | +| Offset | Bytes | Type | Field | Description | | ||
| 106 | +|:------:|:-----:|------|-------|-------------| | ||
| 107 | +| 0 | 2 | uint16 | dataType | Data type | | ||
| 108 | +| 2 | 4 | uint32 | count | Number of ranks | | ||
| 109 | +| 6 | 8 × count | uint64[] | displs | Data offset for each rank | | ||
| 110 | +| 6+8×n | 8 × count | uint64[] | counts | Data size for each rank | | ||
| 111 | + | ||
| 112 | +#### 3.2.3 All2AllDataDesTag | ||
| 113 | + | ||
| 114 | +**Purpose**: All2All / All2AllV operations. | ||
| 115 | + | ||
| 116 | +**Size**: Variable. | ||
| 117 | + | ||
| 118 | +| Offset | Bytes | Type | Field | Description | | ||
| 119 | +|:------:|:-----:|------|-------|-------------| | ||
| 120 | +| 0 | 2 | uint16 | sendType | Send data type | | ||
| 121 | +| 2 | 2 | uint16 | recvType | Receive data type | | ||
| 122 | +| 4 | 8 | uint64 | sendCount | Send data count | | ||
| 123 | +| 12 | 8 | uint64 | recvCount | Receive data count | | ||
| 124 | +| 20 | 4 | uint32 | count | Matrix size (= rankSize²) | | ||
| 125 | +| 24 | 8 × count | uint64[] | sendCountMatrix | Send matrix | | ||
| 126 | + | ||
| 127 | +### 3.3 ChannelInfo Structure | ||
| 128 | + | ||
| 129 | +**Applicable**: `op_expansion_mode = 0` (CCU mode) | ||
| 130 | + | ||
| 131 | +#### 3.3.1 ChannelInfo Header | ||
| 132 | + | ||
| 133 | +| Offset | Bytes | Type | Field | Description | | ||
| 134 | +|:------:|:-----:|------|-------|-------------| | ||
| 135 | +| 0 | 4 | uint32 | count | Number of channels | | ||
| 136 | +| 4 | Variable | ChannelData[] | data | Channel data array | | ||
| 137 | + | ||
| 138 | +#### 3.3.2 ChannelData | ||
| 139 | + | ||
| 140 | +**Size**: 152 bytes. | ||
| 141 | + | ||
| 142 | +| Offset | Bytes | Type | Field | Description | | ||
| 143 | +|:------:|:-----:|------|-------|-------------| | ||
| 144 | +| 0 | 2 | uint16 | channelId | Channel ID | | ||
| 145 | +| 2 | 1 | uint8 | srcDieId | Source Die ID | | ||
| 146 | +| 3 | 1 | uint8 | dstDieId | Destination Die ID | | ||
| 147 | +| 4 | 4 | uint32 | srcRank | Source Rank ID | | ||
| 148 | +| 8 | 4 | uint32 | dstRank | Destination Rank ID | | ||
| 149 | +| 12 | 16 | uint8[16] | leid | Local EID | | ||
| 150 | +| 28 | 16 | uint8[16] | reid | Remote EID | | ||
| 151 | +| 44 | 2 | uint16 | protocol | Protocol type | | ||
| 152 | +| 46 | 2 | uint16 | jettyNum | Number of jetties | | ||
| 153 | +| 48 | 128 | uint32[32] | jettyId | Jetty ID array | | ||
| 154 | + | ||
| 155 | +### 3.4 JettyInfo Structure | ||
| 156 | + | ||
| 157 | +**Applicable**: `op_expansion_mode = 1` (AICPU mode) | ||
| 158 | + | ||
| 159 | +#### 3.4.1 JettyInfo Header | ||
| 160 | + | ||
| 161 | +| Offset | Bytes | Type | Field | Description | | ||
| 162 | +|:------:|:-----:|------|-------|-------------| | ||
| 163 | +| 0 | 4 | uint32 | count | Number of jetties | | ||
| 164 | +| 4 | Variable | JettyData[] | data | Jetty data array | | ||
| 165 | + | ||
| 166 | +#### 3.4.2 JettyData | ||
| 167 | + | ||
| 168 | +**Size**: 48 bytes. | ||
| 169 | + | ||
| 170 | +| Offset | Bytes | Type | Field | Description | | ||
| 171 | +|:------:|:-----:|------|-------|-------------| | ||
| 172 | +| 0 | 4 | uint32 | jettyId | Jetty ID | | ||
| 173 | +| 4 | 1 | uint8 | srcDieId | Source Die ID | | ||
| 174 | +| 5 | 1 | uint8 | dstDieId | Destination Die ID | | ||
| 175 | +| 6 | 2 | uint16 | protocol | Protocol type | | ||
| 176 | +| 8 | 4 | uint32 | srcRank | Source Rank ID | | ||
| 177 | +| 12 | 4 | uint32 | dstRank | Destination Rank ID | | ||
| 178 | +| 16 | 16 | uint8[16] | leid | Local EID | | ||
| 179 | +| 32 | 16 | uint8[16] | reid | Remote EID | | ||
| 180 | + | ||
| 181 | +### 3.5 MemLayoutInfo Structure | ||
| 182 | + | ||
| 183 | +#### 3.5.1 MemLayoutInfo Header | ||
| 184 | + | ||
| 185 | +| Offset | Bytes | Type | Field | Description | | ||
| 186 | +|:------:|:-----:|------|-------|-------------| | ||
| 187 | +| 0 | 4 | uint32 | count | Number of memory blocks | | ||
| 188 | +| 4 | Variable | MemLayoutData[] | data | Memory layout array | | ||
| 189 | + | ||
| 190 | +#### 3.5.2 MemLayoutData | ||
| 191 | + | ||
| 192 | +**Size**: 32 bytes. | ||
| 193 | + | ||
| 194 | +| Offset | Bytes | Type | Field | Description | | ||
| 195 | +|:------:|:-----:|------|-------|-------------| | ||
| 196 | +| 0 | 4 | uint32 | rank_id | Rank ID | | ||
| 197 | +| 4 | 1 | uint8 | buffer_type | Buffer type (see enum) | | ||
| 198 | +| 5 | 1 | uint8 | reserved | Reserved | | ||
| 199 | +| 6 | 8 | uint64 | start_addr | Physical start address | | ||
| 200 | +| 14 | 8 | uint64 | size | Block size | | ||
| 201 | +| 22 | 8 | uint64 | global_offset | Total offset for type | | ||
| 202 | + | ||
| 203 | +--- | ||
| 204 | + | ||
| 205 | +## 4. Microcode Instruction File (`*_hcclvm_instr_data.bin`) | ||
| 206 | + | ||
| 207 | +### 4.1 Overall Structure | ||
| 208 | + | ||
| 209 | +| No. | Data Block | Size | Description | | ||
| 210 | +|:---:|------------|------|-------------| | ||
| 211 | +| 1 | FileHeader | 20 Bytes | File header | | ||
| 212 | +| 2 | MicrocodeInstrInner × count | Variable | Microcode instruction data (including Desc and Instr) | | ||
| 213 | + | ||
| 214 | +### 4.2 MicrocodeInstrDesc | ||
| 215 | + | ||
| 216 | +**Size**: 8 bytes. | ||
| 217 | + | ||
| 218 | +| Offset | Bytes | Type | Field | Description | | ||
| 219 | +|:------:|:-----:|------|-------|-------------| | ||
| 220 | +| 0 | 4 | uint32 | rank_id | Rank ID | | ||
| 221 | +| 4 | 1 | uint8 | die_id | Die ID | | ||
| 222 | +| 5 | 1 | uint8 | reserved | Reserved | | ||
| 223 | +| 6 | 2 | uint16 | count | Number of instructions | | ||
| 224 | + | ||
| 225 | +### 4.3 Microcode Instruction Data | ||
| 226 | + | ||
| 227 | +Each MicrocodeInstrInner contains: | ||
| 228 | + | ||
| 229 | +| Data Block | Size | Description | | ||
| 230 | +|------------|------|-------------| | ||
| 231 | +| MicrocodeInstrDesc | 8 Bytes | Instruction descriptor | | ||
| 232 | +| CcuInstr[desc.count] | 32 × count Bytes | Microcode instruction array | | ||
| 233 | + | ||
| 234 | +--- | ||
| 235 | + | ||
| 236 | +## 5. Task Metadata File (`*_hcclvm_task_data.bin`) | ||
| 237 | + | ||
| 238 | +### 5.1 Overall Structure | ||
| 239 | + | ||
| 240 | +| No. | Data Block | Size | Description | | ||
| 241 | +|:---:|------------|------|-------------| | ||
| 242 | +| 1 | FileHeader | 20 Bytes | File header | | ||
| 243 | +| 2 | HcclTaskMetaData × count | Variable | Task list | | ||
| 244 | + | ||
| 245 | +### 5.2 HcclTaskMetaData Structure | ||
| 246 | + | ||
| 247 | +**Size**: Approximately 136 bytes (including a union) | ||
| 248 | + | ||
| 249 | +| Offset | Bytes | Type | Field | Description | | ||
| 250 | +|:------:|:-----:|------|-------|-------------| | ||
| 251 | +| 0 | 1 | int8 | taskType | Task type (see enum) | | ||
| 252 | +| 1 | 2 | uint16 | commId | Communication domain ID | | ||
| 253 | +| 3 | 4 | uint32 | rankId | Rank ID | | ||
| 254 | +| 7 | 8 | uint64 | streamId | Stream ID | | ||
| 255 | +| 15 | 4 | uint32 | jettyId | Jetty ID | | ||
| 256 | +| 19 | Variable | union | taskData | Task data (parsed by type) | | ||
| 257 | + | ||
| 258 | +### 5.3 taskData Union | ||
| 259 | + | ||
| 260 | +#### 5.3.1 TransMemTask | ||
| 261 | + | ||
| 262 | +**Applicable task type**: `MEM_CPY (3)`. | ||
| 263 | + | ||
| 264 | +**Size**: 33 bytes. | ||
| 265 | + | ||
| 266 | +| Offset | Bytes | Type | Field | Description | | ||
| 267 | +|:------:|:-----:|------|-------|-------------| | ||
| 268 | +| 0 | 4 | uint32 | srcRankId | Source Rank ID | | ||
| 269 | +| 4 | 8 | uint64 | srcOffset | Source offset address | | ||
| 270 | +| 12 | 4 | uint32 | dstRankId | Destination Rank ID | | ||
| 271 | +| 16 | 8 | uint64 | dstOffset | Destination offset address | | ||
| 272 | +| 24 | 8 | uint64 | len | Data length | | ||
| 273 | +| 32 | 1 | uint8 | protocol | Protocol type | | ||
| 274 | + | ||
| 275 | +#### 5.3.2 ReduceTask | ||
| 276 | + | ||
| 277 | +**Applicable task type**: `REDUCE (2)`. | ||
| 278 | + | ||
| 279 | +**Size**: 35 bytes. | ||
| 280 | + | ||
| 281 | +| Offset | Bytes | Type | Field | Description | | ||
| 282 | +|:------:|:-----:|------|-------|-------------| | ||
| 283 | +| 0 | 4 | uint32 | srcRankId | Source Rank ID | | ||
| 284 | +| 4 | 8 | uint64 | srcOffset | Source offset address | | ||
| 285 | +| 12 | 4 | uint32 | dstRankId | Destination Rank ID | | ||
| 286 | +| 16 | 8 | uint64 | dstOffset | Destination offset address | | ||
| 287 | +| 24 | 8 | uint64 | dataCount | Number of data elements | | ||
| 288 | +| 32 | 1 | uint8 | dataType | Data type | | ||
| 289 | +| 33 | 1 | uint8 | reduceOp | Reduce operation type | | ||
| 290 | +| 34 | 1 | uint8 | protocol | Protocol type | | ||
| 291 | + | ||
| 292 | +#### 5.3.3 NotifyTask | ||
| 293 | + | ||
| 294 | +**Applicable task types**: `NOTIFY_WAIT (0)` / `NOTIFY_RECORD (1)`. | ||
| 295 | + | ||
| 296 | +**Size**: 18 bytes. | ||
| 297 | + | ||
| 298 | +| Offset | Bytes | Type | Field | Description | | ||
| 299 | +|:------:|:-----:|------|-------|-------------| | ||
| 300 | +| 0 | 4 | uint32 | srcRankId | Source Rank ID | | ||
| 301 | +| 4 | 8 | uint64 | notifyId | Notification ID | | ||
| 302 | +| 12 | 4 | uint32 | dstRankId | Destination Rank ID | | ||
| 303 | +| 16 | 1 | uint8 | notifyCount | Notification count | | ||
| 304 | +| 17 | 1 | uint8 | protocol | Protocol type | | ||
| 305 | + | ||
| 306 | +--- | ||
| 307 | + | ||
| 308 | +## 6. Enum Definitions | ||
| 309 | + | ||
| 310 | +### 6.1 Task Type (HccLTaskMetaType) | ||
| 311 | + | ||
| 312 | +| Value | Name | Description | | ||
| 313 | +|:----:|------|-------------| | ||
| 314 | +| 0 | NOTIFY_WAIT | Wait for notification | | ||
| 315 | +| 1 | NOTIFY_RECORD | Record notification | | ||
| 316 | +| 2 | REDUCE | Reduce operation | | ||
| 317 | +| 3 | MEM_CPY | Memory copy | | ||
| 318 | +| 4 | CCU_GRAPH | CCU graph execution | | ||
| 319 | +| 5 | AIV_GRAPH | AIV graph execution | | ||
| 320 | +| 6 | EVENT_WAIT | Event wait | | ||
| 321 | +| 7 | EVENT_RECORD | Event record | | ||
| 322 | + | ||
| 323 | +### 6.2 Protocol Type (ProtocolType) | ||
| 324 | + | ||
| 325 | +| Value | Name | Description | | ||
| 326 | +|:----:|------|-------------| | ||
| 327 | +| 0 | HCCS | High-speed chip interconnect | | ||
| 328 | +| 1 | ROCE | RDMA over Converged Ethernet | | ||
| 329 | +| 2 | PCIE | PCIe communication | | ||
| 330 | +| 3 | SIO | Socket I/O | | ||
| 331 | +| 4 | UBC_CTP | UBC CTP protocol | | ||
| 332 | +| 5 | UBC_TP | UBC TP protocol | | ||
| 333 | +| 6 | UB_MEM | UB memory protocol | | ||
| 334 | + | ||
| 335 | +### 6.3 Buffer Type (BufferType) | ||
| 336 | + | ||
| 337 | +| Value | Name | Description | | ||
| 338 | +|:----:|------|-------------| | ||
| 339 | +| 0 | INPUT | Input buffer | | ||
| 340 | +| 1 | OUTPUT | Output buffer | | ||
| 341 | +| 2 | CCL | Communication buffer | | ||
| 342 | +| 3 | RESERVED | Reserved | | ||
| 343 | + | ||
| 344 | +### 6.4 Expansion Mode (SimOpExpansionMode) | ||
| 345 | + | ||
| 346 | +| Value | Name | Description | | ||
| 347 | +|:----:|------|-------------| | ||
| 348 | +| 0 | SIM_OP_EXPANSION_MODE_CCU | CCU mode, uses ChannelInfo | | ||
| 349 | +| 1 | SIM_OP_EXPANSION_MODE_AICPU | AICPU mode, uses JettyInfo | | ||
| 350 | + | ||
| 351 | +--- | ||
| 352 | + | ||
| 353 | +## 7. C/C++ Reading Examples | ||
| 354 | + | ||
| 355 | +### 7.1 Structure Definitions | ||
| 356 | + | ||
| 357 | +```cpp | ||
| 358 | +#include <cstdint> | ||
| 359 | +#include <vector> | ||
| 360 | +#include <cstring> | ||
| 361 | + | ||
| 362 | +#pragma pack(push, 1) | ||
| 363 | + | ||
| 364 | +// ============================================================================ | ||
| 365 | +// File header | ||
| 366 | +// ============================================================================ | ||
| 367 | +struct FileHeader { | ||
| 368 | + uint32_t magic; | ||
| 369 | + uint16_t version; | ||
| 370 | + uint16_t header_size; | ||
| 371 | + uint32_t flags; | ||
| 372 | + uint32_t count; | ||
| 373 | + uint32_t checksum; | ||
| 374 | +}; | ||
| 375 | + | ||
| 376 | +// ============================================================================ | ||
| 377 | +// Model information | ||
| 378 | +// ============================================================================ | ||
| 379 | +struct ModelInfoCommInner { | ||
| 380 | + uint32_t src_rank; | ||
| 381 | + uint32_t dst_rank; | ||
| 382 | + uint32_t root; | ||
| 383 | + uint32_t rank_size; | ||
| 384 | + uint16_t chip_type; | ||
| 385 | + uint16_t op_type; | ||
| 386 | + uint16_t reduce_op; | ||
| 387 | + uint16_t data_type; | ||
| 388 | + uint64_t data_count; | ||
| 389 | + uint32_t op_expansion_mode; | ||
| 390 | + uint64_t ccu0_resource_base_addr; | ||
| 391 | + uint64_t ccu1_resource_base_addr; | ||
| 392 | +}; | ||
| 393 | + | ||
| 394 | +// ============================================================================ | ||
| 395 | +// Channel data (CCU mode) | ||
| 396 | +// ============================================================================ | ||
| 397 | +struct ChannelData { | ||
| 398 | + uint16_t channelId; | ||
| 399 | + uint8_t srcDieId; | ||
| 400 | + uint8_t dstDieId; | ||
| 401 | + uint32_t srcRank; | ||
| 402 | + uint32_t dstRank; | ||
| 403 | + uint8_t leid[16]; | ||
| 404 | + uint8_t reid[16]; | ||
| 405 | + uint16_t protocol; | ||
| 406 | + uint16_t jettyNum; | ||
| 407 | + uint32_t jettyId[32]; | ||
| 408 | +}; | ||
| 409 | + | ||
| 410 | +// ============================================================================ | ||
| 411 | +// Jetty data (AICPU mode) | ||
| 412 | +// ============================================================================ | ||
| 413 | +struct JettyData { | ||
| 414 | + uint32_t jettyId; | ||
| 415 | + uint8_t srcDieId; | ||
| 416 | + uint8_t dstDieId; | ||
| 417 | + uint16_t protocol; | ||
| 418 | + uint32_t srcRank; | ||
| 419 | + uint32_t dstRank; | ||
| 420 | + uint8_t leid[16]; | ||
| 421 | + uint8_t reid[16]; | ||
| 422 | +}; | ||
| 423 | + | ||
| 424 | +// ============================================================================ | ||
| 425 | +// Memory layout | ||
| 426 | +// ============================================================================ | ||
| 427 | +struct MemLayoutData { | ||
| 428 | + uint32_t rank_id; | ||
| 429 | + uint8_t buffer_type; | ||
| 430 | + uint8_t reserved; | ||
| 431 | + uint64_t start_addr; | ||
| 432 | + uint64_t size; | ||
| 433 | + uint64_t global_offset; | ||
| 434 | +}; | ||
| 435 | + | ||
| 436 | +// ============================================================================ | ||
| 437 | +// Task type enum | ||
| 438 | +// ============================================================================ | ||
| 439 | +enum class HccLTaskMetaType : int8_t { | ||
| 440 | + NOTIFY_WAIT = 0, | ||
| 441 | + NOTIFY_RECORD = 1, | ||
| 442 | + REDUCE = 2, | ||
| 443 | + MEM_CPY = 3, | ||
| 444 | + CCU_GRAPH = 4, | ||
| 445 | + AIV_GRAPH = 5, | ||
| 446 | + EVENT_WAIT = 6, | ||
| 447 | + EVENT_RECORD = 7 | ||
| 448 | +}; | ||
| 449 | + | ||
| 450 | +// ============================================================================ | ||
| 451 | +// Task data union | ||
| 452 | +// ============================================================================ | ||
| 453 | +struct TransMemTask { | ||
| 454 | + uint32_t srcRankId; | ||
| 455 | + uint64_t srcOffset; | ||
| 456 | + uint32_t dstRankId; | ||
| 457 | + uint64_t dstOffset; | ||
| 458 | + uint64_t len; | ||
| 459 | + uint8_t protocol; | ||
| 460 | +}; | ||
| 461 | + | ||
| 462 | +struct ReduceTask { | ||
| 463 | + uint32_t srcRankId; | ||
| 464 | + uint64_t srcOffset; | ||
| 465 | + uint32_t dstRankId; | ||
| 466 | + uint64_t dstOffset; | ||
| 467 | + uint64_t dataCount; | ||
| 468 | + uint8_t dataType; | ||
| 469 | + uint8_t reduceOp; | ||
| 470 | + uint8_t protocol; | ||
| 471 | +}; | ||
| 472 | + | ||
| 473 | +struct NotifyTask { | ||
| 474 | + uint32_t srcRankId; | ||
| 475 | + uint64_t notifyId; | ||
| 476 | + uint32_t dstRankId; | ||
| 477 | + uint8_t notifyCount; | ||
| 478 | + uint8_t protocol; | ||
| 479 | +}; | ||
| 480 | + | ||
| 481 | +// ============================================================================ | ||
| 482 | +// Task metadata | ||
| 483 | +// ============================================================================ | ||
| 484 | +struct HcclTaskMetaData { | ||
| 485 | + HccLTaskMetaType taskType; | ||
| 486 | + uint16_t commId; | ||
| 487 | + uint32_t rankId; | ||
| 488 | + uint64_t streamId; | ||
| 489 | + uint32_t jettyId; | ||
| 490 | + union { | ||
| 491 | + TransMemTask transMem; | ||
| 492 | + ReduceTask reduce; | ||
| 493 | + NotifyTask notify; | ||
| 494 | + } taskData; | ||
| 495 | +}; | ||
| 496 | + | ||
| 497 | +#pragma pack(pop) | ||
| 498 | +``` | ||
| 499 | + | ||
| 500 | +### 7.2 Reading Synthesis Data Example | ||
| 501 | + | ||
| 502 | +```cpp | ||
| 503 | +#include <cstdio> | ||
| 504 | +#include <iostream> | ||
| 505 | + | ||
| 506 | +// Magic number definitions | ||
| 507 | +constexpr uint32_t HCCLVM_SYN_FILE_MAGIC = 0x48564D44; | ||
| 508 | + | ||
| 509 | +bool ReadSynthesisData(const char* filename) { | ||
| 510 | + FILE* fp = fopen(filename, "rb"); | ||
| 511 | + if (!fp) { | ||
| 512 | + std::cerr << "Failed to open file: " << filename << std::endl; | ||
| 513 | + return false; | ||
| 514 | + } | ||
| 515 | + | ||
| 516 | + // ===== Step 1: Read file header ===== | ||
| 517 | + FileHeader header; | ||
| 518 | + if (fread(&header, sizeof(FileHeader), 1, fp) != 1) { | ||
| 519 | + std::cerr << "Failed to read header" << std::endl; | ||
| 520 | + fclose(fp); | ||
| 521 | + return false; | ||
| 522 | + } | ||
| 523 | + | ||
| 524 | + // Validate magic number | ||
| 525 | + if (header.magic != HCCLVM_SYN_FILE_MAGIC) { | ||
| 526 | + std::cerr << "Invalid magic number: 0x" << std::hex << header.magic << std::endl; | ||
| 527 | + fclose(fp); | ||
| 528 | + return false; | ||
| 529 | + } | ||
| 530 | + | ||
| 531 | + std::cout << "File version: " << header.version << std::endl; | ||
| 532 | + std::cout << "Data count: " << header.count << std::endl; | ||
| 533 | + | ||
| 534 | + // ===== Step 2: Read model information ===== | ||
| 535 | + ModelInfoCommInner modelComm; | ||
| 536 | + fread(&modelComm, sizeof(ModelInfoCommInner), 1, fp); | ||
| 537 | + | ||
| 538 | + std::cout << "\n[Model Info]" << std::endl; | ||
| 539 | + std::cout << " Rank Size: " << modelComm.rank_size << std::endl; | ||
| 540 | + std::cout << " Op Type: " << modelComm.op_type << std::endl; | ||
| 541 | + std::cout << " Data Count: " << modelComm.data_count << std::endl; | ||
| 542 | + std::cout << " Expansion Mode: " << modelComm.op_expansion_mode << std::endl; | ||
| 543 | + | ||
| 544 | + // ===== Step 3: Read VDataDesTag ===== | ||
| 545 | + uint16_t vDataType; | ||
| 546 | + uint32_t vDataCount; | ||
| 547 | + fread(&vDataType, sizeof(uint16_t), 1, fp); | ||
| 548 | + fread(&vDataCount, sizeof(uint32_t), 1, fp); | ||
| 549 | + | ||
| 550 | + if (vDataCount > 0) { | ||
| 551 | + std::vector<uint64_t> displs(vDataCount); | ||
| 552 | + std::vector<uint64_t> counts(vDataCount); | ||
| 553 | + fread(displs.data(), sizeof(uint64_t), vDataCount, fp); | ||
| 554 | + fread(counts.data(), sizeof(uint64_t), vDataCount, fp); | ||
| 555 | + std::cout << "\n[VDataDes] Count: " << vDataCount << std::endl; | ||
| 556 | + } | ||
| 557 | + | ||
| 558 | + // ===== Step 4: Read All2AllDataDesTag ===== | ||
| 559 | + uint16_t sendType, recvType; | ||
| 560 | + uint64_t sendCount, recvCount; | ||
| 561 | + uint32_t matrixCount; | ||
| 562 | + | ||
| 563 | + fread(&sendType, sizeof(uint16_t), 1, fp); | ||
| 564 | + fread(&recvType, sizeof(uint16_t), 1, fp); | ||
| 565 | + fread(&sendCount, sizeof(uint64_t), 1, fp); | ||
| 566 | + fread(&recvCount, sizeof(uint64_t), 1, fp); | ||
| 567 | + fread(&matrixCount, sizeof(uint32_t), 1, fp); | ||
| 568 | + | ||
| 569 | + if (matrixCount > 0) { | ||
| 570 | + std::vector<uint64_t> sendCountMatrix(matrixCount); | ||
| 571 | + fread(sendCountMatrix.data(), sizeof(uint64_t), matrixCount, fp); | ||
| 572 | + std::cout << "\n[All2All] Matrix Count: " << matrixCount << std::endl; | ||
| 573 | + } | ||
| 574 | + | ||
| 575 | + // ===== Step 5: Read Channel or Jetty info based on op_expansion_mode ===== | ||
| 576 | + if (modelComm.op_expansion_mode == 0) { | ||
| 577 | + // CCU mode - read ChannelInfo | ||
| 578 | + uint32_t channelCount; | ||
| 579 | + fread(&channelCount, sizeof(uint32_t), 1, fp); | ||
| 580 | + | ||
| 581 | + std::vector<ChannelData> channels(channelCount); | ||
| 582 | + fread(channels.data(), sizeof(ChannelData), channelCount, fp); | ||
| 583 | + | ||
| 584 | + std::cout << "\n[ChannelInfo] Count: " << channelCount << std::endl; | ||
| 585 | + for (size_t i = 0; i < channels.size() && i < 3; ++i) { | ||
| 586 | + std::cout << " Channel[" << i << "]: ID=" << channels[i].channelId | ||
| 587 | + << ", srcRank=" << channels[i].srcRank | ||
| 588 | + << ", dstRank=" << channels[i].dstRank << std::endl; | ||
| 589 | + } | ||
| 590 | + } else { | ||
| 591 | + // AICPU mode - read JettyInfo | ||
| 592 | + uint32_t jettyCount; | ||
| 593 | + fread(&jettyCount, sizeof(uint32_t), 1, fp); | ||
| 594 | + | ||
| 595 | + std::vector<JettyData> jetties(jettyCount); | ||
| 596 | + fread(jetties.data(), sizeof(JettyData), jettyCount, fp); | ||
| 597 | + | ||
| 598 | + std::cout << "\n[JettyInfo] Count: " << jettyCount << std::endl; | ||
| 599 | + for (size_t i = 0; i < jetties.size() && i < 3; ++i) { | ||
| 600 | + std::cout << " Jetty[" << i << "]: ID=" << jetties[i].jettyId | ||
| 601 | + << ", srcRank=" << jetties[i].srcRank | ||
| 602 | + << ", dstRank=" << jetties[i].dstRank << std::endl; | ||
| 603 | + } | ||
| 604 | + } | ||
| 605 | + | ||
| 606 | + // ===== Step 6: Read memory layout ===== | ||
| 607 | + uint32_t memCount; | ||
| 608 | + fread(&memCount, sizeof(uint32_t), 1, fp); | ||
| 609 | + | ||
| 610 | + std::vector<MemLayoutData> memLayouts(memCount); | ||
| 611 | + fread(memLayouts.data(), sizeof(MemLayoutData), memCount, fp); | ||
| 612 | + | ||
| 613 | + std::cout << "\n[MemLayout] Count: " << memCount << std::endl; | ||
| 614 | + for (size_t i = 0; i < memLayouts.size() && i < 3; ++i) { | ||
| 615 | + std::cout << " Mem[" << i << "]: rank=" << memLayouts[i].rank_id | ||
| 616 | + << ", type=" << (int)memLayouts[i].buffer_type | ||
| 617 | + << ", size=" << memLayouts[i].size << std::endl; | ||
| 618 | + } | ||
| 619 | + | ||
| 620 | + fclose(fp); | ||
| 621 | + std::cout << "\nRead synthesis data success!" << std::endl; | ||
| 622 | + return true; | ||
| 623 | +} | ||
| 624 | +``` | ||
| 625 | + | ||
| 626 | +### 7.3 Reading Task Metadata Example | ||
| 627 | + | ||
| 628 | +```cpp | ||
| 629 | +constexpr uint32_t HCCLVM_TASK_FILE_MAGIC = 0x48565444; | ||
| 630 | + | ||
| 631 | +bool ReadTaskMetaData(const char* filename) { | ||
| 632 | + FILE* fp = fopen(filename, "rb"); | ||
| 633 | + if (!fp) { | ||
| 634 | + std::cerr << "Failed to open file: " << filename << std::endl; | ||
| 635 | + return false; | ||
| 636 | + } | ||
| 637 | + | ||
| 638 | + // ===== Step 1: Read file header ===== | ||
| 639 | + FileHeader header; | ||
| 640 | + fread(&header, sizeof(FileHeader), 1, fp); | ||
| 641 | + | ||
| 642 | + if (header.magic != HCCLVM_TASK_FILE_MAGIC) { | ||
| 643 | + std::cerr << "Invalid magic number" << std::endl; | ||
| 644 | + fclose(fp); | ||
| 645 | + return false; | ||
| 646 | + } | ||
| 647 | + | ||
| 648 | + std::cout << "Task count: " << header.count << std::endl; | ||
| 649 | + | ||
| 650 | + // ===== Step 2: Read all tasks ===== | ||
| 651 | + std::vector<HcclTaskMetaData> tasks(header.count); | ||
| 652 | + fread(tasks.data(), sizeof(HcclTaskMetaData), header.count, fp); | ||
| 653 | + | ||
| 654 | + // ===== Step 3: Parse and print tasks ===== | ||
| 655 | + std::cout << "\n[Task List]" << std::endl; | ||
| 656 | + for (size_t i = 0; i < tasks.size(); ++i) { | ||
| 657 | + const auto& task = tasks[i]; | ||
| 658 | + std::cout << "Task[" << i << "] "; | ||
| 659 | + | ||
| 660 | + switch (task.taskType) { | ||
| 661 | + case HccLTaskMetaType::MEM_CPY: | ||
| 662 | + std::cout << "MEM_CPY: " | ||
| 663 | + << "src=" << task.taskData.transMem.srcRankId | ||
| 664 | + << ", dst=" << task.taskData.transMem.dstRankId | ||
| 665 | + << ", len=" << task.taskData.transMem.len; | ||
| 666 | + break; | ||
| 667 | + | ||
| 668 | + case HccLTaskMetaType::REDUCE: | ||
| 669 | + std::cout << "REDUCE: " | ||
| 670 | + << "src=" << task.taskData.reduce.srcRankId | ||
| 671 | + << ", dst=" << task.taskData.reduce.dstRankId | ||
| 672 | + << ", count=" << task.taskData.reduce.dataCount; | ||
| 673 | + break; | ||
| 674 | + | ||
| 675 | + case HccLTaskMetaType::NOTIFY_WAIT: | ||
| 676 | + std::cout << "NOTIFY_WAIT: " | ||
| 677 | + << "notifyId=" << task.taskData.notify.notifyId; | ||
| 678 | + break; | ||
| 679 | + | ||
| 680 | + case HccLTaskMetaType::NOTIFY_RECORD: | ||
| 681 | + std::cout << "NOTIFY_RECORD: " | ||
| 682 | + << "notifyId=" << task.taskData.notify.notifyId; | ||
| 683 | + break; | ||
| 684 | + | ||
| 685 | + default: | ||
| 686 | + std::cout << "UNKNOWN: type=" << static_cast<int>(task.taskType); | ||
| 687 | + break; | ||
| 688 | + } | ||
| 689 | + std::cout << std::endl; | ||
| 690 | + } | ||
| 691 | + | ||
| 692 | + fclose(fp); | ||
| 693 | + std::cout << "\nRead task meta data success!" << std::endl; | ||
| 694 | + return true; | ||
| 695 | +} | ||
| 696 | +``` | ||
| 697 | + | ||
| 698 | +--- | ||
| 699 | + | ||
| 700 | +## 8. Notes | ||
| 701 | + | ||
| 702 | +| No. | Note | Description | | ||
| 703 | +|:---:|------|-------------| | ||
| 704 | +| 1 | Byte order | All multi-byte fields use **Little-Endian** | | ||
| 705 | +| 2 | Memory alignment | Structures use `#pragma pack(1)` for **1-byte alignment** | | ||
| 706 | +| 3 | Variable-length field reading | Read the `count` field first, then read the corresponding number of data items based on count | | ||
| 707 | +| 4 | Mode determination | Read ChannelInfo or JettyInfo based on `op_expansion_mode` | | ||
| 708 | +| 5 | Union parsing | In task metadata, parse `taskData` according to `taskType` to select the correct structure | | ||
| 709 | +| 6 | Magic number validation | Always verify the magic number when reading a file to ensure the correct file type | | ||
| 710 | + | ||
| 711 | +--- | ||
| 712 | + | ||
| 713 | +## Appendix A: Magic Number Quick Reference | ||
| 714 | + | ||
| 715 | +| File Type | Magic Number | ASCII | Description | | ||
| 716 | +|-----------|-------------|-------|-------------| | ||
| 717 | +| Synthesis data | `0x48564D44` | "HVMD" | Hccl VM Data | | ||
| 718 | +| Microcode instruction | `0x434D4349` | "CMCI" | CCU Microcode Instruction | | ||
| 719 | +| Task metadata | `0x48565444` | "HVTM" | Hccl VM Task Metadata | | ||
| 720 | + | ||
| 721 | +--- | ||
| 722 | + | ||
| 723 | +## Appendix B: File Reading Flow | ||
| 724 | + | ||
| 725 | +**Step 1**: Open the binary file. | ||
| 726 | + | ||
| 727 | +**Step 2**: Read FileHeader (20 Bytes) | ||
| 728 | + | ||
| 729 | +**Step 3**: Validate magic number | ||
| 730 | + | ||
| 731 | +- Magic mismatch → Return error, close file. | ||
| 732 | +- Magic matches → Continue. | ||
| 733 | + | ||
| 734 | +**Step 4**: Read data content based on count. | ||
| 735 | + | ||
| 736 | +**Step 5**: Close file. | ||
| 737 | + | ||
| 738 | +--- | ||
| 739 | + | ||
| 740 | +## Appendix C: Contact | ||
| 741 | + | ||
| 742 | +For questions, please contact HCCL VM Team. | ||
| 743 | + | ||
| 744 | +--- | ||
| 745 | + | ||
| 746 | +## End of Document | ||
| @@ -0,0 +1,343 @@ | |||
| 1 | +# HVRM Insight V3 User Guide | ||
| 2 | + | ||
| 3 | +## 1. Overview | ||
| 4 | + | ||
| 5 | +HVRM Insight V3 currently mainly supports `DAGView` task graph viewing. This document focuses on common operations around DAGView, including data preparation, page access, dataset selection, DAG graph browsing, and node search. | ||
| 6 | + | ||
| 7 | +Other pages and advanced linked capabilities are not yet implemented. | ||
| 8 | + | ||
| 9 | +--- | ||
| 10 | + | ||
| 11 | +## 2. Prerequisites | ||
| 12 | + | ||
| 13 | +Before using Insight V3, ensure Checker has been executed and Insight data has been generated. | ||
| 14 | + | ||
| 15 | +To configure Checker data output, ensure Insight dump is enabled in the Checker configuration file: | ||
| 16 | + | ||
| 17 | +```json | ||
| 18 | +# Configuration file located at /pathto/hccl_vm_install/plugin/checker/manifest.json | ||
| 19 | + | ||
| 20 | +{ | ||
| 21 | + ... | ||
| 22 | + "setting": { // Checker plugin configuration | ||
| 23 | + ... | ||
| 24 | + "enable_insight_dump": true, // Whether to enable visualization data output (disabled by default) | ||
| 25 | + "enable_memory_snapshot_dump": false // Whether to enable visualization memory snapshot data output (disabled by default, only supported by old Checker, requires "enable_insight_dump" to be enabled first) | ||
| 26 | + } | ||
| 27 | +} | ||
| 28 | +``` | ||
| 29 | + | ||
| 30 | +DAGView requires at least the following data files: | ||
| 31 | + | ||
| 32 | +```text | ||
| 33 | +<dataset_name>/ | ||
| 34 | +├── manifest.json | ||
| 35 | +└── graph/ | ||
| 36 | + ├── graph.msgpack | ||
| 37 | + └── layout.msgpack | ||
| 38 | +``` | ||
| 39 | + | ||
| 40 | +`manifest.json` is used for reading dataset metadata, while `graph.msgpack` and `layout.msgpack` are used for rendering the DAG task graph. | ||
| 41 | + | ||
| 42 | +--- | ||
| 43 | + | ||
| 44 | +## 3. Compilation and Installation of Insight Plugin | ||
| 45 | + | ||
| 46 | +The Insight V3 frontend source code is located at: | ||
| 47 | + | ||
| 48 | +```text | ||
| 49 | +{hccl_vm directory}/src/plugin/insight/frontend_v3 | ||
| 50 | +``` | ||
| 51 | + | ||
| 52 | +Before installing the Insight plugin, first complete the frontend compilation, then execute the HCCL VM build and installation process. | ||
| 53 | + | ||
| 54 | +Note: Compiling the Insight visualization frontend requires `Node.js` and `npm`. It is recommended to use `Node.js 20.19.0` or later and `npm 10.x` or later. | ||
| 55 | + | ||
| 56 | +Recommended steps: | ||
| 57 | + | ||
| 58 | +1. Enter the frontend directory: | ||
| 59 | + | ||
| 60 | +```bash | ||
| 61 | +cd {hccl_vm directory}/src/plugin/insight/frontend_v3 | ||
| 62 | +``` | ||
| 63 | + | ||
| 64 | +2. Install frontend dependencies: | ||
| 65 | + | ||
| 66 | +```bash | ||
| 67 | +npm install | ||
| 68 | +``` | ||
| 69 | + | ||
| 70 | +3. Compile the frontend artifacts: | ||
| 71 | + | ||
| 72 | +```bash | ||
| 73 | +npm run build | ||
| 74 | +``` | ||
| 75 | + | ||
| 76 | +After frontend compilation completes, the `src/plugin/insight/dist/` directory is generated. | ||
| 77 | + | ||
| 78 | +4. Return to the HCCL_VM root directory and execute `build.sh` to compile and install hccl-vm: | ||
| 79 | + | ||
| 80 | +```bash | ||
| 81 | +cd {HCCL_VM directory} | ||
| 82 | +bash build.sh --package-path <ASCEND_CANN_PATH> --hcomm-path <HCOMM_CODE_PATH> | ||
| 83 | +``` | ||
| 84 | + | ||
| 85 | +To package the installation directory, append `--pkg`. If the current build scenario requires AICPU / AIV / FULL mode, append `--aicpu`, `--aiv`, or `--full` as per the project's standard process. | ||
| 86 | + | ||
| 87 | +--- | ||
| 88 | + | ||
| 89 | +## 4. Insight Plugin Configuration, Start and Stop | ||
| 90 | + | ||
| 91 | +The Insight plugin configuration file is located at: | ||
| 92 | + | ||
| 93 | +```text | ||
| 94 | +hccl_vm_install/plugin/visualization/insight/manifest.json | ||
| 95 | +``` | ||
| 96 | + | ||
| 97 | +The current default configuration is: | ||
| 98 | + | ||
| 99 | +```json | ||
| 100 | +{ | ||
| 101 | + "name": "insight", | ||
| 102 | + "version": "1.0.0", | ||
| 103 | + "entry": "python3 server.py", | ||
| 104 | + "dependency": { | ||
| 105 | + "min_core_version": "1.0.0" | ||
| 106 | + }, | ||
| 107 | + "setting": { | ||
| 108 | + "dist_path": "./dist", | ||
| 109 | + "data_path": "../../checker/data/insight", | ||
| 110 | + "topo_config_path": "../../../../../asset/cluster_model/config/cluster", | ||
| 111 | + "port": 8080 | ||
| 112 | + } | ||
| 113 | +} | ||
| 114 | +``` | ||
| 115 | + | ||
| 116 | +Common field descriptions: | ||
| 117 | + | ||
| 118 | +| Field | Description | | ||
| 119 | +|-------|-------------| | ||
| 120 | +| `entry` | Insight plugin startup command; currently starts the service via `python3 server.py` | | ||
| 121 | +| `setting.dist_path` | Frontend static resource directory; reads `./dist` by default | | ||
| 122 | +| `setting.data_path` | Insight data directory; defaults to Checker output `data/insight` | | ||
| 123 | +| `setting.topo_config_path` | Cluster topology configuration directory | | ||
| 124 | +| `setting.port` | Insight service port; defaults to `8080` | | ||
| 125 | + | ||
| 126 | +To change the port or data directory, directly edit the `setting` fields in this `manifest.json`. | ||
| 127 | + | ||
| 128 | +> Warning: Insight binds to `127.0.0.1` by default via `python3 server.py`. To listen on a specific IP instead of the default `localhost (127.0.0.1)`, enter the `hccl_vm_install/plugin/visualization/insight` directory and manually execute: | ||
| 129 | +> | ||
| 130 | +> ```bash | ||
| 131 | +> python3 server.py --host <ip> --port <port> | ||
| 132 | +> ``` | ||
| 133 | +> | ||
| 134 | +> For example: | ||
| 135 | +> | ||
| 136 | +> ```bash | ||
| 137 | +> cd hccl_vm_install/plugin/visualization/insight | ||
| 138 | +> python3 server.py --host 0.0.0.0 --port 8080 | ||
| 139 | +> ``` | ||
| 140 | + | ||
| 141 | +Plugin installation and uninstallation commands: | ||
| 142 | + | ||
| 143 | +```bash | ||
| 144 | +# Install and start the Insight plugin | ||
| 145 | +hccl-vm plugin install @insight | ||
| 146 | + | ||
| 147 | +# Uninstall the Insight plugin | ||
| 148 | +hccl-vm plugin uninstall @insight | ||
| 149 | +``` | ||
| 150 | + | ||
| 151 | +--- | ||
| 152 | + | ||
| 153 | +## 5. Opening Insight | ||
| 154 | + | ||
| 155 | +If the Insight service is already started, open the service address directly in your browser, for example: | ||
| 156 | + | ||
| 157 | +```text | ||
| 158 | +http://localhost:8080 | ||
| 159 | +``` | ||
| 160 | + | ||
| 161 | +To start Insight via the plugin, execute: | ||
| 162 | + | ||
| 163 | +```bash | ||
| 164 | +hccl-vm plugin install @insight | ||
| 165 | +``` | ||
| 166 | + | ||
| 167 | +After the plugin starts, the terminal outputs the access address. Open the address in your browser to enter the Insight page. | ||
| 168 | + | ||
| 169 | + | ||
| 170 | + | ||
| 171 | +--- | ||
| 172 | + | ||
| 173 | +## 6. Selecting a Dataset | ||
| 174 | + | ||
| 175 | +After opening Insight, you enter the `Overview` page by default. Follow these steps to select a dataset for analysis: | ||
| 176 | + | ||
| 177 | +1. View the dataset list in the center dataset table. | ||
| 178 | +2. Click the target dataset. | ||
| 179 | +3. Confirm the ranks to view in the Rank tree on the left. | ||
| 180 | +4. If the DAG graph is large, reduce the rank selection range first. | ||
| 181 | +5. Click `Enter Correlation View` on the right. | ||
| 182 | + | ||
| 183 | + | ||
| 184 | + | ||
| 185 | +It is recommended to start with a small number of ranks for initial analysis, then expand the rank range after confirming the analysis direction. | ||
| 186 | + | ||
| 187 | +--- | ||
| 188 | + | ||
| 189 | +## 7. Entering DAGView | ||
| 190 | + | ||
| 191 | +After entering the `Correlation` page, focus on the lower half's `DAGView Task View`. | ||
| 192 | + | ||
| 193 | +The main areas of the page are: | ||
| 194 | + | ||
| 195 | +| Area | Function | | ||
| 196 | +|------|----------| | ||
| 197 | +| Left panel | Select ranks, search nodes | | ||
| 198 | +| Center bottom | Display the DAG task graph | | ||
| 199 | +| Right details panel | View current node details | | ||
| 200 | + | ||
| 201 | +If the upper memory view is empty, it does not affect the viewing and use of the DAGView below. | ||
| 202 | + | ||
| 203 | +--- | ||
| 204 | + | ||
| 205 | +## 8. Viewing the DAG Task Graph | ||
| 206 | + | ||
| 207 | +The DAG task graph consists of swimlanes, nodes, and arrows: | ||
| 208 | + | ||
| 209 | +| Element | Description | | ||
| 210 | +|---------|-------------| | ||
| 211 | +| Swimlane | Represents an execution queue, typically displayed as `Rank / Stream / Queue` | | ||
| 212 | +| Node | Represents a task, such as data copy, Reduce, Record, Wait, etc. | | ||
| 213 | +| Arrow | Represents dependency relationships between tasks | | ||
| 214 | +| Loop dashed box | Represents a Loop region, enclosed by a dashed box with a `Loop` label outside | | ||
| 215 | + | ||
| 216 | +Common node types: | ||
| 217 | + | ||
| 218 | +| Node Type | Meaning | | ||
| 219 | +|-----------|---------| | ||
| 220 | +| `TRANS_MEM` | Data copy task | | ||
| 221 | +| `REDUCE` | Reduce task | | ||
| 222 | +| `RECORD` | Notify record task | | ||
| 223 | +| `WAIT` | Notify wait task | | ||
| 224 | +| `CCU_GRAPH` | CCU graph task | | ||
| 225 | +| `AIV_GRAPH` | AIV graph task | | ||
| 226 | + | ||
| 227 | +Recommended viewing approach: | ||
| 228 | + | ||
| 229 | +1. Follow the arrow direction to view task dependency order. | ||
| 230 | +2. Click on a task node of interest. | ||
| 231 | +3. View the node's Rank, Stream, Queue, and detailed information in the right details panel. | ||
| 232 | +4. Continue viewing the node's parent and child nodes to trace upstream and downstream dependencies. | ||
| 233 | + | ||
| 234 | +For Loop display, pay additional attention to the following: | ||
| 235 | + | ||
| 236 | +1. If a Loop exists in the DAG, the page automatically overlays a Loop dashed box on the task graph. | ||
| 237 | +2. The dashed box covers the Loop's start, end, and internal nodes, making it easy to identify loop boundaries. | ||
| 238 | +3. Clicking the Loop capsule directly locates the corresponding Loop Start node. | ||
| 239 | +4. After selecting a Loop internal node, the right details panel shows the Loop information and nested chain for that node. | ||
| 240 | + | ||
| 241 | + | ||
| 242 | + | ||
| 243 | +--- | ||
| 244 | + | ||
| 245 | +## 9. Canvas Operations | ||
| 246 | + | ||
| 247 | +The DAGView canvas supports the following operations: | ||
| 248 | + | ||
| 249 | +| Operation | Description | | ||
| 250 | +|-----------|-------------| | ||
| 251 | +| Move canvas | Drag the blank area | | ||
| 252 | +| Zoom in / out | Use the mouse wheel, or click `+` / `-` in the bottom right | | ||
| 253 | +| Reset zoom | Click the percentage button in the bottom right | | ||
| 254 | +| Select node | Click the target node | | ||
| 255 | + | ||
| 256 | +When the graph is large, first reduce the rank selection range, then zoom into the local area to view dependencies. | ||
| 257 | + | ||
| 258 | +--- | ||
| 259 | + | ||
| 260 | +## 10. Viewing Node Details | ||
| 261 | + | ||
| 262 | +After clicking a DAG node, the right details panel displays node information. Common information includes: | ||
| 263 | + | ||
| 264 | +| Section | Description | | ||
| 265 | +|---------|-------------| | ||
| 266 | +| Node Overview | Node ID, task type, Rank, Stream, Queue | | ||
| 267 | +| Loop Info | The Loop the current node belongs to, Loop count, instruction range, Loop boundaries, etc. | | ||
| 268 | +| Parent Nodes | Upstream nodes the current node depends on | | ||
| 269 | +| Child Nodes | Downstream nodes that depend on the current node | | ||
| 270 | +| Node Semantics | Notify, Task metadata, Memory Slices, etc. | | ||
| 271 | +| Raw JSON | Complete raw information for the current node | | ||
| 272 | + | ||
| 273 | +When investigating dependency relationships, prioritize checking `Parent Nodes` and `Child Nodes`. When investigating loop structures, focus on `Loop Info`. When investigating data copy issues, focus on `Memory Slices`. | ||
| 274 | + | ||
| 275 | +The `Memory Slices` section has the following display rules: | ||
| 276 | + | ||
| 277 | +1. Normal memory slices display `rank / type / offset / size`. | ||
| 278 | +2. If the slice belongs to `MS_CCU`, Insight V3 preferentially displays the `MSID` instead of the underlying abstract offset. | ||
| 279 | +3. For batch tasks with multiple `MS_CCU` slices, they are automatically merged into a summary card. | ||
| 280 | +4. The summary card shows a description like `8 MSIDs used in total`. | ||
| 281 | +5. Expanding `MSID Details` shows each `MSID`'s corresponding `id` and `size`. | ||
| 282 | + | ||
| 283 | +--- | ||
| 284 | + | ||
| 285 | +## 11. Searching for Nodes | ||
| 286 | + | ||
| 287 | +The left `Search` panel can be used to quickly locate DAG nodes. | ||
| 288 | + | ||
| 289 | +Supported search fields: | ||
| 290 | + | ||
| 291 | +| Field | Use Case | | ||
| 292 | +|-------|----------| | ||
| 293 | +| `taskId` | Use when the node ID is known | | ||
| 294 | +| `taskType` | Search by task type, e.g., `TRANS_MEM` | | ||
| 295 | +| `notifyId` | Search by notify ID | | ||
| 296 | + | ||
| 297 | +Steps: | ||
| 298 | + | ||
| 299 | +1. Select a search field in the left search panel. | ||
| 300 | +2. Enter the complete keyword. | ||
| 301 | +3. Click the search result. | ||
| 302 | +4. DAGView automatically locates and selects the corresponding node. | ||
| 303 | + | ||
| 304 | +The current search is exact match. If no results are found, confirm the keyword is complete and that the target rank is selected. | ||
| 305 | + | ||
| 306 | +--- | ||
| 307 | + | ||
| 308 | +## 12. Recommended Analysis Flow | ||
| 309 | + | ||
| 310 | +We recommend the following DAGView analysis flow: | ||
| 311 | + | ||
| 312 | +1. Select a dataset on the `Overview` page. | ||
| 313 | +2. Select a small number of ranks in the left Rank tree. | ||
| 314 | +3. Click `Enter Correlation View`. | ||
| 315 | +4. View the overall nodes and dependency arrows in DAGView. | ||
| 316 | +5. Click nodes of interest and view details on the right. | ||
| 317 | +6. Trace upstream and downstream dependencies through parent and child nodes. | ||
| 318 | +7. Use the search function to quickly locate known nodes. | ||
| 319 | + | ||
| 320 | +--- | ||
| 321 | + | ||
| 322 | +## 13. Frequently Asked Questions | ||
| 323 | + | ||
| 324 | +### 13.1 No Data on the Page | ||
| 325 | + | ||
| 326 | +Confirm the Insight service address is correct and that Checker has generated Insight data. | ||
| 327 | + | ||
| 328 | +### 13.2 DAG is Empty After Entering Correlation Page | ||
| 329 | + | ||
| 330 | +Confirm the following files exist in the dataset: | ||
| 331 | + | ||
| 332 | +```text | ||
| 333 | +graph/graph.msgpack | ||
| 334 | +graph/layout.msgpack | ||
| 335 | +``` | ||
| 336 | + | ||
| 337 | +### 13.3 DAG Graph is Too Dense | ||
| 338 | + | ||
| 339 | +First reduce the number of selected ranks on the left, view only some ranks, then zoom into local areas for analysis. | ||
| 340 | + | ||
| 341 | +### 13.4 Cannot Find a Node via Search | ||
| 342 | + | ||
| 343 | +Confirm the search keyword is complete and that the target node's rank is selected. | ||
| @@ -0,0 +1,226 @@ | |||
| 1 | +# HVRM Insight User Guide | ||
| 2 | + | ||
| 3 | +## 1. Overview | ||
| 4 | + | ||
| 5 | +HVRM Insight is a distributed operator visualization analysis tool for unified display and linked analysis of graph structures, memory snapshots, and verification error information during the execution of HCCL collective communication operators. The tool includes three pages: | ||
| 6 | + | ||
| 7 | +| Page | Tab | Purpose | | ||
| 8 | +|------|-----|---------| | ||
| 9 | +| **Dashboard** | Overview | Browse datasets, select ranks, view statistics; the starting point for using the tool | | ||
| 10 | +| **MemView** | Correlation | Linked display of memory timeline and DAG task graph; the core analysis page for troubleshooting | | ||
| 11 | +| **Analytic** | Errors | Centralized display of verification errors, supporting one-click navigation to the problem site | | ||
| 12 | + | ||
| 13 | +--- | ||
| 14 | + | ||
| 15 | +## 2. Installation and Startup | ||
| 16 | + | ||
| 17 | +### 2.1 Prerequisites | ||
| 18 | + | ||
| 19 | +HVRM Insight depends on data files output by the Checker plugin. Before use, ensure Checker has been fully executed and the following two data output switches are enabled in the Checker configuration file: | ||
| 20 | + | ||
| 21 | +The Checker configuration file is located at `/path/to/hccl_vm_install/plugin/checker/manifest.json` by default. | ||
| 22 | + | ||
| 23 | +```json | ||
| 24 | +"setting": { | ||
| 25 | + "enable_insight_dump": true, | ||
| 26 | + "enable_memory_snapshot_dump": true | ||
| 27 | +} | ||
| 28 | +``` | ||
| 29 | + | ||
| 30 | +- **`enable_insight_dump`**: When enabled, Checker outputs the graph structure and verification result files required by Insight during execution. | ||
| 31 | +- **`enable_memory_snapshot_dump`**: When enabled, Checker outputs memory snapshot files for each rank during execution. | ||
| 32 | + | ||
| 33 | +Both switches default to `false`. If not enabled, Insight will not be able to read the complete analysis data. | ||
| 34 | + | ||
| 35 | +### 2.2 Data Directory Structure | ||
| 36 | + | ||
| 37 | +The tool reads datasets from the `data/` directory. Each dataset is an independent subdirectory: | ||
| 38 | + | ||
| 39 | +```text | ||
| 40 | +data/ | ||
| 41 | +└── <dataset_name>/ | ||
| 42 | + ├── manifest.json # Dataset metadata (required) | ||
| 43 | + ├── graph/ # Graph structure files (DAG) | ||
| 44 | + ├── memory/ # Memory snapshot files | ||
| 45 | + └── validation/ | ||
| 46 | + └── issues.msgpack # Verification error records (optional) | ||
| 47 | +``` | ||
| 48 | + | ||
| 49 | +When optional directories are missing, the corresponding page displays an empty state. | ||
| 50 | + | ||
| 51 | +### 2.3 Method 1: Via hccl-vm Plugin (Recommended) | ||
| 52 | + | ||
| 53 | +```bash | ||
| 54 | +# Install and start the backend service | ||
| 55 | +hccl-vm plugin install insight | ||
| 56 | +# After successful installation, the terminal outputs the access URL (default `http://localhost:8000`). Open it in your browser. | ||
| 57 | + | ||
| 58 | +# View the access URL again | ||
| 59 | +hccl-vm plugin run @insight | ||
| 60 | + | ||
| 61 | +# Uninstall the plugin (also stops the backend service) | ||
| 62 | +hccl-vm plugin uninstall @insight | ||
| 63 | +``` | ||
| 64 | + | ||
| 65 | +### 2.4 Method 2: Local Direct Startup | ||
| 66 | + | ||
| 67 | +```bash | ||
| 68 | +# Enter the insight plugin directory | ||
| 69 | +python3 serve.py | ||
| 70 | +``` | ||
| 71 | + | ||
| 72 | +The backend listens on `http://localhost:8000` by default, serving both the frontend page and the data API. | ||
| 73 | + | ||
| 74 | +### 2.5 Verify Successful Startup | ||
| 75 | + | ||
| 76 | +After opening the URL in the browser, if you can see the dataset list on the Dashboard page, startup was successful. If the page is blank, check whether the `data/` directory exists and contains the `manifest.json` file. | ||
| 77 | + | ||
| 78 | +--- | ||
| 79 | + | ||
| 80 | +## 3. Scenario Demonstrations | ||
| 81 | + | ||
| 82 | +### 3.1 Scenario 1: Select Dataset and View Overview (Dashboard) | ||
| 83 | + | ||
| 84 | +Browse all datasets, click to select a target dataset, view operator information and statistics, then enter the correlation analysis page. | ||
| 85 | + | ||
| 86 | + | ||
| 87 | + | ||
| 88 | +### 3.2 Scenario 2: Linked Analysis of Memory and Task Graph (MemView) | ||
| 89 | + | ||
| 90 | +Through the linked display of the memory timeline and DAG task graph, locate memory operations for a specific step, view node details, and trace data sources. | ||
| 91 | + | ||
| 92 | + | ||
| 93 | + | ||
| 94 | +### 3.3 Scenario 3: View Errors and Navigate to the Problem Site (Analytic) | ||
| 95 | + | ||
| 96 | +View error details on the Analytic page or the error list on the left side of MemView, then one-click navigate to the associated DAG node or memory context. | ||
| 97 | + | ||
| 98 | + | ||
| 99 | + | ||
| 100 | +--- | ||
| 101 | + | ||
| 102 | +## 4. Page Details | ||
| 103 | + | ||
| 104 | +### 4.1 Dashboard Page | ||
| 105 | + | ||
| 106 | +Dashboard is the starting point for using the tool. The page is divided into three areas from left to right: | ||
| 107 | + | ||
| 108 | +- **Left**: Operator summary and Rank tree. The Rank tree supports selection/deselection, and the selection is synchronized to all pages. | ||
| 109 | +- **Center**: Dataset list. Click a row to select a dataset; the left and right panels refresh simultaneously. | ||
| 110 | +- **Right**: Details panel for the selected dataset, showing statistics and navigation buttons. | ||
| 111 | + | ||
| 112 | + | ||
| 113 | + | ||
| 114 | +After selecting a dataset, detailed statistics are displayed on the right: | ||
| 115 | + | ||
| 116 | + | ||
| 117 | + | ||
| 118 | +The bottom of the details panel provides two navigation entry points: | ||
| 119 | + | ||
| 120 | +- **`Enter Correlation View`**: Navigate to MemView with the current dataset and rank selection. | ||
| 121 | +- **`View Diagnostics`**: Navigate to Analytic to view all verification errors for this dataset. | ||
| 122 | + | ||
| 123 | +### 4.2 MemView Page | ||
| 124 | + | ||
| 125 | +MemView displays the memory timeline and DAG task graph in the same view for linked analysis. It is the core page for troubleshooting. | ||
| 126 | + | ||
| 127 | + | ||
| 128 | + | ||
| 129 | +**Page Layout:** | ||
| 130 | + | ||
| 131 | +- **Left**: Rank selection tree, issue list, search panel. | ||
| 132 | +- **Top**: Memory timeline — shows the buffer state evolution for each rank by step. | ||
| 133 | +- **Bottom**: DAG task graph — displays task nodes and dependencies in Rank/Stream swimlanes. | ||
| 134 | +- **Right**: Details panel — shows detailed information for the selected step or node. | ||
| 135 | + | ||
| 136 | + | ||
| 137 | + | ||
| 138 | +#### 4.3 Core Coordination Mechanism: Step, Rank, and Task Interplay | ||
| 139 | + | ||
| 140 | +- Select a step on the timeline → DAG highlights the corresponding node synchronously. | ||
| 141 | +- Click a node in the DAG → the right panel switches to node details. | ||
| 142 | +- Click a navigation button on the right → the timeline locates the corresponding step synchronously. | ||
| 143 | + | ||
| 144 | +**The right details panel has two modes**, automatically switching based on whether a DAG node is selected: | ||
| 145 | + | ||
| 146 | +- **Step Detail Mode**: Displays the memory operation list (Task Memory Ops) and Buffer slice details (Buffer Layout) for the current step. Each slice in Buffer Layout can be expanded to view data source semantics, with click navigation support. | ||
| 147 | +- **Node Detail Mode**: Displays node attributes, parent/child node relationships, cross-stage node mapping, and memory operations associated with the node. | ||
| 148 | + | ||
| 149 | +**Other Features:** | ||
| 150 | + | ||
| 151 | +- **Bottom Player**: Supports browsing by step order, with a draggable slider for quick jumping. Playback speed is 1 second/step. | ||
| 152 | +- **Focus View**: When a step is selected, the DAG automatically enters a partial view showing only nodes related to the current step and their context. Suitable for narrowing down when nodes are dense. | ||
| 153 | +- **Subgraph Browsing**: When a node contains a subgraph, click `Open Subgraph` on the right to view a finer-grained task structure. | ||
| 154 | +- **Stage Switching**: Switch graph stages via the top dropdown (e.g., `input_graph` / `input_task_queues`). | ||
| 155 | +- **Search**: Supports searching DAG nodes by taskId / taskType / notifyId. Clicking a result locates the node. | ||
| 156 | + | ||
| 157 | +### 4.4 Analytic Page | ||
| 158 | + | ||
| 159 | +Analytic centrally displays all errors found during verification. The page is divided into three areas: | ||
| 160 | + | ||
| 161 | + | ||
| 162 | + | ||
| 163 | +- **Left**: Dataset and rank selection. | ||
| 164 | +- **Center**: Error list, each showing title, severity label (color-coded), key field summary, and original error code. | ||
| 165 | +- **Right**: Details panel for the selected error, including basic error information, involved ranks, associated node information, slice details (precise location of data inconsistencies), supplementary information, and raw JSON. | ||
| 166 | + | ||
| 167 | +The top of the details panel provides navigation buttons: | ||
| 168 | + | ||
| 169 | +- **`View memView`**: Navigate to MemView, locating the memory step associated with the error. | ||
| 170 | +- **`View dagView`**: Navigate to MemView, directly locating the DAG node associated with the error. | ||
| 171 | + | ||
| 172 | +Navigation automatically carries the dataset, rank, and node location information, so no re-selection is needed after arriving at MemView. | ||
| 173 | + | ||
| 174 | +--- | ||
| 175 | + | ||
| 176 | +## 5. Common Usage Flows | ||
| 177 | + | ||
| 178 | +### 5.1 Starting Analysis from Dashboard | ||
| 179 | + | ||
| 180 | +1. Click the target dataset in the dataset list. | ||
| 181 | +2. Select the ranks to view in the Rank tree on the left. | ||
| 182 | +3. Confirm the dataset's file count and error statistics on the right. | ||
| 183 | +4. Click `Enter Correlation View` to enter MemView. | ||
| 184 | + | ||
| 185 | +### 5.2 Locating Memory Issues in MemView | ||
| 186 | + | ||
| 187 | +1. Click the target step on the timeline (or use the bottom player to jump). | ||
| 188 | +2. View the step's memory operations in the right `Task Memory Ops` section. | ||
| 189 | +3. Select a rank and buffer type in `Buffer Layout`, expand a slice to view the data source. | ||
| 190 | +4. Click `Locate Task` or the semantic card to jump to the corresponding DAG node. | ||
| 191 | + | ||
| 192 | +### 5.3 Reverse Tracing Memory from a DAG Node | ||
| 193 | + | ||
| 194 | +1. Click the target node in the DAG. | ||
| 195 | +2. View the node's `Memory Ops` on the right. | ||
| 196 | +3. Click `Jump to Step` to return to the timeline, or click parent/child nodes to continue tracing along the dependency chain. | ||
| 197 | + | ||
| 198 | +### 5.4 Quick Location from Errors | ||
| 199 | + | ||
| 200 | +1. Click an error in the Analytic page or the error list on the left of MemView. | ||
| 201 | +2. View the associated node and slice information in the details. | ||
| 202 | +3. Click `View memView` or `View dagView` to jump to the problem site. | ||
| 203 | + | ||
| 204 | +--- | ||
| 205 | + | ||
| 206 | +## 6. Usage Tips | ||
| 207 | + | ||
| 208 | +- First confirm the dataset and rank range in Dashboard, then enter MemView for analysis. | ||
| 209 | +- When investigating memory issues, use `Task Memory Ops` and `Buffer Layout` together. | ||
| 210 | +- When investigating node dependency relationships, first select the node in the DAG, then use the parent/child relationship and node mapping on the right to navigate. | ||
| 211 | +- When DAG nodes are too dense, first use Focus View to narrow down, then expand gradually. | ||
| 212 | +- The error list on the left of MemView allows quick browsing and error node location without leaving the current page. | ||
| 213 | + | ||
| 214 | +--- | ||
| 215 | + | ||
| 216 | +## 7. Terminology | ||
| 217 | + | ||
| 218 | +| Term | Description | | ||
| 219 | +|------|-------------| | ||
| 220 | +| **Rank** | A single compute unit (e.g., NPU). Each rank independently executes computation | | ||
| 221 | +| **DAG** | Directed Acyclic Graph, describing dependency relationships between task nodes | | ||
| 222 | +| **Step** | A time step on the timeline, corresponding to a memory state snapshot | | ||
| 223 | +| **Buffer** | A data area in memory for storing computation or communication data | | ||
| 224 | +| **Stage** | A phase of the graph, such as `input_graph` or `input_task_queues` | | ||
| 225 | +| **Stream** | Task execution flow. Tasks within the same stream execute sequentially | | ||
| 226 | +| **Issue** | An error or anomaly record found during verification | | ||
| @@ -0,0 +1,1833 @@ | |||
| 1 | +# HCCL Simulator Runner Data Model Design | ||
| 2 | + | ||
| 3 | +## 1. Hardware-Software Resource Interaction Modeling | ||
| 4 | + | ||
| 5 | +Data flow interaction diagram. | ||
| 6 | + | ||
| 7 | +```mermaid | ||
| 8 | +sequenceDiagram | ||
| 9 | + participant HostThread as Host CPU (Runner) | ||
| 10 | + participant StreamQueue as Stream (Memory Queue) | ||
| 11 | + participant DeviceScheduler as Device Hardware (TS) | ||
| 12 | + participant EventMem as Event Status (Memory) | ||
| 13 | + | ||
| 14 | + Note over HostThread: 1. aclrtRecordEvent(evt1) | ||
| 15 | + HostThread->>StreamQueue: Push CMD: [Write Event1=Done] | ||
| 16 | + | ||
| 17 | + Note over HostThread: 2. aclrtStreamWaitEvent(evt1) | ||
| 18 | + HostThread->>StreamQueue: Push CMD: [Wait Event1==Done] | ||
| 19 | + | ||
| 20 | + Note over HostThread: 3. aclrtLaunchKernel(MatMul) | ||
| 21 | + HostThread->>StreamQueue: Push CMD: [Execute MatMul] | ||
| 22 | + | ||
| 23 | + Note over DeviceScheduler: Async execution phase (Device side) | ||
| 24 | + | ||
| 25 | + StreamQueue->>DeviceScheduler: Pop CMD: [Write Event1] | ||
| 26 | + DeviceScheduler->>EventMem: Update Status to DONE | ||
| 27 | + | ||
| 28 | + StreamQueue->>DeviceScheduler: Pop CMD: [Wait Event1] | ||
| 29 | + DeviceScheduler->>EventMem: Check Status? | ||
| 30 | + Note right of DeviceScheduler: Found DONE, pass!<br/>(If NotReady, hardware spins here waiting) | ||
| 31 | + | ||
| 32 | + StreamQueue->>DeviceScheduler: Pop CMD: [MatMul] | ||
| 33 | + DeviceScheduler->>DeviceScheduler: Start AI Core Computing... | ||
| 34 | +``` | ||
| 35 | + | ||
| 36 | +Everything placed in a Stream is executed by the Device hardware. | ||
| 37 | + | ||
| 38 | +## 2. Device, Context, Stream and Other Hardware Resource Relationship Modeling | ||
| 39 | + | ||
| 40 | +Relationship between Device, Context, Stream, and user host threads. | ||
| 41 | + | ||
| 42 | +```mermaid | ||
| 43 | +graph TD | ||
| 44 | + subgraph Server1[Server 1] | ||
| 45 | + Host1 | ||
| 46 | + Device | ||
| 47 | + Device2 | ||
| 48 | + end | ||
| 49 | + | ||
| 50 | + subgraph Server2[Server 2] | ||
| 51 | + Host3 | ||
| 52 | + Host2 | ||
| 53 | + Device3 | ||
| 54 | + Device4 | ||
| 55 | + | ||
| 56 | + end | ||
| 57 | + | ||
| 58 | + subgraph Host1[Host1] | ||
| 59 | + Runner1[runner<br>user thread 1] | ||
| 60 | + Runner2[runner<br>user thread 2] | ||
| 61 | + end | ||
| 62 | + | ||
| 63 | + subgraph Host2[Host2] | ||
| 64 | + Runner3[Runner...] | ||
| 65 | + end | ||
| 66 | + | ||
| 67 | + subgraph Host3[Device CPU<br>Edge computing/embedded: atlas 500] | ||
| 68 | + embedRunner[Embed Runner...] | ||
| 69 | + end | ||
| 70 | + | ||
| 71 | + subgraph Device[Device 1] | ||
| 72 | + Context1 | ||
| 73 | + end | ||
| 74 | + | ||
| 75 | + subgraph Context1[run-Context1] | ||
| 76 | + Stream1 | ||
| 77 | + end | ||
| 78 | + | ||
| 79 | + subgraph Stream1[ctx-Stream1] | ||
| 80 | + TaskKernel | ||
| 81 | + end | ||
| 82 | + | ||
| 83 | + subgraph TaskKernel[Task/Kernel] | ||
| 84 | + | ||
| 85 | + end | ||
| 86 | + | ||
| 87 | + subgraph Device2[Device 2] | ||
| 88 | + Context2 | ||
| 89 | + end | ||
| 90 | + | ||
| 91 | + subgraph Context2[run-Context2] | ||
| 92 | + Stream2 | ||
| 93 | + end | ||
| 94 | + | ||
| 95 | + subgraph Stream2[ctx-Stream2] | ||
| 96 | + Kernel | ||
| 97 | + end | ||
| 98 | + | ||
| 99 | + subgraph Kernel[Kernel] | ||
| 100 | + end | ||
| 101 | + | ||
| 102 | + subgraph Device3[Device 3] | ||
| 103 | + ctxM[...] | ||
| 104 | + end | ||
| 105 | + | ||
| 106 | + subgraph Device4[Device 4] | ||
| 107 | + ctxN[...] | ||
| 108 | + end | ||
| 109 | + | ||
| 110 | + Runner1-->Device | ||
| 111 | + Runner2-->Device | ||
| 112 | + Runner2-->Device2 | ||
| 113 | + Runner3-->Device3 | ||
| 114 | + Runner3-.->|???sdid|Device | ||
| 115 | + embedRunner-->Device4 | ||
| 116 | + | ||
| 117 | +``` | ||
| 118 | + | ||
| 119 | +### 2.1 Basic Device Relationship Modeling | ||
| 120 | + | ||
| 121 | +#### 2.1.1 Hierarchy Overview | ||
| 122 | + | ||
| 123 | +```mermaid | ||
| 124 | +erDiagram | ||
| 125 | + %% ========================================== | ||
| 126 | + %% Layer 1: Physical Topology Layer (Server -> Host/Device) | ||
| 127 | + %% ========================================== | ||
| 128 | + Server { | ||
| 129 | + typ server-id PK | ||
| 130 | + typ pod-id | ||
| 131 | + typ version | ||
| 132 | + } | ||
| 133 | + Host { | ||
| 134 | + typ host-id PK | ||
| 135 | + typ server-id FK | ||
| 136 | + typ ip | ||
| 137 | + typ arch | ||
| 138 | + } | ||
| 139 | + Device { | ||
| 140 | + typ device-id PK | ||
| 141 | + typ server-id FK | ||
| 142 | + typ logic-id "Sequence number of currently available devices" | ||
| 143 | + typ physical-id | ||
| 144 | + typ ccu-die-num "910D currently dual-die" | ||
| 145 | + typ overflow-mode | ||
| 146 | + typ status | ||
| 147 | + typ soc-version "A3" | ||
| 148 | + typ max-stream-cnt "1984" | ||
| 149 | + } | ||
| 150 | + Server ||--|{ Host : contains | ||
| 151 | + Server ||--o{ Device : contains | ||
| 152 | + | ||
| 153 | + %% ========================================== | ||
| 154 | + %% Layer 2: Process and Context Layer (Runner -> Context -> Stream) | ||
| 155 | + %% ========================================== | ||
| 156 | + Runner { | ||
| 157 | + typ run-id PK | ||
| 158 | + typ host-id FK | ||
| 159 | + typ pid | ||
| 160 | + typ timeout-config-ms | ||
| 161 | + typ current-ctx-id FK | ||
| 162 | + } | ||
| 163 | + Context { | ||
| 164 | + typ ctx-id PK | ||
| 165 | + typ run-id FK | ||
| 166 | + typ thread-id | ||
| 167 | + typ device-id FK | ||
| 168 | + typ is-default | ||
| 169 | + typ float-overflow-addr | ||
| 170 | + typ capture-mode | ||
| 171 | + } | ||
| 172 | + Stream { | ||
| 173 | + typ stream-id PK | ||
| 174 | + typ ctx-id FK | ||
| 175 | + typ sq-base-addr | ||
| 176 | + typ is-primary-default | ||
| 177 | + typ is-other-default | ||
| 178 | + typ priority | ||
| 179 | + typ schedule-strategy | ||
| 180 | + typ failure-mode | ||
| 181 | + typ user-tag | ||
| 182 | + typ overflow-switch | ||
| 183 | + typ activated | ||
| 184 | + typ capture-status | ||
| 185 | + typ task-complete-status | ||
| 186 | + } | ||
| 187 | + Host ||--o{ Runner : runs | ||
| 188 | + Runner ||--o{ Context : "creates/owns" | ||
| 189 | + Runner |o--o{ Context : "current activates" | ||
| 190 | + Context }o--|| Device : binds | ||
| 191 | + Context ||--|{ Stream : owns | ||
| 192 | + Device ||..|{ Stream : "hardware constraint" | ||
| 193 | + | ||
| 194 | + %% ========================================== | ||
| 195 | + %% Layer 3: Device Internal Resource Layer (Port/EndPoint/Ccu) | ||
| 196 | + %% ========================================== | ||
| 197 | + Port { | ||
| 198 | + typ port-id PK | ||
| 199 | + typ device-id FK | ||
| 200 | + typ die-id "die Id" | ||
| 201 | + typ status "0:unused/1:in use" | ||
| 202 | + typ name "0/0, 0/1" | ||
| 203 | + } | ||
| 204 | + Rank { | ||
| 205 | + typ id PK | ||
| 206 | + typ device-id FK | ||
| 207 | + typ rank-id | ||
| 208 | + typ commId | ||
| 209 | + } | ||
| 210 | + EndPoint { | ||
| 211 | + typ endpoint-id PK | ||
| 212 | + typ rank-id FK | ||
| 213 | + typ addr "IP address/EID" | ||
| 214 | + typ type "IPV4/IPV6/EID" | ||
| 215 | + } | ||
| 216 | + Ccu { | ||
| 217 | + typ ccu-id PK | ||
| 218 | + typ device-id FK | ||
| 219 | + typ resource-addr | ||
| 220 | + type die-id | ||
| 221 | + typ status | ||
| 222 | + } | ||
| 223 | + DeviceConnection { | ||
| 224 | + typ connection-id PK | ||
| 225 | + typ src-dev-id FK | ||
| 226 | + typ dst-dev-id FK | ||
| 227 | + typ link-type | ||
| 228 | + typ access-by-remote | ||
| 229 | + } | ||
| 230 | + Device ||--o{ Port : "has" | ||
| 231 | + Device ||--o{ Rank : "has" | ||
| 232 | + Device ||--o{ EndPoint : "has" | ||
| 233 | + Device ||--o{ Ccu : "1:2 dual-die" | ||
| 234 | + Device ||--o{ DeviceConnection : "peer access" | ||
| 235 | +``` | ||
| 236 | + | ||
| 237 | +#### 2.1.2 Network Communication Resources | ||
| 238 | + | ||
| 239 | +```mermaid | ||
| 240 | +erDiagram | ||
| 241 | + %% ========================================== | ||
| 242 | + %% Network Topology and Connection Layer | ||
| 243 | + %% ========================================== | ||
| 244 | + EndPoint { | ||
| 245 | + typ endpoint-id PK | ||
| 246 | + typ rank-id FK | ||
| 247 | + typ func_id "used by ccu" | ||
| 248 | + typ addr "IP address/EID" | ||
| 249 | + typ type "IPV4/IPV6/EID" | ||
| 250 | + } | ||
| 251 | + Port { | ||
| 252 | + typ port-id PK | ||
| 253 | + typ device-id FK | ||
| 254 | + typ die-id "die Id" | ||
| 255 | + typ name "0/0, 0/1" | ||
| 256 | + } | ||
| 257 | + EndPoint-Port-Mapping { | ||
| 258 | + typ mapping-id PK | ||
| 259 | + typ port-id FK | ||
| 260 | + typ endpoint-id FK | ||
| 261 | + } | ||
| 262 | + %% Physical connections defined by topo.json | ||
| 263 | + Link { | ||
| 264 | + typ link-id PK | ||
| 265 | + typ local-endpoint-id FK | ||
| 266 | + typ remote-endpoint-id FK | ||
| 267 | + typ net-layer "network layer" | ||
| 268 | + typ type "connection type" | ||
| 269 | + } | ||
| 270 | + Link-Protocol-Mapping { | ||
| 271 | + typ link-protocol-mapping-id PK | ||
| 272 | + typ link-id FK | ||
| 273 | + typ protocol "UB_CTP/UB_MEM/..." | ||
| 274 | + } | ||
| 275 | + %% | ||
| 276 | + EndPoint-Pair { | ||
| 277 | + typ endpoint-pair-id PK | ||
| 278 | + typ local-endpoint-id FK | ||
| 279 | + typ remote-endpoint-id FK | ||
| 280 | + } | ||
| 281 | + CcuChannel { | ||
| 282 | + typ ccu-channel-id PK | ||
| 283 | + typ channel-id FK "assigned by business" | ||
| 284 | + typ local-endpoint-id FK | ||
| 285 | + typ remote-endpoint-id FK | ||
| 286 | + typ protocol "communication protocol" | ||
| 287 | + typ src-jetty-start "jetty start Id" | ||
| 288 | + typ jetty-num "number of jetties" | ||
| 289 | + } | ||
| 290 | + | ||
| 291 | + EndPoint-Port-Mapping }|--|| Port : maps | ||
| 292 | + EndPoint-Port-Mapping }|--|| EndPoint : maps | ||
| 293 | + Link ||--o{ EndPoint : connects | ||
| 294 | + Link ||--|{ Link-Protocol-Mapping : "supports" | ||
| 295 | + EndPoint-Pair ||--|{ EndPoint : mapping | ||
| 296 | + EndPoint-Pair }o..|| Link : "based-on" | ||
| 297 | + CcuChannel ||--|{ EndPoint : uses | ||
| 298 | + CcuChannel }o..|| Link : "based-on" | ||
| 299 | +``` | ||
| 300 | + | ||
| 301 | +#### 2.1.3 Key Relationship Descriptions | ||
| 302 | + | ||
| 303 | +##### Layer 1: Physical Topology Layer | ||
| 304 | + | ||
| 305 | +| Relationship | Meaning | Notes | | ||
| 306 | +| --- | --- | --- | | ||
| 307 | +| Server → Host | A Server may contain multiple Host instances | e.g., multi-socket CPU or virtualized environment | | ||
| 308 | +| Server → Device | A Server contains multiple AI devices | Corresponds to /dev/davinci0, /dev/davinci1, etc. | | ||
| 309 | + | ||
| 310 | +##### Layer 2: Process and Context Layer | ||
| 311 | + | ||
| 312 | +| Relationship | Meaning | Notes | | ||
| 313 | +| --- | --- | --- | | ||
| 314 | +| Host → Runner | Multiple application threads (Runners) on each host | Each Runner can create multiple Contexts | | ||
| 315 | +| Runner → Context | Thread creates or switches to a different Context | Uses aclrtCreateContext() and aclrtSetCurrentContext() | | ||
| 316 | +| Context → Device | Context binds to a Device | Cannot cross devices once created | | ||
| 317 | +| Context → Stream | Each Context can create multiple Streams | Corresponds to aclrtCreateStream() | | ||
| 318 | +| Runner ↔ Context (current) | Current context activation state | aclrtGetCurrentContext(), aclrtSetCurrentContext() | | ||
| 319 | +| Device .. Stream | Resource upper limit constraint | Number of Streams is limited by hardware (max-stream-cnt) | | ||
| 320 | + | ||
| 321 | +##### Layer 3: Device Internal Resource Layer | ||
| 322 | + | ||
| 323 | +| Relationship | Meaning | Notes | | ||
| 324 | +| --- | --- | --- | | ||
| 325 | +| Device → Port | Device contains multiple communication ports | Used for network topology connections, format like "0/0, 0/1" | | ||
| 326 | +| Device → Rank | Device is associated with a communication Rank | Rank identifies a participating node in the communication domain | | ||
| 327 | +| Device → EndPoint | Device contains multiple endpoints | IP address/EID addressing identifiers, used for network communication | | ||
| 328 | +| Device → Ccu | Device contains multiple CCU units | 910D uses dual-die architecture, one CCU per die | | ||
| 329 | +| Device → DeviceConnection | Inter-device communication channels | aclrtDeviceCanAccessPeer(), aclrtDeviceEnablePeerAccess() | | ||
| 330 | + | ||
| 331 | +##### Network Communication Resource Layer | ||
| 332 | + | ||
| 333 | +| Relationship | Meaning | Notes | | ||
| 334 | +| --- | --- | --- | | ||
| 335 | +| EndPoint ↔ Port | Endpoint-to-port mapping | Many-to-many mapping via EndPoint-Port-Mapping | | ||
| 336 | +| Link → EndPoint | Physical connection associated endpoints | Defined by topo.json, describes physical topology | | ||
| 337 | +| Link → Link-Protocol-Mapping | Protocols supported by the connection | A Link can support multiple protocols (UB_CTP/UB_MEM, etc.) | | ||
| 338 | + | ||
| 339 | +#### 2.1.4 Corresponding Interface Mapping Table | ||
| 340 | + | ||
| 341 | +##### Basic and Device Layer (Device / Server) | ||
| 342 | + | ||
| 343 | +| Entity/Attribute | Key API Interface | | ||
| 344 | +| --------------------------- | ---------------------------------------------------------------------------------------------------- | | ||
| 345 | +| Server/Host | `aclInit`, `aclFinalize` | | ||
| 346 | +| Runner.pid | `rtDeviceGetBareTgid` | | ||
| 347 | +| Device | `rtGetDeviceCount` | | ||
| 348 | +| Device.logic-id | `rtSetDevice`,`rtResetDevice`,`rtGetDevice`,`rtsGetLogicDevIdByPhyDevId` | | ||
| 349 | +| Device.physical-id | `rtGetPhyDevIdByLogicDevId` | | ||
| 350 | +| Device.soc-name | `rtGetSocName` | | ||
| 351 | +| Device.overflow-mode | `rtSetDeviceSatMode`,`rtGetDeviceSatMode` | | ||
| 352 | +| DeviceConnection | `rtGetDevicesTopo`,`rtDeviceDisablePeerAccess`,`rtDeviceEnablePeerAccess`,`rtDevicePeerAccessStatus` | | ||
| 353 | +| Context.context-id | `rtCreateContext`,`rtDestroyContext` | | ||
| 354 | +| Context.is-default | `rtSetCurrentContext`,`rtGetCurrentContext` | | ||
| 355 | +| Context.float-overflow-addr | `rtCtxGetFloatOverflowAddr` | | ||
| 356 | +| Stream table | `aclrtGetStreamAvailableNum` | | ||
| 357 | +| Stream.stream-id | `rtCreateStream`,`rtCreateStreamWithConfig`,`rtDestroyStream`,`rtDestroyStreamForce` | | ||
| 358 | +| Stream.task-complete-status | `rtSynchronizeStream`, `rtSynchronizeStreamWithTimeout` | | ||
| 359 | +| Stream.activated | `rtStreamStop` | | ||
| 360 | +| Stream.failure-mode | `rtSetStreamAttribute`,`rtGetStreamAttribute` | | ||
| 361 | + | ||
| 362 | +### 2.2 Basic Memory Management Relationship Modeling | ||
| 363 | + | ||
| 364 | +```mermaid | ||
| 365 | +erDiagram | ||
| 366 | + PhyMemBlock ||--o{ VirtualPointerTable : "physical to virtual mapping" | ||
| 367 | + PhyMemBlock ||--o{ FdMemRecord : "file descriptor mapping" | ||
| 368 | + PhyMemBlock { | ||
| 369 | + typ phy-mem-id PK "auto-increment ID" | ||
| 370 | + typ device-id FK "0,1...or -1(host)" | ||
| 371 | + typ size | ||
| 372 | + typ type | ||
| 373 | + typ ref-count | ||
| 374 | + } | ||
| 375 | + | ||
| 376 | + VirtualPointerTable { | ||
| 377 | + typ start-ptr PK | ||
| 378 | + typ size | ||
| 379 | + typ context-id FK | ||
| 380 | + typ phy-mem-id FK | ||
| 381 | + typ owner-pid "creating process" | ||
| 382 | + typ source-type "" | ||
| 383 | + typ policy | ||
| 384 | + } | ||
| 385 | + | ||
| 386 | + VirtualPointerTable ||--o{ IpcMemRecord : "shared memory registration" | ||
| 387 | + IpcMemRecord { | ||
| 388 | + typ ipc-id PK | ||
| 389 | + typ vir_mem_id FK | ||
| 390 | + typ phy_mem_id FK | ||
| 391 | + typ name-or-key | ||
| 392 | + typ create-pid | ||
| 393 | + } | ||
| 394 | + | ||
| 395 | + IpcMemRecord ||--o{ IpcMemWhiteList : "process whitelist" | ||
| 396 | + IpcMemWhiteList { | ||
| 397 | + typ ipc-id FK | ||
| 398 | + typ pid | ||
| 399 | + typ create-pid | ||
| 400 | + } | ||
| 401 | + | ||
| 402 | + FdMemRecord { | ||
| 403 | + typ fd PK | ||
| 404 | + typ phy-mem-id FK | ||
| 405 | + typ name | ||
| 406 | + typ type | ||
| 407 | + } | ||
| 408 | + | ||
| 409 | + FdMemRecord ||--o{ FdMemWhiteList : "process whitelist" | ||
| 410 | + FdMemWhiteList { | ||
| 411 | + typ fd-id FK | ||
| 412 | + typ pid | ||
| 413 | + typ create-pid | ||
| 414 | + } | ||
| 415 | + | ||
| 416 | + %%VirtualPointerTable ||--o{ MemMapRecord : "mapping relationship" | ||
| 417 | + %%MemMapRecord { | ||
| 418 | + %% typ ptr FK | ||
| 419 | + %% typ phy-mem-id FK | ||
| 420 | + %%} | ||
| 421 | +``` | ||
| 422 | + | ||
| 423 | +#### 2.2.1 Key Relationship Descriptions (Basic Memory Management Relationship Modeling) | ||
| 424 | + | ||
| 425 | +1. **Physical memory is core**. | ||
| 426 | + `PhyMemBlock` serves as the base entity, associated with all other entities through `phy_mem_id`, reflecting Huawei Ascend's "physical memory pooling" design philosophy[3]. | ||
| 427 | +2. **Three-layer mapping system**: | ||
| 428 | + | ||
| 429 | + - Physical → Virtual (`VirtualPointerTable`) | ||
| 430 | + - Physical → IPC Shared (`IpcMemRecord`) | ||
| 431 | + - Physical → File Descriptor (`fdMemRecord`) | ||
| 432 | +3. **Security control**: | ||
| 433 | + `IpcMemWhiteList` implements secure sharing for Huawei HCCS (Huawei Collective Communication Service) through process PID whitelisting[3]. | ||
| 434 | +4. **Special mapping types**: | ||
| 435 | + `MemMapRecord` records dual virtual address mapping scenarios (e.g., mappings created by `aclrtMapMem`), supporting Huawei NPU's zero-copy data transfer[3]. | ||
| 436 | + | ||
| 437 | +#### 2.1.2 Corresponding Interface Mapping Table (Basic Memory Management Relationship Modeling) | ||
| 438 | + | ||
| 439 | +| Entity | Key Management Interface | | ||
| 440 | +| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| 441 | +| PhyMemBlock | `rtMallocPhysical`, `rtFreePhysical` | | ||
| 442 | +| VirtualPointerTable | `rtMallocWithCfg`,`rtMallocForTaskScheduler`,`rtMallocHostWithCfg`,`rtFree`,`rtReserveMemAddress`,`ReleaseMemAddress`, `rtMapMem`, `rtUnmapMem` | | ||
| 443 | +| VirtualPointerTable.context-id | `rtPointerGetAttributes` | | ||
| 444 | +| FdMemRecord.fd | `rtMemExportToShareableHandle`, `rtMemImportFromShareableHandle` | | ||
| 445 | +| FdMemWhiteList.pid | `rtMemSetPidToShareableHandle` | | ||
| 446 | +| IpcMemRecord.name-or-key | `rtIpcMemGetExportKey` | | ||
| 447 | +| IpcMemRecord.ipc-id | `rtIpcMemImportByKey`,`IpcMemClose` | | ||
| 448 | +| IpcMemWhiteList.pid | `rtIpcMemSetImportPid` | | ||
| 449 | +| MemMapRecord | `rtHostRegister`, `rtHostUnRegister` | | ||
| 450 | + | ||
| 451 | +## 3. Extending the Basic Model with a Data Task Model | ||
| 452 | + | ||
| 453 | +### 3.1 Data / Task Flow Modeling | ||
| 454 | + | ||
| 455 | +```mermaid | ||
| 456 | +erDiagram | ||
| 457 | + Context { | ||
| 458 | + typ ctx-id PK | ||
| 459 | + typ run-id FK | ||
| 460 | + } | ||
| 461 | + | ||
| 462 | + Stream { | ||
| 463 | + typ stream-id PK | ||
| 464 | + typ ctx-id FK | ||
| 465 | + typ state "Running/Idle" | ||
| 466 | + } | ||
| 467 | + | ||
| 468 | + %% Stream contains an ordered list of tasks | ||
| 469 | + Context ||--o{ Stream : "manages/submits" | ||
| 470 | + Stream ||--o{ Task : "queues [1..*]" | ||
| 471 | + | ||
| 472 | + Task { | ||
| 473 | + typ task-id PK | ||
| 474 | + typ stream-id FK | ||
| 475 | + typ seq-number "auto-increment within stream" | ||
| 476 | + typ type "Kernel/Memcpy/Callback" | ||
| 477 | + } | ||
| 478 | + | ||
| 479 | + %% Various specific Task types (logical inheritance relationship) | ||
| 480 | + MemcpyTask { | ||
| 481 | + typ task-id FK | ||
| 482 | + typ src-addr | ||
| 483 | + typ dst-addr | ||
| 484 | + typ size | ||
| 485 | + } | ||
| 486 | + | ||
| 487 | + %% Logical expression of inheritance (Task is divided into multiple types) | ||
| 488 | + Task ||--|{ MemcpyTask : "is a" | ||
| 489 | + | ||
| 490 | + %% MemcpyTask address should be addressable in VirtualPointerTable | ||
| 491 | + MemcpyTask }o..|{ VirtualPointerTable : "Range Constraint" | ||
| 492 | + VirtualPointerTable { | ||
| 493 | + typ start-ptr PK | ||
| 494 | + typ ctx-id FK | ||
| 495 | + } | ||
| 496 | + | ||
| 497 | + VirtualPointerTable }o--|| Context : "belongs to" | ||
| 498 | + | ||
| 499 | +``` | ||
| 500 | + | ||
| 501 | +### 3.2 CCU Resource Modeling | ||
| 502 | + | ||
| 503 | +An NPU device contains 2 CCUs, die0 and die1 respectively. | ||
| 504 | + | ||
| 505 | +```mermaid | ||
| 506 | +graph RL | ||
| 507 | + subgraph DavidDevice0[David 0] | ||
| 508 | + direction RL | ||
| 509 | + Memory0[Memory] | ||
| 510 | + David0Die0[Die0_ccu] | ||
| 511 | + David0Die1[Die1_ccu] | ||
| 512 | + end | ||
| 513 | + | ||
| 514 | + subgraph David0Die0[Die0_ccu] | ||
| 515 | + | ||
| 516 | + CcuBuf00[CcuBuf] | ||
| 517 | + Variable00[Variable] | ||
| 518 | + Notify00[Notify] | ||
| 519 | + CompletedEvent00[CompletedEvent] | ||
| 520 | + Local/Rmt-Addr00[Local/Rmt-Addr] | ||
| 521 | + end | ||
| 522 | + | ||
| 523 | + subgraph David0Die1[Die1_ccu] | ||
| 524 | + | ||
| 525 | + CcuBuf01[CcuBuf] | ||
| 526 | + Variable01[Variable] | ||
| 527 | + Notify01[Notify] | ||
| 528 | + CompletedEvent01[CompletedEvent] | ||
| 529 | + Local/Rmt-Addr01[Local/Rmt-Addr] | ||
| 530 | + end | ||
| 531 | + | ||
| 532 | + David0Die0---Memory0 | ||
| 533 | + David0Die1---Memory0 | ||
| 534 | + | ||
| 535 | +``` | ||
| 536 | + | ||
| 537 | +#### 3.2.1 Corresponding Interface Mapping Table (CCU Resource Modeling) | ||
| 538 | + | ||
| 539 | +| Entity | Key Management Interface | | ||
| 540 | +| ------------------ | -------------------------------------------------------- | | ||
| 541 | +| CcuBuf | `rtCcuBufAlloc`, `rtCcuBufFree`, `rtCcuBufGetAddr` | | ||
| 542 | +| Variable | `rtVariableCreate`, `rtVariableDestroy`, `rtVariableSet` | | ||
| 543 | +| Notify | `rtCreateNotify`, `rtDestroyNotify` | | ||
| 544 | +| CompletedEvent | `rtCreateEvent`, `rtDestroyEvent` | | ||
| 545 | +| Local/Rmt-Addr | `rtGetDeviceLocalAddr`, `rtGetDeviceRemoteAddr` | | ||
| 546 | +| CCU resource query | `rtGetCcudieInfo`, `rtGetCcudieNum` | | ||
| 547 | + | ||
| 548 | +#### 3.2.2 CCU Resource Lifecycle Description | ||
| 549 | + | ||
| 550 | +**CCU initialization flow**: | ||
| 551 | + | ||
| 552 | +1. When the Device starts, two CCUs (die0/die1) initialize automatically. | ||
| 553 | +2. Each CCU has its own independent CcuBuf, Variable, and Notify resource pools. | ||
| 554 | +3. CompletedEvent is used to notify task completion status. | ||
| 555 | + | ||
| 556 | +**Resource constraints**: | ||
| 557 | + | ||
| 558 | +- Each CCU has a limited number of CcuBuf entries (related to Device.soc-version) | ||
| 559 | +- Variable is used to store shared variables during communication. | ||
| 560 | +- Notify is used for cross-CCU synchronization notification mechanisms. | ||
| 561 | +- Local/Rmt-Addr is used for address translation during cross-die communication. | ||
| 562 | + | ||
| 563 | +### 3.3 Async / Sync Execution Modeling | ||
| 564 | + | ||
| 565 | +#### 3.3.1 [Notify Resource Management](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html) | ||
| 566 | + | ||
| 567 | +```mermaid | ||
| 568 | +erDiagram | ||
| 569 | + Device ||..o{ Notify : "Hardware Limit" | ||
| 570 | + Device ||--o{ Context : "referred by" | ||
| 571 | + Device { | ||
| 572 | + typ device-id PK | ||
| 573 | + typ device-type "A3" | ||
| 574 | + typ max-notify-cnt "8192" | ||
| 575 | + } | ||
| 576 | + Context { | ||
| 577 | + typ ctx-id PK | ||
| 578 | + typ device-id FK | ||
| 579 | + } | ||
| 580 | + | ||
| 581 | + Notify ||--o| IpcNotify : "is a" | ||
| 582 | + Notify o|--|| Context : "record" | ||
| 583 | + Notify { | ||
| 584 | + typ notify-id PK | ||
| 585 | + typ create-ctx-id FK | ||
| 586 | + typ device-notify-seq "0~8191" | ||
| 587 | + typ value "notify read/write register" | ||
| 588 | + } | ||
| 589 | + | ||
| 590 | + IpcNotify { | ||
| 591 | + typ ipc-id PK | ||
| 592 | + typ notify-id FK | ||
| 593 | + typ name-or-key | ||
| 594 | + typ create-pid | ||
| 595 | + } | ||
| 596 | + | ||
| 597 | + IpcNotify ||--o{ IpcNotifyVistorList : "has" | ||
| 598 | + IpcNotifyVistorList { | ||
| 599 | + typ ipc-id FK | ||
| 600 | + typ vistor-pid | ||
| 601 | + } | ||
| 602 | +``` | ||
| 603 | + | ||
| 604 | +##### Key Relationship Descriptions ([Notify Resource Management](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html)) | ||
| 605 | + | ||
| 606 | +**Notify and Device hardware constraints**: | ||
| 607 | + | ||
| 608 | +- Each Device has an upper limit of `max-notify-cnt` Notifies (e.g., 8192 for A3 chip) | ||
| 609 | +- `device-notify-seq` is the physical sequence number of a Notify within the Device (0~8191) | ||
| 610 | +- A Notify must specify its owning Context when created, and the Context binds to a specific Device. | ||
| 611 | + | ||
| 612 | +**Notify IPC sharing mechanism**: | ||
| 613 | + | ||
| 614 | +- `IpcNotify` allows cross-process sharing of Notify instances. | ||
| 615 | +- `name-or-key` is the sharing identifier, obtained via `rtNotifyGetExportKey`. | ||
| 616 | +- Other processes import and use it via `rtNotifyImportByKey`. | ||
| 617 | +- `IpcNotifyVistorList` records the process PIDs authorized to access this Notify. | ||
| 618 | + | ||
| 619 | +**Notify state management**: | ||
| 620 | + | ||
| 621 | +- The `value` field maps to hardware registers for read/write status. | ||
| 622 | +- `rtWaitAndResetNotify` waits for a Notify to become Ready and resets it. | ||
| 623 | +- Notify is used for inter-Stream synchronization and cross-process synchronization scenarios. | ||
| 624 | + | ||
| 625 | +##### Corresponding Interface Mapping Table ([Notify Resource Management](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html)) | ||
| 626 | + | ||
| 627 | +| Entity | Key Management Interface | | ||
| 628 | +| -------------------------- | --------------------------------------------------- | | ||
| 629 | +| Notify.notify-id | `rtCreateNotify`,`rtDestroyNotify`, `rtGetNotifyId` | | ||
| 630 | +| Notify.state | `lrtWaitAndResetNotify`, `rtWaitAndResetNotify` | | ||
| 631 | +| IpcNotify.name-or-key | `rtNotifyGetExportKey`,`rtNotifyImportByKey` | | ||
| 632 | +| IpcNotifyVistorList.ipc-id | `rtNotifySetImportPid` | | ||
| 633 | + | ||
| 634 | +#### 3.3.2 Notify Synchronization Control | ||
| 635 | + | ||
| 636 | +```mermaid | ||
| 637 | +erDiagram | ||
| 638 | + Context { | ||
| 639 | + typ ctx-id PK | ||
| 640 | + typ run-id FK | ||
| 641 | + } | ||
| 642 | + | ||
| 643 | + Stream { | ||
| 644 | + typ stream-id PK | ||
| 645 | + typ ctx-id FK | ||
| 646 | + typ state "Running/Idle" | ||
| 647 | + } | ||
| 648 | + | ||
| 649 | + %% Stream contains an ordered list of tasks | ||
| 650 | + Context ||--o{ Stream : "manages/submits" | ||
| 651 | + Stream ||--o{ Task : "queues [1..*]" | ||
| 652 | + | ||
| 653 | + Task { | ||
| 654 | + typ task-id PK | ||
| 655 | + typ stream-id FK | ||
| 656 | + typ seq-number "auto-increment within stream" | ||
| 657 | + typ type "Notify" | ||
| 658 | + } | ||
| 659 | + | ||
| 660 | + %% Various specific Task types (logical inheritance relationship) | ||
| 661 | + NotifyRecordTask { | ||
| 662 | + typ notify-id FK | ||
| 663 | + | ||
| 664 | + } | ||
| 665 | + | ||
| 666 | + NotifyWaitTask { | ||
| 667 | + typ notify-id FK | ||
| 668 | + } | ||
| 669 | + | ||
| 670 | + NotifyRecordTask }o..|| Notify : "use" | ||
| 671 | + NotifyWaitTask }o..|| Notify : "use" | ||
| 672 | + Notify { | ||
| 673 | + typ notify-id PK | ||
| 674 | + typ value | ||
| 675 | + } | ||
| 676 | + %% Logical expression of inheritance (Task is divided into multiple types) | ||
| 677 | + Task ||--|{ NotifyRecordTask : "is a" | ||
| 678 | + Task ||--|{ NotifyWaitTask : "is a" | ||
| 679 | +``` | ||
| 680 | + | ||
| 681 | +##### Key Relationship Descriptions (Notify Synchronization Control) | ||
| 682 | + | ||
| 683 | +**Notify task types**: | ||
| 684 | + | ||
| 685 | +- `NotifyRecordTask`: Sets the Notify state to Ready, indicating an event has completed. | ||
| 686 | +- `NotifyWaitTask`: Waits for the Notify state to become Ready, implementing inter-Stream synchronization. | ||
| 687 | + | ||
| 688 | +**Task execution order**: | ||
| 689 | + | ||
| 690 | +- NotifyRecordTask executes on StreamA, setting Notify to Ready. | ||
| 691 | +- NotifyWaitTask executes on StreamB, waiting for the same Notify. | ||
| 692 | +- Once Notify becomes Ready, subsequent tasks on StreamB can proceed. | ||
| 693 | + | ||
| 694 | +**Cross-Stream synchronization example**: | ||
| 695 | + | ||
| 696 | +```text | ||
| 697 | +StreamA: Task1 -> NotifyRecordTask(notify-id=1) -> Task2 | ||
| 698 | +StreamB: NotifyWaitTask(notify-id=1) -> Task3 | ||
| 699 | +// Task3 must wait for Task1 to complete before executing | ||
| 700 | +``` | ||
| 701 | + | ||
| 702 | +##### Corresponding Interface Mapping Table (Notify Synchronization Control) | ||
| 703 | + | ||
| 704 | +| Entity | Key Management Interface | | ||
| 705 | +| ---------------- | ----------------------- | | ||
| 706 | +| NotifyRecordTask | `rtRecordNotify` | | ||
| 707 | +| NotifyWaitTask | `lrtWaitAndResetNotify` | | ||
| 708 | + | ||
| 709 | +#### 3.3.3 Event Resource Management | ||
| 710 | + | ||
| 711 | +```mermaid | ||
| 712 | +erDiagram | ||
| 713 | + Device ||..|{ Event : "Hardware Limit" | ||
| 714 | + Device ||--o{ Context : "refered by" | ||
| 715 | + Device { | ||
| 716 | + typ device-id PK | ||
| 717 | + typ device-type "A3" | ||
| 718 | + typ max-event-cnt "65536" | ||
| 719 | + } | ||
| 720 | + | ||
| 721 | + Event }o..|| Context : "created by" | ||
| 722 | + Event { | ||
| 723 | + typ event-id PK | ||
| 724 | + typ created-ctx-id FK | ||
| 725 | + typ event-flag | ||
| 726 | + typ device-res-seq "0~65535" | ||
| 727 | + typ created-time | ||
| 728 | + typ status | ||
| 729 | + } | ||
| 730 | +``` | ||
| 731 | + | ||
| 732 | +##### Key Relationship Descriptions (Event Resource Management) | ||
| 733 | + | ||
| 734 | +**Event and Device hardware constraints**: | ||
| 735 | + | ||
| 736 | +- Each Device has an upper limit of `max-event-cnt` Events (e.g., 65536 for A3 chip) | ||
| 737 | +- `device-res-seq` is the physical sequence number of an Event within the Device (0~65535) | ||
| 738 | +- An Event must specify its owning Context when created, and the Context binds to a specific Device. | ||
| 739 | + | ||
| 740 | +**Event and Context relationship**: | ||
| 741 | + | ||
| 742 | +- `created-ctx-id` records the Context that created the Event. | ||
| 743 | +- An Event can be shared across multiple Streams, but must belong to the same Context. | ||
| 744 | +- Cross-Context Event sharing requires IPC mechanisms (similar to Notify) | ||
| 745 | + | ||
| 746 | +**Event state management**: | ||
| 747 | + | ||
| 748 | +- The `status` field indicates the current Event state: NotRecorded/Recorded/Completed. | ||
| 749 | +- `event-flag` controls Event behavior (e.g., auto-reset) | ||
| 750 | +- `created-time` is used for performance statistics. | ||
| 751 | + | ||
| 752 | +##### Corresponding Interface Mapping Table (Event Resource Management) | ||
| 753 | + | ||
| 754 | +| Entity | Key Management Interface | | ||
| 755 | +| -------------- | ------------------------------------------------------------------------ | | ||
| 756 | +| Event.event-id | `rtCreateEvent`, `rtCreateEventWithFlag`,`rtDestroyEvent`,`rtGetEventId` | | ||
| 757 | +| Event.status | `rtRecordEvent`,`rtQueryEventStatus` | | ||
| 758 | + | ||
| 759 | +#### 3.3.4 Event Flow Control | ||
| 760 | + | ||
| 761 | +```mermaid | ||
| 762 | +erDiagram | ||
| 763 | + Context { | ||
| 764 | + typ ctx-id PK | ||
| 765 | + typ run-id FK | ||
| 766 | + } | ||
| 767 | + | ||
| 768 | + Stream { | ||
| 769 | + typ stream-id PK | ||
| 770 | + typ ctx-id FK | ||
| 771 | + typ state "Running/Idle" | ||
| 772 | + } | ||
| 773 | + | ||
| 774 | + %% Stream contains an ordered list of tasks | ||
| 775 | + Context ||--o{ Stream : "manages/submits" | ||
| 776 | + Stream ||--o{ Task : "queues [1..*]" | ||
| 777 | + | ||
| 778 | + Task { | ||
| 779 | + typ task-id PK "auto-increment" | ||
| 780 | + typ stream-id FK | ||
| 781 | + typ seq-number "auto-increment within stream" | ||
| 782 | + typ type "EVENT" | ||
| 783 | + } | ||
| 784 | + | ||
| 785 | + %% Various specific Task types (logical inheritance relationship) | ||
| 786 | + EventTask { | ||
| 787 | + typ task-id FK | ||
| 788 | + typ event-id FK | ||
| 789 | + typ excute-time | ||
| 790 | + typ finish-time | ||
| 791 | + typ first-capture-taskid FK | ||
| 792 | + } | ||
| 793 | + | ||
| 794 | + EventRICaptureTask { | ||
| 795 | + typ updated-time | ||
| 796 | + } | ||
| 797 | + | ||
| 798 | + EventSyncTask { | ||
| 799 | + typ event-id FK | ||
| 800 | + typ excute-time | ||
| 801 | + typ finish-time | ||
| 802 | + typ op-timeout-s | ||
| 803 | + } | ||
| 804 | + EventRecordTask { | ||
| 805 | + typ event-id FK | ||
| 806 | + typ excute-time | ||
| 807 | + typ finish-time | ||
| 808 | + } | ||
| 809 | + EventWaitTask { | ||
| 810 | + typ event-id FK | ||
| 811 | + typ excute-time | ||
| 812 | + typ finish-time | ||
| 813 | + } | ||
| 814 | + EventTimeTask { | ||
| 815 | + typ event-id FK | ||
| 816 | + typ excute-time | ||
| 817 | + } | ||
| 818 | + EventTraceTask { | ||
| 819 | + typ event-id FK | ||
| 820 | + typ start-task-id FK | ||
| 821 | + } | ||
| 822 | + | ||
| 823 | + %% Logical expression of inheritance (Task is divided into multiple types) | ||
| 824 | + Task ||--|{ EventTask : "is a " | ||
| 825 | + EventTask ||--|{ EventRICaptureTask : "is a EXTERNAL" | ||
| 826 | + EventTask ||--|{ EventSyncTask : "is a EX" | ||
| 827 | + EventTask ||--|{ EventTimeTask : "is a EX" | ||
| 828 | + EventTask ||--|{ EventTraceTask : "is a EX" | ||
| 829 | + EventSyncTask ||--|{ EventRecordTask : "is a EX" | ||
| 830 | + EventSyncTask ||--|{ EventWaitTask : "is a EX" | ||
| 831 | +``` | ||
| 832 | + | ||
| 833 | +##### Key Relationship Descriptions (Event Flow Control) | ||
| 834 | + | ||
| 835 | +**Event task type classification**: | ||
| 836 | + | ||
| 837 | +- `EventRecordTask`: Sets the Event state to Recorded/Completed. | ||
| 838 | +- `EventWaitTask`: Waits for the Event state to become Completed. | ||
| 839 | +- `EventSyncTask`: Synchronously waits for Event completion (blocking call) | ||
| 840 | +- `EventRICaptureTask`: Special record task in RI Capture mode. | ||
| 841 | +- `EventTimeTask`: Timestamp-related task. | ||
| 842 | +- `EventTraceTask`: Task record for performance tracing. | ||
| 843 | + | ||
| 844 | +**Event task inheritance relationship**: | ||
| 845 | + | ||
| 846 | +- `EventTask` is the base class, containing task-id, event-id, execute-time, finish-time. | ||
| 847 | +- `EventSyncTask` inherits EventTask, adding an op-timeout-s timeout parameter. | ||
| 848 | +- `EventRecordTask` and `EventWaitTask` inherit EventSyncTask. | ||
| 849 | +- `EventRICaptureTask`, `EventTimeTask`, `EventTraceTask` inherit EventTask directly. | ||
| 850 | + | ||
| 851 | +**Event execution flow**: | ||
| 852 | + | ||
| 853 | +```text | ||
| 854 | +StreamA: KernelTask -> EventRecordTask(event-id=1) | ||
| 855 | +StreamB: EventWaitTask(event-id=1) -> KernelTask2 | ||
| 856 | +// StreamB's KernelTask2 must wait for StreamA's KernelTask to complete | ||
| 857 | +``` | ||
| 858 | + | ||
| 859 | +**Task trace relationship**: | ||
| 860 | + | ||
| 861 | +- `first-capture-task-id` records the Task ID of the first Capture. | ||
| 862 | +- `EventTraceTask.start-task-id` associates the tracking start task. | ||
| 863 | +- EventRecordTask achieves cross-Stream synchronization through EventWaitTask mapping. | ||
| 864 | + | ||
| 865 | +##### Corresponding Interface Mapping Table (Event Flow Control) | ||
| 866 | + | ||
| 867 | +| Entity | Key Management Interface | | ||
| 868 | +| --------------- | ---------------------------------------------------- | | ||
| 869 | +| EventTask | `rtRecordEvent`, `rtResetEvent`,`rtSynchronizeEvent` | | ||
| 870 | +| EventRecordTask | `rtRecordEvent`, `rtResetEvent` | | ||
| 871 | +| EventWaitTask | `rtStreamWaitEvent`, `rtQueryEventWaitStatus` | | ||
| 872 | +| EventTimeTask | `rtResetEvent`, `rtRecordEvent` | | ||
| 873 | +| EventTraceTask | `rtResetEvent`, `rtRecordEvent` | | ||
| 874 | + | ||
| 875 | +## 4. Communication Domain Modeling | ||
| 876 | + | ||
| 877 | +Cross-machine communication involves multi-communication-domain mixed task orchestration. | ||
| 878 | +The essence of a communication domain is a multi-card network topology maintained by HCCL at the framework layer through the link establishment capability provided by Rdma_Agent, stored in the host process. | ||
| 879 | + | ||
| 880 | +### 4.1 Core Concepts of Communication Domain | ||
| 881 | + | ||
| 882 | +A communicator is the basic abstraction of HCCL collective communication. Each communicator defines a set of Ranks participating in communication and their topological relationships. | ||
| 883 | + | ||
| 884 | +#### 4.1.1 Basic Communicator Definition | ||
| 885 | + | ||
| 886 | +```mermaid | ||
| 887 | +erDiagram | ||
| 888 | + %% ========================================== | ||
| 889 | + %% Communicator Definition (HCCL Communicator) | ||
| 890 | + %% ========================================== | ||
| 891 | + Communicator { | ||
| 892 | + typ comm-id PK "Communicator ID" | ||
| 893 | + typ run-id FK "Owning Runner process" | ||
| 894 | + typ world-size "Total number of ranks" | ||
| 895 | + typ my-rank "Current rank" | ||
| 896 | + typ color "Sub-communicator color identifier" | ||
| 897 | + typ new-comm-id FK "Derived new communicator" | ||
| 898 | + } | ||
| 899 | + Rank { | ||
| 900 | + typ rank-id PK "Rank number(0~world-size-1)" | ||
| 901 | + typ device-id FK "Bound device" | ||
| 902 | + typ comm-id FK "Owning communicator" | ||
| 903 | + } | ||
| 904 | + Runner ||--o{ Communicator : "creates/owns" | ||
| 905 | + Communicator ||--|{ Rank : "contains" | ||
| 906 | + Rank }o--|| Device : "binds" | ||
| 907 | + Communicator ||--o{ Communicator : "derives(MPI_Comm_split)" | ||
| 908 | +``` | ||
| 909 | + | ||
| 910 | +#### 4.1.2 Control Plane: Socket Communication | ||
| 911 | + | ||
| 912 | +```mermaid | ||
| 913 | +erDiagram | ||
| 914 | + %% ========================================== | ||
| 915 | + %% Socket Communication (RaSocket series interfaces) | ||
| 916 | + %% ========================================== | ||
| 917 | + Device ||--o{ RaSocket : creates | ||
| 918 | + RaSocket { | ||
| 919 | + typ socket-handle PK "Socket handle" | ||
| 920 | + typ rdev-handle FK "Owning RaDevice" | ||
| 921 | + typ state "LISTENING/CONNECTED/DISCONNECTED" | ||
| 922 | + typ role "SERVER/CLIENT" | ||
| 923 | + typ rank-id FK "Associated rank" | ||
| 924 | + typ peer-rank-id "Peer rank ID" | ||
| 925 | + } | ||
| 926 | + RaSocketPair { | ||
| 927 | + typ pair-id PK "Connection pair ID" | ||
| 928 | + typ client-socket-handle FK "Client socket" | ||
| 929 | + typ server-socket-handle FK "Server socket" | ||
| 930 | + typ connect-time "Connection establishment time" | ||
| 931 | + typ status "ACTIVE/CLOSED" | ||
| 932 | + } | ||
| 933 | + RaSocket ||--o| RaSocketPair : "participates in connection" | ||
| 934 | + RaSocketPair ||--o{ VirtualPointerTable : "associated communication memory" | ||
| 935 | + | ||
| 936 | + %% Socket event management (Epoll mechanism) | ||
| 937 | + RaSocketEvent { | ||
| 938 | + typ event-handle PK "Event handle" | ||
| 939 | + typ max-events "Maximum events" | ||
| 940 | + typ timeout "Timeout (ms)" | ||
| 941 | + } | ||
| 942 | + RaEpoll { | ||
| 943 | + typ epoll-id PK "Epoll ID" | ||
| 944 | + typ event-handle FK "Associated event handle" | ||
| 945 | + typ socket-handle FK "Monitored socket handle" | ||
| 946 | + typ events "Event types of interest" | ||
| 947 | + } | ||
| 948 | + RaSocketEvent ||--o{ RaEpoll : "manages" | ||
| 949 | + RaSocket ||--o{ RaEpoll : "monitored" | ||
| 950 | +``` | ||
| 951 | + | ||
| 952 | +#### 4.1.3 Data Plane: RDMA Communication | ||
| 953 | + | ||
| 954 | +```mermaid | ||
| 955 | +erDiagram | ||
| 956 | + %% ========================================== | ||
| 957 | + %% RDMA Devices and Resources (RaRdev, RaQp, RaMr interfaces) | ||
| 958 | + %% ========================================== | ||
| 959 | + Device ||--|{ RaDevice : "has virtual NIC" | ||
| 960 | + RaDevice { | ||
| 961 | + typ rdev-handle PK "RDMA device handle" | ||
| 962 | + typ device-id FK "Associated NPU Device" | ||
| 963 | + typ mac-addr "MAC address" | ||
| 964 | + typ ip-addr "IP address" | ||
| 965 | + typ state "UP/DOWN" | ||
| 966 | + typ port-num "Physical port number" | ||
| 967 | + typ link-speed "Link speed" | ||
| 968 | + typ mtu "Maximum transmission unit" | ||
| 969 | + } | ||
| 970 | + | ||
| 971 | + %% RDMA core: QP (Queue Pair) | ||
| 972 | + RaQP { | ||
| 973 | + typ qp-handle PK "QP handle" | ||
| 974 | + typ ra-dev-handle FK "Owning RaDevice" | ||
| 975 | + typ qp-num "QPN (Queue Pair Number)" | ||
| 976 | + typ type "RC/UC/UD" | ||
| 977 | + typ state "RESET/INIT/RTR/RTS/SQD/SQE/Error" | ||
| 978 | + typ peer-qpn "Peer QPN" | ||
| 979 | + typ send-cq-handle FK "Send completion queue" | ||
| 980 | + typ recv-cq-handle FK "Receive completion queue" | ||
| 981 | + typ srq-handle FK "Shared receive queue (optional)" | ||
| 982 | + } | ||
| 983 | + RaDevice ||--o{ RaQP : "owns QP" | ||
| 984 | + RaQP ||--|| RaCQ : "send_cq" | ||
| 985 | + RaQP ||--|| RaCQ : "recv_cq" | ||
| 986 | + RaQP |o--o| RaQP : "logical link" | ||
| 987 | + RaQP ||--o| RaSRQ : "uses shared RQ" | ||
| 988 | + | ||
| 989 | + %% Completion Queue CQ | ||
| 990 | + RaCQ { | ||
| 991 | + typ cq-handle PK "CQ handle" | ||
| 992 | + typ ra-dev-handle FK "Owning RaDevice" | ||
| 993 | + typ cqn "CQN" | ||
| 994 | + typ size "Queue depth" | ||
| 995 | + typ policy "CQ completion policy" | ||
| 996 | + } | ||
| 997 | + RaCQe { | ||
| 998 | + typ cqe-id PK "CQE ID" | ||
| 999 | + typ cq-handle FK "Owning CQ" | ||
| 1000 | + typ wr-id "Work Request ID" | ||
| 1001 | + typ status "SUCCESS/FLUSH_ERR/..." | ||
| 1002 | + typ opcode "SEND/RECV/READ/WRITE" | ||
| 1003 | + typ byte-len "Transfer bytes" | ||
| 1004 | + } | ||
| 1005 | + RaDevice ||--o{ RaCQ : "owns CQ" | ||
| 1006 | + RaCQ ||--o{ RaCQe : "contains" | ||
| 1007 | + | ||
| 1008 | + %% Memory Registration MR | ||
| 1009 | + RaMR { | ||
| 1010 | + typ mr-handle PK "MR handle" | ||
| 1011 | + typ ra-dev-handle FK "Owning RaDevice" | ||
| 1012 | + typ lkey "Local Key" | ||
| 1013 | + typ rkey "Remote Key" | ||
| 1014 | + typ addr "Start address" | ||
| 1015 | + typ length "Memory length (bytes)" | ||
| 1016 | + typ access "Access permissions" | ||
| 1017 | + } | ||
| 1018 | + RaDevice ||--o{ RaMR : "registers memory" | ||
| 1019 | + RaMR ||--|{ VirtualPointerTable : "maps to virtual memory" | ||
| 1020 | + | ||
| 1021 | + %% Shared Receive Queue SRQ | ||
| 1022 | + RaSRQ { | ||
| 1023 | + typ srq-handle PK "SRQ handle" | ||
| 1024 | + typ ra-dev-handle FK "Owning RaDevice" | ||
| 1025 | + typ srq-num "SRQN" | ||
| 1026 | + typ max-wr "Maximum WR count" | ||
| 1027 | + typ max-sge "Maximum SGE count" | ||
| 1028 | + } | ||
| 1029 | + RaDevice ||--o{ RaSRQ : "owns SRQ" | ||
| 1030 | + | ||
| 1031 | + %% NDA Direct Access | ||
| 1032 | + RaNdaCQ { | ||
| 1033 | + typ nda-cq-handle PK "NDA CQ handle" | ||
| 1034 | + typ rdma-handle FK "Owning RDMA handle" | ||
| 1035 | + typ cqn "CQN" | ||
| 1036 | + typ depth "Queue depth" | ||
| 1037 | + } | ||
| 1038 | + RaNdaQP { | ||
| 1039 | + typ nda-qp-handle PK "NDA QP handle" | ||
| 1040 | + typ rdma-handle FK "Owning RDMA handle" | ||
| 1041 | + typ qp-num "QPN" | ||
| 1042 | + typ nda-cq-handle FK "Associated NDA CQ" | ||
| 1043 | + } | ||
| 1044 | + RaDevice ||--o{ RaNdaCQ : "creates NDA CQ" | ||
| 1045 | + RaDevice ||--o{ RaNdaQP : "creates NDA QP" | ||
| 1046 | + RaNdaQP ||--|| RaNdaCQ : "uses" | ||
| 1047 | +``` | ||
| 1048 | + | ||
| 1049 | +#### 4.1.4 Data Plane: UB Unified Bus | ||
| 1050 | + | ||
| 1051 | +```mermaid | ||
| 1052 | +erDiagram | ||
| 1053 | + %% ========================================== | ||
| 1054 | + %% UB Context (RaContext series interfaces) | ||
| 1055 | + %% ========================================== | ||
| 1056 | + Device ||--o{ RaContext : creates | ||
| 1057 | + RaContext { | ||
| 1058 | + typ ctx-handle PK "UB context handle" | ||
| 1059 | + typ device-id FK "Associated device ID" | ||
| 1060 | + typ mode "Mode:RDMA/UB/UB_PLUS" | ||
| 1061 | + typ local-endpoint FK "Local endpoint, EID mapping" | ||
| 1062 | + typ max-jetty-num "Maximum Jetty count" | ||
| 1063 | + typ max-jfc-num "Maximum JFC count" | ||
| 1064 | + } | ||
| 1065 | + | ||
| 1066 | + EndPoint-Pair { | ||
| 1067 | + typ endpoint-pair-id PK | ||
| 1068 | + typ local-endpoint-id FK | ||
| 1069 | + typ remote-endpoint-id FK | ||
| 1070 | + } | ||
| 1071 | + | ||
| 1072 | + %% UB core resources | ||
| 1073 | + RaContext ||--o{ RaJetty : "creates Jetty" | ||
| 1074 | + RaContext ||--o{ RaJfc : "creates JFC" | ||
| 1075 | + RaContext ||--o{ RaLmem : "registers local memory" | ||
| 1076 | + RaContext ||--o{ RaRmem : "imports remote memory" | ||
| 1077 | + RaContext ||--o{ RaTp : "manages transport paths" | ||
| 1078 | + RaContext ||--o{ RaTokenId : "allocates TokenID" | ||
| 1079 | + RaContext ||--o{ RaChan : "creates channels" | ||
| 1080 | + RaContext ||--o{ EndPoint-Pair : "associates EndPoint-Pair" | ||
| 1081 | + | ||
| 1082 | + %% Jetty (QP equivalent) | ||
| 1083 | + RaJetty { | ||
| 1084 | + typ jetty-handle PK "Jetty handle" | ||
| 1085 | + typ ctx-handle FK "Owning UB context" | ||
| 1086 | + typ jetty-id "Jetty ID" | ||
| 1087 | + typ mode "URMA_NORMAL/CACHE_LOCK_DWQE/CCU/..." | ||
| 1088 | + typ sq-depth "Send queue depth" | ||
| 1089 | + typ rq-depth "Receive queue depth" | ||
| 1090 | + typ state "RESET/READY/SUSPENDED/ERROR" | ||
| 1091 | + typ peer-jetty-handle FK "Peer Jetty" | ||
| 1092 | + } | ||
| 1093 | + RaJetty ||--o| RaJfc : "send_jfc" | ||
| 1094 | + RaJetty ||--o| RaJfc : "recv_jfc" | ||
| 1095 | + RaJetty |o--o| RaJetty : "logical binding" | ||
| 1096 | + | ||
| 1097 | + %% JFC (CQ equivalent) | ||
| 1098 | + RaJfc { | ||
| 1099 | + typ jfc-handle PK "JFC handle" | ||
| 1100 | + typ ctx-handle FK "Owning UB context" | ||
| 1101 | + typ jfc-id "JFC ID" | ||
| 1102 | + typ depth "Queue depth" | ||
| 1103 | + typ mode "NORMAL/STARS_POLL/CCU_POLL" | ||
| 1104 | + typ policy "Completion policy" | ||
| 1105 | + } | ||
| 1106 | + RaCr { | ||
| 1107 | + typ cr-id PK "Completion request ID" | ||
| 1108 | + typ jfc-handle FK "Owning JFC" | ||
| 1109 | + typ status "SUCCESS/FLUSH_ERR/..." | ||
| 1110 | + typ opcode "SEND/RECV/READ/WRITE" | ||
| 1111 | + typ byte-len "Transfer bytes" | ||
| 1112 | + } | ||
| 1113 | + RaJfc ||--o{ RaCr : "contains" | ||
| 1114 | + | ||
| 1115 | + %% Local memory registration | ||
| 1116 | + RaLmem { | ||
| 1117 | + typ lmem-handle PK "Local memory handle" | ||
| 1118 | + typ ctx-handle FK "Owning UB context" | ||
| 1119 | + typ addr "Memory address" | ||
| 1120 | + typ size "Memory size (bytes)" | ||
| 1121 | + typ mem-key "Memory key" | ||
| 1122 | + typ token-id FK "Associated TokenID" | ||
| 1123 | + } | ||
| 1124 | + RaLmem ||--|{ VirtualPointerTable : "maps" | ||
| 1125 | + | ||
| 1126 | + %% Remote memory import | ||
| 1127 | + RaRmem { | ||
| 1128 | + typ rmem-handle PK "Remote memory handle" | ||
| 1129 | + typ ctx-handle FK "Owning UB context" | ||
| 1130 | + typ mem-key "Remote memory key" | ||
| 1131 | + typ target-seg-handle FK "Target segment handle" | ||
| 1132 | + typ remote-eid "Remote EID" | ||
| 1133 | + } | ||
| 1134 | + | ||
| 1135 | + %% Transport path | ||
| 1136 | + RaTp { | ||
| 1137 | + typ tp-handle PK "Transport path handle" | ||
| 1138 | + typ ctx-handle FK "Owning UB context" | ||
| 1139 | + typ tp-type "RTP/CTP/UTP" | ||
| 1140 | + typ tpn "Transport path number" | ||
| 1141 | + typ speed "Link speed" | ||
| 1142 | + typ status "UP/DOWN" | ||
| 1143 | + } | ||
| 1144 | + RaJetty ||--o{ RaTp : "uses" | ||
| 1145 | + | ||
| 1146 | + %% TokenID | ||
| 1147 | + RaTokenId { | ||
| 1148 | + typ token-handle PK "Token handle" | ||
| 1149 | + typ ctx-handle FK "Owning UB context" | ||
| 1150 | + typ token-id "Token ID" | ||
| 1151 | + typ ref-count "Reference count" | ||
| 1152 | + } | ||
| 1153 | + | ||
| 1154 | + %% | ||
| 1155 | + RaChan { | ||
| 1156 | + typ chan-handle PK "Channel handle" | ||
| 1157 | + typ ctx-handle FK "Owning UB context" | ||
| 1158 | + typ chan-id "Channel ID" | ||
| 1159 | + typ mode "Channel mode" | ||
| 1160 | + } | ||
| 1161 | +``` | ||
| 1162 | + | ||
| 1163 | +#### 4.1.5 Async Request Management | ||
| 1164 | + | ||
| 1165 | +```mermaid | ||
| 1166 | +erDiagram | ||
| 1167 | + AsyncRequest { | ||
| 1168 | + typ req-handle PK "Async request handle" | ||
| 1169 | + typ req-type "CONNECT/LISTEN/CLOSE/QP_CREATE/..." | ||
| 1170 | + typ status "PENDING/COMPLETED/FAILED" | ||
| 1171 | + typ submit-time "Submission time" | ||
| 1172 | + typ complete-time "Completion time" | ||
| 1173 | + } | ||
| 1174 | +``` | ||
| 1175 | + | ||
| 1176 | +#### 4.1.6 Communication Domain Architecture Summary | ||
| 1177 | + | ||
| 1178 | +```text | ||
| 1179 | +┌─────────────────────────────────────────────────────────────────┐ | ||
| 1180 | +│ HCCL Communication Domain Architecture │ | ||
| 1181 | +├─────────────────────────────────────────────────────────────────┤ | ||
| 1182 | +│ Application Layer │ | ||
| 1183 | +│ ┌──────────────────────────────────────────────────────────┐ │ | ||
| 1184 | +│ │ Communicator │ │ | ||
| 1185 | +│ │ └── Rank[0..N] (Participating nodes, each bound to a Device) │ | ||
| 1186 | +│ └──────────────────────────────────────────────────────────┘ │ | ||
| 1187 | +├─────────────────────────────────────────────────────────────────┤ | ||
| 1188 | +│ Control Plane (Link Establishment/Handshake) │ | ||
| 1189 | +│ ┌──────────────────────────────────────────────────────────┐ │ | ||
| 1190 | +│ │ RaSocket (Socket Communication) │ │ | ||
| 1191 | +│ │ ├── RaSocketPair (Connection pair) │ │ | ||
| 1192 | +│ │ └── RaEpoll (Event monitoring) │ │ | ||
| 1193 | +│ └──────────────────────────────────────────────────────────┘ │ | ||
| 1194 | +├─────────────────────────────────────────────────────────────────┤ | ||
| 1195 | +│ Data Plane (Data Transfer) │ | ||
| 1196 | +│ ┌─────────────────────┐ ┌─────────────────────┐ │ | ||
| 1197 | +│ │ RDMA (Traditional) │ │ UB (Unified Bus) │ │ | ||
| 1198 | +│ │ ├── RaDevice │ │ ├── RaContext │ │ | ||
| 1199 | +│ │ ├── RaQP (Queue Pair)│ │ ├── RaJetty (QP) │ │ | ||
| 1200 | +│ │ ├── RaCQ (Completion Queue)│ ├── RaJfc (CQ) │ │ | ||
| 1201 | +│ │ ├── RaMR (Memory Registration)│ ├── RaLmem/Rmem │ │ | ||
| 1202 | +│ │ └── RaSRQ (Shared RQ)│ │ └── RaTp (Transport Path) │ │ | ||
| 1203 | +│ └─────────────────────┘ └─────────────────────┘ │ | ||
| 1204 | +└─────────────────────────────────────────────────────────────────┘ | ||
| 1205 | +``` | ||
| 1206 | + | ||
| 1207 | +#### 4.1.7 Key Entity Comparison Table | ||
| 1208 | + | ||
| 1209 | +| Concept | RDMA Mode | UB Mode | Description | | ||
| 1210 | +| --- | --- | --- | --- | | ||
| 1211 | +| Context | RaDevice | RaContext | Device/context handle | | ||
| 1212 | +| Queue Pair | RaQP | RaJetty | Data transfer channel | | ||
| 1213 | +| Completion Queue | RaCQ | RaJfc | Completion notification | | ||
| 1214 | +| Completion Entry | RaCQe | RaCr | Completion status | | ||
| 1215 | +| Local Memory | RaMR | RaLmem | Memory registration | | ||
| 1216 | +| Remote Memory | - | RaRmem | Remote memory import | | ||
| 1217 | +| Transport Path | - | RaTp | Physical path management | | ||
| 1218 | +| Security Token | - | RaTokenId | Access control | | ||
| 1219 | + | ||
| 1220 | +#### 4.1.8 HCCP Interface Classification | ||
| 1221 | + | ||
| 1222 | +```text | ||
| 1223 | +HCCP Network API | ||
| 1224 | +├── Control Plane (Socket Communication) | ||
| 1225 | +| ├── Initialization: RaSocketInit/RaSocketDeinit (Socket) | ||
| 1226 | +│ ├── Connection Management: RaSocketBatchConnect/Close/Abort | ||
| 1227 | +│ ├── Listen Management: RaSocketListenStart/Stop | ||
| 1228 | +│ ├── Data Send/Receive: RaSocketSend/Recv | ||
| 1229 | +│ ├── Status Query: RaGetSockets | ||
| 1230 | +│ └── Event Management: RaEpollCtlAdd/Mod/Del | ||
| 1231 | +├── Data Plane - RDMA | ||
| 1232 | +| ├── Initialization: RaRdevInit/RaRdevDeinit (RDMA device) | ||
| 1233 | +│ ├── QP Management: RaQpCreate/Destroy/ConnectAsync | ||
| 1234 | +│ ├── CQ Management: RaCqCreate/Destroy | ||
| 1235 | +│ ├── MR Management: RaMrReg/Dereg | ||
| 1236 | +│ ├── Work Request: RaSendWr/RaRecvWrlist | ||
| 1237 | +│ └── Completion Polling: RaPollCq | ||
| 1238 | +├── Data Plane - UB | ||
| 1239 | +| ├── Initialization: RaCtxInit/RaCtxDeinit (Unified context) | ||
| 1240 | +│ ├── Jetty Management: RaCtxQpCreate/Destroy/Import/Bind | ||
| 1241 | +│ ├── JFC Management: RaCtxCqCreate/Destroy | ||
| 1242 | +│ ├── Memory Management: RaCtxLmemRegister/RmemImport | ||
| 1243 | +│ ├── Token Management: RaCtxTokenIdAlloc/Free | ||
| 1244 | +│ └── Work Request: RaBatchSendWr | ||
| 1245 | +├── Async Operations | ||
| 1246 | +│ ├── RaSocketBatchConnectAsync | ||
| 1247 | +│ ├── RaCtxQpCreateAsync/DestroyAsync | ||
| 1248 | +│ └── RaGetAsyncReqResult | ||
| 1249 | +├── Network Diagnostics | ||
| 1250 | +| ├── RaPingInit/RaPingDeinit (Ping) | ||
| 1251 | +│ ├── RaPingTargetAdd/Del | ||
| 1252 | +│ ├── RaPingTaskStart/Stop | ||
| 1253 | +│ └── RaPingGetResults | ||
| 1254 | +└── TLV Messages | ||
| 1255 | +| ├── RaTlvInit/RaTlvDeinit (TLV) | ||
| 1256 | + └── RaTlvRequest | ||
| 1257 | +``` | ||
| 1258 | + | ||
| 1259 | +##### Socket Communication Interfaces | ||
| 1260 | + | ||
| 1261 | +| Interface | Function | Key Parameters | | ||
| 1262 | +| ---------------------- | -------------- | ------------------------------------------------------------------------ | | ||
| 1263 | +| `RaSocketInit` | Socket initialization | `mode`, `rdevInfo`, `socketHandle` | | ||
| 1264 | +| `RaSocketDeinit` | Socket deinitialization | `socketHandle` | | ||
| 1265 | +| `RaSocketBatchConnect` | Batch connect | `SocketConnectInfoT[]`, `num` | | ||
| 1266 | +| `RaSocketBatchClose` | Batch close | `SocketCloseInfoT[]`, `num` | | ||
| 1267 | +| `RaSocketBatchAbort` | Batch abort | `SocketConnectInfoT[]`, `num` | | ||
| 1268 | +| `RaSocketListenStart` | Start listening | `SocketListenInfoT[]`, `num` | | ||
| 1269 | +| `RaSocketListenStop` | Stop listening | `SocketListenInfoT[]`, `num` | | ||
| 1270 | +| `RaGetSockets` | Get socket status | `role`, `SocketInfoT[]`, `num`, `connectedNum` | | ||
| 1271 | +| `RaSocketSend` | Send data | `fdHandle`, `data`, `size`, `sentSize` | | ||
| 1272 | +| `RaSocketRecv` | Receive data | `fdHandle`, `data`, `size`, `receivedSize` | | ||
| 1273 | +| `RaEpollCtlAdd` | Add epoll event | `fdHandle`, `event` | | ||
| 1274 | +| `RaEpollCtlMod` | Modify epoll event | `fdHandle`, `event` | | ||
| 1275 | +| `RaEpollCtlDel` | Delete epoll event | `fdHandle` | | ||
| 1276 | +| `RaCreateEventHandle` | Create event handle | `eventHandle` | | ||
| 1277 | +| `RaWaitEventHandle` | Wait for event | `eventHandle`, `SocketEventInfoT[]`, `timeout`, `maxevents`, `eventsNum` | | ||
| 1278 | +| `RaDestroyEventHandle` | Destroy event handle | `eventHandle` | | ||
| 1279 | +| `RaSocketWhiteListAdd` | Add whitelist | `socketHandle`, `SocketWlistInfoT[]`, `num` | | ||
| 1280 | +| `RaSocketWhiteListDel` | Delete whitelist | `socketHandle`, `SocketWlistInfoT[]`, `num` | | ||
| 1281 | + | ||
| 1282 | +###### RDMA Operation Interfaces | ||
| 1283 | + | ||
| 1284 | +| Interface | Function | Key Parameters | | ||
| 1285 | +| --------------------- | -------------- | ----------------------------------------------------------------------- | | ||
| 1286 | +| `RaRdevInit` | RDMA device initialization | `mode`, `notifyType`, `rdevInfo`, `rdmaHandle` | | ||
| 1287 | +| `RaRdevInitV2` | RDMA device init (extended) | `RdevInitInfo`, `rdevInfo`, `rdmaHandle` | | ||
| 1288 | +| `RaRdevInitWithBackup` | Init with backup | `initInfo`, `rdevInfo`, `backupRdevInfo` | | ||
| 1289 | +| `RaRdevDeinit` | RDMA device deinitialization | `rdmaHandle`, `notifyType` | | ||
| 1290 | +| `RaQpCreate` | Create QP | `rdevHandle`, `flag`, `qpMode`, `qpHandle` | | ||
| 1291 | +| `RaQpCreateWithAttrs` | Create QP (with attributes) | `rdevHandle`, `QpExtAttrs`, `qpHandle` | | ||
| 1292 | +| `RaAiQpCreate` | Create AI QP | `rdevHandle`, `QpExtAttrs`, `AiQpInfo`, `qpHandle` | | ||
| 1293 | +| `RaLoopbackQpCreate` | Create loopback QP | `rdevHandle`, `LoopbackQpPair`, `qpHandle` | | ||
| 1294 | +| `RaTypicalQpCreate` | Create typical QP | `rdevHandle`, `flag`, `qpMode`, `TypicalQp`, `qpHandle` | | ||
| 1295 | +| `RaQpDestroy` | Destroy QP | `qpHandle` | | ||
| 1296 | +| `RaQpConnectAsync` | Async connect QP | `qpHandle`, `fdHandle` | | ||
| 1297 | +| `RaGetQpStatus` | Get QP status | `qpHandle`, `status` | | ||
| 1298 | +| `RaTypicalQpModify` | Modify typical QP | `qpHandle`, `localQpInfo`, `remoteQpInfo` | | ||
| 1299 | +| `RaMrReg` | Register MR | `qpHandle`, `MrInfoT` | | ||
| 1300 | +| `RaMrDereg` | Deregister MR | `qpHandle`, `MrInfoT` | | ||
| 1301 | +| `RaRegisterMr` | Register MR (standalone) | `rdmaHandle`, `MrInfoT`, `mrHandle` | | ||
| 1302 | +| `RaDeregisterMr` | Deregister MR (standalone) | `rdmaHandle`, `mrHandle` | | ||
| 1303 | +| `RaRemapMr` | Remap MR | `rdmaHandle`, `MemRemapInfo[]`, `num` | | ||
| 1304 | +| `RaGetNotifyMrInfo` | Get notify MR info | `rdevHandle`, `MrInfoT` | | ||
| 1305 | +| `RaSendWr` | Send work request | `qpHandle`, `SendWr`, `SendWrRsp` | | ||
| 1306 | +| `RaSendWrV2` | Send work request V2 | `qpHandle`, `SendWrV2`, `SendWrRsp` | | ||
| 1307 | +| `RaSendWrlist` | Batch send | `qpHandle`, `SendWrlistData[]`, `SendWrRsp[]`, `sendNum`, `completeNum` | | ||
| 1308 | +| `RaRecvWrlist` | Batch receive | `qpHandle`, `RecvWrlistData`, `recvNum`, `completeNum` | | ||
| 1309 | +| `RaPollCq` | Poll CQ | `qpHandle`, `isSendCq`, `numEntries`, `wc` | | ||
| 1310 | +| `RaCqCreate` | Create CQ | `rdevHandle`, `CqAttr` | | ||
| 1311 | +| `RaCqDestroy` | Destroy CQ | `rdevHandle`, `CqAttr` | | ||
| 1312 | +| `RaCreateSrq` | Create SRQ | `rdmaHandle`, `SrqAttr` | | ||
| 1313 | +| `RaDestroySrq` | Destroy SRQ | `rdmaHandle`, `SrqAttr` | | ||
| 1314 | +| `RaSetQpAttrQos` | Set QP QoS | `qpHandle`, `QosAttr` | | ||
| 1315 | +| `RaSetQpAttrTimeout` | Set QP timeout | `qpHandle`, `timeout` | | ||
| 1316 | +| `RaSetQpAttrRetryCnt` | Set QP retry count | `qpHandle`, `retryCnt` | | ||
| 1317 | +| `RaGetQpAttr` | Get QP attributes | `qpHandle`, `QpAttr` | | ||
| 1318 | +| `RaGetQpContext` | Get QP context | `qpHandle`, `qp`, `sendCq`, `recvCq` | | ||
| 1319 | + | ||
| 1320 | +###### UB Unified Bus Interfaces | ||
| 1321 | + | ||
| 1322 | +| Interface | Function | Key Parameters | | ||
| 1323 | +| ----------------------- | ---------------- | ---------------------------------------------------------------- | | ||
| 1324 | +| `RaCtxInit` | Context initialization | `CtxInitCfg`, `CtxInitAttr`, `ctxHandle` | | ||
| 1325 | +| `RaCtxDeinit` | Context deinitialization | `ctxHandle` | | ||
| 1326 | +| `RaGetDevEidInfoNum` | Get EID count | `RaInfo`, `num` | | ||
| 1327 | +| `RaGetDevEidInfoList` | Get EID list | `RaInfo`, `HccpDevEidInfo[]`, `num` | | ||
| 1328 | +| `RaGetEidByIp` | Get EID by IP | `ctxHandle`, `IpInfo[]`, `HccpEid[]`, `num` | | ||
| 1329 | +| `RaGetDevBaseAttr` | Get device attributes | `ctxHandle`, `DevBaseAttr` | | ||
| 1330 | +| `RaCtxGetAsyncEvents` | Get async events | `ctxHandle`, `AsyncEvent[]`, `num` | | ||
| 1331 | +| `RaCtxTokenIdAlloc` | Allocate TokenID | `ctxHandle`, `HccpTokenId`, `tokenIdHandle` | | ||
| 1332 | +| `RaCtxTokenIdFree` | Free TokenID | `ctxHandle`, `tokenIdHandle` | | ||
| 1333 | +| `RaCtxLmemRegister` | Register local memory | `ctxHandle`, `MrRegInfoT`, `lmemHandle` | | ||
| 1334 | +| `RaCtxLmemUnregister` | Unregister local memory | `ctxHandle`, `lmemHandle` | | ||
| 1335 | +| `RaCtxRmemImport` | Import remote memory | `ctxHandle`, `MrImportInfoT`, `rmemHandle` | | ||
| 1336 | +| `RaCtxRmemUnimport` | Unimport remote memory | `ctxHandle`, `rmemHandle` | | ||
| 1337 | +| `RaCtxChanCreate` | Create channel | `ctxHandle`, `ChanInfoT`, `chanHandle` | | ||
| 1338 | +| `RaCtxChanDestroy` | Destroy channel | `ctxHandle`, `chanHandle` | | ||
| 1339 | +| `RaCtxCqCreate` | Create CQ | `ctxHandle`, `CqInfoT`, `cqHandle` | | ||
| 1340 | +| `RaCtxCqDestroy` | Destroy CQ | `ctxHandle`, `cqHandle` | | ||
| 1341 | +| `RaCtxQpCreate` | Create QP/Jetty | `ctxHandle`, `QpCreateAttr`, `QpCreateInfo`, `qpHandle` | | ||
| 1342 | +| `RaCtxQpQueryBatch` | Batch query QP | `qpHandle[]`, `JettyAttr[]`, `num` | | ||
| 1343 | +| `RaCtxQpDestroy` | Destroy QP/Jetty | `qpHandle` | | ||
| 1344 | +| `RaCtxQpImport` | Import Jetty | `ctxHandle`, `QpImportInfoT`, `remQpHandle` | | ||
| 1345 | +| `RaCtxQpUnimport` | Unimport Jetty | `ctxHandle`, `remQpHandle` | | ||
| 1346 | +| `RaCtxQpBind` | Bind Jetty | `qpHandle`, `remQpHandle` | | ||
| 1347 | +| `RaCtxQpUnbind` | Unbind Jetty | `qpHandle` | | ||
| 1348 | +| `RaBatchSendWr` | Batch send | `qpHandle`, `SendWrData[]`, `SendWrResp[]`, `num`, `completeNum` | | ||
| 1349 | +| `RaCtxUpdateCi` | Update CI | `qpHandle`, `ci` | | ||
| 1350 | +| `RaCtxGetAuxInfo` | Get auxiliary info | `ctxHandle`, `HccpAuxInfoIn`, `HccpAuxInfoOut` | | ||
| 1351 | +| `RaCtxGetCrErrInfoList` | Get CR errors | `ctxHandle`, `CrErrInfo[]`, `num` | | ||
| 1352 | + | ||
| 1353 | +###### Async Operation Interfaces | ||
| 1354 | + | ||
| 1355 | +| Interface | Function | Key Parameters | | ||
| 1356 | +| --------------------------- | -------------- | -------------------------------------------------------------------- | | ||
| 1357 | +| `RaGetAsyncReqResult` | Get async result | `reqHandle`, `reqResult` | | ||
| 1358 | +| `RaSocketBatchConnectAsync` | Async batch connect | `SocketConnectInfoT[]`, `num`, `reqHandle` | | ||
| 1359 | +| `RaSocketListenStartAsync` | Async start listening | `SocketListenInfoT[]`, `num`, `reqHandle` | | ||
| 1360 | +| `RaSocketListenStopAsync` | Async stop listening | `SocketListenInfoT[]`, `num`, `reqHandle` | | ||
| 1361 | +| `RaSocketBatchCloseAsync` | Async batch close | `SocketCloseInfoT[]`, `num`, `reqHandle` | | ||
| 1362 | +| `RaSocketSendAsync` | Async send | `fdHandle`, `data`, `size`, `sentSize`, `reqHandle` | | ||
| 1363 | +| `RaSocketRecvAsync` | Async receive | `fdHandle`, `data`, `size`, `receivedSize`, `reqHandle` | | ||
| 1364 | +| `RaCtxLmemRegisterAsync` | Async register memory | `ctxHandle`, `MrRegInfoT`, `lmemHandle`, `reqHandle` | | ||
| 1365 | +| `RaCtxLmemUnregisterAsync` | Async unregister memory | `ctxHandle`, `lmemHandle`, `reqHandle` | | ||
| 1366 | +| `RaCtxQpCreateAsync` | Async create QP | `ctxHandle`, `QpCreateAttr`, `QpCreateInfo`, `qpHandle`, `reqHandle` | | ||
| 1367 | +| `RaCtxQpDestroyAsync` | Async destroy QP | `qpHandle`, `reqHandle` | | ||
| 1368 | +| `RaCtxQpDestroyBatchAsync` | Async batch destroy | `ctxHandle`, `qpHandle[]`, `num`, `reqHandle` | | ||
| 1369 | +| `RaCtxQpImportAsync` | Async import Jetty | `ctxHandle`, `QpImportInfoT`, `remQpHandle`, `reqHandle` | | ||
| 1370 | +| `RaGetTpInfoListAsync` | Async get TP info | `ctxHandle`, `GetTpCfg`, `HccpTpInfo[]`, `num`, `reqHandle` | | ||
| 1371 | +| `RaGetEidByIpAsync` | Async get EID | `ctxHandle`, `IpInfo[]`, `HccpEid[]`, `num`, `reqHandle` | | ||
| 1372 | +| `RaGetTpAttrAsync` | Async get TP attributes | `ctxHandle`, `tpHandle`, `attrBitmap`, `TpAttr`, `reqHandle` | | ||
| 1373 | +| `RaSetTpAttrAsync` | Async set TP attributes | `ctxHandle`, `tpHandle`, `attrBitmap`, `TpAttr`, `reqHandle` | | ||
| 1374 | + | ||
| 1375 | +###### Network Diagnostic Interfaces | ||
| 1376 | + | ||
| 1377 | +| Interface | Function | Key Parameters | | ||
| 1378 | +| ------------------ | ------------ | --------------------------------------------| | ||
| 1379 | +| `RaPingInit` | Ping initialization | `PingInitAttr`, `PingInitInfo`, `pingHandle`| | ||
| 1380 | +| `RaPingDeinit` | Ping deinitialization | `pingHandle` | | ||
| 1381 | +| `RaPingTargetAdd` | Add ping target | `pingHandle`, `PingTargetInfo[]`, `num` | | ||
| 1382 | +| `RaPingTargetDel` | Delete ping target | `pingHandle`, `PingTargetCommInfo[]`, `num` | | ||
| 1383 | +| `RaPingTaskStart` | Start ping task | `pingHandle`, `PingTaskAttr` | | ||
| 1384 | +| `RaPingTaskStop` | Stop ping task | `pingHandle` | | ||
| 1385 | +| `RaPingGetResults` | Get ping results | `pingHandle`, `PingTargetResult[]`, `num` | | ||
| 1386 | + | ||
| 1387 | +###### TLV Message Interfaces | ||
| 1388 | + | ||
| 1389 | +| Interface | Function | Key Parameters | | ||
| 1390 | +| -------------- | ----------- | --------------------------------------------- | | ||
| 1391 | +| `RaTlvInit` | TLV initialization | `TlvInitInfo`, `bufferSize`, `tlvHandle` | | ||
| 1392 | +| `RaTlvDeinit` | TLV deinitialization | `tlvHandle` | | ||
| 1393 | +| `RaTlvRequest` | TLV request processing | `tlvHandle`, `moduleType`, `TlvMsg`, `TlvMsg` | | ||
| 1394 | + | ||
| 1395 | +###### NDA (Network Direct Access) Direct Access Interfaces | ||
| 1396 | + | ||
| 1397 | +| Interface | Function | Key Parameters | | ||
| 1398 | +| -------------------- | ---------------- | ------------------------------------------------------ | | ||
| 1399 | +| `RaNdaGetDirectFlag` | Get direct access flag | `rdmaHandle`, `directFlag` | | ||
| 1400 | +| `RaNdaCqCreate` | Create NDA CQ | `rdmaHandle`, `NdaCqInitAttr`, `NdaCqInfo`, `cqHandle` | | ||
| 1401 | +| `RaNdaCqDestroy` | Destroy NDA CQ | `rdmaHandle`, `cqHandle` | | ||
| 1402 | +| `RaNdaQpCreate` | Create NDA QP | `rdmaHandle`, `NdaQpInitAttr`, `NdaQpInfo`, `qpHandle` | | ||
| 1403 | + | ||
| 1404 | +###### General Query Interfaces | ||
| 1405 | + | ||
| 1406 | +| Interface | Function | Key Parameters | | ||
| 1407 | +| ------------------------ | -------------- | ----------------------------------------------| | ||
| 1408 | +| `RaGetIfnum` | Get interface count | `RaGetIfattr`, `num` | | ||
| 1409 | +| `RaGetIfaddrs` | Get interface addresses | `RaGetIfattr`, `InterfaceInfo[]`, `num` | | ||
| 1410 | +| `RaSocketGetVnicIpInfos` | Get virtual NIC IP | `phyId`, `IdType`, `ids[]`, `num`, `IpInfo[]` | | ||
| 1411 | +| `RaGetTlsEnable` | Get TLS status | `RaInfo`, `tlsEnable` | | ||
| 1412 | +| `RaGetHccnCfg` | Get HCCN configuration | `RaInfo`, `HccnCfgKey`, `value`, `valueLen` | | ||
| 1413 | +| `RaGetInterfaceVersion` | Get interface version | `phyId`, `interfaceOpcode`, `interfaceVersion` | | ||
| 1414 | +| `RaRdevGetHandle` | Get Rdev handle | `phyId`, `rdmaHandle` | | ||
| 1415 | +| `RaRdevGetSupportLite` | Get Lite support | `rdmaHandle`, `supportLite` | | ||
| 1416 | +| `RaSaveSnapshot` | Save snapshot | `RaInfo`, `SaveSnapshotAction` | | ||
| 1417 | +| `RaRestoreSnapshot` | Restore snapshot | `RaInfo` | | ||
| 1418 | +| `RaGetSecRandom` | Get secure random | `RaInfo`, `value` | | ||
| 1419 | + | ||
| 1420 | +##### Key Relationship Descriptions (HCCP Interface Classification) | ||
| 1421 | + | ||
| 1422 | +**Communication domain hierarchy**: | ||
| 1423 | + | ||
| 1424 | +- `Communicator` is the core abstraction of HCCL collective communication, defining a set of Ranks participating in communication. | ||
| 1425 | +- `Rank` is a participating node in the communication domain, each Rank bound to a specific Device. | ||
| 1426 | +- A parent communicator derives child communicators (e.g., grouping via the `color` attribute) | ||
| 1427 | + | ||
| 1428 | +**Control plane and data plane separation**: | ||
| 1429 | + | ||
| 1430 | +- **Control Plane (Socket)**: Used for link establishment, QP information exchange, control signaling, based on TCP protocol. | ||
| 1431 | +- **Data Plane (RDMA/UB)**: Used for high-performance data transfer, based on RDMA Verbs or UB protocol. | ||
| 1432 | + | ||
| 1433 | +**RDMA resource hierarchy**: | ||
| 1434 | + | ||
| 1435 | +- `RaDevice` is a virtual NIC abstraction; one Device can create multiple RaDevices. | ||
| 1436 | +- `RaQP` is a queue pair, containing a Send Queue and a Receive Queue. | ||
| 1437 | +- `RaCQ` is a completion queue, used for polling WR completion status. | ||
| 1438 | +- `RaMR` is memory registration, mapping virtual memory to RDMA-accessible physical memory. | ||
| 1439 | +- `RaSRQ` is a shared receive queue; multiple QPs can share the same SRQ to improve resource utilization. | ||
| 1440 | + | ||
| 1441 | +**UB resource hierarchy**: | ||
| 1442 | + | ||
| 1443 | +- `RaContext` is the UB unified context, replacing RaDevice as the device abstraction. | ||
| 1444 | +- `RaJetty` is the QP equivalent, supporting multiple modes (URMA_NORMAL/CCU, etc.) | ||
| 1445 | +- `RaJfc` is the CQ equivalent, used for completion request management. | ||
| 1446 | +- `RaLmem/RaRmem` are local/remote memory management, replacing RaMR. | ||
| 1447 | +- `RaTp` is transport path management, supporting RTP/CTP/UTP types. | ||
| 1448 | +- `RaTokenId` is a secure communication token for cross-process memory access control. | ||
| 1449 | + | ||
| 1450 | +**Entity association highlights**: | ||
| 1451 | + | ||
| 1452 | +1. `RaSocketPair` requires two `RaSocket`s (client and server) to establish a connection. | ||
| 1453 | +2. `RaQP.state` must go through the state transition RESET→INIT→RTR→RTS for normal communication. | ||
| 1454 | +3. `RaMR.lkey` is used for local access, `RaMR.rkey` for remote RDMA access. | ||
| 1455 | +4. `RaJetty` establishes a logical connection with the peer Jetty through the `Bind` operation. | ||
| 1456 | +5. `RaLmem` must be registered before `RaRmem` can import and access it from the peer. | ||
| 1457 | + | ||
| 1458 | +**NDA (Network Direct Access) mechanism**: | ||
| 1459 | + | ||
| 1460 | +- `RaNdaQP` and `RaNdaCQ` are QP/CQ variants for network direct access. | ||
| 1461 | +- NDA mode allows bypassing parts of the protocol stack, reducing latency. | ||
| 1462 | +- `RaNdaGetDirectFlag` checks whether the device supports NDA mode. | ||
| 1463 | + | ||
| 1464 | +**Async request management**: | ||
| 1465 | + | ||
| 1466 | +- `AsyncRequest` uniformly manages all async operation request handles. | ||
| 1467 | +- Async operations include: connection, listening, QP create/destroy, memory registration, etc. | ||
| 1468 | +- `RaGetAsyncReqResult` polls async operation results. | ||
| 1469 | + | ||
| 1470 | +**Socket event mechanism**: | ||
| 1471 | + | ||
| 1472 | +- `RaSocketEvent` is the handle for the event waiting mechanism. | ||
| 1473 | +- `RaEpoll` implements an event monitoring mechanism similar to Linux Epoll. | ||
| 1474 | +- Supports adding, modifying, and deleting monitored socket events. | ||
| 1475 | + | ||
| 1476 | +**Network interfaces and configuration**: | ||
| 1477 | + | ||
| 1478 | +- `InterfaceInfo` describes network interface IP/MAC/MTU and other attributes. | ||
| 1479 | +- `HccnConfig` stores HCCN network configuration key-value pairs. | ||
| 1480 | +- `Snapshot` supports saving and restoring device state. | ||
| 1481 | + | ||
| 1482 | +##### Control Plane vs Data Plane | ||
| 1483 | + | ||
| 1484 | +| Dimension | Control Plane | Data Plane | | ||
| 1485 | +| ------------ | ------------------------------------------------------------- | ------------------------------------------ | | ||
| 1486 | +| **Core function** | Link establishment, QP info exchange, control signaling | Data transfer, RDMA operations | | ||
| 1487 | +| **Key entities** | RaSocket, SocketConnection, RaEpoll | RaQP, RaCQ, RaMR, RaJetty | | ||
| 1488 | +| **Key interfaces** | `RaSocketBatchConnect`, `RaSocketListenStart`, `RaGetSockets` | `RaSendWr`, `RaPollCq`, `RaQpConnectAsync` | | ||
| 1489 | +| **Communication method** | TCP Socket | RDMA Verbs / UB | | ||
| 1490 | + | ||
| 1491 | +##### RDMA Mode vs UB Mode | ||
| 1492 | + | ||
| 1493 | +| Comparison | RDMA Mode | UB Mode | | ||
| 1494 | +| ------------ | ---------------- | ---------------------- | | ||
| 1495 | +| **Device abstraction** | RaDevice | RaContext | | ||
| 1496 | +| **Queue pair** | RaQP (QP) | RaJetty (Jetty) | | ||
| 1497 | +| **Completion queue** | RaCQ (CQ) | RaJfc (JFC) | | ||
| 1498 | +| **Memory registration** | RaMR (lkey/rkey) | RaLmem/RaRmem (MemKey) | | ||
| 1499 | +| **Address identifier** | IP + GID | EID (Endpoint ID) | | ||
| 1500 | +| **Transport path** | QPN + GID | RaTp (TPN) | | ||
| 1501 | + | ||
| 1502 | +##### Key Point Descriptions | ||
| 1503 | + | ||
| 1504 | +1. **RaContext/RaDevice**: Unified context entity, supports both RDMA and UB modes. | ||
| 1505 | +2. **RaJetty/RaJfc/RaQP/RaCQ**: QP/CQ equivalents in UB mode. | ||
| 1506 | +3. **RaLmem/RaRmem/RaMR**: Local/remote memory management in UB mode. | ||
| 1507 | +4. **RaTp**: UB transport path management. | ||
| 1508 | +5. **RaTokenId**: Secure communication token. | ||
| 1509 | + | ||
| 1510 | +##### Key Attribute Supplements | ||
| 1511 | + | ||
| 1512 | +- **QP Mode**: `NOR` (normal), `GDR_TMPL` (template), `OP` (operation), `GDR_ASYN` (async GDR) | ||
| 1513 | +- **Transport Mode**: `RC` (reliable connection), `RM` (reliable message, UB only) | ||
| 1514 | +- **Jetty Mode**: `URMA_NORMAL`, `CACHE_LOCK_DWQE`, `CCU`, `USER_CTL_NORMAL` | ||
| 1515 | +- **JFC Mode**: `NORMAL`, `STARS_POLL`, `CCU_POLL` | ||
| 1516 | + | ||
| 1517 | +##### Corresponding Interface Mapping Table (HCCP Interface Classification) | ||
| 1518 | + | ||
| 1519 | +| Entity | Initialization Interface | Creation Interface | Operation Interface | Destroy/Cleanup Interface | | ||
| 1520 | +| --------------- | ---------------------------- | -------------------------------------------------------- | ------------------------------------------ | ------------------------------------------ | | ||
| 1521 | +| **Communicator** | - | `HcclCommInitRankInfo`, `HcclCommInitClusterInfo` | `HcclGetRankId`, `HcclGetRankSize` | `HcclCommDestroy` | | ||
| 1522 | +| **Rank** | - | - | - | - | | ||
| 1523 | +| **RaSocket** | `RaSocketInit` | `RaSocketBatchConnect` | `RaSocketSend`, `RaSocketRecv`, `RaGetSockets` | `RaSocketDeinit`, `RaSocketBatchClose`, `RaSocketBatchAbort` | | ||
| 1524 | +| **RaSocketPair** | - | `RaSocketBatchConnect` | `RaGetSockets` | `RaSocketBatchClose` | | ||
| 1525 | +| **RaSocketEvent** | `RaCreateEventHandle` | - | `RaWaitEventHandle` | `RaDestroyEventHandle` | | ||
| 1526 | +| **RaEpoll** | - | `RaEpollCtlAdd` | `RaEpollCtlMod` | `RaEpollCtlDel` | | ||
| 1527 | +| **RaDevice** | `RaRdevInit`, `RaRdevInitV2`, `RaRdevInitWithBackup` | - | `RaRdevGetHandle`, `RaRdevGetSupportLite` | `RaRdevDeinit` | | ||
| 1528 | +| **RaQP** | - | `RaQpCreate`, `RaQpCreateWithAttrs`, `RaTypicalQpCreate`, `RaAiQpCreate`, `RaLoopbackQpCreate` | `RaQpConnectAsync`, `RaSendWr`, `RaSendWrV2`, `RaSendWrlist`, `RaRecvWrlist`, `RaPollCq`, `RaGetQpStatus`, `RaTypicalQpModify` | `RaQpDestroy` | | ||
| 1529 | +| **RaCQ** | - | `RaCqCreate` | `RaPollCq` | `RaCqDestroy` | | ||
| 1530 | +| **RaSRQ** | - | `RaCreateSrq` | `RaModifySrq` | `RaDestroySrq` | | ||
| 1531 | +| **RaMR** | - | `RaMrReg`, `RaRegisterMr` | `RaRemapMr`, `RaGetNotifyMrInfo` | `RaMrDereg`, `RaDeregisterMr` | | ||
| 1532 | +| **RaCQe** | - | - | `RaPollCq` | - | | ||
| 1533 | +| **RaQPAttr** | - | - | `RaGetQpAttr`, `RaSetQpAttrQos`, `RaSetQpAttrTimeout`, `RaSetQpAttrRetryCnt`, `RaGetQpContext` | - | | ||
| 1534 | +| **RaNdaQP** | - | `RaNdaQpCreate` | - | - | | ||
| 1535 | +| **RaNdaCQ** | - | `RaNdaCqCreate` | - | `RaNdaCqDestroy` | | ||
| 1536 | +| **RaContext** | `RaCtxInit` | - | `RaGetDevBaseAttr`, `RaGetDevEidInfoList`, `RaGetDevEidInfoNum`, `RaGetEidByIp`, `RaCtxGetAsyncEvents` | `RaCtxDeinit` | | ||
| 1537 | +| **RaJetty** | - | `RaCtxQpCreate` | `RaBatchSendWr`, `RaCtxUpdateCi`, `RaCtxQpQueryBatch` | `RaCtxQpDestroy` | | ||
| 1538 | +| **EndPointPair** | - | - | `RaCtxQpBind`, `RaCtxQpUnbind`, `RaCtxQpImport`, `RaCtxQpUnimport` | - | | ||
| 1539 | +| **RaJfc** | - | `RaCtxCqCreate` | `RaCtxGetAuxInfo`, `RaCtxGetCrErrInfoList` | `RaCtxCqDestroy` | | ||
| 1540 | +| **RaCr** | - | - | `RaCtxGetAuxInfo` | - | | ||
| 1541 | +| **RaLmem** | - | `RaCtxLmemRegister` | - | `RaCtxLmemUnregister` | | ||
| 1542 | +| **RaRmem** | - | `RaCtxRmemImport` | - | `RaCtxRmemUnimport` | | ||
| 1543 | +| **RaTp** | - | `RaGetTpInfoListAsync` | `RaGetTpAttrAsync`, `RaSetTpAttrAsync` | - | | ||
| 1544 | +| **RaTokenId** | - | `RaCtxTokenIdAlloc` | - | `RaCtxTokenIdFree` | | ||
| 1545 | +| **RaChan** | - | `RaCtxChanCreate` | - | `RaCtxChanDestroy` | | ||
| 1546 | +| **RaPing** | `RaPingInit` | `RaPingTargetAdd` | `RaPingTaskStart`, `RaPingGetResults` | `RaPingDeinit`, `RaPingTargetDel` | | ||
| 1547 | +| **RaTlv** | `RaTlvInit` | - | `RaTlvRequest` | `RaTlvDeinit` | | ||
| 1548 | +| **AsyncRequest** | - | `RaSocketBatchConnectAsync`, `RaCtxQpCreateAsync`, `RaCtxQpDestroyAsync`, `RaCtxQpDestroyBatchAsync`, `RaCtxQpImportAsync`, `RaCtxLmemRegisterAsync`, `RaSocketListenStartAsync`, `RaSocketListenStopAsync`, `RaSocketBatchCloseAsync`, `RaSocketSendAsync`, `RaSocketRecvAsync`, `RaGetTpInfoListAsync`, `RaGetEidByIpAsync`, `RaGetTpAttrAsync`, `RaSetTpAttrAsync` | `RaGetAsyncReqResult` | - | | ||
| 1549 | +| **InterfaceInfo** | - | - | `RaGetIfnum`, `RaGetIfaddrs`, `RaSocketGetVnicIpInfos` | - | | ||
| 1550 | +| **HccnConfig** | - | - | `RaGetHccnCfg`, `RaGetTlsEnable`, `RaGetInterfaceVersion`, `RaGetSecRandom` | - | | ||
| 1551 | +| **Snapshot** | - | - | `RaSaveSnapshot`, `RaRestoreSnapshot` | - | | ||
| 1552 | + | ||
| 1553 | +##### Entity Attribute and Interface Mapping Supplementary Table | ||
| 1554 | + | ||
| 1555 | +| Entity Attribute | Corresponding Interface | | ||
| 1556 | +| -------------------------- | ------------------------------------------------------------------------ | | ||
| 1557 | +| RaSocket.state | `RaGetSockets` returns state | | ||
| 1558 | +| RaSocket.white-list | `RaSocketWhiteListAdd`, `RaSocketWhiteListDel` | | ||
| 1559 | +| RaQP.peer-qpn/peer-lid | `RaQpConnectAsync` exchanges peer info | | ||
| 1560 | +| RaQP.attr | `RaGetQpAttr`, `RaSetQpAttrQos`, `RaSetQpAttrTimeout`, `RaSetQpAttrRetryCnt` | | ||
| 1561 | +| RaQP.context | `RaGetQpContext` returns QP's send_cq and recv_cq | | ||
| 1562 | +| RaCQe.wr-id | Set by `RaSendWr`, `RaSendWrV2`, `RaSendWrlist`, `RaRecvWrlist` | | ||
| 1563 | +| RaCQe.status | Returned by `RaPollCq` | | ||
| 1564 | +| RaJetty.state | Returned by `RaCtxQpQueryBatch` | | ||
| 1565 | +| RaJetty.peer-jetty-handle | Set by `RaCtxQpBind`, `RaCtxQpImport` | | ||
| 1566 | +| RaMR.access | Parameter setting in `RaMrReg` | | ||
| 1567 | +| RaLmem.token-id | Pre-allocated by `RaCtxTokenIdAlloc` | | ||
| 1568 | +| RaTp.tp-type | Returned by `RaGetTpInfoListAsync` | | ||
| 1569 | +| RaContext.local-eid | Obtained via `RaGetDevEidInfoList`, `RaGetEidByIp` | | ||
| 1570 | +| RaContext.mode | Returned by `RaGetDevBaseAttr` | | ||
| 1571 | +| AsyncRequest.status | Returned by `RaGetAsyncReqResult` | | ||
| 1572 | +| InterfaceInfo.* | Obtained via `RaGetIfnum`, `RaGetIfaddrs`, `RaSocketGetVnicIpInfos` | | ||
| 1573 | +| HccnConfig.value | Obtained via `RaGetHccnCfg` | | ||
| 1574 | +| RaNdaQP.nda-direct-flag | `RaNdaGetDirectFlag` checks NDA support | | ||
| 1575 | +| Snapshot.data | Saved by `RaSaveSnapshot`, restored by `RaRestoreSnapshot` | | ||
| 1576 | +| RaSocketEvent.events | Managed by `RaEpollCtlAdd`, `RaEpollCtlMod`, `RaEpollCtlDel` | | ||
| 1577 | +| RaDevice.direct-flag | `RaNdaGetDirectFlag` checks NDA support | | ||
| 1578 | + | ||
| 1579 | +## 5. Callback and Report Relationship Modeling | ||
| 1580 | + | ||
| 1581 | +```mermaid | ||
| 1582 | +erDiagram | ||
| 1583 | + Runner ||--o{ Context: "creates" | ||
| 1584 | + Runner { | ||
| 1585 | + typ run-id PK | ||
| 1586 | + } | ||
| 1587 | + | ||
| 1588 | + Context { | ||
| 1589 | + typ ctx-id PK | ||
| 1590 | + typ run-id FK | ||
| 1591 | + } | ||
| 1592 | + | ||
| 1593 | + Stream { | ||
| 1594 | + int stream-id PK | ||
| 1595 | + int ctx-id FK | ||
| 1596 | + string state "Running/Idle" | ||
| 1597 | + } | ||
| 1598 | + | ||
| 1599 | + %% Stream contains an ordered list of tasks | ||
| 1600 | + Context ||--o{ Stream : "manages/submits" | ||
| 1601 | + Stream ||--o{ Task : "queues [1..*]" | ||
| 1602 | + | ||
| 1603 | + Task { | ||
| 1604 | + int task-id PK | ||
| 1605 | + int stream-id FK | ||
| 1606 | + typ seq-number "auto-increment within stream" | ||
| 1607 | + string type "Kernel/Memcpy/Callback" | ||
| 1608 | + } | ||
| 1609 | + | ||
| 1610 | + %% Various specific Task types (logical inheritance relationship) | ||
| 1611 | + CallbackTask { | ||
| 1612 | + typ report-id FK | ||
| 1613 | + typ callback-fn | ||
| 1614 | + typ user-data | ||
| 1615 | + } | ||
| 1616 | + | ||
| 1617 | + %% Logical expression of inheritance (Task is divided into multiple types) | ||
| 1618 | + Task ||--|{ CallbackTask : "is a" | ||
| 1619 | + | ||
| 1620 | + ReportChannel { | ||
| 1621 | + typ report-id | ||
| 1622 | + typ stream-id | ||
| 1623 | + typ run-id | ||
| 1624 | + } | ||
| 1625 | + | ||
| 1626 | + CallbackTask }o ..|| ReportChannel : "push" | ||
| 1627 | + ReportChannel ||--o{ Runner : "trigger and called by" | ||
| 1628 | + | ||
| 1629 | +``` | ||
| 1630 | + | ||
| 1631 | +### 5.1 Key Relationship Descriptions (Callback and Report Relationship Modeling) | ||
| 1632 | + | ||
| 1633 | +When the **device** executes a CallbackTask, it triggers the Host Runner thread to execute the callback. | ||
| 1634 | + | ||
| 1635 | +#### 5.1.1 Corresponding Interface Mapping Table (Callback and Report Relationship Modeling) | ||
| 1636 | + | ||
| 1637 | +| Entity | Key Management Interface | | ||
| 1638 | +| ------------- | --------------------------------------------------------------------- | | ||
| 1639 | +| CallbackTask | `rtSetExceptionInfoCallback`, `rtLaunchCallback`,`rtSynchronizeEvent` | | ||
| 1640 | +| ReportChannel | `rtSubscribeReport`, `rtUnSubscribeReport` | | ||
| 1641 | +| Runner | `rtProcessReport` | | ||
| 1642 | + | ||
| 1643 | +## 6. Fine-grained Low-level Extension of the Basic Device Model | ||
| 1644 | + | ||
| 1645 | +```mermaid | ||
| 1646 | +erDiagram | ||
| 1647 | + | ||
| 1648 | + Device ||--|| DeviceStatus : "has a" | ||
| 1649 | + Device ||--|{ TaskSchedulerDevice : "has " | ||
| 1650 | + Device { | ||
| 1651 | + typ device-id PK | ||
| 1652 | + } | ||
| 1653 | + DeviceStatus { | ||
| 1654 | + typ device-id FK | ||
| 1655 | + typ overflow-status | ||
| 1656 | + typ synchronize-strategy | ||
| 1657 | + typ synchronize-timeout | ||
| 1658 | + typ capability-mask | ||
| 1659 | + typ run-by-host | ||
| 1660 | + typ ts-core | ||
| 1661 | + typ online-status | ||
| 1662 | + } | ||
| 1663 | + | ||
| 1664 | + TaskSchedulerDevice ||--|| Scalar :"is a" | ||
| 1665 | + TaskSchedulerDevice ||--|| CCU :"is a" | ||
| 1666 | + TaskSchedulerDevice ||--|| CPU :"is a" | ||
| 1667 | + TaskSchedulerDevice { | ||
| 1668 | + typ ts-id PK | ||
| 1669 | + typ device-id FK | ||
| 1670 | + typ type "Scalar" | ||
| 1671 | + } | ||
| 1672 | + | ||
| 1673 | + CPU { | ||
| 1674 | + } | ||
| 1675 | + | ||
| 1676 | + Scalar ||--|| ComputeDie :"schedule" | ||
| 1677 | + Scalar { | ||
| 1678 | + } | ||
| 1679 | + | ||
| 1680 | + CCU { | ||
| 1681 | + typ ccu-id PK | ||
| 1682 | + typ ts-id FK | ||
| 1683 | + typ version "v1/v2" | ||
| 1684 | + typ xnNum | ||
| 1685 | + typ ckeNum | ||
| 1686 | + typ msNum | ||
| 1687 | + typ channelNum | ||
| 1688 | + } | ||
| 1689 | + | ||
| 1690 | + ComputeDie ||--|| Cube :"is a" | ||
| 1691 | + ComputeDie ||--|| Vector :"is a" | ||
| 1692 | + ComputeDie ||--|| HybridComputeDie :"is a (vector+cube)" | ||
| 1693 | + ComputeDie { | ||
| 1694 | + typ compute-id PK | ||
| 1695 | + typ ts-id FK | ||
| 1696 | + typ type | ||
| 1697 | + } | ||
| 1698 | + Cube { | ||
| 1699 | + } | ||
| 1700 | + Vector { | ||
| 1701 | + } | ||
| 1702 | +``` | ||
| 1703 | + | ||
| 1704 | +### 6.1 Key Relationship Descriptions (Fine-grained Low-level Extension of the Basic Device Model) | ||
| 1705 | + | ||
| 1706 | +**Device scheduler hierarchy**: | ||
| 1707 | + | ||
| 1708 | +- `TaskSchedulerDevice` is the abstraction of a device scheduler; one Device may contain multiple schedulers. | ||
| 1709 | +- Scheduler types include: `Scalar` (scalar processor), `CCU` (collective communication unit), `CPU` (AI CPU) | ||
| 1710 | +- Different scheduler types handle different computation task types. | ||
| 1711 | + | ||
| 1712 | +**ComputeDie compute unit**: | ||
| 1713 | + | ||
| 1714 | +- `ComputeDie` is the abstraction of a compute unit; the inheritance relationship indicates the compute unit type. | ||
| 1715 | +- `Vector`: Vector compute unit, handles vector operations. | ||
| 1716 | +- `Cube`: Cube compute unit, handles matrix operations. | ||
| 1717 | +- `HybridComputeDie`: Hybrid compute unit, supports both Vector and Cube. | ||
| 1718 | + | ||
| 1719 | +**CCU internal structure**: | ||
| 1720 | + | ||
| 1721 | +- `xnNum`: Number of XN nodes (cross-node communication) | ||
| 1722 | +- `ckeNum`: Number of CKE engines (Checksum engine) | ||
| 1723 | +- `msNum`: Number of MS modules (Memory Scheduler) | ||
| 1724 | +- `channelNum`: Number of communication channels. | ||
| 1725 | +- `version`: CCU version (v1/v2, determines feature differences) | ||
| 1726 | + | ||
| 1727 | +**DeviceStatus state management**: | ||
| 1728 | + | ||
| 1729 | +- `overflow-status`: Overflow state. | ||
| 1730 | +- `synchronize-strategy`: Synchronization strategy configuration. | ||
| 1731 | +- `synchronize-timeout`: Synchronization timeout setting. | ||
| 1732 | +- `capability-mask`: Device capability mask. | ||
| 1733 | +- `run-by-host`: Whether running in Host mode. | ||
| 1734 | +- `ts-core`: Number of scheduler cores. | ||
| 1735 | +- `online-status`: Device online status. | ||
| 1736 | + | ||
| 1737 | +**Scalar scheduling relationship**: | ||
| 1738 | + | ||
| 1739 | +- The `Scalar` scheduler manages the execution of `ComputeDie`. | ||
| 1740 | +- Different ComputeDie types handle different computational workloads. | ||
| 1741 | + | ||
| 1742 | +#### 6.1.1 Corresponding Interface Mapping Table (Fine-grained Low-level Extension of the Basic Device Model) | ||
| 1743 | + | ||
| 1744 | +| Entity | Key Management Interface | | ||
| 1745 | +| ------------------- | ---------------------------------- | | ||
| 1746 | +| TaskSchedulerDevice | `rtGetDeviceInfo`, `rtSetTsDevice` | | ||
| 1747 | +| DeviceStatus | `rtGetRunMode` | | ||
| 1748 | + | ||
| 1749 | +## 7. Kernel Runtime Relationship Modeling | ||
| 1750 | + | ||
| 1751 | +```mermaid | ||
| 1752 | +erDiagram | ||
| 1753 | + KernelBinary { | ||
| 1754 | + typ id PK | ||
| 1755 | + typ file | ||
| 1756 | + typ create-pid | ||
| 1757 | + } | ||
| 1758 | + | ||
| 1759 | + KernelBinary ||--|| KernelBinaryHandle :"loaded" | ||
| 1760 | + KernelBinaryHandle ||--o{ KernelFuncHandle :"contains" | ||
| 1761 | + KernelBinaryHandle { | ||
| 1762 | + typ handle-id PK | ||
| 1763 | + typ kernel-id FK | ||
| 1764 | + } | ||
| 1765 | + | ||
| 1766 | + KernelFuncHandle ||--o{ KernelFuncArgsHandle :"has a" | ||
| 1767 | + KernelFuncHandle }o--|| Task :"Called " | ||
| 1768 | + KernelFuncHandle }o--|| KernelLaunchCfg :"launch config" | ||
| 1769 | + KernelFuncHandle { | ||
| 1770 | + typ handle-id PK | ||
| 1771 | + typ binary-id FK | ||
| 1772 | + typ func-name | ||
| 1773 | + typ kernel-name | ||
| 1774 | + typ aic-addr | ||
| 1775 | + typ aiv-addr | ||
| 1776 | + } | ||
| 1777 | + | ||
| 1778 | + KernelFuncArgsHandle ||--o{ KernelFuncArgsParamHandle :"append" | ||
| 1779 | + KernelFuncArgsHandle { | ||
| 1780 | + typ args-handle-id PK | ||
| 1781 | + typ func-id FK | ||
| 1782 | + typ args-size | ||
| 1783 | + typ type "device/host" | ||
| 1784 | + } | ||
| 1785 | + | ||
| 1786 | + KernelFuncArgsParamHandle { | ||
| 1787 | + typ args-param-id PK | ||
| 1788 | + typ args-id FK | ||
| 1789 | + typ param-size | ||
| 1790 | + typ is-place-holder | ||
| 1791 | + } | ||
| 1792 | +``` | ||
| 1793 | + | ||
| 1794 | +### 7.1 Key Relationship Descriptions (Kernel Runtime Relationship Modeling) | ||
| 1795 | + | ||
| 1796 | +**KernelBinary loading flow**: | ||
| 1797 | + | ||
| 1798 | +1. `KernelBinary` stores the binary file path (.o/.so, etc.) and the creating process PID. | ||
| 1799 | +2. `rtBinaryLoadFromFile` or `rtBinaryLoadFromData` loads the binary into device memory. | ||
| 1800 | +3. After loading, a `KernelBinaryHandle` is generated, containing handle-id and kernel-id. | ||
| 1801 | +4. `KernelFuncHandle` represents a specific function in the binary, containing func-name, kernel-name, and address info. | ||
| 1802 | + | ||
| 1803 | +**Kernel function call relationship**: | ||
| 1804 | + | ||
| 1805 | +- `aic-addr` is the AI Core function address. | ||
| 1806 | +- `aiv-addr` is the AI Vector function address. | ||
| 1807 | +- `KernelFuncArgsHandle` stores function parameter information. | ||
| 1808 | +- `KernelFuncArgsParamHandle` records parameter size and whether it is a placeholder. | ||
| 1809 | +- `KernelLaunchCfg` configures Kernel launch parameters (block/grid, etc.) | ||
| 1810 | + | ||
| 1811 | +**Kernel lifecycle**: | ||
| 1812 | + | ||
| 1813 | +```text | ||
| 1814 | +rtCreateBinary -> rtBinaryLoad -> rtBinaryGetFunction | ||
| 1815 | + -> rtLaunchKernel(funcHandle, argsHandle, cfg) | ||
| 1816 | + -> rtBinaryUnLoad -> rtDestroyBinary | ||
| 1817 | +``` | ||
| 1818 | + | ||
| 1819 | +**Parameter management mechanism**: | ||
| 1820 | + | ||
| 1821 | +- args-handle supports two types: device and host. | ||
| 1822 | +- is-place-holder identifies whether the parameter is a placeholder (late binding) | ||
| 1823 | +- param-size records the size of an individual parameter. | ||
| 1824 | + | ||
| 1825 | +#### 7.1.1 Corresponding Interface Mapping Table (Kernel Runtime Relationship Modeling) | ||
| 1826 | + | ||
| 1827 | +| Entity | Key Management Interface | | ||
| 1828 | +| ------------------ | --------------------------------------------------------------------------------------------------------------- | | ||
| 1829 | +| KernelBinary | `rtCreateBinary`, `rtDestroyBinary` | | ||
| 1830 | +| KernelBinaryHandle | `rtBinaryLoad`, `rtBinaryUnLoad`,`rtBinaryLoadFromFile`,`rtBinaryLoadFromData` | | ||
| 1831 | +| KernelFuncHandle | `rtBinaryGetFunction`, `rtBinaryGetFunctionByEntry`,`rtGetFunctionAddr`,`rtGetFunctionName`,`rtRegisterCpuFunc` | | ||
| 1832 | + | ||
| 1833 | +## 8. Model Loading Relationship Modeling | ||
| @@ -0,0 +1,55 @@ | |||
| 1 | +# TABLE CMD | ||
| 2 | + | ||
| 3 | +[toc] | ||
| 4 | + | ||
| 5 | +## 1 Core Features | ||
| 6 | + | ||
| 7 | +### 1.1 Starting the Simulation Environment | ||
| 8 | + | ||
| 9 | +#### 1.1.1 Viewing All Table Names | ||
| 10 | + | ||
| 11 | +```bash | ||
| 12 | +(hvm)$> hccl-vm table show all | ||
| 13 | +all | ||
| 14 | +Server | ||
| 15 | +Host | ||
| 16 | +Runner | ||
| 17 | +Device | ||
| 18 | +... | ||
| 19 | +(hvm)$> | ||
| 20 | +``` | ||
| 21 | + | ||
| 22 | +#### 1.1.2 Viewing Specified Table Contents | ||
| 23 | + | ||
| 24 | +```bash | ||
| 25 | +(hvm)$> hccl-vm table show Device | ||
| 26 | +| id | server_id | logic_id | physical_id | overflow_mode | soc_version | status | | ||
| 27 | +| 1 | 1 | 0 | 0 | 0 | Ascend950 | 0 | | ||
| 28 | +| 2 | 1 | 1 | 1 | 0 | Ascend950 | 0 | | ||
| 29 | +| 3 | 1 | 2 | 2 | 0 | Ascend950 | 0 | | ||
| 30 | +| 4 | 1 | 3 | 3 | 0 | Ascend950 | 0 | | ||
| 31 | + | ||
| 32 | +(hvm)$> | ||
| 33 | +``` | ||
| 34 | + | ||
| 35 | +#### 1.1.3 Updating Specified Content in a Specified Table Row (currently only supports modifying device soc_version) | ||
| 36 | + | ||
| 37 | +```bash | ||
| 38 | +(hvm)$> hccl-vm table show Device | ||
| 39 | +| id | server_id | logic_id | physical_id | overflow_mode | soc_version | status | | ||
| 40 | +| 1 | 1 | 0 | 0 | 0 | Ascend950 | 0 | | ||
| 41 | +| 2 | 1 | 1 | 1 | 0 | Ascend950 | 0 | | ||
| 42 | +| 3 | 1 | 2 | 2 | 0 | Ascend950 | 0 | | ||
| 43 | +| 4 | 1 | 3 | 3 | 0 | Ascend950 | 0 | | ||
| 44 | +(hvm)$> hccl-vm table update device 1 soc_version Ascend951 | ||
| 45 | +update device 1 soc_version Ascend951 | ||
| 46 | +Updating device [id=1].soc_version = "Ascend951" | ||
| 47 | +(hvm)$> hccl-vm table show Device | ||
| 48 | +| id | server_id | logic_id | physical_id | overflow_mode | soc_version | status | | ||
| 49 | +| 1 | 1 | 0 | 0 | 0 | Ascend951 | 0 | | ||
| 50 | +| 2 | 1 | 1 | 1 | 0 | Ascend950 | 0 | | ||
| 51 | +| 3 | 1 | 2 | 2 | 0 | Ascend950 | 0 | | ||
| 52 | +| 4 | 1 | 3 | 3 | 0 | Ascend950 | 0 | | ||
| 53 | + | ||
| 54 | +(hvm)$> | ||
| 55 | +``` | ||
| @@ -0,0 +1,298 @@ | |||
| 1 | +# HCCL_VM UT Test Execution Script | ||
| 2 | + | ||
| 3 | +## Overview | ||
| 4 | + | ||
| 5 | +`run_ut.sh` is the unit test execution script for the HCCL_VM project, used to automate the compilation and execution of test cases. | ||
| 6 | + | ||
| 7 | +## Three-Step Process | ||
| 8 | + | ||
| 9 | +The script automatically completes the following steps during execution: | ||
| 10 | + | ||
| 11 | +| Step | Description | Log Output | | ||
| 12 | +|------|-------------|------------| | ||
| 13 | +| Step 1 | CMake configuration + make compilation | `build.log` | | ||
| 14 | +| Step 2 | Generate executable files | Lists all binaries with size/time | | ||
| 15 | +| Step 3 | Execute test cases and display results | `run.log` + `summary.log` | | ||
| 16 | +| Step 4 | Generate gcov/lcov coverage HTML report (requires `--cov`) | `coverage.log` | | ||
| 17 | + | ||
| 18 | +## Usage | ||
| 19 | + | ||
| 20 | +```bash | ||
| 21 | +cd {HCCL_VM_PATH}/test | ||
| 22 | + | ||
| 23 | +# Basic usage | ||
| 24 | +./run_ut.sh # Full compilation + execute all tests | ||
| 25 | +./run_ut.sh --cov # Full compilation + execution + generate gcov/lcov coverage HTML report | ||
| 26 | +./run_ut.sh <directory> # Compile + execute all tests in specified directory (recursive) | ||
| 27 | +./run_ut.sh <binary_name> # Compile + execute specified test binary | ||
| 28 | +./run_ut.sh <test_file_name> # Compile + execute the binary corresponding to the specified test file | ||
| 29 | +./run_ut.sh <file> <test_case_name> # Compile + execute a single test case in the specified file | ||
| 30 | + | ||
| 31 | +# Other commands | ||
| 32 | +./run_ut.sh -l, --list # List all available tests | ||
| 33 | +./run_ut.sh -h, --help # Display help information | ||
| 34 | +``` | ||
| 35 | + | ||
| 36 | +## Command Details | ||
| 37 | + | ||
| 38 | +### 1. Full Execution | ||
| 39 | + | ||
| 40 | +```bash | ||
| 41 | +./run_ut.sh | ||
| 42 | +``` | ||
| 43 | + | ||
| 44 | +- Executes all tests in the `test` directory. | ||
| 45 | +- Complete three-step process: compilation → generate executables → run all tests. | ||
| 46 | +- Suitable for full regression testing. | ||
| 47 | + | ||
| 48 | +### 2. Coverage Report | ||
| 49 | + | ||
| 50 | +```bash | ||
| 51 | +./run_ut.sh --cov | ||
| 52 | +``` | ||
| 53 | + | ||
| 54 | +- Full compilation + execute all tests + generate gcov/lcov coverage HTML report. | ||
| 55 | +- Automatically enables the `--coverage` compilation flag, generating `.gcno`/`.gcda` files. | ||
| 56 | +- Four-step process: compilation (with coverage instrumentation) → execution → collect coverage data → generate HTML report. | ||
| 57 | +- Report output path: `$CODE_DIR/coverage_report/html/index.html`. | ||
| 58 | +- Automatically filters system headers, third-party libraries, stub files, and other non-business code. | ||
| 59 | + | ||
| 60 | +### 3. Directory Execution | ||
| 61 | + | ||
| 62 | +```bash | ||
| 63 | +./run_ut.sh plugin/checker | ||
| 64 | +./run_ut.sh plugin/ccu_executor | ||
| 65 | +./run_ut.sh store | ||
| 66 | +``` | ||
| 67 | + | ||
| 68 | +- Recursively finds all `*_test.cc` files in the specified directory. | ||
| 69 | +- Compiles and executes all tests in that directory. | ||
| 70 | +- Suitable for module-level testing. | ||
| 71 | + | ||
| 72 | +### 4. Binary Execution | ||
| 73 | + | ||
| 74 | +```bash | ||
| 75 | +./run_ut.sh test_checker | ||
| 76 | +./run_ut.sh test_allgather_semantics_checker | ||
| 77 | +``` | ||
| 78 | + | ||
| 79 | +- Compiles and executes the specified test binary. | ||
| 80 | +- Binary names start with `test_`. | ||
| 81 | + | ||
| 82 | +### 5. File Execution | ||
| 83 | + | ||
| 84 | +```bash | ||
| 85 | +./run_ut.sh checker_test.cc | ||
| 86 | +./run_ut.sh allgather_semantics_checker_test.cc | ||
| 87 | +``` | ||
| 88 | + | ||
| 89 | +- Automatically matches the corresponding binary based on the test file name. | ||
| 90 | +- Compiles and executes. | ||
| 91 | + | ||
| 92 | +### 6. Single Test Case Execution | ||
| 93 | + | ||
| 94 | +```bash | ||
| 95 | +./run_ut.sh checker_test.cc CheckerTest.GenAndCheckGraph_EmptyQueues | ||
| 96 | +./run_ut.sh allgather_semantics_checker_test.cc AllgatherSemanticsCheckerTest.CheckBasic | ||
| 97 | +``` | ||
| 98 | + | ||
| 99 | +- Executes a single test case in the specified test file. | ||
| 100 | +- Test case name format: `TestSuiteName.TestCaseName`. | ||
| 101 | + | ||
| 102 | +### 7. List Tests | ||
| 103 | + | ||
| 104 | +```bash | ||
| 105 | +./run_ut.sh -l | ||
| 106 | +./run_ut.sh --list | ||
| 107 | +``` | ||
| 108 | + | ||
| 109 | +- Lists all available test files and their status. | ||
| 110 | +- Displays the number of test cases and corresponding binary names. | ||
| 111 | + | ||
| 112 | +## Examples | ||
| 113 | + | ||
| 114 | +```bash | ||
| 115 | +# Example 1: Full test | ||
| 116 | +./run_ut.sh | ||
| 117 | + | ||
| 118 | +# Example 2: Full test + coverage report | ||
| 119 | +./run_ut.sh --cov | ||
| 120 | + | ||
| 121 | +# Example 3: Execute all tests in the plugin/checker directory | ||
| 122 | +./run_ut.sh plugin/checker | ||
| 123 | + | ||
| 124 | +# Example 4: Compile and execute test_checker | ||
| 125 | +./run_ut.sh test_checker | ||
| 126 | + | ||
| 127 | +# Example 5: Compile and execute the binary corresponding to checker_test.cc | ||
| 128 | +./run_ut.sh checker_test.cc | ||
| 129 | + | ||
| 130 | +# Example 6: Execute a single test case | ||
| 131 | +./run_ut.sh checker_test.cc CheckerTest.GenAndCheckGraph_EmptyQueues | ||
| 132 | + | ||
| 133 | +# Example 7: List all tests | ||
| 134 | +./run_ut.sh -l | ||
| 135 | +``` | ||
| 136 | + | ||
| 137 | +## Log Directory | ||
| 138 | + | ||
| 139 | +Each execution generates log files under `ut_logs/<timestamp>/`: | ||
| 140 | + | ||
| 141 | +```text | ||
| 142 | +{HCCL_VM_PATH}/ut_logs/20260425_142048/ | ||
| 143 | +├── build.log # Detailed compilation log (cmake + make output) | ||
| 144 | +├── run.log # Detailed execution log (full output of each test) | ||
| 145 | +└── summary.log # Summary log (execution result of each test) | ||
| 146 | +``` | ||
| 147 | + | ||
| 148 | +### Log File Description | ||
| 149 | + | ||
| 150 | +| File | Content | | ||
| 151 | +|------|---------| | ||
| 152 | +| `build.log` | CMake configuration output, make compilation output, list of generated executables | | ||
| 153 | +| `run.log` | Full output of each test (including gtest details) | | ||
| 154 | +| `summary.log` | Execution status, pass/fail count, and time summary for each test | | ||
| 155 | + | ||
| 156 | +## Execution Results | ||
| 157 | + | ||
| 158 | +After the script finishes, summary information is displayed: | ||
| 159 | + | ||
| 160 | +```text | ||
| 161 | +======================================== | ||
| 162 | + Directory Test Result Summary: plugin/checker | ||
| 163 | +======================================== | ||
| 164 | + Executed: 16 | ||
| 165 | + PASSED: 185 | ||
| 166 | + FAILED: 8 | ||
| 167 | + CRASHED: 0 | ||
| 168 | + TIMEOUT: 0 | ||
| 169 | +``` | ||
| 170 | + | ||
| 171 | +### Status Description | ||
| 172 | + | ||
| 173 | +| Status | Description | | ||
| 174 | +|--------|-------------| | ||
| 175 | +| `PASS` | Test passed | | ||
| 176 | +| `FAIL` | Test failed (assertion failure) | | ||
| 177 | +| `CRASH` | Test crashed (core dump) | | ||
| 178 | +| `TIMEOUT` | Test timed out (default 60 seconds) | | ||
| 179 | + | ||
| 180 | +## Directory Structure | ||
| 181 | + | ||
| 182 | +```text | ||
| 183 | +test/ | ||
| 184 | +├── run_ut.sh # This script | ||
| 185 | +├── cmd/ # Command-related tests | ||
| 186 | +│ ├── base/ | ||
| 187 | +│ ├── subcmds/ | ||
| 188 | +│ └── utils/ | ||
| 189 | +├── device_arm/ # Device-related tests | ||
| 190 | +├── device_vir/ | ||
| 191 | +├── ipc/ # IPC-related tests | ||
| 192 | +│ └── shm/ | ||
| 193 | +├── log/ # Log-related tests | ||
| 194 | +├── plugin/ # Plugin-related tests | ||
| 195 | +│ ├── ccu_executor/ | ||
| 196 | +│ │ ├── control_type/ | ||
| 197 | +│ │ ├── load_type/ | ||
| 198 | +│ │ ├── reduce_type/ | ||
| 199 | +│ │ └── trans_type/ | ||
| 200 | +│ └── checker/ | ||
| 201 | +│ └── framework/ | ||
| 202 | +│ ├── mem_conflict_check/ | ||
| 203 | +│ ├── semantics_check/ | ||
| 204 | +│ └── singletask_check/ | ||
| 205 | +├── proxy/ # Proxy-related tests | ||
| 206 | +│ ├── level1/ | ||
| 207 | +│ └── level2/ | ||
| 208 | +├── runnerdb/ # Database-related tests | ||
| 209 | +├── store/ # Storage-related tests | ||
| 210 | +│ └── hccl_shm/ | ||
| 211 | +└── src_root/ # Source root directory tests | ||
| 212 | +``` | ||
| 213 | + | ||
| 214 | +## Environment Requirements | ||
| 215 | + | ||
| 216 | +**Before executing the script, you must modify the following environment variables in `run_ut.sh` to use the actual paths:** | ||
| 217 | + | ||
| 218 | +```bash | ||
| 219 | +# The following are example paths. Modify them according to your actual environment. | ||
| 220 | +export HCOMM_CODE_HOME=/home/q30033976/checker/hcomm # hcomm source code path | ||
| 221 | +export HCCL_CODE_HOME=/home/q30033976/checker/hccl # hccl source code path (required for AIV/AICPU mode) | ||
| 222 | +source /home/q30033976/checker/Ascend/cann/set_env.sh # CANN environment script path | ||
| 223 | +``` | ||
| 224 | + | ||
| 225 | +**Example**: If your working directory is `/home/workspace`, modify the settings to: | ||
| 226 | + | ||
| 227 | +```bash | ||
| 228 | +export HCOMM_CODE_HOME=/home/workspace/hcomm | ||
| 229 | +export HCCL_CODE_HOME=/home/workspace/hccl | ||
| 230 | +source /home/workspace/Ascend/cann/set_env.sh | ||
| 231 | +``` | ||
| 232 | + | ||
| 233 | +The script will automatically load these environment variables. Failure to modify them will result in compilation errors. | ||
| 234 | + | ||
| 235 | +## Configuration Parameters | ||
| 236 | + | ||
| 237 | +The script has the following built-in configuration (can be modified at the beginning of the script): | ||
| 238 | + | ||
| 239 | +| Parameter | Default Value | Description | | ||
| 240 | +|-----------|---------------|-------------| | ||
| 241 | +| `CMAKE_BUILD_TYPE` | `Debug` | CMake build type | | ||
| 242 | +| `MAKE_JOBS` | `8` | Number of parallel make jobs | | ||
| 243 | +| `LOG_DIR` | `$CODE_DIR/ut_logs` | Log output directory | | ||
| 244 | + | ||
| 245 | +## Notes | ||
| 246 | + | ||
| 247 | +1. **First execution**: The first execution will perform a complete CMake configuration, which takes a longer time. | ||
| 248 | +2. **Compilation failure**: If compilation fails, check `build.log` for detailed error information. | ||
| 249 | +3. **Test failure**: If tests fail, check `run.log` for the specific failing test cases. | ||
| 250 | +4. **Log cleanup**: Log directories are named by timestamp. Clean up old logs periodically to save space. | ||
| 251 | + | ||
| 252 | +## Frequently Asked Questions | ||
| 253 | + | ||
| 254 | +### Q: How to compile without executing? | ||
| 255 | + | ||
| 256 | +A: The current script integrates compilation and execution. To compile only, use cmake and make directly: | ||
| 257 | + | ||
| 258 | +```bash | ||
| 259 | +cd {HCCL_VM_PATH}/build | ||
| 260 | +cmake .. && make -j8 test_checker | ||
| 261 | +``` | ||
| 262 | + | ||
| 263 | +### Q: How to view detailed output of a specific test? | ||
| 264 | + | ||
| 265 | +A: Check the `run.log` file, which contains the complete gtest output for each test. | ||
| 266 | + | ||
| 267 | +### Q: What to do if a test times out? | ||
| 268 | + | ||
| 269 | +A: The default timeout is 60 seconds. You can modify the `timeout_sec` parameter in the script. | ||
| 270 | + | ||
| 271 | +### Q: How to add new tests? | ||
| 272 | + | ||
| 273 | +A: Create a `*_test.cc` file in the corresponding directory and add the build target to the corresponding `CMakeLists.txt`. | ||
| 274 | + | ||
| 275 | +## Viewing Coverage Reports | ||
| 276 | + | ||
| 277 | +After generating the HTML coverage report on Linux, start an HTTP server to view it: | ||
| 278 | + | ||
| 279 | +```bash | ||
| 280 | +cd <coverage_report/html directory> | ||
| 281 | +python3 -m http.server 8080 | ||
| 282 | +``` | ||
| 283 | + | ||
| 284 | +- Access via local browser: `http://localhost:8080`. | ||
| 285 | +- For remote servers, use SSH port forwarding to access locally: | ||
| 286 | + | ||
| 287 | +```bash | ||
| 288 | +ssh -L 8080:localhost:8080 <user>@<server_ip> | ||
| 289 | +# Then open http://localhost:8080 in your local browser | ||
| 290 | +``` | ||
| 291 | + | ||
| 292 | +Press `^C` to stop the server. | ||
| 293 | + | ||
| 294 | +## Version History | ||
| 295 | + | ||
| 296 | +- v1.0 - Initial version, supports full/directory/file/test-case-level test execution. | ||
| 297 | +- v2.0 - Optimized command-line parameters, supports automatic path derivation, three-step process logging. | ||
| 298 | +- v2.1 - Added `--cov` parameter, supports gcov/lcov coverage HTML report generation. | ||
| @@ -0,0 +1,38 @@ | |||
| 1 | +# Algorithm Analyzer Tool Usage Guide | ||
| 2 | + | ||
| 3 | +## Introduction | ||
| 4 | + | ||
| 5 | +This document is only used to guide users in compiling and running the algorithm analyzer test cases in this directory. For detailed tool principles, test case writing, parameter settings, result analysis, and issue location, please refer to the [Algorithm Analyzer Tool User Guide](../../../st/algorithm/README_en.md). | ||
| 6 | + | ||
| 7 | +## Environment Setup | ||
| 8 | + | ||
| 9 | +Refer to the environment setup in [Source Build](../../../../docs/en/build/build.md), install the CANN Toolkit development package, and prepare the prerequisites for compiling the algorithm analyzer. | ||
| 10 | + | ||
| 11 | +## Test Execution | ||
| 12 | + | ||
| 13 | +Run the following commands from the hcomm source code root directory to compile and execute algorithm analyzer test cases: | ||
| 14 | + | ||
| 15 | +```bash | ||
| 16 | +# Compile all test suite cases and execute automatically | ||
| 17 | +bash build.sh --legacy_all_testcase | ||
| 18 | + | ||
| 19 | +# Compile individual test suite cases and execute automatically | ||
| 20 | +bash build.sh --legacy_aicpu_2d_testcase | ||
| 21 | +bash build.sh --legacy_ccu_2d_testcase | ||
| 22 | +bash build.sh --legacy_ccu_1d_hf16p_testcase | ||
| 23 | +bash build.sh --legacy_ccu_1d_testcase_part1 | ||
| 24 | +bash build.sh --legacy_ccu_1d_testcase_part2 | ||
| 25 | +bash build.sh --legacy_alg_ccu_reduce | ||
| 26 | +bash build.sh --legacy_function_ut_testcase | ||
| 27 | +bash build.sh --legacy_alg_testcase | ||
| 28 | + | ||
| 29 | +# Manually re-execute test cases | ||
| 30 | +./build/test/legacy/st/algorithm/testcase/aicpu_2d_testcase/legacy_alg_aicpu_2d_testcase | ||
| 31 | +./build/test/legacy/st/algorithm/testcase/ccu_2d_testcase/legacy_alg_ccu_2d_testcase | ||
| 32 | +./build/test/legacy/st/algorithm/testcase/ccu_1d_hf16p_testcase/legacy_alg_ccu_1d_hf16p_testcase | ||
| 33 | +./build/test/legacy/st/algorithm/testcase/ccu_1d_testcase_part1/legacy_alg_ccu_1d_testcase_part1 | ||
| 34 | +./build/test/legacy/st/algorithm/testcase/ccu_1d_testcase_part2/legacy_alg_ccu_1d_testcase_part2 | ||
| 35 | +./build/test/legacy/st/algorithm/testcase/ccu_reduce_testcase/legacy_alg_ccu_reduce | ||
| 36 | +./build/test/legacy/st/algorithm/testcase/function_ut_testcase/legacy_alg_function_ut_testcase | ||
| 37 | +./build/test/legacy/st/algorithm/testcase/legacy_alg_testcase/legacy_alg_testcase | ||
| 38 | +``` | ||
| @@ -0,0 +1,320 @@ | |||
| 1 | +# Algorithm Analyzer Tool User Guide | ||
| 2 | + | ||
| 3 | +## Introduction | ||
| 4 | + | ||
| 5 | +The HCCL algorithm analyzer is used to simulate the execution of HCCL algorithms in an offline environment, verifying algorithm logic and memory operations. The HCCL algorithm analyzer provides efficient and fast batch execution of test tasks to meet developer needs. | ||
| 6 | + | ||
| 7 | +## Principles | ||
| 8 | + | ||
| 9 | + | ||
| 10 | + | ||
| 11 | +**Key points:** | ||
| 12 | + | ||
| 13 | +1. The algorithm analyzer stubs the platform and framework layers to obtain the task sequence for all ranks during algorithm execution. | ||
| 14 | +2. The task information of all ranks is organized into a **Directed Acyclic Graph (DAG)**. | ||
| 15 | +3. Validations are performed based on **graph algorithms**, such as memory read/write conflict detection and semantic validation. 1) Memory conflict detection analyzes whether there are potential read/write conflicts based on synchronization within the graph. 2) Semantic validation simulates the execution of the task graph, records **data movement information**, and checks whether the **data movement information** in the UserOutput memory meets the operator requirements after simulation. | ||
| 16 | + | ||
| 17 | +## Environment Setup | ||
| 18 | + | ||
| 19 | +Refer to the environment setup and source code download in [Source Build](../../../docs/en/build/build.md) to prepare the prerequisites for compiling the algorithm analyzer. | ||
| 20 | + | ||
| 21 | +## Test Case Writing | ||
| 22 | + | ||
| 23 | +### LLT Test Case Overview | ||
| 24 | + | ||
| 25 | +An algorithm checker test case consists of 5 steps, as shown in the 5 boxes in the figure below. The following sections describe how to write each step to accommodate different operator requirements, as well as how to use the checker tool for issue location when problems arise. | ||
| 26 | + | ||
| 27 | + | ||
| 28 | + | ||
| 29 | +### LLT Test Case Step Details | ||
| 30 | + | ||
| 31 | +#### Topology Generation | ||
| 32 | + | ||
| 33 | +- TopoMeta structure introduction | ||
| 34 | + | ||
| 35 | +  | ||
 [MEDIUM] 英文文档图片本地化不一致：部分图无 _en 版本 本英文文档混用了带 _en 后缀的英文图与无后缀的原图： 第 35 行：compile_testcase2.png 问题分析：
建议修复：确认上述原图是否含中文标注；若含，补充对应 _en.png 英文版。 | |||
| 36 | + | ||
| 37 | + checker: TopoMeta is used to represent a topology structure. TopoMeta is a three-layer vector structure. | ||
| 38 | + | ||
| 39 | + PhyDeviceId: Represents the physical ID of an NPU. | ||
| 40 | + | ||
| 41 | + ServerMeta: Composed of PhyDeviceId, representing the number of cards on a server and their corresponding PhyDeviceIds. | ||
| 42 | + | ||
| 43 | + SuperPodMeta: Composed of ServerMeta, representing the server composition of a super node. | ||
| 44 | + | ||
| 45 | + TopoMeta: Represents the overall cluster topology. | ||
| 46 | + | ||
| 47 | +- TopoMeta generation methods | ||
| 48 | + | ||
| 49 | + There are two ways to generate TopoMeta: | ||
| 50 | + | ||
| 51 | + 1. Specify the number of super nodes, the number of servers, and the number of cards per server, then use the `GenTopoMeta` function provided by the `RankTable_For_LLT` class to generate it. Suitable for symmetric topology scenarios. | ||
| 52 | + | ||
| 53 | +  | ||
| 54 | + | ||
| 55 | + 2. Fully customize super nodes, servers, and card counts for asymmetric topology scenarios. As shown below, the TopoMeta has one super node with two servers inside — one server has 2 cards and the other has 3 cards. | ||
| 56 | + | ||
| 57 | +  | ||
| 58 | + | ||
| 59 | +- Rank table generation | ||
| 60 | + | ||
| 61 | + Once TopoMeta is available, use the `GenRankTable` function provided by the `RankTable_For_LLT` class to generate the rank table. | ||
| 62 | + | ||
| 63 | +#### Environment Variable Configuration | ||
| 64 | + | ||
| 65 | +- Setting environment variables | ||
| 66 | + | ||
| 67 | + Environment variables affect the logic flow in the code. Use the `setenv` function to configure the required conditions before test case execution. | ||
| 68 | + | ||
| 69 | +- Cleaning environment variables | ||
| 70 | + | ||
| 71 | + Since environment variables are process-level and an LLT task runs in the same process, environment variable usage may affect other test cases. To clean up environment variables, the `TearDown` function in the test suite currently calls the environment variable cleanup function. | ||
| 72 | + | ||
| 73 | +  | ||
| 74 | + | ||
| 75 | + The current cleanup function handles the following environment variables. If new environment variables are added in the future, they need to be added to this function. | ||
| 76 | + | ||
| 77 | +  | ||
| 78 | + | ||
| 79 | +#### Log Level Configuration | ||
| 80 | + | ||
| 81 | +- Checker log level (default: ERROR) | ||
| 82 | + | ||
| 83 | + 1. Call the `setCheckerLogWarn()` interface to set the log level | ||
| 84 | + | ||
| 85 | + Set the checker log level to WARNING: | ||
| 86 | + | ||
| 87 | +  | ||
| 88 | + | ||
| 89 | + 2. Set the log level via environment variable | ||
| 90 | + | ||
| 91 | + Enable WARNING level logging: | ||
| 92 | + | ||
| 93 | + ```bash | ||
| 94 | + export CHECK_LOG_LEVEL=2 | ||
| 95 | + ``` | ||
| 96 | + | ||
| 97 | + Enable ERROR level logging: | ||
| 98 | + | ||
| 99 | + ```bash | ||
| 100 | + export CHECK_LOG_LEVEL=3 | ||
| 101 | + ``` | ||
| 102 | + | ||
| 103 | +- HCCL log level (default: ERROR) | ||
| 104 | + | ||
| 105 | + The current environment variable method is deprecated. Use the following approach to set the log level and print logs. | ||
| 106 | + | ||
| 107 | +  | ||
| 108 | + | ||
| 109 | + Enable DEBUG level logging: | ||
| 110 | + | ||
| 111 | + ```bash | ||
| 112 | + export ASCEND_LOG_LEVEL=0 | ||
| 113 | + ``` | ||
| 114 | + | ||
| 115 | + Enable INFO level logging: | ||
| 116 | + | ||
| 117 | + ```bash | ||
| 118 | + export ASCEND_LOG_LEVEL=1 | ||
| 119 | + ``` | ||
| 120 | + | ||
| 121 | + Enable WARNING level logging: | ||
| 122 | + | ||
| 123 | + ```bash | ||
| 124 | + export ASCEND_LOG_LEVEL=2 | ||
| 125 | + ``` | ||
| 126 | + | ||
| 127 | + Enable ERROR level logging: | ||
| 128 | + | ||
| 129 | + ```bash | ||
| 130 | + export ASCEND_LOG_LEVEL=3 | ||
| 131 | + ``` | ||
| 132 | + | ||
| 133 | +#### Operator Parameter Configuration | ||
| 134 | + | ||
| 135 | +`TestOpParam` is used to configure test parameters. The table below describes the main parameters used. | ||
| 136 | + | ||
| 137 | +| Parameter | Required or Optional | Description | Remarks | | ||
| 138 | +| ----------- | -------------------- | ------------------------------------------------------------- | ----------------------------------------------- | | ||
| 139 | +| opType | Required | Specifies the operator type to be tested | batchsendrecv is currently under development | | ||
| 140 | +| tag | Required | The tag of the operator execution entry | Can be set arbitrarily | | ||
| 141 | +| algName | Optional | Specifies the algorithm name to execute | If specified, skips algorithm selection; otherwise auto-selects | | ||
| 142 | +| opMode | Required | Specifies single operator mode or graph mode | | | ||
| 143 | +| reduceType | Optional | Required when a reduce type is involved | | | ||
| 144 | +| devtype | Required | The hardware type for execution | Supports 310P3 V / 310P3 Duo / 910A / 910B / 910C | | ||
| 145 | +| is310P3V | Optional | Must be set to true when running on 310P3 V hardware | | | ||
| 146 | +| count | Required | Number of data elements | | | ||
| 147 | +| dataType | Required | Data type | | | ||
| 148 | + | ||
| 149 | +#### Running the Checker | ||
| 150 | + | ||
| 151 | +Pass the `TestOpParam`, `rankTable`, `TopoMeta`, and other parameters generated in the previous steps to the `Check` function of the `Checker` object for execution. | ||
| 152 | + | ||
| 153 | +#### Verifying Check Results | ||
| 154 | + | ||
| 155 | +Check that the return value of `Check` is `HcclResult::HCCL_SUCCESS`. | ||
| 156 | + | ||
| 157 | +### LLT Test Case Filtering | ||
| 158 | + | ||
| 159 | +When there are many test cases and you only need to execute a specific one, modify the test case name in `main.cc`. | ||
| 160 | + | ||
| 161 | + | ||
| 162 | + | ||
| 163 | +## Test Execution | ||
| 164 | + | ||
| 165 | +Run the following commands from the source code root directory to compile and execute algorithm analyzer test cases: | ||
| 166 | + | ||
| 167 | +```bash | ||
| 168 | +# Compile all test suite cases and execute automatically | ||
| 169 | +bash build.sh --st | ||
| 170 | + | ||
| 171 | +# Compile individual test suite cases and execute automatically | ||
| 172 | +bash build.sh --open_hccl_test | ||
| 173 | +bash build.sh --executor_hccl_test | ||
| 174 | +bash build.sh --executor_reduce_hccl_test | ||
| 175 | +bash build.sh --executor_pipeline_hccl_test | ||
| 176 | + | ||
| 177 | +# Manually execute test cases | ||
| 178 | +./build/test/st/algorithm/testcase/testcase/open_hccl_test | ||
| 179 | +./build/test/st/algorithm/testcase/testcase/executor_hccl_test | ||
| 180 | +./build/test/st/algorithm/testcase/testcase/executor_reduce_hccl_test | ||
| 181 | +./build/test/st/algorithm/testcase/testcase/executor_pipeline_hccl_test | ||
| 182 | +``` | ||
| 183 | + | ||
| 184 | +## Result Examples | ||
| 185 | + | ||
| 186 | +### Result Analysis | ||
| 187 | + | ||
| 188 | +The test case execution results are shown below: | ||
| 189 | + | ||
| 190 | + | ||
| 191 | + | ||
| 192 | +The meaning of each field is as follows: | ||
| 193 | + | ||
| 194 | +`[run]`: Indicates the test case being executed | ||
| 195 | + | ||
| 196 | +`[OK]`: Indicates successful execution and verification passed | ||
| 197 | + | ||
| 198 | +`[FAIL]`: Indicates execution failure. Analyze the specific cause based on the printed logs. | ||
| 199 | + | ||
| 200 | +## Issue Location | ||
| 201 | + | ||
| 202 | +### Memory Conflict Detection | ||
| 203 | + | ||
| 204 | +#### Symptoms | ||
| 205 | + | ||
| 206 | +When a region of memory between two synchronization signals is concurrently written by multiple tasks, or is read while being written, a memory conflict occurs. In real runtime environments, this typically manifests as random accuracy issues. | ||
| 207 | + | ||
| 208 | +Under the current Mesh structure, false positives may occur when Reduce operators are present. This is because, in a Mesh structure, a memory block may be simultaneously written by other cards within one synchronization interval. In this case, the hardware can ensure the atomicity of the Reduce operation and no accuracy issues occur in actual execution. However, from the checker's perspective, multiple reads and writes to the same memory occur between two synchronizations, so it is flagged as an error. | ||
| 209 | + | ||
| 210 | +Except for the scenario above, the following error output indicates a risk of memory conflicts in the task scheduling: | ||
| 211 | + | ||
| 212 | +```text | ||
| 213 | +[1]there is memory use conflict in two SliceMemoryStatus | ||
| 214 | +[2]one is startAddr is 0, size is 3200, status is WRITE. | ||
| 215 | +[3]another is startAddr is 0, size is 3200, status is WRITE. | ||
| 216 | +[4]failed to check memory BufferType::OUTPUT_CCL | ||
| 217 | +[5]memory conflict between node [rankId:1, queueId:0, index:1] and node [rankId:2, queueId:0, index:1] | ||
| 218 | +[6]check rank memory conflict failed for rank 0 | ||
| 219 | +``` | ||
| 220 | + | ||
| 221 | +- Lines 2 and 3 indicate the start address, size, and read/write status of the two conflicting memory blocks. | ||
| 222 | + | ||
| 223 | + The status can be READ or WRITE. READ means the memory block is being read; WRITE means it is being written. Being read and being written are abstract memory operation semantics, not limited to write tasks and read tasks. | ||
| 224 | + | ||
| 225 | + Memory blocks that may have READ status include the src of a localcopy task, the src of a read task, and the src of a write task. Memory blocks that may have WRITE status include the dst of a localcopy task, the dst of a read task, and the dst of a write task. | ||
| 226 | + | ||
| 227 | +- Line 4 indicates the type of the conflicting memory block. | ||
| 228 | +- Line 5 indicates which two tasks caused the memory conflict. | ||
| 229 | +- Line 6 indicates the rank ID where the memory conflict occurred. | ||
| 230 | + | ||
| 231 | +The above error log indicates that two tasks are simultaneously performing write operations in the range 0 to 3200 of the `OUTPUT_CCL` type. | ||
| 232 | + | ||
| 233 | +#### Debugging Method | ||
| 234 | + | ||
| 235 | +1. Enable task printing before calling the `Check` function. | ||
| 236 | + | ||
| 237 | + ```text | ||
| 238 | + checker.EnableTaskPrint(); | ||
| 239 | + ``` | ||
| 240 | + | ||
| 241 | +2. Based on the error log, locate the two tasks that caused the memory conflict and check the synchronization arrangement before and after these two tasks. | ||
| 242 | + | ||
| 243 | + The error log in the [Symptoms](#symptoms) section indicates that two tasks are simultaneously performing write operations in the range 0 to 3200 of the `OUTPUT_CCL` type. | ||
| 244 | + | ||
| 245 | +### Semantic Validation Failure | ||
| 246 | + | ||
| 247 | +#### Basic Concepts | ||
| 248 | + | ||
| 249 | +The algorithm analyzer uses relative addresses to represent memory, consisting of three fields: memory type, offset address, and size, represented by the `DataSlice` structure: | ||
| 250 | + | ||
| 251 | +```c | ||
| 252 | +class DataSlice { | ||
| 253 | +public: | ||
| 254 | + // Some method functions | ||
| 255 | + | ||
| 256 | +private: | ||
| 257 | + BufferType type; | ||
| 258 | + u64 offset; | ||
| 259 | + u64 size; | ||
| 260 | +} | ||
| 261 | +``` | ||
| 262 | + | ||
| 263 | +The supported memory types include Input, Output, CCL_Input, CCL_Output, Scratch, etc. | ||
| 264 | + | ||
| 265 | +During collective communication algorithm execution, complex data movement and reduction operations are involved. The algorithm analyzer uses **BufferSemantic** to record **data movement relationships**, which includes a destination memory expression and multiple source memory expressions. The destination memory is represented by the member variables `startAddr` and `Size`. The source memory is represented by the `SrcBufDes` structure, which is defined as follows: | ||
| 266 | + | ||
| 267 | +```c | ||
| 268 | +struct BufferSemantic { | ||
| 269 | + u64 startAddr; | ||
| 270 | + mutable u64 size; // Size, shared between source and destination memory | ||
| 271 | + mutable bool isReduce; // Whether a reduce operation was performed; when multiple srcBufs exist, it must be a reduce scenario | ||
| 272 | + mutable HcclReduce0p reduceType; // Type of reduce operation | ||
| 273 | + mutable std::set<SrcBufDes> srcBufs; // The rank or ranks from which this data originates | ||
| 274 | +}; | ||
| 275 | + | ||
| 276 | +struct SrcBufDes { | ||
| 277 | + RankId rankId; // Rank ID of the data source | ||
| 278 | + BufferType bufType; // Memory type of the data source | ||
| 279 | + mutable u64 srcAddr; // Offset address relative to the source memory type | ||
| 280 | +}; | ||
| 281 | +``` | ||
| 282 | + | ||
| 283 | +#### Semantic Calculation Example | ||
| 284 | + | ||
| 285 | +The following example illustrates what semantic calculation is. | ||
| 286 | + | ||
| 287 | +1. Initial state: There are two ranks, Rank0 and Rank1, with two memory types: Input and Output. | ||
| 288 | + | ||
| 289 | +  | ||
| 290 | + | ||
| 291 | +2. Action of state 1: Copy the data block from rank0's Input at offset address 20, size 30, to rank0's Output at offset address 35. Result: A semantic block is generated on rank0's Output, recording the movement information. | ||
| 292 | + | ||
| 293 | +  | ||
| 294 | + | ||
| 295 | +3. Action of state 2: Copy the data block from rank1's Input at offset address 70, size 15, to rank0's Output at offset address 50. Result: The destination memory overlaps with an existing semantic block, so the existing semantic block needs to be split, producing two semantic blocks. | ||
| 296 | + | ||
| 297 | +  | ||
| 298 | + | ||
| 299 | +#### Result Validation | ||
| 300 | + | ||
| 301 | +During semantic analysis execution, many semantic blocks are generated (recording many data movement relationships). After execution, the semantic blocks in the Output memory are checked against expectations. | ||
| 302 | + | ||
| 303 | +The following uses a 2-rank AllGather example to illustrate the normal and abnormal scenarios for the semantic blocks in Rank0's Output memory. Assume the input data size is 100 bytes. | ||
| 304 | + | ||
| 305 | +- **Correct scenario:** | ||
| 306 | + | ||
| 307 | +  | ||
| 308 | + | ||
| 309 | +- **Incorrect scenario:** | ||
| 310 | + | ||
| 311 | +  | ||
| 312 | + | ||
| 313 | +#### Debugging Approach | ||
| 314 | + | ||
| 315 | +The semantic validation phase can detect two types of errors: | ||
| 316 | + | ||
| 317 | +- Missing data. | ||
| 318 | +- Incorrect data source. | ||
| 319 | + | ||
| 320 | +For reduction scenarios, similar issues can occur, such as missing participating ranks or different data offset addresses among participating ranks. Typically, the semantic check provides hints when reporting errors. Analysis should be performed using the hints along with the task sequence printed by the algorithm analyzer. | ||
| @@ -0,0 +1,3 @@ | |||
| 1 | +version https://git-lfs.github.com/spec/v1 | ||
| 2 | +oid sha256:f865223adb9926179552564b294c049019b9e82bcbf2399332a7be9e58ad20dd | ||
| 3 | +size 9105 | ||
| @@ -0,0 +1,3 @@ | |||
| 1 | +version https://git-lfs.github.com/spec/v1 | ||
| 2 | +oid sha256:ce05d9f486f674115ab9a743a0e648a72b6561714d96a3ece7a9e34314d16bcb | ||
| 3 | +size 35930 | ||
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
| 2 | -oid sha256:7651608d859af4c98ca2efbe693ec6fccb179ba2204f9ea4edd538840ee288f0 | 2 | +oid sha256:21b279f31e18493e4d1979b286c8a8cbf8df90000ddffb97d4378a08e8bb3a55 |
| 3 | -size 31696 | 3 | +size 17536 |
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
| 2 | -oid sha256:df673edfe1722cc3771cc52fd8a4f710abf468ab57ae5a0cebcf4c4ba18bfba4 | 2 | +oid sha256:08212de0e6773ac8d81478ee6d850e076afda8f9b788d919ef814dce861f14e3 |
| 3 | -size 50926 | 3 | +size 28997 |
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
| 2 | -oid sha256:8d31e92a63c9ec528c19e2fc053214c479ae59cfe9fd2df8bd0c700eed82a81d | 2 | +oid sha256:3d2bec60b9bc9b49d80132c87b61f971e7c7020a76e99870523d1de467d9d74a |
| 3 | -size 29353 | 3 | +size 16727 |
| @@ -1,3 +1,3 @@ | |||
| 1 | version https://git-lfs.github.com/spec/v1 | 1 | version https://git-lfs.github.com/spec/v1 |
| 2 | -oid sha256:7cba534dafabe6c596c36c9efabd5b2d30acb5a6104989542e25b0011e6fb3d9 | 2 | +oid sha256:8176a42227c2c88f7568bf2582d54d96375696783fd9091a1ca9b82a2dc1e7ed |
| 3 | -size 32227 | 3 | +size 17319 |
| @@ -0,0 +1,3 @@ | |||
| 1 | +version https://git-lfs.github.com/spec/v1 | ||
| 2 | +oid sha256:e7f48474510b0a4681bc58214efbf5efb464f29541866516c6d806d294cb0b99 | ||
| 3 | +size 66092 | ||
| @@ -0,0 +1,3 @@ | |||
| 1 | +version https://git-lfs.github.com/spec/v1 | ||
| 2 | +oid sha256:e54925bcadd5de339195278b714087924e52d937808ba8562978c7ec7bf9df56 | ||
| 3 | +size 14220 | ||
⚠️ [中] [图片替换] ccubuffer.png
三张 CCU 图片（ccubuffer.png、ccu_arch.png、ccu_in_950.png）被替换且体积大幅缩小（ccubuffer 206K→8K、ccu_arch 36K→14K、ccu_in_950 43K→10K）。
建议：请确认新图片内容正确、清晰度满足要求，非低质量误替换。