已合并
同步 CheckerL2 最新代码到 hccl_vm #3274
c15029001705创建于 7月1日
同步 CheckerL2 最新代码到 hccl_vm #3274
已合并
共 175 个文件变更+10697-6590
| @@ -93,6 +93,7 @@ include_directories( | |||
| 93 | 93 | ||
| 94 | function(config_hcclvm_install) | 94 | function(config_hcclvm_install) |
| 95 | install(DIRECTORY DESTINATION "data") | 95 | install(DIRECTORY DESTINATION "data") |
| 96 | + install(DIRECTORY DESTINATION "archive") | ||
| 96 | install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/config/" | 97 | install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/config/" |
| 97 | DESTINATION "config") | 98 | DESTINATION "config") |
| 98 | install(FILES | 99 | install(FILES |
| @@ -100,6 +101,7 @@ function(config_hcclvm_install) | |||
| 100 | "${CMAKE_CURRENT_SOURCE_DIR}/asset/generate_cluster_topo.sh" | 101 | "${CMAKE_CURRENT_SOURCE_DIR}/asset/generate_cluster_topo.sh" |
| 101 | "${CMAKE_CURRENT_SOURCE_DIR}/asset/generate_server_topo.sh" | 102 | "${CMAKE_CURRENT_SOURCE_DIR}/asset/generate_server_topo.sh" |
| 102 | "${CMAKE_CURRENT_SOURCE_DIR}/asset/hccl_config.sh" | 103 | "${CMAKE_CURRENT_SOURCE_DIR}/asset/hccl_config.sh" |
| 104 | + "${CMAKE_CURRENT_SOURCE_DIR}/asset/archive.sh" | ||
| 103 | DESTINATION "script" | 105 | DESTINATION "script" |
| 104 | PERMISSIONS | 106 | PERMISSIONS |
| 105 | OWNER_READ OWNER_WRITE OWNER_EXECUTE | 107 | OWNER_READ OWNER_WRITE OWNER_EXECUTE |
| @@ -31,7 +31,7 @@ chmod +x Ascend-cann-950-ops_9.1.0_linux-x86_64.run | |||
| 31 | 31 | ||
| 32 | hccl_test是昇腾官方提供的HCCL性能测试工具,详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html),HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。 | 32 | hccl_test是昇腾官方提供的HCCL性能测试工具,详见[HCCL性能测试工具](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910beta1/devaids/hccltool/HCCLpertest_16_0001.html),HCCL-VM支持在虚拟环境中运行hccl_test用例。请先参照[hccl_test用例构建](#42-hccl-test用例构建)章节进行用例二进制程序的编译。 |
| 33 | 33 | ||
| 34 | -备注:可选,未来支持Pytorch用例 | 34 | +备注:可选,未来支持Pytorch用例。 |
| 35 | 35 | ||
| 36 | --- | 36 | --- |
| 37 | 37 | ||
| @@ -64,11 +64,11 @@ bash ./build.sh --full | |||
| 64 | 64 | ||
| 65 | #### 3.2.1 环境配置 | 65 | #### 3.2.1 环境配置 |
| 66 | 66 | ||
| 67 | -请参照[hccl_rootinfo文件内容](#47-hccl_rootinfojson文件),创建并配置hccl_rootinfo.json文件, | 67 | +请参照[hccl_rootinfo文件内容](#47-hccl_rootinfojson文件),创建并配置hccl_rootinfo.json文件。 |
| 68 | 68 | ||
| 69 | #### 3.2.2 CCU模式 | 69 | #### 3.2.2 CCU模式 |
| 70 | 70 | ||
| 71 | -1. 环境变量配置 | 71 | +1. 环境变量配置。 |
| 72 | 72 | ||
| 73 | ```bash | 73 | ```bash |
| 74 | # 进入工具安装目录 | 74 | # 进入工具安装目录 |
| @@ -103,22 +103,24 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | |||
| 103 | (hvm)$> exit | 103 | (hvm)$> exit |
| 104 | ``` | 104 | ``` |
| 105 | 105 | ||
| 106 | -3. 验证hccl_test用例运行结果 [Runner结果查看](#491-runner插件结果) [Checker结果查看](#492-checker插件结果) | 106 | +3. 验证hccl_test用例运行结果 |
| 107 | +[Runner结果查看](#491-runner插件结果) | ||
| 108 | +[Checker结果查看](#492-checker插件结果) | ||
| 107 | 109 | ||
| 108 | #### 3.2.3 AICPU模式 | 110 | #### 3.2.3 AICPU模式 |
| 109 | 111 | ||
| 110 | -AICPU展开模式需要将算法展开步骤放到设备侧执行,因此hccl-vm工具需要将HCCL的设备侧的符号编译并模拟执行。由于设备侧符号是ARM架构的,因此在X86环境上编译时需要借助交叉编译器,运行时需要借助QEMU实现AICPU模式的模拟运行 | 112 | +AICPU展开模式需要将算法展开步骤放到设备侧执行,因此hccl-vm工具需要将HCCL的设备侧的符号编译并模拟执行。由于设备侧符号是ARM架构的,因此在X86环境上编译时需要借助交叉编译器,运行时需要借助QEMU实现AICPU模式的模拟运行。 |
| 111 | 113 | ||
| 112 | -设备侧符号使用hccl和hcomm的源码编译,为了保证Host与Device通信协议正确,需要同时编译Host侧的安装包并进行替换安装 | 114 | +设备侧符号使用hccl和hcomm的源码编译,为了保证Host与Device通信协议正确,需要同时编译Host侧的安装包并进行替换安装。 |
| 113 | 115 | ||
| 114 | -1. HCCL设备侧符号编译、安装、拷贝等 | 116 | +1. HCCL设备侧符号编译、安装、拷贝等。 |
| 115 | 117 | ||
| 116 | ```bash | 118 | ```bash |
| 117 | cd /home/workspace/hcomm/test/hccl_vm/ | 119 | cd /home/workspace/hcomm/test/hccl_vm/ |
| 118 | bash ./build_pkg.sh | 120 | bash ./build_pkg.sh |
| 119 | ``` | 121 | ``` |
| 120 | 122 | ||
| 121 | -2. 环境变量配置 | 123 | +2. 环境变量配置。 |
| 122 | 124 | ||
| 123 | ```bash | 125 | ```bash |
| 124 | # 进入工具安装目录 | 126 | # 进入工具安装目录 |
| @@ -156,9 +158,9 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | |||
| 156 | 158 | ||
| 157 | #### 3.2.4 AIV模式 | 159 | #### 3.2.4 AIV模式 |
| 158 | 160 | ||
| 159 | -**当前仅支持Runner插件** | 161 | +**当前仅支持Runner插件**。 |
| 160 | 162 | ||
| 161 | -1. 环境变量配置 | 163 | +1. 环境变量配置。 |
| 162 | 164 | ||
| 163 | ```bash | 165 | ```bash |
| 164 | # 进入工具安装目录 | 166 | # 进入工具安装目录 |
| @@ -193,7 +195,7 @@ cd /home/workspace/hcomm/test/hccl_vm/hccl_vm_install/bin | |||
| 193 | 195 | ||
| 194 | ### 3.3 Pytorch用例示例 | 196 | ### 3.3 Pytorch用例示例 |
| 195 | 197 | ||
| 196 | -暂不支持 | 198 | +暂不支持。 |
| 197 | 199 | ||
| 198 | ### 3.4 hccl代码修改验证示例 | 200 | ### 3.4 hccl代码修改验证示例 |
| 199 | 201 | ||
| @@ -208,8 +210,8 @@ export HCCL_CODE_HOME=/home/workspace/hccl | |||
| 208 | export HCOMM_CODE_HOME=/home/workspace/hcomm | 210 | export HCOMM_CODE_HOME=/home/workspace/hcomm |
| 209 | ``` | 211 | ``` |
| 210 | 212 | ||
| 211 | -1. 若您修改了CANN hccl仓代码,请执行bash build_pkg.sh --install hccl | 213 | +1. 若您修改了CANN hccl仓代码,请执行bash build_pkg.sh --install hccl。 |
| 212 | -2. 若您修改了CANN hcomm仓代码,请执行bash build_pkg.sh --install hcomm | 214 | +2. 若您修改了CANN hcomm仓代码,请执行bash build_pkg.sh --install hcomm。 |
| 213 | 3. 参考[使用示例](#32-使用示例)步骤,重新运行用例。 | 215 | 3. 参考[使用示例](#32-使用示例)步骤,重新运行用例。 |
| 214 | 216 | ||
| 215 | --- | 217 | --- |
| @@ -262,7 +264,7 @@ MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi make ASCEND_DIR=${ASCEND_HOME_PATH} | |||
| 262 | 264 | ||
| 263 | #### 4.2.2 MPICH环境编译 | 265 | #### 4.2.2 MPICH环境编译 |
| 264 | 266 | ||
| 265 | -假设mpich的路径为: `/usr/lib/mpich` | 267 | +假设mpich的路径为: `/usr/lib/mpich`。 |
| 266 | 268 | ||
| 267 | ```bash | 269 | ```bash |
| 268 | # 进入hccl_test用例源码目录 | 270 | # 进入hccl_test用例源码目录 |
| @@ -414,7 +416,7 @@ topology: | |||
| 414 | `topo.json` 和 `ranktable.json` 文件不需要手动创建,工具会根据以下信息自动生成: | 416 | `topo.json` 和 `ranktable.json` 文件不需要手动创建,工具会根据以下信息自动生成: |
| 415 | 417 | ||
| 416 | - **拓扑配置编号**:用户在启动时指定的编号(如 112、113 等) | 418 | - **拓扑配置编号**:用户在启动时指定的编号(如 112、113 等) |
| 417 | -- **芯片类型**:根据运行环境自动识别的芯片类型 | 419 | +- **芯片类型**:根据运行环境自动识别的芯片类型。 |
| 418 | 420 | ||
| 419 | 虽然配置文件由工具自动生成,但了解其结构有助于理解拓扑配置。 | 421 | 虽然配置文件由工具自动生成,但了解其结构有助于理解拓扑配置。 |
| 420 | 422 | ||
| @@ -478,11 +480,11 @@ topology: | |||
| 478 | 480 | ||
| 479 | **字段说明**: | 481 | **字段说明**: |
| 480 | 482 | ||
| 481 | -- `server.device_count`:设备总数 | 483 | +- `server.device_count`:设备总数。 |
| 482 | -- `server.groups`:设备分组信息 | 484 | +- `server.groups`:设备分组信息。 |
| 483 | -- `ports`:端口配置 | 485 | +- `ports`:端口配置。 |
| 484 | - `usage`:端口用途(`peer2peer` 表示设备间连接,`peer2net` 表示与外部连接) | 486 | - `usage`:端口用途(`peer2peer` 表示设备间连接,`peer2net` 表示与外部连接) |
| 485 | -- `links`:链路配置 | 487 | +- `links`:链路配置。 |
| 486 | - `link_type`:链路类型(`PEER2PEER` 或 `PEER2NET`) | 488 | - `link_type`:链路类型(`PEER2PEER` 或 `PEER2NET`) |
| 487 | - `topo_type`:拓扑类型(`1DMESH`、`CLOS` 等) | 489 | - `topo_type`:拓扑类型(`1DMESH`、`CLOS` 等) |
| 488 | 490 | ||
| @@ -508,11 +510,11 @@ topology: | |||
| 508 | 510 | ||
| 509 | **字段说明**: | 511 | **字段说明**: |
| 510 | 512 | ||
| 511 | -- `server_count`:服务器数量 | 513 | +- `server_count`:服务器数量。 |
| 512 | -- `device_count`:设备总数 | 514 | +- `device_count`:设备总数。 |
| 513 | -- `server_list`:服务器和设备列表 | 515 | +- `server_list`:服务器和设备列表。 |
| 514 | - - `device_ip`:设备 IP 地址 | 516 | + - `device_ip`:设备 IP 地址。 |
| 515 | - - `port`:设备端口号 | 517 | + - `port`:设备端口号。 |
| 516 | 518 | ||
| 517 | ### 4.4 hccl\_config.sh文件说明 | 519 | ### 4.4 hccl\_config.sh文件说明 |
| 518 | 520 | ||
| @@ -756,7 +758,7 @@ data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result: | |||
| 756 | 758 | ||
| 757 | ### 开源第三方软件依赖 | 759 | ### 开源第三方软件依赖 |
| 758 | 760 | ||
| 759 | -编译本项目时,依赖的第三方开源软件列表如下,离线编译场景可下载并重命名软件包后放置在本项目内的third_party目录下 | 761 | +编译本项目时,依赖的第三方开源软件列表如下,离线编译场景可下载并重命名软件包后放置在本项目内的third_party目录下。 |
| 760 | 762 | ||
| 761 | | 开源软件 | 版本 |下载地址 | | 763 | | 开源软件 | 版本 |下载地址 | |
| 762 | | ------------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | 764 | | ------------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | |
| @@ -781,5 +783,5 @@ data_size(Bytes): | aveg_time(us): | alg_bandwidth(GB/s): | check_result: | |||
| 781 | 783 | ||
| 782 | --- | 784 | --- |
| 783 | 785 | ||
| 784 | -**文档版本**:v1.0 | 786 | +**文档版本**:v1.1。 |
| 785 | -**最后更新**:2026-06-26 | 787 | +**最后更新**:2026-06-30。 |
| @@ -0,0 +1,101 @@ | |||
| 1 | +#!/bin/bash | ||
| 2 | +# Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | +# CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | +# Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | +# See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | + | ||
| 10 | +set -e | ||
| 11 | + | ||
| 12 | +MAX_ARCHIVE_COUNT=10 | ||
| 13 | + | ||
| 14 | +RED='\033[0;31m' | ||
| 15 | +GREEN='\033[0;32m' | ||
| 16 | +YELLOW='\033[1;33m' | ||
| 17 | +BLUE='\033[0;34m' | ||
| 18 | +NC='\033[0m' | ||
| 19 | + | ||
| 20 | +error_exit() { | ||
| 21 | + echo -e "${RED}[ERROR] $1${NC}" >&2 | ||
| 22 | + exit 1 | ||
| 23 | +} | ||
| 24 | + | ||
| 25 | +success_msg() { | ||
| 26 | + echo -e "${GREEN}[SUCCESS] $1${NC}" | ||
| 27 | +} | ||
| 28 | + | ||
| 29 | +info_msg() { | ||
| 30 | + echo -e "${BLUE}[INFO] $1${NC}" | ||
| 31 | +} | ||
| 32 | + | ||
| 33 | +warn_msg() { | ||
| 34 | + echo -e "${YELLOW}[WARN] $1${NC}" | ||
| 35 | +} | ||
| 36 | + | ||
| 37 | +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| 38 | +case "$(basename "${SCRIPT_DIR}")" in | ||
| 39 | + bin|script) | ||
| 40 | + INSTALL_DIR="$(dirname "${SCRIPT_DIR}")" | ||
| 41 | + ;; | ||
| 42 | + *) | ||
| 43 | + INSTALL_DIR="${SCRIPT_DIR}" | ||
| 44 | + ;; | ||
| 45 | +esac | ||
| 46 | + | ||
| 47 | +ARCHIVE_DIR="${INSTALL_DIR}/archive" | ||
| 48 | +LOGS_DIR="${INSTALL_DIR}/logs" | ||
| 49 | +DATA_DIR="${INSTALL_DIR}/data" | ||
| 50 | + | ||
| 51 | +mkdir -p "${ARCHIVE_DIR}" | ||
| 52 | + | ||
| 53 | +TIMESTAMP=$(date +"%Y%m%d_%H%M%S") | ||
| 54 | +ARCHIVE_SUBDIR="${ARCHIVE_DIR}/${TIMESTAMP}" | ||
| 55 | +mkdir -p "${ARCHIVE_SUBDIR}/logs" | ||
| 56 | +mkdir -p "${ARCHIVE_SUBDIR}/data" | ||
| 57 | + | ||
| 58 | +info_msg "Archive directory created: ${ARCHIVE_SUBDIR}" | ||
| 59 | + | ||
| 60 | +ARCHIVE_DIRS=($(ls -1dt "${ARCHIVE_DIR}"/*/ 2>/dev/null)) | ||
| 61 | +ARCHIVE_COUNT=${#ARCHIVE_DIRS[@]} | ||
| 62 | + | ||
| 63 | +if [[ ${ARCHIVE_COUNT} -gt ${MAX_ARCHIVE_COUNT} ]]; then | ||
| 64 | + DELETE_COUNT=$((ARCHIVE_COUNT - MAX_ARCHIVE_COUNT)) | ||
| 65 | + info_msg "Archive count (${ARCHIVE_COUNT}) exceeds limit (${MAX_ARCHIVE_COUNT}), removing ${DELETE_COUNT} oldest" | ||
| 66 | + for ((i=0; i<DELETE_COUNT; i++)); do | ||
| 67 | + OLDEST="${ARCHIVE_DIRS[$((ARCHIVE_COUNT - 1 - i))]}" | ||
| 68 | + warn_msg "Removing old archive: ${OLDEST}" | ||
| 69 | + rm -rf "${OLDEST}" | ||
| 70 | + done | ||
| 71 | +fi | ||
| 72 | + | ||
| 73 | +if [[ -d "${LOGS_DIR}" ]]; then | ||
| 74 | + shopt -s dotglob nullglob | ||
| 75 | + LOGS_CONTENTS=("${LOGS_DIR}"/*) | ||
| 76 | + if [[ ${#LOGS_CONTENTS[@]} -gt 0 ]]; then | ||
| 77 | + mv "${LOGS_DIR}"/* "${ARCHIVE_SUBDIR}/logs/" | ||
| 78 | + info_msg "Moved logs contents to ${ARCHIVE_SUBDIR}/logs/" | ||
| 79 | + else | ||
| 80 | + info_msg "Logs directory is empty, nothing to move" | ||
| 81 | + fi | ||
| 82 | + shopt -u dotglob nullglob | ||
| 83 | +else | ||
| 84 | + warn_msg "Logs directory not found: ${LOGS_DIR}" | ||
| 85 | +fi | ||
| 86 | + | ||
| 87 | +if [[ -d "${DATA_DIR}" ]]; then | ||
| 88 | + shopt -s dotglob nullglob | ||
| 89 | + DATA_CONTENTS=("${DATA_DIR}"/*) | ||
| 90 | + if [[ ${#DATA_CONTENTS[@]} -gt 0 ]]; then | ||
| 91 | + mv "${DATA_DIR}"/* "${ARCHIVE_SUBDIR}/data/" | ||
| 92 | + info_msg "Moved data contents to ${ARCHIVE_SUBDIR}/data/" | ||
| 93 | + else | ||
| 94 | + info_msg "Data directory is empty, nothing to move" | ||
| 95 | + fi | ||
| 96 | + shopt -u dotglob nullglob | ||
| 97 | +else | ||
| 98 | + warn_msg "Data directory not found: ${DATA_DIR}" | ||
| 99 | +fi | ||
| 100 | + | ||
| 101 | +success_msg "Archive completed: ${ARCHIVE_SUBDIR}" | ||
| @@ -0,0 +1,1229 @@ | |||
| 1 | +# HCCL-VM FAQ 测试文档 | ||
| 2 | + | ||
| 3 | +> 本文档用于测试FAQ HTML生成框架。 | ||
| 4 | + | ||
| 5 | +--- | ||
| 6 | + | ||
| 7 | +## 模块:HCCL-VM | ||
| 8 | + | ||
| 9 | +### 子模块:命令行 | ||
| 10 | + | ||
| 11 | +--- | ||
| 12 | + | ||
| 13 | +#### FAQ-E001 | ||
| 14 | + | ||
| 15 | +**标题:** 未配置通信域 | ||
| 16 | + | ||
| 17 | +**错误码:** `NA` (4) | ||
| 18 | + | ||
| 19 | +**错误函数:** `db_sim_runner_common.cc::GetDeviceByRankId()` | ||
| 20 | + | ||
| 21 | +**关键日志:** | ||
| 22 | +``` | ||
| 23 | +[error][PID:173579][TID:173579][db_sim_runner_common.cc][GetDeviceByRankId] cannot find rank by rank id 0 | ||
| 24 | +[error][PID:173579][TID:173579][aclrt_device_stub.cc][aclrtSetDevice] [DEVICE_STUB]device not found by rankId:0 | ||
| 25 | +acl interface return err ./common/src/hccl_test_common.cc:861, retcode: 100000. | ||
| 26 | +This is an error in device_init. | ||
| 27 | +``` | ||
| 28 | + | ||
| 29 | +**问题现象:** 执行业务用例,报找不到rank id为0的设备。 | ||
| 30 | + | ||
| 31 | +**可能原因:** | ||
| 32 | + 在执行业务用例前,需要用户自行判断本次算子所使用的通信域规模,并通过hccl-vm mock-comm aa命令配置通信域。其中aa.yaml文件所在路径为$HCCL_VM_INSTALL_DIR/config/topo_meta/aa.yaml。 | ||
| 33 | + | ||
| 34 | +--- | ||
| 35 | + | ||
| 36 | +#### FAQ-E002 | ||
| 37 | + | ||
| 38 | +**标题:** RANK_TABLE_FILE 未设置 | ||
| 39 | + | ||
| 40 | +**错误码:** `HCCL_SIM_E_PARA` (1) | ||
| 41 | + | ||
| 42 | +**错误函数:** `hccl_comm_stub.cc::HcclCommInitRootInfo()` | ||
| 43 | + | ||
| 44 | +**关键日志:** | ||
| 45 | +``` | ||
| 46 | +RANK_TABLE_FILE env not set, please check your config. | ||
| 47 | +``` | ||
| 48 | + | ||
| 49 | +**问题现象:** 通信域初始化时找不到 rank table 配置文件。 | ||
| 50 | + | ||
| 51 | +**可能原因:** | ||
| 52 | +1. 环境变量未设置 | ||
| 53 | +2. 文件路径错误 | ||
| 54 | + | ||
| 55 | +**解决方案:** | ||
| 56 | +```bash | ||
| 57 | +export RANK_TABLE_FILE=/path/to/rank_table.json | ||
| 58 | +``` | ||
| 59 | + | ||
| 60 | +--- | ||
| 61 | + | ||
| 62 | +#### FAQ-E003 | ||
| 63 | + | ||
| 64 | +**标题:** HCCL_VM_INSTALL_DIR 未设置 | ||
| 65 | + | ||
| 66 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 67 | + | ||
| 68 | +**错误函数:** `hccl_op_stub.cc::VirtualExecuteAivKernel()` | ||
| 69 | + | ||
| 70 | +**关键日志:** | ||
| 71 | +``` | ||
| 72 | +[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set, can not locate <path> for kernel <name> | ||
| 73 | +``` | ||
| 74 | + | ||
| 75 | +**问题现象:** AIV kernel 虚拟执行失败,找不到对应的 .so 文件。 | ||
| 76 | + | ||
| 77 | +**解决方案:** | ||
| 78 | +```bash | ||
| 79 | +export HCCL_VM_INSTALL_DIR=/path/to/hccl_vm/install/dir | ||
| 80 | +``` | ||
| 81 | + | ||
| 82 | +--- | ||
| 83 | + | ||
| 84 | +#### FAQ-E004 | ||
| 85 | + | ||
| 86 | +**标题:** 在子shell中重复执行start命令 | ||
| 87 | + | ||
| 88 | +**错误码:** `NA` (无错误码,仅WARNING) | ||
| 89 | + | ||
| 90 | +**错误函数:** `subcmd_start.cc::StartCommand::Execute()` | ||
| 91 | + | ||
| 92 | +**关键日志:** | ||
| 93 | +``` | ||
| 94 | +[warning][PID:<PID>][TID:<TID>][subcmd_start.cc][Execute] hccl-vm has already started. Please do not start it again in a sub-bash. | ||
| 95 | +``` | ||
| 96 | + | ||
| 97 | +**问题现象:** 在hvm子shell环境中再次执行`hccl-vm start`命令,系统提示已启动并忽略本次操作。 | ||
| 98 | + | ||
| 99 | +**可能原因:** `hccl-vm start`会fork一个子bash进程,用户在该子bash(提示符为`(hvm)$>`)内再次输入`hccl-vm start`时,系统拒绝重复启动。 | ||
| 100 | + | ||
| 101 | +**解决方法:** | ||
| 102 | + 不要在子shell内重复执行`hccl-vm start`。如需重新启动仿真环境,先退出当前子shell(输入`exit`),再重新执行`hccl-vm start`。 | ||
| 103 | + | ||
| 104 | +--- | ||
| 105 | + | ||
| 106 | +#### FAQ-E005 | ||
| 107 | + | ||
| 108 | +**标题:** fork子进程失败 | ||
| 109 | + | ||
| 110 | +**错误码:** `HCCL_SIM_HOST_ERROR_CMD` (无标准错误码) | ||
| 111 | + | ||
| 112 | +**错误函数:** `cmd_base_utils.cc::StartHvmCmd()` | ||
| 113 | + | ||
| 114 | +**关键日志:** | ||
| 115 | +``` | ||
| 116 | +fork failed: Resource temporarily unavailable | ||
| 117 | +``` | ||
| 118 | + | ||
| 119 | +**问题现象:** 执行`hccl-vm start`命令后,系统无法创建子shell进程,仿真环境启动失败。 | ||
| 120 | + | ||
| 121 | +**可能原因:** | ||
| 122 | +1. 系统用户进程数已达上限(ulimit -u) | ||
| 123 | +2. 系统内存不足,无法为新进程分配资源 | ||
| 124 | +3. PID资源耗尽(/proc/sys/kernel/pid_max) | ||
| 125 | + | ||
| 126 | +**排查步骤:** | ||
| 127 | +```bash | ||
| 128 | +ulimit -u | ||
| 129 | +cat /proc/sys/kernel/pid_max | ||
| 130 | +free -m | ||
| 131 | +ps -eLf | wc -l | ||
| 132 | +``` | ||
| 133 | + | ||
| 134 | +**解决方法:** | ||
| 135 | +1. 增大用户进程数限制:`ulimit -u <更大的值>` | ||
| 136 | +2. 清理系统中残留的僵尸进程 | ||
| 137 | +3. 检查是否有其他程序占用过多系统资源 | ||
| 138 | + | ||
| 139 | +--- | ||
| 140 | + | ||
| 141 | +#### FAQ-E006 | ||
| 142 | + | ||
| 143 | +**标题:** 插件名称格式错误 | ||
| 144 | + | ||
| 145 | +**错误码:** `NA` (CLI参数校验) | ||
| 146 | + | ||
| 147 | +**错误函数:** `subcmd_plugin.cc::PluginCommand::Setup()` | ||
| 148 | + | ||
| 149 | +**关键日志:** | ||
| 150 | +``` | ||
| 151 | +[HVM] [ERROR] Install plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 152 | +[HVM] [ERROR] Uninstall plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 153 | +[HVM] [ERROR] Run plugin : Invalid format! Plugin name must start with '@' (e.g., @myplugin). | ||
| 154 | +``` | ||
| 155 | + | ||
| 156 | +**问题现象:** 执行`hccl-vm plugin install/run/uninstall`命令时,CLI参数校验失败,拒绝执行。 | ||
| 157 | + | ||
| 158 | +**可能原因:** 插件名称未以`@`符号开头。例如输入`hccl-vm plugin install runner`而非`hccl-vm plugin install @runner`。 | ||
| 159 | + | ||
| 160 | +**解决方法:** | ||
| 161 | + 确保插件名称以`@`开头,例如: | ||
| 162 | +```bash | ||
| 163 | +hccl-vm plugin install @runner | ||
| 164 | +hccl-vm plugin install @checker | ||
| 165 | +hccl-vm plugin uninstall @runner | ||
| 166 | +``` | ||
| 167 | + | ||
| 168 | +--- | ||
| 169 | + | ||
| 170 | +#### FAQ-E007 | ||
| 171 | + | ||
| 172 | +**标题:** 拓扑配置文件不存在 | ||
| 173 | + | ||
| 174 | +**错误码:** `NA` (CLI参数校验) | ||
| 175 | + | ||
| 176 | +**错误函数:** `cmd_base_utils.cc::FileInModelDir()` | ||
| 177 | + | ||
| 178 | +**关键日志:** | ||
| 179 | +``` | ||
| 180 | +[HVM] model File not found: <install_path>/config/topo_meta/<name>.yaml | ||
| 181 | +``` | ||
| 182 | + | ||
| 183 | +**问题现象:** 执行`hccl-vm mock-comm <name>`命令时,指定的拓扑yaml配置文件不存在,CLI参数校验直接拒绝。通信域配置文件是用于描述算子执行通信域的规模(如通信域包含几个超节点,几个server,以及每个server内取哪几张卡,具体详见文件描述)。 | ||
| 184 | + | ||
| 185 | +**可能原因:** | ||
| 186 | +1. 指定的拓扑名称拼写错误 | ||
| 187 | +2. 对应的yaml文件未放置在`$HCCL_VM_INSTALL_DIR/config/topo_meta/`目录下 | ||
| 188 | +3. 文件扩展名错误(应为`.yaml`) | ||
| 189 | + | ||
| 190 | +**排查步骤:** | ||
| 191 | +```bash | ||
| 192 | +ls $HCCL_VM_INSTALL_DIR/config/topo_meta/ | ||
| 193 | +``` | ||
| 194 | + | ||
| 195 | +**解决方法:** | ||
| 196 | + 确认拓扑yaml文件已放置在正确目录中,且文件名与命令参数一致。例如执行`hccl-vm mock-comm 121`需要`config/topo_meta/121.yaml`文件存在。 | ||
| 197 | + | ||
| 198 | +--- | ||
| 199 | + | ||
| 200 | +#### FAQ-E008 | ||
| 201 | + | ||
| 202 | +**标题:** YAML拓扑文件格式解析异常 | ||
| 203 | + | ||
| 204 | +**错误码:** `NA` (运行时解析错误) | ||
| 205 | + | ||
| 206 | +**错误函数:** `cmd_cluster_model_utils.cc::ParseYamlTopoImpl()` | ||
| 207 | + | ||
| 208 | +**关键日志:** | ||
| 209 | +``` | ||
| 210 | +[error][PID:<PID>][TID:<TID>][cmd_cluster_model_utils.cc][ParseYamlTopoImpl] Exception when parsing YAML: <detail> | ||
| 211 | +``` | ||
| 212 | + | ||
| 213 | +**问题现象:** 执行`hccl-vm mock-comm <name>`命令时,YAML拓扑配置文件解析失败,通信域初始化中断。 | ||
| 214 | + | ||
| 215 | +**可能原因:** | ||
| 216 | +1. YAML文件存在语法错误(如缩进不正确、冒号后缺少空格、非法字符等) | ||
| 217 | +2. YAML文件中包含不支持的字段类型或格式 | ||
| 218 | +3. YAML文件编码非UTF-8 | ||
| 219 | + | ||
| 220 | +**排查步骤:** | ||
| 221 | +```bash | ||
| 222 | +# 使用python验证yaml格式 | ||
| 223 | +python3 -c "import yaml; yaml.safe_load(open('$HCCL_VM_INSTALL_DIR/config/topo_meta/<name>.yaml'))" | ||
| 224 | +``` | ||
| 225 | + | ||
| 226 | +**解决方法:** | ||
| 227 | + 根据日志中的`<detail>`信息修正YAML文件的语法错误。常见问题包括: | ||
| 228 | +1. 缩进必须使用空格,不能使用Tab | ||
| 229 | +2. 键值对的冒号后需要有空格 | ||
| 230 | +3. 列表项(`-`)的缩进需要与所在层级一致 | ||
| 231 | + | ||
| 232 | +--- | ||
| 233 | + | ||
| 234 | +### 子模块:内存管理 | ||
| 235 | + | ||
| 236 | +--- | ||
| 237 | + | ||
| 238 | +#### FAQ-M001 | ||
| 239 | + | ||
| 240 | +**标题:** 设备内存分配超限 | ||
| 241 | + | ||
| 242 | +**错误码:** `HCCL_SIM_E_MEMORY` (3) | ||
| 243 | + | ||
| 244 | +**错误函数:** `store_sim_device_memory_manager.cc::AllocPhyMem()` | ||
| 245 | + | ||
| 246 | +**关键日志:** | ||
| 247 | +``` | ||
| 248 | +dev:<N> alloc phy mem:<ADDR> size:<SIZE> exceeds pool ceiling:<CEILING>, reject | ||
| 249 | +``` | ||
| 250 | + | ||
| 251 | +**问题现象:** 设备内存分配请求超出模拟的内存池上限。 | ||
| 252 | + | ||
| 253 | +**图示说明:** | ||
| 254 | +```mermaid | ||
| 255 | +graph LR | ||
| 256 | + A[内存分配请求] --> B{检查池上限} | ||
| 257 | + B -->|未超限| C[分配成功] | ||
| 258 | + B -->|超限| D[拒绝分配] | ||
| 259 | + D --> E[报错:exceeds pool ceiling] | ||
| 260 | +``` | ||
| 261 | + | ||
| 262 | +--- | ||
| 263 | + | ||
| 264 | +#### FAQ-M002 | ||
| 265 | + | ||
| 266 | +**标题:** 共享内存创建失败 | ||
| 267 | + | ||
| 268 | +**错误码:** `HCCL_SIM_E_SYSCALL` (8) | ||
| 269 | + | ||
| 270 | +**错误函数:** `store_sim_shm_ops.cc::ShmCreate()` | ||
| 271 | + | ||
| 272 | +**关键日志:** | ||
| 273 | +``` | ||
| 274 | +[SHM_OPS] create: shm_open failed, name: <name> | ||
| 275 | +[SHM_OPS] create: ftruncate failed, name: <name> | ||
| 276 | +[SHM_OPS] create: mmap failed, name: <name> | ||
| 277 | +``` | ||
| 278 | + | ||
| 279 | +**问题现象:** 无法创建共享内存段。 | ||
| 280 | + | ||
| 281 | +**可能原因:** | ||
| 282 | +1. `/dev/shm` 空间不足 | ||
| 283 | +2. 权限不足 | ||
| 284 | +3. 同名共享内存已存在且冲突 | ||
| 285 | + | ||
| 286 | +**排查步骤:** | ||
| 287 | +```bash | ||
| 288 | +df -h /dev/shm | ||
| 289 | +ls /dev/shm/ | grep hccl | ||
| 290 | +``` | ||
| 291 | + | ||
| 292 | +--- | ||
| 293 | + | ||
| 294 | +#### FAQ-M003 | ||
| 295 | + | ||
| 296 | +**标题:** 通信内存分配失败 | ||
| 297 | + | ||
| 298 | +**错误码:** `HCCL_SIM_E_NOT_FOUND` (6) | ||
| 299 | + | ||
| 300 | +**错误函数:** `store_sim_comm_memory_manager.cc` | ||
| 301 | + | ||
| 302 | +**关键日志:** | ||
| 303 | +``` | ||
| 304 | +[COMM_MEM] alloc failed, name: <name> | ||
| 305 | +[COMM_MEM] acquire failed, name: <name> | ||
| 306 | +[COMM_MEM] write size too large, size: <N>, max: <MAX> | ||
| 307 | +``` | ||
| 308 | + | ||
| 309 | +**问题现象:** 跨进程通信内存操作失败。 | ||
| 310 | + | ||
| 311 | +--- | ||
| 312 | + | ||
| 313 | +### 子模块:桩代理(proxy) | ||
| 314 | + | ||
| 315 | +--- | ||
| 316 | + | ||
| 317 | +#### FAQ-PX001 | ||
| 318 | + | ||
| 319 | +**标题:** AIV Kernel虚拟执行失败 | ||
| 320 | + | ||
| 321 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 322 | + | ||
| 323 | +**错误函数:** `hccl_op_stub.cc::VirtualExecuteAivKernel()` | ||
| 324 | + | ||
| 325 | +**关键日志:** | ||
| 326 | +``` | ||
| 327 | +[virtual-aiv] env HCCL_VM_INSTALL_DIR is not set | ||
| 328 | +[virtual-aiv] missing aiv stub shared library, kernel=<name> | ||
| 329 | +[virtual-aiv] dlopen <so> failed, err = <error> | ||
| 330 | +[virtual-aiv] dlsym <symbol> from <so> failed, err = <error> | ||
| 331 | +``` | ||
| 332 | + | ||
| 333 | +**问题现象:** AIV kernel在虚拟环境中执行失败。 | ||
| 334 | + | ||
| 335 | +**排查步骤:** | ||
| 336 | +```bash | ||
| 337 | +echo $HCCL_VM_INSTALL_DIR | ||
| 338 | +ls -la $HCCL_VM_INSTALL_DIR/lib/aiv/ | ||
| 339 | +nm -D $HCCL_VM_INSTALL_DIR/lib/aiv/<kernel>.so | grep <symbol> | ||
| 340 | +``` | ||
| 341 | + | ||
| 342 | +--- | ||
| 343 | + | ||
| 344 | +#### FAQ-PX002 | ||
| 345 | + | ||
| 346 | +**标题:** 算子数据库记录失败 | ||
| 347 | + | ||
| 348 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 349 | + | ||
| 350 | +**错误函数:** `hccl_op_stub.cc::RecordOpDbInfo()` | ||
| 351 | + | ||
| 352 | +**关键日志:** | ||
| 353 | +``` | ||
| 354 | +[RecordOpDbInfo] insert op detail+mem failed | ||
| 355 | +[HcclAllReduce] record op db info failed | ||
| 356 | +``` | ||
| 357 | + | ||
| 358 | +**问题现象:** HCCL集合通信算子的参数无法写入仿真数据库。 | ||
| 359 | + | ||
| 360 | +**影响的算子:** AlltoAll, AlltoAllV, AllGather, Broadcast, AllReduce, Scatter, Reduce, ReduceScatter | ||
| 361 | + | ||
| 362 | +--- | ||
| 363 | + | ||
| 364 | +#### FAQ-PX003 | ||
| 365 | + | ||
| 366 | +**标题:** QP未找到或状态错误 | ||
| 367 | + | ||
| 368 | +**错误码:** `HCCL_SIM_E_NOT_FOUND` (6) | ||
| 369 | + | ||
| 370 | +**错误函数:** `hccp_stub.cc::RaSendWr()` | ||
| 371 | + | ||
| 372 | +**关键日志:** | ||
| 373 | +``` | ||
| 374 | +[HCCP] RaSendWr: QP <N> not found | ||
| 375 | +[HCCP] RaSendWr: QP <N> not in RTS state, current state:<N> | ||
| 376 | +``` | ||
| 377 | + | ||
| 378 | +**问题现象:** RDMA QP操作失败——QP不存在或未达到RTS状态。 | ||
| 379 | + | ||
| 380 | +**图示说明:** | ||
| 381 | +```mermaid | ||
| 382 | +stateDiagram-v2 | ||
| 383 | + [*] --> INIT | ||
| 384 | + INIT --> RTR: RaQpConnect | ||
| 385 | + RTR --> RTS: RaTypicalQpModify | ||
| 386 | + RTS --> [*]: 可以发送数据 | ||
| 387 | + RTS --> ERROR: 状态异常 | ||
| 388 | + INIT --> ERROR: 未正确初始化 | ||
| 389 | +``` | ||
| 390 | + | ||
| 391 | +--- | ||
| 392 | + | ||
| 393 | +#### FAQ-PX004 | ||
| 394 | + | ||
| 395 | +**标题:** EndPoint查找失败 | ||
| 396 | + | ||
| 397 | +**错误码:** `HCCL_SIM_E_NOT_FOUND` (6) | ||
| 398 | + | ||
| 399 | +**错误函数:** `hccp_stub.cc::RaCtxQpImport()` | ||
| 400 | + | ||
| 401 | +**关键日志:** | ||
| 402 | +``` | ||
| 403 | +[HCCP] cannot find endpoint addr:<IP> | ||
| 404 | +Get remote endpoint failed. ip:<IP>, eid:<EID> | ||
| 405 | +``` | ||
| 406 | + | ||
| 407 | +**问题现象:** 网络端点查找失败。 | ||
| 408 | + | ||
| 409 | +**可能原因:** IP地址不在rank table配置的端点列表中。 | ||
| 410 | + | ||
| 411 | +--- | ||
| 412 | + | ||
| 413 | +#### FAQ-PX005 | ||
| 414 | + | ||
| 415 | +**标题:** CCU微码加载失败 | ||
| 416 | + | ||
| 417 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 418 | + | ||
| 419 | +**错误函数:** `hccp_ccu_stub.cc::LoadMicrocodeInstruction()` | ||
| 420 | + | ||
| 421 | +**关键日志:** | ||
| 422 | +``` | ||
| 423 | +[LoadMicrocodeInstruction] get device by logic id <N> failed. | ||
| 424 | +[LoadMicrocodeInstruction] get ccu from device by die id <N> failed. | ||
| 425 | +[LoadMicrocodeInstruction] insert instr failed | ||
| 426 | +``` | ||
| 427 | + | ||
| 428 | +**问题现象:** CCU微码指令加载到模拟器失败。 | ||
| 429 | + | ||
| 430 | +--- | ||
| 431 | + | ||
| 432 | +#### FAQ-PX006 | ||
| 433 | + | ||
| 434 | +**标题:** 无法获取当前Context | ||
| 435 | + | ||
| 436 | +**错误码:** `HCCL_SIM_E_NOT_FOUND` (6) | ||
| 437 | + | ||
| 438 | +**错误函数:** `hccp_stub.cc::RaRdevInit()` | ||
| 439 | + | ||
| 440 | +**关键日志:** | ||
| 441 | +``` | ||
| 442 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaRdevInit] can not get CurrContext: <N> | ||
| 443 | +``` | ||
| 444 | + | ||
| 445 | +**问题现象:** 在RDMA设备初始化时,无法通过当前Runner获取活跃的Context,导致RDMA设备创建失败。 | ||
| 446 | + | ||
| 447 | +**可能原因:** | ||
| 448 | +1. 应用层未调用`aclrtSetDevice`/`aclrtCreateContext`初始化设备和上下文 | ||
| 449 | +2. Context已被提前销毁 | ||
| 450 | +3. Runner的TLS(线程局部存储)中current_ctx_id无效 | ||
| 451 | +4. 应用层在调用`aclrtSetDevice`初始化设备上下文前,调用了其他runtime接口获取了上下文 | ||
| 452 | + | ||
| 453 | +**排查步骤:** | ||
| 454 | +```bash | ||
| 455 | +# 检查Context表 | ||
| 456 | +hccl-vm table show Context | ||
| 457 | +# 检查Runner表中的current_ctx_id | ||
| 458 | +hccl-vm table show Runner | ||
| 459 | +``` | ||
| 460 | + | ||
| 461 | +**解决方法:** | ||
| 462 | + 确认应用层在调用RDMA操作前已正确调用`aclrtSetDevice`和`aclrtCreateContext`,且Context未被提前销毁。 | ||
| 463 | + | ||
| 464 | +--- | ||
| 465 | + | ||
| 466 | +#### FAQ-PX007 | ||
| 467 | + | ||
| 468 | +**标题:** 找不到AICPU二进制文件 | ||
| 469 | + | ||
| 470 | +**错误码:** `ACL_ERROR_RT_FEATURE_NOT_SUPPORT` | ||
| 471 | + | ||
| 472 | +**错误函数:** `aclrt_kernel_stub.cc::aclrtDestroyBinary()` | ||
| 473 | + | ||
| 474 | +**关键日志:** | ||
| 475 | +``` | ||
| 476 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][aclrtDestroyBinary] can not find this binary | ||
| 477 | +``` | ||
| 478 | + | ||
| 479 | +**问题现象:** 销毁AICPU二进制对象时,在全局kernel binary注册表中找不到对应的二进制句柄。 | ||
| 480 | + | ||
| 481 | +**可能原因:** | ||
| 482 | +1. 该二进制文件未被正确加载(`aclrtLoadBinary`未执行或失败) | ||
| 483 | +2. 二进制句柄已被重复销毁(double-free) | ||
| 484 | +3. 二进制对象在多线程环境下被并发操作导致状态不一致 | ||
| 485 | + | ||
| 486 | +**排查步骤:** | ||
| 487 | +```bash | ||
| 488 | +# 检查是否存在重复的destroy调用 | ||
| 489 | +# 确认aclrtLoadBinary的返回值 | ||
| 490 | +``` | ||
| 491 | + | ||
| 492 | +**解决方法:** | ||
| 493 | + 确保`aclrtLoadBinary`成功返回后再调用`aclrtDestroyBinary`,且不要对同一二进制对象重复销毁。 | ||
| 494 | + | ||
| 495 | +--- | ||
| 496 | + | ||
| 497 | +#### FAQ-PX008 | ||
| 498 | + | ||
| 499 | +**标题:** AICPU设备进程异常退出 | ||
| 500 | + | ||
| 501 | +**错误码:** `NA` (进程级错误) | ||
| 502 | + | ||
| 503 | +**错误函数:** `aclrt_kernel_stub.cc::WaitAicpuProcess()` | ||
| 504 | + | ||
| 505 | +**关键日志:** | ||
| 506 | +``` | ||
| 507 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] exited with status <N> | ||
| 508 | +[error][PID:<PID>][TID:<TID>][aclrt_kernel_stub.cc][WaitAicpuProcess] device process[<PID>] killed by signal <N> | ||
| 509 | +``` | ||
| 510 | + | ||
| 511 | +**问题现象:** AICPU设备子进程异常退出或被信号杀死,导致主进程随后也退出(`exit(EXIT_FAILURE)`)。 | ||
| 512 | + | ||
| 513 | +**可能原因:** | ||
| 514 | +1. AICPU进程内部发生未捕获异常或段错误 | ||
| 515 | +2. 系统资源不足(内存、文件描述符等)导致子进程被OOM killer杀死 | ||
| 516 | +3. AICPU二进制文件本身存在bug | ||
| 517 | +4. 子进程依赖的共享库缺失 | ||
| 518 | + | ||
| 519 | +**排查步骤:** | ||
| 520 | +```bash | ||
| 521 | +# 检查系统日志是否有OOM记录 | ||
| 522 | +dmesg | grep -i "oom\|killed" | ||
| 523 | +# 确认AICPU二进制文件是否完整 | ||
| 524 | +ls -la $HCCL_VM_INSTALL_DIR/bin/ | ||
| 525 | +# 检查系统资源 | ||
| 526 | +ulimit -a | ||
| 527 | +free -m | ||
| 528 | +``` | ||
| 529 | + | ||
| 530 | +**解决方法:** | ||
| 531 | +1. 检查AICPU二进制文件是否正确编译和部署 | ||
| 532 | +2. 确认系统资源充足(内存、文件描述符限制等) | ||
| 533 | +3. 如为信号杀死,根据信号编号(如11=SIGSEGV, 9=SIGKILL)进一步定位原因 | ||
| 534 | + | ||
| 535 | +--- | ||
| 536 | + | ||
| 537 | +#### FAQ-PX009 | ||
| 538 | + | ||
| 539 | +**标题:** CCU加载微码时找不到任何rank | ||
| 540 | + | ||
| 541 | +**错误码:** `HCCL_SIM_E_NOT_FOUND` (6) | ||
| 542 | + | ||
| 543 | +**错误函数:** `hccp_ccu_stub.cc::LoadMicrocodeInstruction()` | ||
| 544 | + | ||
| 545 | +**关键日志:** | ||
| 546 | +``` | ||
| 547 | +[error][PID:<PID>][TID:<TID>][hccp_ccu_stub.cc][LoadMicrocodeInstruction] can not find any rank | ||
| 548 | +``` | ||
| 549 | + | ||
| 550 | +**问题现象:** CCU微码指令加载过程中,无法在当前设备对应的Rank表中找到任何rank记录。 | ||
| 551 | + | ||
| 552 | +**可能原因:** | ||
| 553 | +1. 通信域未通过`mock-comm`命令初始化,Rank表为空 | ||
| 554 | +2. 当前设备ID在通信域配置中不存在 | ||
| 555 | + | ||
| 556 | +**排查步骤:** | ||
| 557 | +```bash | ||
| 558 | +# 检查Rank表是否有数据 | ||
| 559 | +hccl-vm table show Rank | ||
| 560 | +# 检查设备表 | ||
| 561 | +hccl-vm table show Device | ||
| 562 | +``` | ||
| 563 | + | ||
| 564 | +**解决方法:** | ||
| 565 | + 确保在执行CCU相关操作前,已通过`hccl-vm mock-comm`命令正确初始化通信域,且通信域配置覆盖当前设备。 | ||
| 566 | + | ||
| 567 | +--- | ||
| 568 | + | ||
| 569 | +#### FAQ-PX010 | ||
| 570 | + | ||
| 571 | +**标题:** 按rankId查找设备失败 | ||
| 572 | + | ||
| 573 | +**错误码:** `HCCL_E_NOT_FOUND` | ||
| 574 | + | ||
| 575 | +**错误函数:** `aclrt_device_stub.cc::hrtSetDevice()` | ||
| 576 | + | ||
| 577 | +**关键日志:** | ||
| 578 | +``` | ||
| 579 | +[error][PID:<PID>][TID:<TID>][aclrt_device_stub.cc][hrtSetDevice] device not found by rankId:<N> | ||
| 580 | +``` | ||
| 581 | + | ||
| 582 | +**问题现象:** 调用`aclrtSetDevice`设置当前设备时,根据rankId查找对应的设备失败。 | ||
| 583 | + | ||
| 584 | +**可能原因:** | ||
| 585 | +1. rankId超出了通信域中实际存在的rank范围 —— 如通信域配置4个NPU,但实际mpirun起了6个NPU进程,导致rankid 4, 5都报找不到device。 | ||
| 586 | +2. 通信域未初始化(未执行`mock-comm`命令) —— 【大概率】初始化通信域后,工具内部才会初始化Rank表 | ||
| 587 | +3. ranktable配置与实际使用的rank数量不匹配 —— 可能是`RANK_TABLE_FILE`配置了错误的文件路径 | ||
| 588 | + | ||
| 589 | +**排查步骤:** | ||
| 590 | +```bash | ||
| 591 | +# 检查rankId是否在有效范围内 | ||
| 592 | +hccl-vm table show Rank | ||
| 593 | +``` | ||
| 594 | + | ||
| 595 | +**解决方法:** | ||
| 596 | + 确认rankId在通信域配置的合法范围内(0 到 rank_count-1),且`RANK_TABLE_FILE`环境变量指向正确的ranktable.json文件。 | ||
| 597 | + | ||
| 598 | +--- | ||
| 599 | + | ||
| 600 | +#### FAQ-PX011 | ||
| 601 | + | ||
| 602 | +**标题:** 桩接口暂未实现 | ||
| 603 | + | ||
| 604 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) 或 `NA` | ||
| 605 | + | ||
| 606 | +**错误函数:** 多个桩函数文件(`hccp_stub.cc`、`ascend_hal_stub.cc`、`aclrt_kernel_stub.cc`等) | ||
| 607 | + | ||
| 608 | +**关键日志:** | ||
| 609 | +``` | ||
| 610 | +[warning][PID:<PID>][TID:<TID>][ascend_hal_stub.cc][*] [STUB] is empty | ||
| 611 | +[warning][PID:<PID>][TID:<TID>][hccp_stub.cc][*] [STUB] is empty | ||
| 612 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetAuxInfo] Not support yet | ||
| 613 | +[error][PID:<PID>][TID:<TID>][hccp_stub.cc][RaCtxGetCrErrInfoList] Not support yet | ||
| 614 | +``` | ||
| 615 | + | ||
| 616 | +**问题现象:** 应用层调用了仿真器尚未实现的底层驱动或运行时接口,日志中出现`[STUB] is empty`或`Not support yet`告警/错误。此类桩函数直接返回默认值(通常为0或成功),不进行任何实际操作。 | ||
| 617 | + | ||
| 618 | +**可能原因:** | ||
| 619 | + 仿真器当前版本仅实现了HCCL集合通信所需的核心接口子集。部分底层驱动接口(如drvGetDeviceCapability、RaCtxGetAuxInfo、drvMemPrefetch等)不在HCCL通信的核心路径上,因此桩函数体为空或标记为不支持。 | ||
| 620 | + 一般情况下,对于HCCL-VM工具支持的流程不会调用这些接口,因此不会出现此类告警。若用户调用错误应用层接口,或进入了错误的HCCL业务流程,可能会出现此类告警。 | ||
| 621 | + | ||
| 622 | +**涉及的主要接口类型:** | ||
| 623 | +1. **驱动层接口**(`ascend_hal_stub.cc`):drvGetDeviceCapability、drvMemPrefetch、drvStreamQuery等约315个接口 | ||
| 624 | +2. **RDMA接口**(`hccp_stub.cc`):RaRestoreSnapshot、RaRdevInitWithBackup、RaCtxGetAuxInfo等约44个接口 | ||
| 625 | +3. **Runtime适配层**(`adapter_rts_stub.cc`):部分aclrt扩展接口 | ||
| 626 | +4. **TSD客户端**(`tsd_client_stub.cc`):TSD相关接口 | ||
| 627 | + | ||
| 628 | +**解决方法:** | ||
| 629 | +1. 此类告警通常不影响HCCL算子的正确性仿真,可安全忽略 | ||
| 630 | +2. 如果该告警伴随功能异常,说明应用依赖了未实现的接口,请反馈给仿真器开发团队 | ||
| 631 | +3. 如需特定接口的桩实现,可联系开发团队优先适配 | ||
| 632 | + | ||
| 633 | +--- | ||
| 634 | + | ||
| 635 | +### 子模块:组网 | ||
| 636 | + | ||
| 637 | +--- | ||
| 638 | + | ||
| 639 | +#### FAQ-N001 | ||
| 640 | + | ||
| 641 | +**标题:** Ranktable环境变量配置错误 | ||
| 642 | + | ||
| 643 | +**错误码:** `NA` (1) | ||
| 644 | + | ||
| 645 | +**错误函数:** `param_check_v2.cc::RanktableRealPath` | ||
| 646 | + | ||
| 647 | +**关键日志:** | ||
| 648 | +``` | ||
| 649 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172019]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path | ||
| 650 | + | ||
| 651 | +[info][PID:172021][TID:172021][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172021][hrtGetDeviceRefresh]deviceLogicId[3] | ||
| 652 | +[error][PID:172020][TID:172020][log_stub.cc][DlogPrintStub] [HCCL_LOG][param_check_v2.cc:457][172020]RanktableRealPath: /home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/ranktable.json is not a valid real path | ||
| 653 | + | ||
| 654 | +[info][PID:172018][TID:172018][log_stub.cc][DlogPrintStub] [HCCL_LOG][adapter_rts.cc:234] [172018][hrtGetDeviceRefresh]deviceLogicId[0] | ||
| 655 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base_v2.cc:294][172019][HcclCommInitClusterInfoV2]call trace: hcclRet -> 1 | ||
| 656 | + | ||
| 657 | +[error][PID:172019][TID:172019][log_stub.cc][DlogPrintStub] [HCCL_LOG][op_base.cc:811] [172019][operator()]call trace: hcclRet -> 1 | ||
| 658 | +``` | ||
| 659 | + | ||
| 660 | +**问题现象:** 运行用例,初始化通信域失败。 | ||
| 661 | + | ||
| 662 | +**可能原因:** ranktable.json文件路径配置错误,请查看RANK_TABLE_FILE环境变量的配置。ranktable.json由工具生成,其路径为$HCCL_VM_INSTALL_DIR/data/ranktable.json。 | ||
| 663 | + | ||
| 664 | +**排查步骤:** | ||
| 665 | + | ||
| 666 | +```bash | ||
| 667 | +echo $RANK_TABLE_FILE | ||
| 668 | +``` | ||
| 669 | + | ||
| 670 | +**解决方法:** | ||
| 671 | + 确认RANK_TABLE_FILE环境变量配置正确,指向ranktable.json文件的路径。 | ||
| 672 | + | ||
| 673 | +--- | ||
| 674 | + | ||
| 675 | +#### FAQ-N002 | ||
| 676 | + | ||
| 677 | +**标题:** topo.json路径配置错误 | ||
| 678 | + | ||
| 679 | +**错误码:** `NA` (1) | ||
| 680 | + | ||
| 681 | +**错误函数:** `communicator_impl.cc::GetTopoFilePath` | ||
| 682 | + | ||
| 683 | +**关键日志:** | ||
| 684 | +``` | ||
| 685 | +[error][PID:172635][TID:172635][log_stub.cc][DlogPrintStub] [HCCL_LOG][communicator_impl.cc:1339][172635][GetTopoFilePath] topo_file_path[/home/teamserver/workspace/CheckerL2_2128/hccl_vm_install/topo.json] is not a valid real path | ||
| 686 | +``` | ||
| 687 | + | ||
| 688 | +**问题现象:** 运行用例,初始化通信域失败。 | ||
| 689 | + | ||
| 690 | +**可能原因:** topo.json文件路径在/etc/hccl_rootinfo.json文件中配置错误,请查看topo_file_path字段。topo.json由工具生成,其路径为$HCCL_VM_INSTALL_DIR/data/topo.json。 | ||
| 691 | + | ||
| 692 | +**排查步骤:** | ||
| 693 | + | ||
| 694 | +```bash | ||
| 695 | +echo $TOPO_FILE_PATH | ||
| 696 | +``` | ||
| 697 | + | ||
| 698 | +**解决方法:** | ||
| 699 | + 确认TOPO_FILE_PATH环境变量配置正确,指向topo.json文件的路径。 | ||
| 700 | + | ||
| 701 | +--- | ||
| 702 | + | ||
| 703 | +#### FAQ-N003 | ||
| 704 | + | ||
| 705 | +**标题:** mock-comm命令报错 | ||
| 706 | + | ||
| 707 | +**错误码:** `NA` (6) | ||
| 708 | + | ||
| 709 | +**错误函数:** `db_sim_runner_ops.cc::GetServerKeyById` | ||
| 710 | + | ||
| 711 | +**关键日志:** | ||
| 712 | +``` | ||
| 713 | +(hvm)$> hccl-vm mock-comm 144 | ||
| 714 | +[error][PID:172799][TID:172875][db_sim_runner_ops.cc][GetServerKeyById] can not find server by id: 0, 2 | ||
| 715 | +[error][PID:172799][TID:172875][topo_ascend_cluster_parser.cc][InitDynamicModelData] cannot find device by physical id 0 | ||
| 716 | +[error][PID:172799][TID:172875][cmd_base_utils.cc][InitHvmCommEnv] [HVM] InitHvmCommEnv failed | ||
| 717 | +[error][PID:172799][TID:172875][subcmd_mock_comm.cc][Execute] [HVM] Failed to initialize mock communication environment. Cleaning up environment. | ||
| 718 | +``` | ||
| 719 | + | ||
| 720 | +**问题现象:** 运行用例前,通过mock-comm命令配置通信域失败。 | ||
| 721 | + | ||
| 722 | +**可能原因:** mock-comm命令配置的通信域144配置,超过了工具启动的集群配置。比如工具启动的集群,一个超节点只有2个server,但通信域144表示该超节点下有4个server。 | ||
| 723 | + | ||
| 724 | +**解决方法:** | ||
| 725 | + 检查工具启动的集群配置,确认每个超节点下的server数量。若确实需要配置144通信域,则确保工具启动时选择一个更大的集群组网配置。 | ||
| 726 | + | ||
| 727 | +**排查步骤:** | ||
| 728 | + 确认工具启动时的集群配置文件和mock-comm命令配置的通信域配置文件。 | ||
| 729 | + | ||
| 730 | +**解决方法:** | ||
| 731 | + 确保mock-comm命令配置的通信域,不超过工具启动的集群配置。 | ||
| 732 | + | ||
| 733 | +--- | ||
| 734 | + | ||
| 735 | +#### FAQ-N004 | ||
| 736 | + | ||
| 737 | +**标题:** EndPoint IP 查找失败 | ||
| 738 | + | ||
| 739 | +**错误码:** `HCCL_SIM_E_NOT_FOUND` (6) | ||
| 740 | + | ||
| 741 | +**错误函数:** `topo_ascend_cluster_parser.cc::AddLinkInfo()` | ||
| 742 | + | ||
| 743 | +**关键日志:** | ||
| 744 | +``` | ||
| 745 | +cannot find endPoint by ip <IP_ADDR> | ||
| 746 | +``` | ||
| 747 | + | ||
| 748 | +**问题现象:** 网络链路配置中引用的 IP 地址在拓扑中不存在。 | ||
| 749 | + | ||
| 750 | +--- | ||
| 751 | + | ||
| 752 | +### 子模块:数据库 | ||
| 753 | + | ||
| 754 | +--- | ||
| 755 | + | ||
| 756 | +#### FAQ-DB001 | ||
| 757 | + | ||
| 758 | +**标题:** SQLite 数据库连接失败 | ||
| 759 | + | ||
| 760 | +**错误码:** `HCCL_SIM_E_OPEN_FILE_FAILURE` (10) | ||
| 761 | + | ||
| 762 | +**错误函数:** `db_hccl_db_sqlite.cc::Connect()` | ||
| 763 | + | ||
| 764 | +**关键日志:** | ||
| 765 | +``` | ||
| 766 | +[dbInit] Connect database failed | ||
| 767 | +Connect database:<path> failed | ||
| 768 | +``` | ||
| 769 | + | ||
| 770 | +**问题现象:** 无法连接到 SQLite 数据库文件。 | ||
| 771 | + | ||
| 772 | +**可能原因:** | ||
| 773 | +1. 数据库文件不存在 | ||
| 774 | +2. 文件权限不足 | ||
| 775 | +3. 文件被其他进程锁定 | ||
| 776 | + | ||
| 777 | +--- | ||
| 778 | + | ||
| 779 | +#### FAQ-DB002 | ||
| 780 | + | ||
| 781 | +**标题:** 数据库备份文件未找到 | ||
| 782 | + | ||
| 783 | +**错误码:** `HCCL_SIM_E_OPEN_FILE_FAILURE` (10) | ||
| 784 | + | ||
| 785 | +**错误函数:** `sim_loader.cc::BackupDatabase()` | ||
| 786 | + | ||
| 787 | +**关键日志:** | ||
| 788 | +``` | ||
| 789 | +[Loader] Backup database file not found: <dbPath> | ||
| 790 | +``` | ||
| 791 | + | ||
| 792 | +**问题现象:** Loader 无法找到仿真数据库文件。 | ||
| 793 | + | ||
| 794 | +**可能原因:** | ||
| 795 | +1. 仿真数据文件路径配置错误 | ||
| 796 | +2. 仿真数据尚未生成 | ||
| 797 | +3. 文件权限不足 | ||
| 798 | + | ||
| 799 | +**排查步骤:** | ||
| 800 | +```bash | ||
| 801 | +ls -la <dbPath> | ||
| 802 | +``` | ||
| 803 | + | ||
| 804 | +--- | ||
| 805 | + | ||
| 806 | +#### FAQ-DB003 | ||
| 807 | + | ||
| 808 | +**标题:** SQLite 查询失败 | ||
| 809 | + | ||
| 810 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 811 | + | ||
| 812 | +**错误函数:** `db_hccl_db_sqlite.cc` | ||
| 813 | + | ||
| 814 | +**关键日志:** | ||
| 815 | +``` | ||
| 816 | +Prepare failed: <error> sql:<SQL> | ||
| 817 | +Step failed: <error>, sql:<SQL> | ||
| 818 | +``` | ||
| 819 | + | ||
| 820 | +**问题现象:** SQL 查询执行失败。 | ||
| 821 | + | ||
| 822 | +**可能原因:** | ||
| 823 | +1. 数据库表结构不匹配(版本不兼容) | ||
| 824 | +2. 数据库文件损坏 | ||
| 825 | +3. 磁盘空间不足 | ||
| 826 | + | ||
| 827 | +--- | ||
| 828 | + | ||
| 829 | +## 模块:插件 | ||
| 830 | + | ||
| 831 | +### 子模块:checker | ||
| 832 | + | ||
| 833 | +--- | ||
| 834 | + | ||
| 835 | +##### HCCL_SIM_E_INTERNAL (4) | ||
| 836 | + | ||
| 837 | +--- | ||
| 838 | + | ||
| 839 | +#### FAQ-C001 | ||
| 840 | + | ||
| 841 | +**标题:** 内存切片溢出 | ||
| 842 | + | ||
| 843 | +**错误函数:** `task_graph_single_task_check_v3.cc::CheckMemorySlice()` | ||
| 844 | + | ||
| 845 | +**关键日志:** | ||
| 846 | +``` | ||
| 847 | +[TaskGraphSingleTaskCheckV3] Memory slice overflow while accumulating coverage, <detail> | ||
| 848 | +[TaskGraphSingleTaskCheckV3] Memory slice overflow, node=<node>, slice=<slice> | ||
| 849 | +``` | ||
| 850 | + | ||
| 851 | +**问题现象:** 单个任务的内存切片覆盖范围超出了目标buffer的总大小。 | ||
| 852 | + | ||
| 853 | +**可能原因:** | ||
| 854 | +1. HCCL算法层计算的内存偏移量有误(HCCL业务问题) | ||
| 855 | +2. 仿真数据中内存布局信息与任务参数不匹配(工具数据问题) | ||
| 856 | + | ||
| 857 | +**图示说明:** | ||
| 858 | +```mermaid | ||
| 859 | +graph LR | ||
| 860 | + A[任务内存切片] --> B[目标Buffer] | ||
| 861 | + A -.->|超出边界| C[溢出报错] | ||
| 862 | + style C fill:#f96,stroke:#333 | ||
| 863 | +``` | ||
| 864 | + | ||
| 865 | +--- | ||
| 866 | + | ||
| 867 | +#### FAQ-C002 | ||
| 868 | + | ||
| 869 | +**标题:** 缓冲区语义不完整 | ||
| 870 | + | ||
| 871 | +**错误函数:** `task_graph_semantic_check_v3.cc::CheckBufferContinuity()` | ||
| 872 | + | ||
| 873 | +**关键日志:** | ||
| 874 | +``` | ||
| 875 | +[TaskGraphSemanticCheckV3] Head gap, expect start=0x<ADDR>, actual start=0x<ADDR> | ||
| 876 | +[TaskGraphSemanticCheckV3] Middle gap, prev end=0x<ADDR>, cur start=0x<ADDR> | ||
| 877 | +[TaskGraphSemanticCheckV3] Tail gap, expect end=0x<ADDR>, actual end=0x<ADDR> | ||
| 878 | +``` | ||
| 879 | + | ||
| 880 | +**问题现象:** 目标buffer的数据语义覆盖存在空洞。 | ||
| 881 | + | ||
| 882 | +**可能原因:** | ||
| 883 | +1. HCCL算法遗漏了部分数据区域 | ||
| 884 | +2. Checker未能正确追踪传输路径 | ||
| 885 | + | ||
| 886 | +**图示说明:** | ||
| 887 | +```mermaid | ||
| 888 | +graph TB | ||
| 889 | + subgraph 目标Buffer | ||
| 890 | + A[地址0x0] --> B[地址0x100] | ||
| 891 | + B -.->|空洞| C[地址0x200] | ||
| 892 | + C --> D[地址0x300] | ||
| 893 | + end | ||
| 894 | + style B fill:#ff9,stroke:#333 | ||
| 895 | + style C fill:#ff9,stroke:#333 | ||
| 896 | +``` | ||
| 897 | + | ||
| 898 | +--- | ||
| 899 | + | ||
| 900 | +#### FAQ-C003 | ||
| 901 | + | ||
| 902 | +**标题:** Reduce语义错误 | ||
| 903 | + | ||
| 904 | +**错误函数:** `task_graph_semantic_check_v3.cc::CheckReduceSemantics()` | ||
| 905 | + | ||
| 906 | +**关键日志:** | ||
| 907 | +``` | ||
| 908 | +[TaskGraphSemanticCheckV3] Reduce type mismatch, pair=<pair> | ||
| 909 | +[TaskGraphSemanticCheckV3] Duplicate reduce source, pair=<pair>, srcOffset=0x<ADDR> | ||
| 910 | +[TaskGraphSemanticCheckV3] Destination reduce semantic incomplete, pair=<pair> | ||
| 911 | +``` | ||
| 912 | + | ||
| 913 | +**问题现象:** Reduce操作的数据语义校验失败。 | ||
| 914 | + | ||
| 915 | +**可能原因:** | ||
| 916 | +1. Reduce操作的数据类型不匹配 | ||
| 917 | +2. 存在重复的reduce源 | ||
| 918 | +3. 目标buffer未被所有reduce源完全覆盖 | ||
| 919 | + | ||
| 920 | +--- | ||
| 921 | + | ||
| 922 | +##### HCCL_SIM_E_PARA (1) | ||
| 923 | + | ||
| 924 | +--- | ||
| 925 | + | ||
| 926 | +#### FAQ-C004 | ||
| 927 | + | ||
| 928 | +**标题:** rankSize 为零 | ||
| 929 | + | ||
| 930 | +**错误函数:** `task_graph_semantic_check_v3.cc` | ||
| 931 | + | ||
| 932 | +**关键日志:** | ||
| 933 | +``` | ||
| 934 | +[TaskGraphSemanticCheckV3] rankSize is zero | ||
| 935 | +``` | ||
| 936 | + | ||
| 937 | +**问题现象:** 语义检查时参与通信的rank数量为0。 | ||
| 938 | + | ||
| 939 | +**可能原因:** 通信域未正确初始化,或rank table解析失败。 | ||
| 940 | + | ||
| 941 | +--- | ||
| 942 | + | ||
| 943 | +#### FAQ-C005 | ||
| 944 | + | ||
| 945 | +**标题:** Batch Trans 配对大小不匹配 | ||
| 946 | + | ||
| 947 | +**错误函数:** `task_graph_single_task_check_v3.cc::CheckBatchTrans()` | ||
| 948 | + | ||
| 949 | +**关键日志:** | ||
| 950 | +``` | ||
| 951 | +[TaskGraphSingleTaskCheckV3] Batch trans slice length mismatch, node=<node>, label=<label>, index=<N> | ||
| 952 | +[TaskGraphSingleTaskCheckV3] Batch trans pair size mismatch, node=<node>, label=<label> | ||
| 953 | +``` | ||
| 954 | + | ||
| 955 | +**问题现象:** 批量传输操作中,切片长度或配对数量不匹配。 | ||
| 956 | + | ||
| 957 | +**可能原因:** AlltoAll等算子的rank间数据分布不均等。 | ||
| 958 | + | ||
| 959 | +--- | ||
| 960 | + | ||
| 961 | +##### HCCL_SIM_E_NOT_SUPPORT (5) | ||
| 962 | + | ||
| 963 | +--- | ||
| 964 | + | ||
| 965 | +#### FAQ-C006 | ||
| 966 | + | ||
| 967 | +**标题:** 不支持的内存类型 | ||
| 968 | + | ||
| 969 | +**错误函数:** `task_graph_single_task_check_v3.cc` | ||
| 970 | + | ||
| 971 | +**关键日志:** | ||
| 972 | +``` | ||
| 973 | +[TaskGraphSingleTaskCheckV3] Unsupported memory type, node=<node>, slice=<slice> | ||
| 974 | +[TaskGraphSingleTaskCheckV3] Invalid memory slice, node=<node>, slice=<slice> | ||
| 975 | +``` | ||
| 976 | + | ||
| 977 | +**问题现象:** 内存切片的类型不在Checker支持的范围内。 | ||
| 978 | + | ||
| 979 | +--- | ||
| 980 | + | ||
| 981 | +##### HCCL_SIM_E_OPEN_FILE_FAILURE (10) | ||
| 982 | + | ||
| 983 | +--- | ||
| 984 | + | ||
| 985 | +#### FAQ-C007 | ||
| 986 | + | ||
| 987 | +**标题:** Dump文件写入失败 | ||
| 988 | + | ||
| 989 | +**错误函数:** `dump_manager.cc`, `dump_v3_manager.cc` | ||
| 990 | + | ||
| 991 | +**关键日志:** | ||
| 992 | +``` | ||
| 993 | +[DumpManager::WriteMsgpackFile] failed to open file: <path> | ||
| 994 | +[DumpManager::WriteJsonFile] json serialize failed: <error>, file: <path> | ||
| 995 | +[DumpV3Manager::WriteMsgpack] failed to open file: <path> | ||
| 996 | +``` | ||
| 997 | + | ||
| 998 | +**问题现象:** Checker的中间结果dump文件写入失败。 | ||
| 999 | + | ||
| 1000 | +**可能原因:** | ||
| 1001 | +1. 磁盘空间不足 | ||
| 1002 | +2. 目录不存在或无写权限 | ||
| 1003 | + | ||
| 1004 | +--- | ||
| 1005 | + | ||
| 1006 | +#### FAQ-C008 | ||
| 1007 | + | ||
| 1008 | +**标题:** 二进制文件magic number不匹配 | ||
| 1009 | + | ||
| 1010 | +**错误函数:** `binary_data_operator.cc::FileHeaderRead()` | ||
| 1011 | + | ||
| 1012 | +**关键日志:** | ||
| 1013 | +``` | ||
| 1014 | +[FileHeaderRead] Unmatched magic number:0x<N>≠0x<M> | ||
| 1015 | +``` | ||
| 1016 | + | ||
| 1017 | +**问题现象:** 读取仿真数据文件时,文件头中的magic number不匹配。 | ||
| 1018 | + | ||
| 1019 | +**可能原因:** | ||
| 1020 | +1. 数据文件版本与工具版本不兼容 | ||
| 1021 | +2. 文件损坏 | ||
| 1022 | + | ||
| 1023 | +--- | ||
| 1024 | + | ||
| 1025 | +### 子模块:runner | ||
| 1026 | + | ||
| 1027 | +--- | ||
| 1028 | + | ||
| 1029 | +#### FAQ-R001 | ||
| 1030 | + | ||
| 1031 | +**标题:** CCU版本不支持 | ||
| 1032 | + | ||
| 1033 | +**错误码:** `HCCL_SIM_E_NOT_SUPPORT` (5) | ||
| 1034 | + | ||
| 1035 | +**错误函数:** `ccu_resource_manager.cc::Init()` | ||
| 1036 | + | ||
| 1037 | +**关键日志:** | ||
| 1038 | +``` | ||
| 1039 | +[CcuResourceManager][Init] ccu version <N> not supported | ||
| 1040 | +Invalid ccu version:<version> | ||
| 1041 | +``` | ||
| 1042 | + | ||
| 1043 | +**问题现象:** CCU执行器不支持当前设备的CCU版本。 | ||
| 1044 | + | ||
| 1045 | +**可能原因:** 仿真数据中的CCU版本与工具支持的版本不匹配。 | ||
| 1046 | + | ||
| 1047 | +**图示说明:** | ||
| 1048 | +```mermaid | ||
| 1049 | +graph TD | ||
| 1050 | + A[加载CCU微码] --> B{版本检查} | ||
| 1051 | + B -->|CCU_V1| C[执行V1指令集] | ||
| 1052 | + B -->|CCU_V2| D[执行V2指令集] | ||
| 1053 | + B -->|其他版本| E[报错:版本不支持] | ||
| 1054 | +``` | ||
| 1055 | + | ||
| 1056 | +--- | ||
| 1057 | + | ||
| 1058 | +#### FAQ-R002 | ||
| 1059 | + | ||
| 1060 | +**标题:** 数据传输大小为0 | ||
| 1061 | + | ||
| 1062 | +**错误码:** `HCCL_SIM_E_PARA` (1) | ||
| 1063 | + | ||
| 1064 | +**错误函数:** `trans_loc_ms_to_loc_mem_executor.cc::Run()` | ||
| 1065 | + | ||
| 1066 | +**关键日志:** | ||
| 1067 | +``` | ||
| 1068 | +[TransLocMSToLocMemExecutor] The size of data transfer is 0. | ||
| 1069 | +``` | ||
| 1070 | + | ||
| 1071 | +**问题现象:** 数据传输操作的数据量为0。 | ||
| 1072 | + | ||
| 1073 | +**可能原因:** | ||
| 1074 | +1. 源地址和目的地址相同 | ||
| 1075 | +2. CCU指令参数计算错误 | ||
| 1076 | + | ||
| 1077 | +--- | ||
| 1078 | + | ||
| 1079 | +#### FAQ-R003 | ||
| 1080 | + | ||
| 1081 | +**标题:** AIV初始化失败 | ||
| 1082 | + | ||
| 1083 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 1084 | + | ||
| 1085 | +**错误函数:** `hccl_task_thread.cc` | ||
| 1086 | + | ||
| 1087 | +**关键日志:** | ||
| 1088 | +``` | ||
| 1089 | +Failed to init AivGraphExecutor, rankId=<N>, streamId=<N>, launchIndex=<N> | ||
| 1090 | +AivGraphExecutor failed, rankId=<N>, streamId=<N>, launchIndex=<N>, ret=<N> | ||
| 1091 | +``` | ||
| 1092 | + | ||
| 1093 | +**问题现象:** AIV图执行器初始化或执行失败。 | ||
| 1094 | + | ||
| 1095 | +--- | ||
| 1096 | + | ||
| 1097 | +#### FAQ-R004 | ||
| 1098 | + | ||
| 1099 | +**标题:** CCU指令执行失败 | ||
| 1100 | + | ||
| 1101 | +**错误码:** `HCCL_SIM_E_INTERNAL` (4) | ||
| 1102 | + | ||
| 1103 | +**错误函数:** `ccu_simulator.cc::ExecuteInstr()` | ||
| 1104 | + | ||
| 1105 | +**关键日志:** | ||
| 1106 | +``` | ||
| 1107 | +[CcuSimulator][ExecuteInstr] locCcu[<rank>:<die>], Invalid curInstrId_[<N>], endInstrId[<M>] | ||
| 1108 | +``` | ||
| 1109 | + | ||
| 1110 | +**问题现象:** CCU微码指令执行时,当前指令ID无效。 | ||
| 1111 | + | ||
| 1112 | +--- | ||
| 1113 | + | ||
| 1114 | +#### FAQ-R005 | ||
| 1115 | + | ||
| 1116 | +**标题:** CCU资源地址越界 | ||
| 1117 | + | ||
| 1118 | +**错误码:** `HCCL_SIM_E_PARA` (1) | ||
| 1119 | + | ||
| 1120 | +**错误函数:** `ccu_resource_manager.cc` | ||
| 1121 | + | ||
| 1122 | +**关键日志:** | ||
| 1123 | +``` | ||
| 1124 | +dieId is out of range, dieId=[<N>] | ||
| 1125 | +msAddr is out of range, msAddr=[<N>],addr=[<N>] | ||
| 1126 | +``` | ||
| 1127 | + | ||
| 1128 | +**问题现象:** CCU资源访问时地址或ID越界。 | ||
| 1129 | + | ||
| 1130 | +--- | ||
| 1131 | + | ||
| 1132 | +#### FAQ-R006 | ||
| 1133 | + | ||
| 1134 | +**标题:** 内存搬运地址无效 | ||
| 1135 | + | ||
| 1136 | +**错误码:** `HCCL_SIM_E_MEMORY` (3) | ||
| 1137 | + | ||
| 1138 | +**错误函数:** `ccu_resource_manager.cc` | ||
| 1139 | + | ||
| 1140 | +**关键日志:** | ||
| 1141 | +``` | ||
| 1142 | +[TransMemToMem] 无法获取srcBuf的设备地址(addr= 0x<ADDR>)! | ||
| 1143 | +[TransMemToMem] 无法获取dstBuf的设备地址(addr= 0x<ADDR>)! | ||
| 1144 | +``` | ||
| 1145 | + | ||
| 1146 | +**问题现象:** CCU执行内存搬运时,源或目标地址无法在设备内存表中找到。 | ||
| 1147 | + | ||
| 1148 | +--- | ||
| 1149 | + | ||
| 1150 | +### 子模块:insight | ||
| 1151 | + | ||
| 1152 | +--- | ||
| 1153 | + | ||
| 1154 | +#### FAQ-I001 | ||
| 1155 | + | ||
| 1156 | +**标题:** Insight前端页面打不开 | ||
| 1157 | + | ||
| 1158 | +**关键日志:** | ||
| 1159 | +``` | ||
| 1160 | +[Insight] Failed to start web server on port <N> | ||
| 1161 | +[Insight] Address already in use | ||
| 1162 | +``` | ||
| 1163 | + | ||
| 1164 | +**问题现象:** Insight可视化界面无法访问。 | ||
| 1165 | + | ||
| 1166 | +**可能原因:** | ||
| 1167 | +1. 端口被占用 | ||
| 1168 | +2. 网络配置问题 | ||
| 1169 | +3. Python依赖未安装 | ||
| 1170 | + | ||
| 1171 | +**排查步骤:** | ||
| 1172 | +```bash | ||
| 1173 | +netstat -tlnp | grep <port> | ||
| 1174 | +``` | ||
| 1175 | + | ||
| 1176 | +--- | ||
| 1177 | + | ||
| 1178 | +#### FAQ-I002 | ||
| 1179 | + | ||
| 1180 | +**标题:** Insight数据加载异常 | ||
| 1181 | + | ||
| 1182 | +**关键日志:** | ||
| 1183 | +``` | ||
| 1184 | +[Insight] Failed to load data from SQLite | ||
| 1185 | +[Insight] JSON parse error: <error> | ||
| 1186 | +``` | ||
| 1187 | + | ||
| 1188 | +**问题现象:** Insight面板中数据无法正常显示。 | ||
| 1189 | + | ||
| 1190 | +**可能原因:** | ||
| 1191 | +1. 数据库连接失败 | ||
| 1192 | +2. 数据文件损坏 | ||
| 1193 | +3. JSON格式错误 | ||
| 1194 | + | ||
| 1195 | +--- | ||
| 1196 | + | ||
| 1197 | +#### FAQ-I003 | ||
| 1198 | + | ||
| 1199 | +**标题:** DAG图渲染问题 | ||
| 1200 | + | ||
| 1201 | +**关键日志:** | ||
| 1202 | +``` | ||
| 1203 | +[Insight] DAG render failed: <error> | ||
| 1204 | +[Insight] Node count exceeds limit: <N> | ||
| 1205 | +``` | ||
| 1206 | + | ||
| 1207 | +**问题现象:** 任务图无法正常渲染或显示不全。 | ||
| 1208 | + | ||
| 1209 | +**可能原因:** | ||
| 1210 | +1. 任务节点数量过多 | ||
| 1211 | +2. 图布局算法超时 | ||
| 1212 | +3. 浏览器内存不足 | ||
| 1213 | + | ||
| 1214 | +--- | ||
| 1215 | + | ||
| 1216 | +## 附录:错误码速查表 | ||
| 1217 | + | ||
| 1218 | +| 错误码 | 枚举值 | 含义 | | ||
| 1219 | +|--------|--------|------| | ||
| 1220 | +| 0 | HCCL_SIM_SUCCESS | 成功 | | ||
| 1221 | +| 1 | HCCL_SIM_E_PARA | 参数错误 | | ||
| 1222 | +| 2 | HCCL_SIM_E_PTR | 空指针 | | ||
| 1223 | +| 3 | HCCL_SIM_E_MEMORY | 内存错误 | | ||
| 1224 | +| 4 | HCCL_SIM_E_INTERNAL | 内部错误 | | ||
| 1225 | +| 5 | HCCL_SIM_E_NOT_SUPPORT | 不支持的特性 | | ||
| 1226 | +| 6 | HCCL_SIM_E_NOT_FOUND | 资源未找到 | | ||
| 1227 | +| 8 | HCCL_SIM_E_SYSCALL | 系统调用错误 | | ||
| 1228 | +| 9 | HCCL_SIM_E_TIMEOUT | 超时 | | ||
| 1229 | +| 10 | HCCL_SIM_E_OPEN_FILE_FAILURE | 文件打开失败 | | ||
| @@ -0,0 +1,605 @@ | |||
| 1 | +#!/usr/bin/env python3 | ||
| 2 | +""" | ||
| 3 | +FAQ Markdown 转 HTML 转换脚本 | ||
| 4 | + | ||
| 5 | +功能: | ||
| 6 | +1. 解析 Markdown FAQ 文档,提取结构化数据(支持模块→子模块→错误码分组的分层结构) | ||
| 7 | +2. 生成 JSON 数据 | ||
| 8 | +3. 将 JSON 嵌入到 HTML 模板中,生成最终的 index.html | ||
| 9 | + | ||
| 10 | +用法: | ||
| 11 | + python3 scripts/generate_faq_html.py docs/faq/test_faq.md docs/faq/index.html | ||
| 12 | +""" | ||
| 13 | + | ||
| 14 | +import re | ||
| 15 | +import sys | ||
| 16 | +import json | ||
| 17 | +from pathlib import Path | ||
| 18 | +from typing import List, Dict | ||
| 19 | + | ||
| 20 | + | ||
| 21 | +def parse_faq_markdown(md_content: str) -> List[Dict]: | ||
| 22 | + """解析Markdown FAQ文档,提取结构化数据""" | ||
| 23 | + faq_items = [] | ||
| 24 | + | ||
| 25 | + lines = md_content.split('\n') | ||
| 26 | + current_module = '' | ||
| 27 | + current_submodule = '' | ||
| 28 | + current_error_code_group = '' | ||
| 29 | + | ||
| 30 | + faq_blocks = re.split(r'\n#### FAQ-', md_content) | ||
| 31 | + | ||
| 32 | + for i, block in enumerate(faq_blocks[1:]): | ||
| 33 | + faq_id_match = re.match(r'([A-Z]+\d+)', block) | ||
| 34 | + if not faq_id_match: | ||
| 35 | + continue | ||
| 36 | + | ||
| 37 | + faq_id = faq_id_match.group(1) | ||
| 38 | + | ||
| 39 | + original_block = '\n#### FAQ-' + block | ||
| 40 | + current_module, current_submodule, current_error_code_group = find_context(md_content, original_block) | ||
| 41 | + | ||
| 42 | + faq_item = parse_faq_block(faq_id, block) | ||
| 43 | + if faq_item: | ||
| 44 | + faq_item['module'] = current_module | ||
| 45 | + faq_item['submodule'] = current_submodule | ||
| 46 | + faq_item['errorCodeGroup'] = current_error_code_group | ||
| 47 | + faq_items.append(faq_item) | ||
| 48 | + | ||
| 49 | + return faq_items | ||
| 50 | + | ||
| 51 | + | ||
| 52 | +def find_context(md_content: str, block: str) -> tuple: | ||
| 53 | + """查找FAQ块在文档中的上下文(模块、子模块、错误码分组)""" | ||
| 54 | + block_start = md_content.find(block) | ||
| 55 | + if block_start == -1: | ||
| 56 | + return '', '', '' | ||
| 57 | + | ||
| 58 | + context_text = md_content[:block_start] | ||
| 59 | + lines = context_text.split('\n') | ||
| 60 | + | ||
| 61 | + module = '' | ||
| 62 | + submodule = '' | ||
| 63 | + error_code_group = '' | ||
| 64 | + | ||
| 65 | + for line in reversed(lines): | ||
| 66 | + line = line.strip() | ||
| 67 | + if line.startswith('##### ') and not error_code_group and not submodule: | ||
| 68 | + match = re.match(r'#####\s*([A-Z_]+)\s*\(\d+\)', line) | ||
| 69 | + if match: | ||
| 70 | + error_code_group = match.group(1) | ||
| 71 | + elif line.startswith('### 子模块:') and not submodule: | ||
| 72 | + submodule = line.replace('### 子模块:', '').strip() | ||
| 73 | + elif line.startswith('## 模块:') and not module: | ||
| 74 | + module = line.replace('## 模块:', '').strip() | ||
| 75 | + break | ||
| 76 | + | ||
| 77 | + return module, submodule, error_code_group | ||
| 78 | + | ||
| 79 | + | ||
| 80 | +def parse_faq_block(faq_id: str, block: str) -> Dict: | ||
| 81 | + """解析单个FAQ块""" | ||
| 82 | + faq_item = { | ||
| 83 | + 'id': f'FAQ-{faq_id}', | ||
| 84 | + 'module': '', | ||
| 85 | + 'submodule': '', | ||
| 86 | + 'errorCodeGroup': '', | ||
| 87 | + 'title': extract_field(block, '标题'), | ||
| 88 | + 'errorCode': extract_field(block, '错误码'), | ||
| 89 | + 'errorFunctions': extract_list_field(block, '错误函数'), | ||
| 90 | + 'keywords': [], | ||
| 91 | + 'logPatterns': extract_code_block(block, '关键日志'), | ||
| 92 | + 'symptom': extract_field(block, '问题现象'), | ||
| 93 | + 'reasons': extract_numbered_list(block, '可能原因'), | ||
| 94 | + 'steps': extract_code_block(block, '排查步骤'), | ||
| 95 | + 'solution': extract_code_block(block, '解决方案'), | ||
| 96 | + 'diagram': '', | ||
| 97 | + 'diagramType': '' | ||
| 98 | + } | ||
| 99 | + | ||
| 100 | + if faq_item['errorCode']: | ||
| 101 | + code_match = re.search(r'HCCL_SIM_\w+', faq_item['errorCode']) | ||
| 102 | + if code_match: | ||
| 103 | + faq_item['keywords'].append(code_match.group()) | ||
| 104 | + | ||
| 105 | + if faq_item['logPatterns']: | ||
| 106 | + keywords = extract_keywords_from_log(faq_item['logPatterns'][0]) | ||
| 107 | + faq_item['keywords'].extend(keywords) | ||
| 108 | + | ||
| 109 | + diagram_info = extract_diagram(block) | ||
| 110 | + faq_item['diagram'] = diagram_info['content'] | ||
| 111 | + faq_item['diagramType'] = diagram_info['type'] | ||
| 112 | + | ||
| 113 | + return faq_item | ||
| 114 | + | ||
| 115 | + | ||
| 116 | +def extract_field(block: str, field_name: str) -> str: | ||
| 117 | + """提取字段值""" | ||
| 118 | + pattern = rf'\*\*{field_name}:\*\*\s*(.+)' | ||
| 119 | + match = re.search(pattern, block) | ||
| 120 | + return match.group(1).strip() if match else '' | ||
| 121 | + | ||
| 122 | + | ||
| 123 | +def extract_list_field(block: str, field_name: str) -> List[str]: | ||
| 124 | + """提取逗号分隔的列表字段""" | ||
| 125 | + value = extract_field(block, field_name) | ||
| 126 | + if not value: | ||
| 127 | + return [] | ||
| 128 | + return [v.strip() for v in value.split(',') if v.strip()] | ||
| 129 | + | ||
| 130 | + | ||
| 131 | +def extract_code_block(block: str, field_name: str) -> List[str]: | ||
| 132 | + """提取代码块""" | ||
| 133 | + pattern = rf'\*\*{field_name}:\*\*\s*\n```.*?\n(.*?)```' | ||
| 134 | + matches = re.findall(pattern, block, re.DOTALL) | ||
| 135 | + return [m.strip() for m in matches] if matches else [] | ||
| 136 | + | ||
| 137 | + | ||
| 138 | +def extract_numbered_list(block: str, field_name: str) -> List[str]: | ||
| 139 | + """提取编号列表""" | ||
| 140 | + pattern = rf'\*\*{field_name}:\*\*\s*\n((?:\d+\..+\n)+)' | ||
| 141 | + match = re.search(pattern, block) | ||
| 142 | + if not match: | ||
| 143 | + return [] | ||
| 144 | + | ||
| 145 | + list_text = match.group(1) | ||
| 146 | + items = re.findall(r'\d+\.\s*(.+)', list_text) | ||
| 147 | + return [item.strip() for item in items] | ||
| 148 | + | ||
| 149 | + | ||
| 150 | +def extract_keywords_from_log(log_content: str) -> List[str]: | ||
| 151 | + """从日志中提取关键词""" | ||
| 152 | + keywords = [] | ||
| 153 | + bracket_keywords = re.findall(r'\[([^\]]+)\]', log_content) | ||
| 154 | + keywords.extend(bracket_keywords[:3]) | ||
| 155 | + | ||
| 156 | + env_vars = re.findall(r'\b[A-Z_]{5,}\b', log_content) | ||
| 157 | + keywords.extend([v for v in env_vars if 'HCCL' in v or 'RANK' in v]) | ||
| 158 | + | ||
| 159 | + return keywords[:5] | ||
| 160 | + | ||
| 161 | + | ||
| 162 | +def extract_diagram(block: str) -> Dict: | ||
| 163 | + """提取图示信息""" | ||
| 164 | + mermaid_pattern = r'\*\*图示说明:\*\*\s*\n```mermaid\s*\n(.*?)```' | ||
| 165 | + mermaid_match = re.search(mermaid_pattern, block, re.DOTALL) | ||
| 166 | + if mermaid_match: | ||
| 167 | + return {'content': mermaid_match.group(1).strip(), 'type': 'mermaid'} | ||
| 168 | + | ||
| 169 | + image_pattern = r'\*\*图示说明:\*\*\s*\n!\[.*?\]\((.*?)\)' | ||
| 170 | + image_match = re.search(image_pattern, block) | ||
| 171 | + if image_match: | ||
| 172 | + return {'content': image_match.group(1), 'type': 'image'} | ||
| 173 | + | ||
| 174 | + return {'content': '', 'type': ''} | ||
| 175 | + | ||
| 176 | + | ||
| 177 | +def generate_html(faq_data: List[Dict], modules: List[str], submodules: Dict[str, List[str]], error_codes: List[str]) -> str: | ||
| 178 | + """生成完整的HTML文件内容""" | ||
| 179 | + html_template = get_html_template() | ||
| 180 | + | ||
| 181 | + json_data = json.dumps(faq_data, ensure_ascii=False, indent=2) | ||
| 182 | + modules_json = json.dumps(modules, ensure_ascii=False) | ||
| 183 | + submodules_json = json.dumps(submodules, ensure_ascii=False) | ||
| 184 | + codes_json = json.dumps(error_codes, ensure_ascii=False) | ||
| 185 | + | ||
| 186 | + html_content = html_template.replace('{{FAQ_DATA}}', json_data) | ||
| 187 | + html_content = html_content.replace('{{MODULES}}', modules_json) | ||
| 188 | + html_content = html_content.replace('{{SUBMODULES}}', submodules_json) | ||
| 189 | + html_content = html_content.replace('{{ERROR_CODES}}', codes_json) | ||
| 190 | + | ||
| 191 | + return html_content | ||
| 192 | + | ||
| 193 | + | ||
| 194 | +def get_html_template() -> str: | ||
| 195 | + """获取HTML模板""" | ||
| 196 | + return '''<!DOCTYPE html> | ||
| 197 | +<html lang="zh-CN"> | ||
| 198 | +<head> | ||
| 199 | + <meta charset="UTF-8"> | ||
| 200 | + <meta name="viewport" content="width=device-width, initial-scale=1.0"> | ||
| 201 | + <title>HCCL-VM FAQ 查询系统</title> | ||
| 202 | + <script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script> | ||
| 203 | + <style> | ||
| 204 | + * { margin: 0; padding: 0; box-sizing: border-box; } | ||
| 205 | + body { | ||
| 206 | + font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; | ||
| 207 | + background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); | ||
| 208 | + min-height: 100vh; padding: 20px; | ||
| 209 | + } | ||
| 210 | + .container { | ||
| 211 | + max-width: 1200px; margin: 0 auto; | ||
| 212 | + background: #fff; border-radius: 12px; | ||
| 213 | + box-shadow: 0 20px 60px rgba(0,0,0,0.3); overflow: hidden; | ||
| 214 | + } | ||
| 215 | + .header { | ||
| 216 | + background: linear-gradient(90deg, #2c3e50, #3498db); | ||
| 217 | + color: white; padding: 30px; text-align: center; | ||
| 218 | + } | ||
| 219 | + .header h1 { font-size: 28px; margin-bottom: 10px; } | ||
| 220 | + .header p { opacity: 0.9; font-size: 14px; } | ||
| 221 | + .search-panel { background: #f8f9fa; padding: 25px; border-bottom: 1px solid #dee2e6; } | ||
| 222 | + .search-row { display: flex; gap: 15px; margin-bottom: 15px; flex-wrap: wrap; } | ||
| 223 | + .search-group { flex: 1; min-width: 180px; } | ||
| 224 | + .search-group label { display: block; font-weight: 600; margin-bottom: 8px; color: #2c3e50; } | ||
| 225 | + .search-group select, .search-group input { | ||
| 226 | + width: 100%; padding: 10px 15px; border: 2px solid #e0e0e0; | ||
| 227 | + border-radius: 8px; font-size: 14px; transition: all 0.3s; | ||
| 228 | + } | ||
| 229 | + .search-group select:focus, .search-group input:focus { | ||
| 230 | + border-color: #3498db; outline: none; | ||
| 231 | + box-shadow: 0 0 0 3px rgba(52, 152, 219, 0.1); | ||
| 232 | + } | ||
| 233 | + .btn-row { display: flex; gap: 10px; justify-content: center; } | ||
| 234 | + .btn { padding: 12px 30px; border: none; border-radius: 8px; | ||
| 235 | + font-size: 14px; font-weight: 600; cursor: pointer; transition: all 0.3s; } | ||
| 236 | + .btn-primary { background: #3498db; color: white; } | ||
| 237 | + .btn-primary:hover { background: #2980b9; transform: translateY(-2px); } | ||
| 238 | + .btn-secondary { background: #95a5a6; color: white; } | ||
| 239 | + .btn-secondary:hover { background: #7f8c8d; } | ||
| 240 | + .stats-bar { background: #ecf0f1; padding: 15px 25px; | ||
| 241 | + display: flex; justify-content: space-between; align-items: center; | ||
| 242 | + font-size: 13px; color: #7f8c8d; } | ||
| 243 | + .results-container { padding: 25px; min-height: 400px; } | ||
| 244 | + .no-results { text-align: center; padding: 60px; color: #95a5a6; } | ||
| 245 | + .no-results h3 { font-size: 20px; margin-bottom: 10px; } | ||
| 246 | + .faq-item { background: #fff; border: 1px solid #e0e0e0; border-radius: 10px; | ||
| 247 | + margin-bottom: 20px; overflow: hidden; transition: all 0.3s; } | ||
| 248 | + .faq-item:hover { box-shadow: 0 5px 15px rgba(0,0,0,0.1); border-color: #3498db; } | ||
| 249 | + .faq-header { background: linear-gradient(90deg, #f8f9fa, #e9ecef); padding: 15px 20px; | ||
| 250 | + cursor: pointer; display: flex; justify-content: space-between; align-items: center; flex-wrap: wrap; gap: 8px; } | ||
| 251 | + .faq-header:hover { background: linear-gradient(90deg, #ecf0f1, #d5dbdb); } | ||
| 252 | + .faq-id { font-weight: 700; color: #3498db; font-size: 14px; } | ||
| 253 | + .faq-title { flex: 1; margin-left: 15px; font-weight: 600; color: #2c3e50; min-width: 200px; } | ||
| 254 | + .faq-tags { display: flex; gap: 8px; flex-wrap: wrap; } | ||
| 255 | + .tag { padding: 4px 10px; border-radius: 12px; font-size: 12px; font-weight: 500; } | ||
| 256 | + .tag-module { background: #e8f5e9; color: #2e7d32; } | ||
| 257 | + .tag-submodule { background: #e3f2fd; color: #1565c0; } | ||
| 258 | + .tag-code { background: #fff3e0; color: #e65100; } | ||
| 259 | + .faq-content { padding: 20px; display: none; } | ||
| 260 | + .faq-content.active { display: block; } | ||
| 261 | + .faq-section { margin-bottom: 15px; } | ||
| 262 | + .faq-section h4 { font-size: 14px; color: #34495e; margin-bottom: 8px; font-weight: 600; } | ||
| 263 | + .faq-section p, .faq-section ul { font-size: 13px; color: #555; line-height: 1.6; } | ||
| 264 | + .faq-section ul { padding-left: 20px; } | ||
| 265 | + .faq-section li { margin-bottom: 5px; } | ||
| 266 | + .code-block { background: #2d3436; color: #dfe6e9; padding: 15px; border-radius: 6px; | ||
| 267 | + font-family: "Consolas", "Monaco", monospace; font-size: 13px; | ||
| 268 | + overflow-x: auto; white-space: pre-wrap; } | ||
| 269 | + .diagram-container { background: #f8f9fa; padding: 15px; border-radius: 6px; | ||
| 270 | + margin-top: 10px; text-align: center; } | ||
| 271 | + .diagram-container img { max-width: 100%; border-radius: 6px; } | ||
| 272 | + .expand-icon { font-size: 18px; color: #95a5a6; transition: transform 0.3s; } | ||
| 273 | + .expand-icon.rotated { transform: rotate(180deg); } | ||
| 274 | + .highlight { background: #fff176; padding: 2px 4px; border-radius: 3px; } | ||
| 275 | + .module-group-header { | ||
| 276 | + background: linear-gradient(90deg, #2c3e50, #3498db); color: white; | ||
| 277 | + padding: 14px 20px; font-size: 17px; font-weight: 700; | ||
| 278 | + margin: 25px 0 12px 0; border-radius: 8px; | ||
| 279 | + display: flex; align-items: center; gap: 10px; | ||
| 280 | + } | ||
| 281 | + .module-group-header:first-child { margin-top: 0; } | ||
| 282 | + .module-count { | ||
| 283 | + background: rgba(255,255,255,0.25); padding: 2px 10px; | ||
| 284 | + border-radius: 10px; font-size: 12px; font-weight: 500; | ||
| 285 | + } | ||
| 286 | + .submodule-group-header { | ||
| 287 | + background: #f0f4f8; border-left: 4px solid #3498db; | ||
| 288 | + padding: 10px 16px; margin: 16px 0 8px 8px; border-radius: 0 6px 6px 0; | ||
| 289 | + cursor: pointer; display: flex; align-items: center; | ||
| 290 | + gap: 8px; transition: background 0.2s; user-select: none; | ||
| 291 | + } | ||
| 292 | + .submodule-group-header:hover { background: #e2e8f0; } | ||
| 293 | + .submodule-group-header .sub-title { font-size: 15px; font-weight: 600; color: #2c3e50; } | ||
| 294 | + .submodule-group-header .sub-count { | ||
| 295 | + background: #3498db; color: white; padding: 2px 8px; | ||
| 296 | + border-radius: 10px; font-size: 11px; font-weight: 500; | ||
| 297 | + } | ||
| 298 | + .submodule-group-header .sub-toggle { color: #7f8c8d; font-size: 12px; margin-left: auto; } | ||
| 299 | + .submodule-group.collapsed .faq-item { display: none; } | ||
| 300 | + .submodule-group.collapsed .submodule-group-header { opacity: 0.65; } | ||
| 301 | + </style> | ||
| 302 | +</head> | ||
| 303 | +<body> | ||
| 304 | + <div class="container"> | ||
| 305 | + <div class="header"> | ||
| 306 | + <h1>HCCL-VM 常见问题查询系统</h1> | ||
| 307 | + <p>按模块、子模块、错误码、关键词快速定位问题 | 支持模糊匹配</p> | ||
| 308 | + </div> | ||
| 309 | + <div class="search-panel"> | ||
| 310 | + <div class="search-row"> | ||
| 311 | + <div class="search-group"> | ||
| 312 | + <label>按模块筛选</label> | ||
| 313 | + <select id="moduleFilter" onchange="onModuleChange()"><option value="">全部模块</option></select> | ||
| 314 | + </div> | ||
| 315 | + <div class="search-group"> | ||
| 316 | + <label>按子模块筛选</label> | ||
| 317 | + <select id="submoduleFilter"><option value="">全部子模块</option></select> | ||
| 318 | + </div> | ||
| 319 | + <div class="search-group"> | ||
| 320 | + <label>按错误码筛选</label> | ||
| 321 | + <select id="errorCodeFilter"><option value="">全部错误码</option></select> | ||
| 322 | + </div> | ||
| 323 | + </div> | ||
| 324 | + <div class="search-row"> | ||
| 325 | + <div class="search-group" style="flex: 2;"> | ||
| 326 | + <label>关键词搜索</label> | ||
| 327 | + <input type="text" id="keywordInput" | ||
| 328 | + placeholder="输入关键词:函数名、日志内容、问题描述..."> | ||
| 329 | + </div> | ||
| 330 | + </div> | ||
| 331 | + <div class="btn-row"> | ||
| 332 | + <button class="btn btn-primary" onclick="searchFAQ()">🔍 搜索</button> | ||
| 333 | + <button class="btn btn-secondary" onclick="clearFilters()">🔄 清空</button> | ||
| 334 | + </div> | ||
| 335 | + </div> | ||
| 336 | + <div class="stats-bar"> | ||
| 337 | + <span id="statsInfo">共 <strong>0</strong> 条FAQ</span> | ||
| 338 | + <span id="matchInfo"></span> | ||
| 339 | + </div> | ||
| 340 | + <div class="results-container" id="resultsContainer"> | ||
| 341 | + <div class="no-results"> | ||
| 342 | + <h3>请输入搜索条件</h3> | ||
| 343 | + <p>选择模块、子模块或错误码,或输入关键词进行搜索</p> | ||
| 344 | + </div> | ||
| 345 | + </div> | ||
| 346 | + </div> | ||
| 347 | + | ||
| 348 | + <script> | ||
| 349 | + const faqData = {{FAQ_DATA}}; | ||
| 350 | + const modules = {{MODULES}}; | ||
| 351 | + const submodules = {{SUBMODULES}}; | ||
| 352 | + const errorCodes = {{ERROR_CODES}}; | ||
| 353 | + let currentKeyword = ''; | ||
| 354 | + | ||
| 355 | + mermaid.initialize({ startOnLoad: false, theme: 'neutral' }); | ||
| 356 | + | ||
| 357 | + function initModuleFilter() { | ||
| 358 | + const select = document.getElementById('moduleFilter'); | ||
| 359 | + modules.forEach(m => { | ||
| 360 | + const option = document.createElement('option'); | ||
| 361 | + option.value = m; option.textContent = m; | ||
| 362 | + select.appendChild(option); | ||
| 363 | + }); | ||
| 364 | + } | ||
| 365 | + | ||
| 366 | + function initSubmoduleFilter(moduleName) { | ||
| 367 | + const select = document.getElementById('submoduleFilter'); | ||
| 368 | + select.innerHTML = '<option value="">全部子模块</option>'; | ||
| 369 | + if (!moduleName || !submodules[moduleName]) return; | ||
| 370 | + submodules[moduleName].forEach(sm => { | ||
| 371 | + const option = document.createElement('option'); | ||
| 372 | + option.value = sm; option.textContent = sm; | ||
| 373 | + select.appendChild(option); | ||
| 374 | + }); | ||
| 375 | + } | ||
| 376 | + | ||
| 377 | + function initErrorCodeFilter() { | ||
| 378 | + const select = document.getElementById('errorCodeFilter'); | ||
| 379 | + errorCodes.forEach(c => { | ||
| 380 | + const option = document.createElement('option'); | ||
| 381 | + option.value = c; option.textContent = c; | ||
| 382 | + select.appendChild(option); | ||
| 383 | + }); | ||
| 384 | + } | ||
| 385 | + | ||
| 386 | + function onModuleChange() { | ||
| 387 | + const moduleName = document.getElementById('moduleFilter').value; | ||
| 388 | + initSubmoduleFilter(moduleName); | ||
| 389 | + document.getElementById('submoduleFilter').value = ''; | ||
| 390 | + searchFAQ(); | ||
| 391 | + } | ||
| 392 | + | ||
| 393 | + function searchFAQ() { | ||
| 394 | + const moduleFilter = document.getElementById('moduleFilter').value; | ||
| 395 | + const submoduleFilter = document.getElementById('submoduleFilter').value; | ||
| 396 | + const errorCodeFilter = document.getElementById('errorCodeFilter').value; | ||
| 397 | + const keyword = document.getElementById('keywordInput').value.trim().toLowerCase(); | ||
| 398 | + currentKeyword = keyword; | ||
| 399 | + | ||
| 400 | + let results = faqData.filter(item => { | ||
| 401 | + if (moduleFilter && item.module !== moduleFilter) return false; | ||
| 402 | + if (submoduleFilter && item.submodule !== submoduleFilter) return false; | ||
| 403 | + if (errorCodeFilter) { | ||
| 404 | + if (item.errorCode && item.errorCode.includes(errorCodeFilter)) return true; | ||
| 405 | + if (item.errorCodeGroup && item.errorCodeGroup === errorCodeFilter) return true; | ||
| 406 | + return false; | ||
| 407 | + } | ||
| 408 | + if (keyword) { | ||
| 409 | + const searchFields = [ | ||
| 410 | + item.title, item.errorCode, item.errorCodeGroup, ...item.errorFunctions, | ||
| 411 | + ...item.keywords, ...item.logPatterns, item.symptom, | ||
| 412 | + ...item.reasons, item.solution, item.id, item.module, item.submodule | ||
| 413 | + ].join(' ').toLowerCase(); | ||
| 414 | + return searchFields.includes(keyword); | ||
| 415 | + } | ||
| 416 | + return true; | ||
| 417 | + }); | ||
| 418 | + displayResults(results); | ||
| 419 | + } | ||
| 420 | + | ||
| 421 | + function displayResults(results) { | ||
| 422 | + const container = document.getElementById('resultsContainer'); | ||
| 423 | + document.getElementById('statsInfo').innerHTML = `共 <strong>${faqData.length}</strong> 条FAQ`; | ||
| 424 | + document.getElementById('matchInfo').innerHTML = results.length > 0 ? | ||
| 425 | + `匹配到 <strong>${results.length}</strong> 条结果` : ''; | ||
| 426 | + | ||
| 427 | + if (results.length === 0) { | ||
| 428 | + container.innerHTML = '<div class="no-results"><h3>未找到匹配结果</h3><p>请调整筛选条件</p></div>'; | ||
| 429 | + return; | ||
| 430 | + } | ||
| 431 | + | ||
| 432 | + // 按模块 → 子模块分组 | ||
| 433 | + const grouped = {}; | ||
| 434 | + results.forEach(item => { | ||
| 435 | + const mod = item.module || '未知模块'; | ||
| 436 | + const sub = item.submodule || '其他'; | ||
| 437 | + if (!grouped[mod]) grouped[mod] = {}; | ||
| 438 | + if (!grouped[mod][sub]) grouped[mod][sub] = []; | ||
| 439 | + grouped[mod][sub].push(item); | ||
| 440 | + }); | ||
| 441 | + | ||
| 442 | + let html = ''; | ||
| 443 | + const moduleOrder = modules.filter(m => grouped[m]); | ||
| 444 | + Object.keys(grouped).filter(m => !modules.includes(m)).forEach(m => moduleOrder.push(m)); | ||
| 445 | + | ||
| 446 | + moduleOrder.forEach(mod => { | ||
| 447 | + const moduleCount = Object.values(grouped[mod]).reduce((sum, arr) => sum + arr.length, 0); | ||
| 448 | + html += `<div class="module-group-header"> | ||
| 449 | + <span>${mod}</span> | ||
| 450 | + <span class="module-count">${moduleCount} 条</span> | ||
| 451 | + </div>`; | ||
| 452 | + | ||
| 453 | + const submoduleOrder = (submodules[mod] || []).filter(sm => grouped[mod][sm]); | ||
| 454 | + Object.keys(grouped[mod]).filter(sm => !(submodules[mod] || []).includes(sm)).forEach(sm => submoduleOrder.push(sm)); | ||
| 455 | + | ||
| 456 | + submoduleOrder.forEach(sub => { | ||
| 457 | + const items = grouped[mod][sub]; | ||
| 458 | + const groupId = `group-${mod}-${sub}`.replace(/[\\s()()]/g, '_'); | ||
| 459 | + html += `<div class="submodule-group" id="${groupId}"> | ||
| 460 | + <div class="submodule-group-header" onclick="toggleSubmoduleGroup('${groupId}')"> | ||
| 461 | + <span class="sub-title">${sub}</span> | ||
| 462 | + <span class="sub-count">${items.length} 条</span> | ||
| 463 | + <span class="sub-toggle">收起/展开</span> | ||
| 464 | + </div>`; | ||
| 465 | + | ||
| 466 | + items.forEach(item => { | ||
| 467 | + html += `<div class="faq-item"> | ||
| 468 | + <div class="faq-header" onclick="toggleFAQ('${item.id}')"> | ||
| 469 | + <span class="faq-id">${item.id}</span> | ||
| 470 | + <span class="faq-title">${highlightText(item.title)}</span> | ||
| 471 | + <div class="faq-tags"> | ||
| 472 | + ${item.errorCode ? `<span class="tag tag-code">${extractErrorCode(item.errorCode)}</span>` : (item.errorCodeGroup ? `<span class="tag tag-code">${item.errorCodeGroup}</span>` : '')} | ||
| 473 | + </div> | ||
| 474 | + <span class="expand-icon" id="icon-${item.id}">▼</span> | ||
| 475 | + </div> | ||
| 476 | + <div class="faq-content" id="content-${item.id}">${renderFAQContent(item)}</div> | ||
| 477 | + </div>`; | ||
| 478 | + }); | ||
| 479 | + html += '</div>'; | ||
| 480 | + }); | ||
| 481 | + }); | ||
| 482 | + container.innerHTML = html; | ||
| 483 | + | ||
| 484 | + setTimeout(() => { | ||
| 485 | + document.querySelectorAll('.mermaid').forEach(el => mermaid.run({ nodes: [el] })); | ||
| 486 | + }, 100); | ||
| 487 | + } | ||
| 488 | + | ||
| 489 | + function toggleSubmoduleGroup(groupId) { | ||
| 490 | + const group = document.getElementById(groupId); | ||
| 491 | + if (group) group.classList.toggle('collapsed'); | ||
| 492 | + } | ||
| 493 | + | ||
| 494 | + function renderFAQContent(item) { | ||
| 495 | + let html = ''; | ||
| 496 | + if (item.module) html += `<div class="faq-section"><h4>所属模块</h4><p><strong>${item.module}</strong>${item.submodule ? ' → ' + item.submodule : ''}</p></div>`; | ||
| 497 | + if (item.errorCode) html += `<div class="faq-section"><h4>错误码</h4><p>${item.errorCode}</p></div>`; | ||
| 498 | + if (item.errorFunctions && item.errorFunctions.length > 0) | ||
| 499 | + html += `<div class="faq-section"><h4>错误函数</h4><ul>${item.errorFunctions.map(f => `<li>${f}</li>`).join('')}</ul></div>`; | ||
| 500 | + if (item.logPatterns && item.logPatterns.length > 0) | ||
| 501 | + html += `<div class="faq-section"><h4>关键日志</h4><div class="code-block">${highlightText(item.logPatterns[0])}</div></div>`; | ||
| 502 | + if (item.symptom) html += `<div class="faq-section"><h4>问题现象</h4><p>${highlightText(item.symptom)}</p></div>`; | ||
| 503 | + if (item.reasons && item.reasons.length > 0) | ||
| 504 | + html += `<div class="faq-section"><h4>可能原因</h4><ul>${item.reasons.map(r => `<li>${highlightText(r)}</li>`).join('')}</ul></div>`; | ||
| 505 | + if (item.steps) html += `<div class="faq-section"><h4>排查步骤</h4><div class="code-block">${item.steps}</div></div>`; | ||
| 506 | + if (item.solution) html += `<div class="faq-section"><h4>解决方案</h4><div class="code-block">${item.solution}</div></div>`; | ||
| 507 | + if (item.diagram) | ||
| 508 | + html += `<div class="faq-section"><h4>图示说明</h4><div class="diagram-container">${item.diagramType === 'mermaid' ? `<pre class="mermaid">${item.diagram}</pre>` : `<img src="${item.diagram}" alt="示意图">`}</div></div>`; | ||
| 509 | + return html; | ||
| 510 | + } | ||
| 511 | + | ||
| 512 | + function toggleFAQ(id) { | ||
| 513 | + const content = document.getElementById('content-' + id); | ||
| 514 | + const icon = document.getElementById('icon-' + id); | ||
| 515 | + content.classList.toggle('active'); | ||
| 516 | + icon.classList.toggle('rotated'); | ||
| 517 | + } | ||
| 518 | + | ||
| 519 | + function highlightText(text) { | ||
| 520 | + if (!currentKeyword || !text) return text; | ||
| 521 | + const regex = new RegExp(currentKeyword, 'gi'); | ||
| 522 | + return text.replace(regex, match => `<span class="highlight">${match}</span>`); | ||
| 523 | + } | ||
| 524 | + | ||
| 525 | + function extractErrorCode(codeStr) { | ||
| 526 | + const match = codeStr.match(/HCCL_SIM_\w+/); | ||
| 527 | + return match ? match[0] : codeStr; | ||
| 528 | + } | ||
| 529 | + | ||
| 530 | + function clearFilters() { | ||
| 531 | + document.getElementById('moduleFilter').value = ''; | ||
| 532 | + document.getElementById('submoduleFilter').innerHTML = '<option value="">全部子模块</option>'; | ||
| 533 | + document.getElementById('errorCodeFilter').value = ''; | ||
| 534 | + document.getElementById('keywordInput').value = ''; | ||
| 535 | + currentKeyword = ''; | ||
| 536 | + displayResults(faqData); | ||
| 537 | + } | ||
| 538 | + | ||
| 539 | + window.onload = function() { | ||
| 540 | + initModuleFilter(); initErrorCodeFilter(); displayResults(faqData); | ||
| 541 | + }; | ||
| 542 | + | ||
| 543 | + document.getElementById('keywordInput').addEventListener('keypress', function(e) { | ||
| 544 | + if (e.key === 'Enter') searchFAQ(); | ||
| 545 | + }); | ||
| 546 | + </script> | ||
| 547 | +</body> | ||
| 548 | +</html>''' | ||
| 549 | + | ||
| 550 | + | ||
| 551 | +def main(): | ||
| 552 | + if len(sys.argv) < 3: | ||
| 553 | + print("用法: python3 scripts/generate_faq_html.py <input_md> <output_html>") | ||
| 554 | + print("示例: python3 scripts/generate_faq_html.py docs/faq/test_faq.md docs/faq/index.html") | ||
| 555 | + sys.exit(1) | ||
| 556 | + | ||
| 557 | + input_md = sys.argv[1] | ||
| 558 | + output_html = sys.argv[2] | ||
| 559 | + | ||
| 560 | + if not Path(input_md).exists(): | ||
| 561 | + print(f"❌ 输入文件不存在: {input_md}") | ||
| 562 | + sys.exit(1) | ||
| 563 | + | ||
| 564 | + output_dir = Path(output_html).parent | ||
| 565 | + output_dir.mkdir(parents=True, exist_ok=True) | ||
| 566 | + | ||
| 567 | + print(f"📖 解析: {input_md}") | ||
| 568 | + md_content = Path(input_md).read_text(encoding='utf-8') | ||
| 569 | + | ||
| 570 | + faq_items = parse_faq_markdown(md_content) | ||
| 571 | + | ||
| 572 | + print(f" 提取到 {len(faq_items)} 条FAQ") | ||
| 573 | + | ||
| 574 | + modules = sorted(set(item['module'] for item in faq_items if item['module'])) | ||
| 575 | + | ||
| 576 | + submodules = {} | ||
| 577 | + for item in faq_items: | ||
| 578 | + if item['module'] and item['submodule']: | ||
| 579 | + if item['module'] not in submodules: | ||
| 580 | + submodules[item['module']] = [] | ||
| 581 | + if item['submodule'] not in submodules[item['module']]: | ||
| 582 | + submodules[item['module']].append(item['submodule']) | ||
| 583 | + for key in submodules: | ||
| 584 | + submodules[key] = sorted(submodules[key]) | ||
| 585 | + | ||
| 586 | + error_codes = [] | ||
| 587 | + for item in faq_items: | ||
| 588 | + if item['errorCode']: | ||
| 589 | + match = re.search(r'HCCL_SIM_\w+', item['errorCode']) | ||
| 590 | + if match and match.group() not in error_codes: | ||
| 591 | + error_codes.append(match.group()) | ||
| 592 | + if item['errorCodeGroup'] and item['errorCodeGroup'] not in error_codes: | ||
| 593 | + error_codes.append(item['errorCodeGroup']) | ||
| 594 | + error_codes = sorted(error_codes) | ||
| 595 | + | ||
| 596 | + print(f"🔧 生成HTML...") | ||
| 597 | + html_content = generate_html(faq_items, modules, submodules, error_codes) | ||
| 598 | + Path(output_html).write_text(html_content, encoding='utf-8') | ||
| 599 | + | ||
| 600 | + print(f"✅ 完成: {output_html}") | ||
| 601 | + print(f"\n请在浏览器中打开查看效果") | ||
| 602 | + | ||
| 603 | + | ||
| 604 | +if __name__ == '__main__': | ||
| 605 | + main() | ||
| @@ -21,7 +21,7 @@ | |||
| 21 | | 2 | `{dataId}_hcclvm_instr_data.bin` | `0x434D4349` | 微码指令数据 | | 21 | | 2 | `{dataId}_hcclvm_instr_data.bin` | `0x434D4349` | 微码指令数据 | |
| 22 | | 3 | `{dataId}_hcclvm_task_data.bin` | `0x48565444` | 任务元数据 | | 22 | | 3 | `{dataId}_hcclvm_task_data.bin` | `0x48565444` | 任务元数据 | |
| 23 | 23 | ||
| 24 | -> **说明**:`{dataId}` 为时间戳格式的唯一标识,例如 `20250115_143052_3847` | 24 | +> **说明**:`{dataId}` 为时间戳格式的唯一标识,例如 `20250115_143052_3847`。 |
| 25 | 25 | ||
| 26 | --- | 26 | --- |
| 27 | 27 | ||
| @@ -79,7 +79,7 @@ struct FileHeader { | |||
| 79 | 79 | ||
| 80 | #### 3.2.1 ModelInfoCommInner | 80 | #### 3.2.1 ModelInfoCommInner |
| 81 | 81 | ||
| 82 | -**大小**:36 字节 | 82 | +**大小**:36 字节。 |
| 83 | 83 | ||
| 84 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 84 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 85 | |:------:|:------:|---------|-------------------|------------------| | 85 | |:------:|:------:|---------|-------------------|------------------| |
| @@ -98,9 +98,9 @@ struct FileHeader { | |||
| 98 | 98 | ||
| 99 | #### 3.2.2 VDataDesTag | 99 | #### 3.2.2 VDataDesTag |
| 100 | 100 | ||
| 101 | -**用途**:ReduceScatterV / AllGatherV 操作 | 101 | +**用途**:ReduceScatterV / AllGatherV 操作。 |
| 102 | 102 | ||
| 103 | -**大小**:变长 | 103 | +**大小**:变长。 |
| 104 | 104 | ||
| 105 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 105 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 106 | |:-------:|:----------:|----------|---------|--------------------| | 106 | |:-------:|:----------:|----------|---------|--------------------| |
| @@ -111,9 +111,9 @@ struct FileHeader { | |||
| 111 | 111 | ||
| 112 | #### 3.2.3 All2AllDataDesTag | 112 | #### 3.2.3 All2AllDataDesTag |
| 113 | 113 | ||
| 114 | -**用途**:All2All / All2AllV 操作 | 114 | +**用途**:All2All / All2AllV 操作。 |
| 115 | 115 | ||
| 116 | -**大小**:变长 | 116 | +**大小**:变长。 |
| 117 | 117 | ||
| 118 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 118 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 119 | |:------:|:----------:|----------|-----------------|------------------------| | 119 | |:------:|:----------:|----------|-----------------|------------------------| |
| @@ -137,7 +137,7 @@ struct FileHeader { | |||
| 137 | 137 | ||
| 138 | #### 3.3.2 ChannelData | 138 | #### 3.3.2 ChannelData |
| 139 | 139 | ||
| 140 | -**大小**:152 字节 | 140 | +**大小**:152 字节。 |
| 141 | 141 | ||
| 142 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 142 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 143 | |:------:|:------:|------------|-----------|---------------| | 143 | |:------:|:------:|------------|-----------|---------------| |
| @@ -165,7 +165,7 @@ struct FileHeader { | |||
| 165 | 165 | ||
| 166 | #### 3.4.2 JettyData | 166 | #### 3.4.2 JettyData |
| 167 | 167 | ||
| 168 | -**大小**:48 字节 | 168 | +**大小**:48 字节。 |
| 169 | 169 | ||
| 170 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 170 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 171 | |:------:|:------:|-----------|----------|--------------| | 171 | |:------:|:------:|-----------|----------|--------------| |
| @@ -189,7 +189,7 @@ struct FileHeader { | |||
| 189 | 189 | ||
| 190 | #### 3.5.2 MemLayoutData | 190 | #### 3.5.2 MemLayoutData |
| 191 | 191 | ||
| 192 | -**大小**:32 字节 | 192 | +**大小**:32 字节。 |
| 193 | 193 | ||
| 194 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 194 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 195 | |:------:|:------:|--------|--------------|--------------------| | 195 | |:------:|:------:|--------|--------------|--------------------| |
| @@ -213,7 +213,7 @@ struct FileHeader { | |||
| 213 | 213 | ||
| 214 | ### 4.2 MicrocodeInstrDesc | 214 | ### 4.2 MicrocodeInstrDesc |
| 215 | 215 | ||
| 216 | -**大小**:8 字节 | 216 | +**大小**:8 字节。 |
| 217 | 217 | ||
| 218 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 218 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 219 | |:------:|:------:|--------|----------|----------| | 219 | |:------:|:------:|--------|----------|----------| |
| @@ -259,9 +259,9 @@ struct FileHeader { | |||
| 259 | 259 | ||
| 260 | #### 5.3.1 TransMemTask | 260 | #### 5.3.1 TransMemTask |
| 261 | 261 | ||
| 262 | -**适用任务类型**:`MEM_CPY (3)` | 262 | +**适用任务类型**:`MEM_CPY (3)`。 |
| 263 | 263 | ||
| 264 | -**大小**:33 字节 | 264 | +**大小**:33 字节。 |
| 265 | 265 | ||
| 266 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 266 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 267 | |:------:|:------:|--------|-----------|--------------| | 267 | |:------:|:------:|--------|-----------|--------------| |
| @@ -274,9 +274,9 @@ struct FileHeader { | |||
| 274 | 274 | ||
| 275 | #### 5.3.2 ReduceTask | 275 | #### 5.3.2 ReduceTask |
| 276 | 276 | ||
| 277 | -**适用任务类型**:`REDUCE (2)` | 277 | +**适用任务类型**:`REDUCE (2)`。 |
| 278 | 278 | ||
| 279 | -**大小**:35 字节 | 279 | +**大小**:35 字节。 |
| 280 | 280 | ||
| 281 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 281 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 282 | |:------:|:------:|--------|-----------|--------------| | 282 | |:------:|:------:|--------|-----------|--------------| |
| @@ -291,9 +291,9 @@ struct FileHeader { | |||
| 291 | 291 | ||
| 292 | #### 5.3.3 NotifyTask | 292 | #### 5.3.3 NotifyTask |
| 293 | 293 | ||
| 294 | -**适用任务类型**:`NOTIFY_WAIT (0)` / `NOTIFY_RECORD (1)` | 294 | +**适用任务类型**:`NOTIFY_WAIT (0)` / `NOTIFY_RECORD (1)`。 |
| 295 | 295 | ||
| 296 | -**大小**:18 字节 | 296 | +**大小**:18 字节。 |
| 297 | 297 | ||
| 298 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | | 298 | | 偏移量 | 字节数 | 类型 | 字段名 | 说明 | |
| 299 | |:------:|:------:|--------|-------------|------------| | 299 | |:------:|:------:|--------|-------------|------------| |
| @@ -722,18 +722,18 @@ bool ReadTaskMetaData(const char* filename) { | |||
| 722 | 722 | ||
| 723 | ## 附录 B:文件读取流程 | 723 | ## 附录 B:文件读取流程 |
| 724 | 724 | ||
| 725 | -**Step 1**:打开二进制文件 | 725 | +**Step 1**:打开二进制文件。 |
| 726 | 726 | ||
| 727 | **Step 2**:读取 FileHeader (20 Bytes) | 727 | **Step 2**:读取 FileHeader (20 Bytes) |
| 728 | 728 | ||
| 729 | **Step 3**:验证魔数 (magic) | 729 | **Step 3**:验证魔数 (magic) |
| 730 | 730 | ||
| 731 | -- 魔数不匹配 → 返回错误,关闭文件 | 731 | +- 魔数不匹配 → 返回错误,关闭文件。 |
| 732 | -- 魔数匹配 → 继续 | 732 | +- 魔数匹配 → 继续。 |
| 733 | 733 | ||
| 734 | -**Step 4**:根据 count 读取数据内容 | 734 | +**Step 4**:根据 count 读取数据内容。 |
| 735 | 735 | ||
| 736 | -**Step 5**:关闭文件 | 736 | +**Step 5**:关闭文件。 |
| 737 | 737 | ||
| 738 | --- | 738 | --- |
| 739 | 739 | ||
| @@ -306,11 +306,11 @@ DAGView 画布支持以下操作: | |||
| 306 | 306 | ||
| 307 | ## 13. 常见问题 | 307 | ## 13. 常见问题 |
| 308 | 308 | ||
| 309 | -### 页面没有数据 | 309 | +### 13.1 页面没有数据 |
| 310 | 310 | ||
| 311 | 请确认 Insight 服务地址是否正确,并确认 Checker 已生成 Insight 数据。 | 311 | 请确认 Insight 服务地址是否正确,并确认 Checker 已生成 Insight 数据。 |
| 312 | 312 | ||
| 313 | -### 进入关联页后 DAG 为空 | 313 | +### 13.2 进入关联页后 DAG 为空 |
| 314 | 314 | ||
| 315 | 请确认数据集中存在以下文件: | 315 | 请确认数据集中存在以下文件: |
| 316 | 316 | ||
| @@ -319,10 +319,10 @@ graph/graph.msgpack | |||
| 319 | graph/layout.msgpack | 319 | graph/layout.msgpack |
| 320 | ``` | 320 | ``` |
| 321 | 321 | ||
| 322 | -### DAG 图过于密集 | 322 | +### 13.3 DAG 图过于密集 |
| 323 | 323 | ||
| 324 | 建议先减少左侧 Rank 勾选数量,只查看部分 Rank,再放大局部区域进行分析。 | 324 | 建议先减少左侧 Rank 勾选数量,只查看部分 Rank,再放大局部区域进行分析。 |
| 325 | 325 | ||
| 326 | -### 搜索不到节点 | 326 | +### 13.4 搜索不到节点 |
| 327 | 327 | ||
| 328 | 请确认搜索关键词完整,并确认目标节点所在 Rank 已被勾选。 | 328 | 请确认搜索关键词完整,并确认目标节点所在 Rank 已被勾选。 |
| @@ -1,6 +1,6 @@ | |||
| 1 | # HVRM Insight 使用指南 | 1 | # HVRM Insight 使用指南 |
| 2 | 2 | ||
| 3 | -## 概述 | 3 | +## 1. 概述 |
| 4 | 4 | ||
| 5 | HVRM Insight 是一个分布式算子可视化分析工具,用于对 HCCL 集合通信算子执行过程中的图结构、内存快照和校验报错信息进行统一展示和联动分析。工具包含三个页面: | 5 | HVRM Insight 是一个分布式算子可视化分析工具,用于对 HCCL 集合通信算子执行过程中的图结构、内存快照和校验报错信息进行统一展示和联动分析。工具包含三个页面: |
| 6 | 6 | ||
| @@ -12,13 +12,13 @@ HVRM Insight 是一个分布式算子可视化分析工具,用于对 HCCL 集 | |||
| 12 | 12 | ||
| 13 | --- | 13 | --- |
| 14 | 14 | ||
| 15 | -## 安装与启动 | 15 | +## 2. 安装与启动 |
| 16 | 16 | ||
| 17 | -### 运行前准备 | 17 | +### 2.1 运行前准备 |
| 18 | 18 | ||
| 19 | HVRM Insight 依赖 Checker 插件输出的数据文件。使用前,请确保已完整运行过 Checker,并在 Checker 的配置文件中开启以下两个数据输出开关: | 19 | HVRM Insight 依赖 Checker 插件输出的数据文件。使用前,请确保已完整运行过 Checker,并在 Checker 的配置文件中开启以下两个数据输出开关: |
| 20 | 20 | ||
| 21 | -Checker配置文件默认位于`/path/to/hccl_vm_install/plugin/checker/manifest.json` | 21 | +Checker配置文件默认位于`/path/to/hccl_vm_install/plugin/checker/manifest.json`。 |
| 22 | 22 | ||
| 23 | ```json | 23 | ```json |
| 24 | "setting": { | 24 | "setting": { |
| @@ -27,12 +27,12 @@ Checker配置文件默认位于`/path/to/hccl_vm_install/plugin/checker/manifest | |||
| 27 | } | 27 | } |
| 28 | ``` | 28 | ``` |
| 29 | 29 | ||
| 30 | -- **`enable_insight_dump`**:开启后,Checker 运行时会输出 Insight 所需的图结构和校验结果文件 | 30 | +- **`enable_insight_dump`**:开启后,Checker 运行时会输出 Insight 所需的图结构和校验结果文件。 |
| 31 | -- **`enable_memory_snapshot_dump`**:开启后,Checker 运行时会输出各 Rank 的内存快照文件 | 31 | +- **`enable_memory_snapshot_dump`**:开启后,Checker 运行时会输出各 Rank 的内存快照文件。 |
| 32 | 32 | ||
| 33 | 这两个开关默认为 `false`。如未开启,Insight 工具将无法读取到完整的分析数据。 | 33 | 这两个开关默认为 `false`。如未开启,Insight 工具将无法读取到完整的分析数据。 |
| 34 | 34 | ||
| 35 | -### 数据目录结构 | 35 | +### 2.2 数据目录结构 |
| 36 | 36 | ||
| 37 | 工具从 `data/` 目录读取数据集,每个数据集是一个独立的子目录: | 37 | 工具从 `data/` 目录读取数据集,每个数据集是一个独立的子目录: |
| 38 | 38 | ||
| @@ -48,7 +48,7 @@ data/ | |||
| 48 | 48 | ||
| 49 | 缺少可选目录时,对应页面会显示空状态。 | 49 | 缺少可选目录时,对应页面会显示空状态。 |
| 50 | 50 | ||
| 51 | -### 方式一:通过 hccl-vm 插件(推荐) | 51 | +### 2.3 方式一:通过 hccl-vm 插件(推荐) |
| 52 | 52 | ||
| 53 | ```bash | 53 | ```bash |
| 54 | # 安装并启动后端服务 | 54 | # 安装并启动后端服务 |
| @@ -62,7 +62,7 @@ hccl-vm plugin run @insight | |||
| 62 | hccl-vm plugin uninstall @insight | 62 | hccl-vm plugin uninstall @insight |
| 63 | ``` | 63 | ``` |
| 64 | 64 | ||
| 65 | -### 方式二:本地直接启动 | 65 | +### 2.4 方式二:本地直接启动 |
| 66 | 66 | ||
| 67 | ```bash | 67 | ```bash |
| 68 | # 进入 insight 插件目录 | 68 | # 进入 insight 插件目录 |
| @@ -71,27 +71,27 @@ python3 serve.py | |||
| 71 | 71 | ||
| 72 | 后端默认监听 `http://localhost:8000`,同时提供前端页面和数据 API。 | 72 | 后端默认监听 `http://localhost:8000`,同时提供前端页面和数据 API。 |
| 73 | 73 | ||
| 74 | -### 验证启动成功 | 74 | +### 2.5 验证启动成功 |
| 75 | 75 | ||
| 76 | 浏览器打开地址后,如果能看到 Dashboard 页面的数据集列表,说明启动成功。如果页面空白,请检查 `data/` 目录是否存在且包含 `manifest.json` 文件。 | 76 | 浏览器打开地址后,如果能看到 Dashboard 页面的数据集列表,说明启动成功。如果页面空白,请检查 `data/` 目录是否存在且包含 `manifest.json` 文件。 |
| 77 | 77 | ||
| 78 | --- | 78 | --- |
| 79 | 79 | ||
| 80 | -## 场景演示 | 80 | +## 3. 场景演示 |
| 81 | 81 | ||
| 82 | -### 场景一:选择数据集并查看概览(Dashboard) | 82 | +### 3.1 场景一:选择数据集并查看概览(Dashboard) |
| 83 | 83 | ||
| 84 | 浏览所有数据集,点击选择目标数据集,查看算子信息和统计数据,然后进入关联分析页面。 | 84 | 浏览所有数据集,点击选择目标数据集,查看算子信息和统计数据,然后进入关联分析页面。 |
| 85 | 85 | ||
| 86 |  | 86 |  |
| 87 | 87 | ||
| 88 | -### 场景二:联动分析内存与任务图(MemView) | 88 | +### 3.2 场景二:联动分析内存与任务图(MemView) |
| 89 | 89 | ||
| 90 | 通过内存时间线和 DAG 任务图的联动,定位某个 Step 的内存操作、查看节点详情、追溯数据来源。 | 90 | 通过内存时间线和 DAG 任务图的联动,定位某个 Step 的内存操作、查看节点详情、追溯数据来源。 |
| 91 | 91 | ||
| 92 |  | 92 |  |
| 93 | 93 | ||
| 94 | -### 场景三:查看报错并跳转到问题现场(Analytic) | 94 | +### 3.3 场景三:查看报错并跳转到问题现场(Analytic) |
| 95 | 95 | ||
| 96 | 在 Analytic 页面或 MemView 左侧报错列表中查看错误详情,然后一键跳转到关联的 DAG 节点或内存上下文。 | 96 | 在 Analytic 页面或 MemView 左侧报错列表中查看错误详情,然后一键跳转到关联的 DAG 节点或内存上下文。 |
| 97 | 97 | ||
| @@ -99,15 +99,15 @@ python3 serve.py | |||
| 99 | 99 | ||
| 100 | --- | 100 | --- |
| 101 | 101 | ||
| 102 | -## 页面详解 | 102 | +## 4.页面详解 |
| 103 | 103 | ||
| 104 | -### Dashboard 页面 | 104 | +### 4.1 Dashboard 页面 |
| 105 | 105 | ||
| 106 | Dashboard 是使用工具的起点,页面从左到右分为三个区域: | 106 | Dashboard 是使用工具的起点,页面从左到右分为三个区域: |
| 107 | 107 | ||
| 108 | -- **左侧**:算子摘要和 Rank 树。Rank 树支持勾选 / 取消勾选,选择结果会同步到所有页面 | 108 | +- **左侧**:算子摘要和 Rank 树。Rank 树支持勾选 / 取消勾选,选择结果会同步到所有页面。 |
| 109 | -- **中间**:数据集列表。点击某一行选中数据集,左侧和右侧同步刷新 | 109 | +- **中间**:数据集列表。点击某一行选中数据集,左侧和右侧同步刷新。 |
| 110 | -- **右侧**:选中数据集的详情面板,展示统计信息和跳转按钮 | 110 | +- **右侧**:选中数据集的详情面板,展示统计信息和跳转按钮。 |
| 111 | 111 | ||
| 112 |  | 112 |  |
| 113 | 113 | ||
| @@ -117,10 +117,10 @@ Dashboard 是使用工具的起点,页面从左到右分为三个区域: | |||
| 117 | 117 | ||
| 118 | 详情面板底部提供两个跳转入口: | 118 | 详情面板底部提供两个跳转入口: |
| 119 | 119 | ||
| 120 | -- **`进入关联视图`**:跳转到 MemView,带入当前数据集和 Rank 选择 | 120 | +- **`进入关联视图`**:跳转到 MemView,带入当前数据集和 Rank 选择。 |
| 121 | -- **`查看诊断`**:跳转到 Analytic,查看该数据集的所有校验错误 | 121 | +- **`查看诊断`**:跳转到 Analytic,查看该数据集的所有校验错误。 |
| 122 | 122 | ||
| 123 | -### MemView 页面 | 123 | +### 4.2 MemView 页面 |
| 124 | 124 | ||
| 125 | MemView 将内存时间线和 DAG 任务图放在同一视图中联动展示,是排查问题的核心页面。 | 125 | MemView 将内存时间线和 DAG 任务图放在同一视图中联动展示,是排查问题的核心页面。 |
| 126 | 126 | ||
| @@ -128,92 +128,92 @@ MemView 将内存时间线和 DAG 任务图放在同一视图中联动展示, | |||
| 128 | 128 | ||
| 129 | **页面布局:** | 129 | **页面布局:** |
| 130 | 130 | ||
| 131 | -- **左侧**:Rank 选择树、报错列表(Issues)、搜索面板 | 131 | +- **左侧**:Rank 选择树、报错列表(Issues)、搜索面板。 |
| 132 | -- **上半区**:内存时间线 —— 按 Step 展示各 Rank 的 buffer 状态演化 | 132 | +- **上半区**:内存时间线 —— 按 Step 展示各 Rank 的 buffer 状态演化。 |
| 133 | -- **下半区**:DAG 任务图 —— 按 Rank / Stream 泳道展示任务节点和依赖关系 | 133 | +- **下半区**:DAG 任务图 —— 按 Rank / Stream 泳道展示任务节点和依赖关系。 |
| 134 | -- **右侧**:详情栏 —— 展示当前选中 Step 或节点的详细信息 | 134 | +- **右侧**:详情栏 —— 展示当前选中 Step 或节点的详细信息。 |
| 135 | 135 | ||
| 136 |  | 136 |  |
| 137 | 137 | ||
| 138 | -#### 核心联动机制:Step、Rank、Task 三者联动 | 138 | +#### 4.3 核心联动机制:Step、Rank、Task 三者联动 |
| 139 | 139 | ||
| 140 | -- 在时间线上选中一个 Step → DAG 同步高亮对应节点 | 140 | +- 在时间线上选中一个 Step → DAG 同步高亮对应节点。 |
| 141 | -- 在 DAG 中点击一个节点 → 右侧切到节点详情 | 141 | +- 在 DAG 中点击一个节点 → 右侧切到节点详情。 |
| 142 | -- 从右侧点击跳转按钮 → 时间线同步定位到对应 Step | 142 | +- 从右侧点击跳转按钮 → 时间线同步定位到对应 Step。 |
| 143 | 143 | ||
| 144 | **右侧详情栏**有两种模式,根据是否选中 DAG 节点自动切换: | 144 | **右侧详情栏**有两种模式,根据是否选中 DAG 节点自动切换: |
| 145 | 145 | ||
| 146 | -- **Step 详情模式**:展示当前 Step 的内存操作列表(Task Memory Ops)和 Buffer 切片详情(Buffer Layout)。Buffer Layout 中的每个切片可展开查看数据来源语义,支持点击跳转 | 146 | +- **Step 详情模式**:展示当前 Step 的内存操作列表(Task Memory Ops)和 Buffer 切片详情(Buffer Layout)。Buffer Layout 中的每个切片可展开查看数据来源语义,支持点击跳转。 |
| 147 | -- **节点详情模式**:展示节点属性、父/子节点关系、跨阶段节点映射、以及该节点关联的内存操作 | 147 | +- **节点详情模式**:展示节点属性、父/子节点关系、跨阶段节点映射、以及该节点关联的内存操作。 |
| 148 | 148 | ||
| 149 | **其他功能:** | 149 | **其他功能:** |
| 150 | 150 | ||
| 151 | -- **底部播放器**:支持按 Step 顺序浏览,可拖拽滑块快速跳转,播放速度为 1 秒 / Step | 151 | +- **底部播放器**:支持按 Step 顺序浏览,可拖拽滑块快速跳转,播放速度为 1 秒 / Step。 |
| 152 | -- **Focus 视图**:选中 Step 后 DAG 自动进入局部视图,只显示当前 Step 相关节点及其上下文,适合节点密集时缩小范围 | 152 | +- **Focus 视图**:选中 Step 后 DAG 自动进入局部视图,只显示当前 Step 相关节点及其上下文,适合节点密集时缩小范围。 |
| 153 | -- **子图浏览**:节点包含子图时,可在右侧点击 `打开子图` 进入查看更细粒度的任务结构 | 153 | +- **子图浏览**:节点包含子图时,可在右侧点击 `打开子图` 进入查看更细粒度的任务结构。 |
| 154 | - **Stage 切换**:通过顶部下拉框切换图阶段(如 `input_graph` / `input_task_queues`) | 154 | - **Stage 切换**:通过顶部下拉框切换图阶段(如 `input_graph` / `input_task_queues`) |
| 155 | -- **搜索**:支持按 taskId / taskType / notifyId 搜索 DAG 节点,结果点击即定位 | 155 | +- **搜索**:支持按 taskId / taskType / notifyId 搜索 DAG 节点,结果点击即定位。 |
| 156 | 156 | ||
| 157 | -### Analytic 页面 | 157 | +### 4.4 Analytic 页面 |
| 158 | 158 | ||
| 159 | Analytic 集中展示校验过程中发现的所有错误,页面分为三个区域: | 159 | Analytic 集中展示校验过程中发现的所有错误,页面分为三个区域: |
| 160 | 160 | ||
| 161 |  | 161 |  |
| 162 | 162 | ||
| 163 | -- **左侧**:数据集与 Rank 选择 | 163 | +- **左侧**:数据集与 Rank 选择。 |
| 164 | -- **中间**:报错列表,每条错误展示标题、严重程度标签(颜色区分)、关键字段摘要和原始错误码 | 164 | +- **中间**:报错列表,每条错误展示标题、严重程度标签(颜色区分)、关键字段摘要和原始错误码。 |
| 165 | -- **右侧**:选中错误的详情面板,包括错误基本信息、涉及的 Rank、关联节点信息、Slice 详情(精确定位数据不一致的内存位置)、补充信息和原始 JSON | 165 | +- **右侧**:选中错误的详情面板,包括错误基本信息、涉及的 Rank、关联节点信息、Slice 详情(精确定位数据不一致的内存位置)、补充信息和原始 JSON。 |
| 166 | 166 | ||
| 167 | 详情面板顶部提供跳转按钮: | 167 | 详情面板顶部提供跳转按钮: |
| 168 | 168 | ||
| 169 | -- **`查看 memView`**:跳转到 MemView,定位到错误关联的内存 Step | 169 | +- **`查看 memView`**:跳转到 MemView,定位到错误关联的内存 Step。 |
| 170 | -- **`查看 dagView`**:跳转到 MemView,直接定位到错误关联的 DAG 节点 | 170 | +- **`查看 dagView`**:跳转到 MemView,直接定位到错误关联的 DAG 节点。 |
| 171 | 171 | ||
| 172 | 跳转时自动带入数据集、Rank 和节点位置信息,到达 MemView 后无需重新选择即可开始排查。 | 172 | 跳转时自动带入数据集、Rank 和节点位置信息,到达 MemView 后无需重新选择即可开始排查。 |
| 173 | 173 | ||
| 174 | --- | 174 | --- |
| 175 | 175 | ||
| 176 | -## 常见使用流程 | 176 | +## 5. 常见使用流程 |
| 177 | 177 | ||
| 178 | -### 从 Dashboard 开始分析 | 178 | +### 5.1 从 Dashboard 开始分析 |
| 179 | 179 | ||
| 180 | -1. 在数据集列表中点击目标数据集 | 180 | +1. 在数据集列表中点击目标数据集。 |
| 181 | -2. 在左侧 Rank 树中勾选需要查看的 Rank | 181 | +2. 在左侧 Rank 树中勾选需要查看的 Rank。 |
| 182 | -3. 在右侧确认数据集的文件数量和错误统计 | 182 | +3. 在右侧确认数据集的文件数量和错误统计。 |
| 183 | -4. 点击 `进入关联视图` 进入 MemView | 183 | +4. 点击 `进入关联视图` 进入 MemView。 |
| 184 | 184 | ||
| 185 | -### 在 MemView 中定位内存问题 | 185 | +### 5.2 在 MemView 中定位内存问题 |
| 186 | 186 | ||
| 187 | 1. 在时间线上点击目标 Step(或使用底部播放器跳转) | 187 | 1. 在时间线上点击目标 Step(或使用底部播放器跳转) |
| 188 | -2. 在右侧 `Task Memory Ops` 查看该 Step 的内存操作 | 188 | +2. 在右侧 `Task Memory Ops` 查看该 Step 的内存操作。 |
| 189 | -3. 在 `Buffer Layout` 中选择 Rank 和 Buffer Type,展开 slice 查看数据来源 | 189 | +3. 在 `Buffer Layout` 中选择 Rank 和 Buffer Type,展开 slice 查看数据来源。 |
| 190 | -4. 点击 `定位 Task` 或语义卡片跳转到对应 DAG 节点 | 190 | +4. 点击 `定位 Task` 或语义卡片跳转到对应 DAG 节点。 |
| 191 | 191 | ||
| 192 | -### 从 DAG 节点反查内存 | 192 | +### 5.3 从 DAG 节点反查内存 |
| 193 | 193 | ||
| 194 | -1. 在 DAG 中点击目标节点 | 194 | +1. 在 DAG 中点击目标节点。 |
| 195 | -2. 在右侧查看 `节点 Memory Ops` | 195 | +2. 在右侧查看 `节点 Memory Ops`。 |
| 196 | -3. 点击 `跳转到 Step` 回到时间线,或点击父子节点继续沿依赖链排查 | 196 | +3. 点击 `跳转到 Step` 回到时间线,或点击父子节点继续沿依赖链排查。 |
| 197 | 197 | ||
| 198 | -### 从报错快速定位 | 198 | +### 5.4 从报错快速定位 |
| 199 | 199 | ||
| 200 | -1. 在 Analytic 页面或 MemView 左侧报错列表中点击一条错误 | 200 | +1. 在 Analytic 页面或 MemView 左侧报错列表中点击一条错误。 |
| 201 | -2. 查看详情中的关联节点和 Slice 信息 | 201 | +2. 查看详情中的关联节点和 Slice 信息。 |
| 202 | -3. 点击 `查看 memView` 或 `查看 dagView` 跳转到问题现场 | 202 | +3. 点击 `查看 memView` 或 `查看 dagView` 跳转到问题现场。 |
| 203 | 203 | ||
| 204 | --- | 204 | --- |
| 205 | 205 | ||
| 206 | -## 使用技巧 | 206 | +## 6. 使用技巧 |
| 207 | 207 | ||
| 208 | -- 先在 Dashboard 确认数据集和 Rank 范围,再进入 MemView 分析 | 208 | +- 先在 Dashboard 确认数据集和 Rank 范围,再进入 MemView 分析。 |
| 209 | -- 看内存问题时,优先用 `Task Memory Ops` 和 `Buffer Layout` 配合排查 | 209 | +- 看内存问题时,优先用 `Task Memory Ops` 和 `Buffer Layout` 配合排查。 |
| 210 | -- 看节点依赖关系时,优先在 DAG 中选中节点,再用右侧父子关系和节点映射跳转 | 210 | +- 看节点依赖关系时,优先在 DAG 中选中节点,再用右侧父子关系和节点映射跳转。 |
| 211 | -- DAG 节点太密集时,先用 Focus 视图缩小范围,再逐步展开 | 211 | +- DAG 节点太密集时,先用 Focus 视图缩小范围,再逐步展开。 |
| 212 | -- MemView 左侧的报错列表可以在不离开当前页面的情况下快速浏览并定位错误节点 | 212 | +- MemView 左侧的报错列表可以在不离开当前页面的情况下快速浏览并定位错误节点。 |
| 213 | 213 | ||
| 214 | --- | 214 | --- |
| 215 | 215 | ||
| 216 | -## 术语说明 | 216 | +## 7. 术语说明 |
| 217 | 217 | ||
| 218 | | 术语 | 说明 | | 218 | | 术语 | 说明 | |
| 219 | |------|------| | 219 | |------|------| |
| @@ -1,8 +1,8 @@ | |||
| 1 | # HCCL 模拟运行器数据模型设计 | 1 | # HCCL 模拟运行器数据模型设计 |
| 2 | 2 | ||
| 3 | -## 软硬件资源交互关系建模 | 3 | +## 1. 软硬件资源交互关系建模 |
| 4 | 4 | ||
| 5 | -数据流交互示意图 | 5 | +数据流交互示意图。 |
| 6 | 6 | ||
| 7 | ```mermaid | 7 | ```mermaid |
| 8 | sequenceDiagram | 8 | sequenceDiagram |
| @@ -35,9 +35,9 @@ sequenceDiagram | |||
| 35 | 35 | ||
| 36 | 只要是塞进 Stream 里的东西,都是由 Device 硬件执行的。 | 36 | 只要是塞进 Stream 里的东西,都是由 Device 硬件执行的。 |
| 37 | 37 | ||
| 38 | -## Device、Context、Stream 等硬件资源关系建模 | 38 | +## 2. Device、Context、Stream 等硬件资源关系建模 |
| 39 | 39 | ||
| 40 | -Device、Context、Stream 与用户主机线程之间的关系 | 40 | +Device、Context、Stream 与用户主机线程之间的关系。 |
| 41 | 41 | ||
| 42 | ```mermaid | 42 | ```mermaid |
| 43 | graph TD | 43 | graph TD |
| @@ -116,9 +116,9 @@ graph TD | |||
| 116 | 116 | ||
| 117 | ``` | 117 | ``` |
| 118 | 118 | ||
| 119 | -### 基础设备关系建模 | 119 | +### 2.1 基础设备关系建模 |
| 120 | 120 | ||
| 121 | -#### 层次结构总览 | 121 | +#### 2.1.1 层次结构总览 |
| 122 | 122 | ||
| 123 | ```mermaid | 123 | ```mermaid |
| 124 | erDiagram | 124 | erDiagram |
| @@ -234,7 +234,7 @@ erDiagram | |||
| 234 | Device ||--o{ DeviceConnection : "peer access" | 234 | Device ||--o{ DeviceConnection : "peer access" |
| 235 | ``` | 235 | ``` |
| 236 | 236 | ||
| 237 | -#### 网络通信资源 | 237 | +#### 2.1.2 网络通信资源 |
| 238 | 238 | ||
| 239 | ```mermaid | 239 | ```mermaid |
| 240 | erDiagram | 240 | erDiagram |
| @@ -298,7 +298,7 @@ erDiagram | |||
| 298 | CcuChannel }o..|| Link : "based-on" | 298 | CcuChannel }o..|| Link : "based-on" |
| 299 | ``` | 299 | ``` |
| 300 | 300 | ||
| 301 | -#### 关键关系说明 | 301 | +#### 2.1.3 关键关系说明 |
| 302 | 302 | ||
| 303 | ##### 第一层:物理拓扑层 | 303 | ##### 第一层:物理拓扑层 |
| 304 | 304 | ||
| @@ -336,7 +336,7 @@ erDiagram | |||
| 336 | | Link → EndPoint | 物理连接关联端点 | 由 topo.json 定义,描述物理拓扑 | | 336 | | Link → EndPoint | 物理连接关联端点 | 由 topo.json 定义,描述物理拓扑 | |
| 337 | | Link → Link-Protocol-Mapping | 连接支持的协议 | 一个Link可支持多种协议(UB_CTP/UB_MEM等) | | 337 | | Link → Link-Protocol-Mapping | 连接支持的协议 | 一个Link可支持多种协议(UB_CTP/UB_MEM等) | |
| 338 | 338 | ||
| 339 | -#### 配套接口映射表 | 339 | +#### 2.1.4 配套接口映射表 |
| 340 | 340 | ||
| 341 | ##### 基础与设备层 (Device / Server) | 341 | ##### 基础与设备层 (Device / Server) |
| 342 | 342 | ||
| @@ -359,7 +359,7 @@ erDiagram | |||
| 359 | | Stream.activated | `rtStreamStop` | | 359 | | Stream.activated | `rtStreamStop` | |
| 360 | | Stream.failure-mode | `rtSetStreamAttribute`,`rtGetStreamAttribute` | | 360 | | Stream.failure-mode | `rtSetStreamAttribute`,`rtGetStreamAttribute` | |
| 361 | 361 | ||
| 362 | -### 基础内存管理关系建模 | 362 | +### 2.2 基础内存管理关系建模 |
| 363 | 363 | ||
| 364 | ```mermaid | 364 | ```mermaid |
| 365 | erDiagram | 365 | erDiagram |
| @@ -420,9 +420,9 @@ erDiagram | |||
| 420 | %%} | 420 | %%} |
| 421 | ``` | 421 | ``` |
| 422 | 422 | ||
| 423 | -#### 关键关系说明(基础内存管理关系建模) | 423 | +#### 2.2.1 关键关系说明(基础内存管理关系建模) |
| 424 | 424 | ||
| 425 | -1. **物理内存核心地位** | 425 | +1. **物理内存核心地位**。 |
| 426 | `PhyMemBlock`作为基础实体,通过`phy_mem_id`与所有其他实体关联,体现华为昇腾"物理内存池化"的设计理念[3]。 | 426 | `PhyMemBlock`作为基础实体,通过`phy_mem_id`与所有其他实体关联,体现华为昇腾"物理内存池化"的设计理念[3]。 |
| 427 | 2. **三层映射体系**: | 427 | 2. **三层映射体系**: |
| 428 | 428 | ||
| @@ -434,7 +434,7 @@ erDiagram | |||
| 434 | 4. **特殊映射类型**: | 434 | 4. **特殊映射类型**: |
| 435 | `MemMapRecord`记录双虚拟地址映射场景(如`aclrtMapMem`产生的映射),支持华为NPU的零拷贝数据传输[3]。 | 435 | `MemMapRecord`记录双虚拟地址映射场景(如`aclrtMapMem`产生的映射),支持华为NPU的零拷贝数据传输[3]。 |
| 436 | 436 | ||
| 437 | -#### 配套接口映射表(基础内存管理关系建模) | 437 | +#### 2.1.2 配套接口映射表(基础内存管理关系建模) |
| 438 | 438 | ||
| 439 | | 实体 | 关键管理接口 | | 439 | | 实体 | 关键管理接口 | |
| 440 | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | 440 | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | |
| @@ -448,9 +448,9 @@ erDiagram | |||
| 448 | | IpcMemWhiteList.pid | `rtIpcMemSetImportPid` | | 448 | | IpcMemWhiteList.pid | `rtIpcMemSetImportPid` | |
| 449 | | MemMapRecord | `rtHostRegister`, `rtHostUnRegister` | | 449 | | MemMapRecord | `rtHostRegister`, `rtHostUnRegister` | |
| 450 | 450 | ||
| 451 | -## 在基础模型上扩展数据任务模型 | 451 | +## 3. 在基础模型上扩展数据任务模型 |
| 452 | 452 | ||
| 453 | -### 数据 / 任务流建模 | 453 | +### 3.1 数据 / 任务流建模 |
| 454 | 454 | ||
| 455 | ```mermaid | 455 | ```mermaid |
| 456 | erDiagram | 456 | erDiagram |
| @@ -498,7 +498,7 @@ erDiagram | |||
| 498 | 498 | ||
| 499 | ``` | 499 | ``` |
| 500 | 500 | ||
| 501 | -### ccu资源建模 | 501 | +### 3.2 ccu资源建模 |
| 502 | 502 | ||
| 503 | 一个NPU device包含2个CCU,分别为die0和die1。 | 503 | 一个NPU device包含2个CCU,分别为die0和die1。 |
| 504 | 504 | ||
| @@ -534,7 +534,7 @@ graph RL | |||
| 534 | 534 | ||
| 535 | ``` | 535 | ``` |
| 536 | 536 | ||
| 537 | -#### 配套接口映射表(ccu资源建模) | 537 | +#### 3.2.1 配套接口映射表(ccu资源建模) |
| 538 | 538 | ||
| 539 | | 实体 | 关键管理接口 | | 539 | | 实体 | 关键管理接口 | |
| 540 | | ------------------ | -------------------------------------------------------- | | 540 | | ------------------ | -------------------------------------------------------- | |
| @@ -545,24 +545,24 @@ graph RL | |||
| 545 | | Local/Rmt-Addr | `rtGetDeviceLocalAddr`, `rtGetDeviceRemoteAddr` | | 545 | | Local/Rmt-Addr | `rtGetDeviceLocalAddr`, `rtGetDeviceRemoteAddr` | |
| 546 | | CCU资源查询 | `rtGetCcudieInfo`, `rtGetCcudieNum` | | 546 | | CCU资源查询 | `rtGetCcudieInfo`, `rtGetCcudieNum` | |
| 547 | 547 | ||
| 548 | -#### CCU资源生命周期说明 | 548 | +#### 3.2.2 CCU资源生命周期说明 |
| 549 | 549 | ||
| 550 | **CCU初始化流程**: | 550 | **CCU初始化流程**: |
| 551 | 551 | ||
| 552 | -1. Device启动时,两个CCU(die0/die1)自动初始化 | 552 | +1. Device启动时,两个CCU(die0/die1)自动初始化。 |
| 553 | -2. 每个CCU分配独立的CcuBuf、Variable、Notify资源池 | 553 | +2. 每个CCU分配独立的CcuBuf、Variable、Notify资源池。 |
| 554 | -3. CompletedEvent用于通知任务完成状态 | 554 | +3. CompletedEvent用于通知任务完成状态。 |
| 555 | 555 | ||
| 556 | **资源约束**: | 556 | **资源约束**: |
| 557 | 557 | ||
| 558 | - 每个CCU的CcuBuf数量有限(与Device.soc-version相关) | 558 | - 每个CCU的CcuBuf数量有限(与Device.soc-version相关) |
| 559 | -- Variable用于存储通信过程中的共享变量 | 559 | +- Variable用于存储通信过程中的共享变量。 |
| 560 | -- Notify用于跨CCU的同步通知机制 | 560 | +- Notify用于跨CCU的同步通知机制。 |
| 561 | -- Local/Rmt-Addr用于跨die通信时的地址转换 | 561 | +- Local/Rmt-Addr用于跨die通信时的地址转换。 |
| 562 | 562 | ||
| 563 | -### 异步 / 同步执行建模 | 563 | +### 3.3 异步 / 同步执行建模 |
| 564 | 564 | ||
| 565 | -#### [Notify资源管理](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html) | 565 | +#### 3.3.1 [Notify资源管理](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html) |
| 566 | 566 | ||
| 567 | ```mermaid | 567 | ```mermaid |
| 568 | erDiagram | 568 | erDiagram |
| @@ -607,20 +607,20 @@ erDiagram | |||
| 607 | 607 | ||
| 608 | - 每个Device的Notify数量上限为`max-notify-cnt`(如A3芯片为8192) | 608 | - 每个Device的Notify数量上限为`max-notify-cnt`(如A3芯片为8192) |
| 609 | - `device-notify-seq`是Notify在Device内的物理序号(0~8191) | 609 | - `device-notify-seq`是Notify在Device内的物理序号(0~8191) |
| 610 | -- Notify创建时必须指定所属Context,Context绑定到特定Device | 610 | +- Notify创建时必须指定所属Context,Context绑定到特定Device。 |
| 611 | 611 | ||
| 612 | **Notify的IPC共享机制**: | 612 | **Notify的IPC共享机制**: |
| 613 | 613 | ||
| 614 | -- `IpcNotify`允许跨进程共享Notify实例 | 614 | +- `IpcNotify`允许跨进程共享Notify实例。 |
| 615 | -- `name-or-key`是共享标识,通过`rtNotifyGetExportKey`获取 | 615 | +- `name-or-key`是共享标识,通过`rtNotifyGetExportKey`获取。 |
| 616 | -- 其他进程通过`rtNotifyImportByKey`导入并使用 | 616 | +- 其他进程通过`rtNotifyImportByKey`导入并使用。 |
| 617 | -- `IpcNotifyVistorList`记录有权访问该Notify的进程PID | 617 | +- `IpcNotifyVistorList`记录有权访问该Notify的进程PID。 |
| 618 | 618 | ||
| 619 | **Notify状态管理**: | 619 | **Notify状态管理**: |
| 620 | 620 | ||
| 621 | -- `value`字段映射到硬件寄存器,用于读写状态 | 621 | +- `value`字段映射到硬件寄存器,用于读写状态。 |
| 622 | -- `rtWaitAndResetNotify`等待Notify变为Ready状态并重置 | 622 | +- `rtWaitAndResetNotify`等待Notify变为Ready状态并重置。 |
| 623 | -- Notify用于Stream间同步、跨进程同步场景 | 623 | +- Notify用于Stream间同步、跨进程同步场景。 |
| 624 | 624 | ||
| 625 | ##### 配套接口映射表([Notify资源管理](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html)) | 625 | ##### 配套接口映射表([Notify资源管理](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/appdevg/acldevg/aclcppdevg_000524.html)) |
| 626 | 626 | ||
| @@ -631,7 +631,7 @@ erDiagram | |||
| 631 | | IpcNotify.name-or-key | `rtNotifyGetExportKey`,`rtNotifyImportByKey` | | 631 | | IpcNotify.name-or-key | `rtNotifyGetExportKey`,`rtNotifyImportByKey` | |
| 632 | | IpcNotifyVistorList.ipc-id | `rtNotifySetImportPid` | | 632 | | IpcNotifyVistorList.ipc-id | `rtNotifySetImportPid` | |
| 633 | 633 | ||
| 634 | -#### Notify同步控制 | 634 | +#### 3.3.2 Notify同步控制 |
| 635 | 635 | ||
| 636 | ```mermaid | 636 | ```mermaid |
| 637 | erDiagram | 637 | erDiagram |
| @@ -682,14 +682,14 @@ erDiagram | |||
| 682 | 682 | ||
| 683 | **Notify任务类型**: | 683 | **Notify任务类型**: |
| 684 | 684 | ||
| 685 | -- `NotifyRecordTask`:将Notify状态设置为Ready,表示某个事件已完成 | 685 | +- `NotifyRecordTask`:将Notify状态设置为Ready,表示某个事件已完成。 |
| 686 | -- `NotifyWaitTask`:等待Notify状态变为Ready,实现Stream间的同步 | 686 | +- `NotifyWaitTask`:等待Notify状态变为Ready,实现Stream间的同步。 |
| 687 | 687 | ||
| 688 | **任务执行顺序**: | 688 | **任务执行顺序**: |
| 689 | 689 | ||
| 690 | -- NotifyRecordTask在StreamA执行,设置Notify为Ready | 690 | +- NotifyRecordTask在StreamA执行,设置Notify为Ready。 |
| 691 | -- NotifyWaitTask在StreamB执行,等待同一个Notify | 691 | +- NotifyWaitTask在StreamB执行,等待同一个Notify。 |
| 692 | -- 当Notify变为Ready后,StreamB后续任务才能继续执行 | 692 | +- 当Notify变为Ready后,StreamB后续任务才能继续执行。 |
| 693 | 693 | ||
| 694 | **跨Stream同步示例**: | 694 | **跨Stream同步示例**: |
| 695 | 695 | ||
| @@ -706,7 +706,7 @@ StreamB: NotifyWaitTask(notify-id=1) -> Task3 | |||
| 706 | | NotifyRecordTask | `rtRecordNotify` | | 706 | | NotifyRecordTask | `rtRecordNotify` | |
| 707 | | NotifyWaitTask | `lrtWaitAndResetNotify` | | 707 | | NotifyWaitTask | `lrtWaitAndResetNotify` | |
| 708 | 708 | ||
| 709 | -#### Event资源管理 | 709 | +#### 3.3.3 Event资源管理 |
| 710 | 710 | ||
| 711 | ```mermaid | 711 | ```mermaid |
| 712 | erDiagram | 712 | erDiagram |
| @@ -735,19 +735,19 @@ erDiagram | |||
| 735 | 735 | ||
| 736 | - 每个Device的Event数量上限为`max-event-cnt`(如A3芯片为65536) | 736 | - 每个Device的Event数量上限为`max-event-cnt`(如A3芯片为65536) |
| 737 | - `device-res-seq`是Event在Device内的物理序号(0~65535) | 737 | - `device-res-seq`是Event在Device内的物理序号(0~65535) |
| 738 | -- Event创建时必须指定所属Context,Context绑定到特定Device | 738 | +- Event创建时必须指定所属Context,Context绑定到特定Device。 |
| 739 | 739 | ||
| 740 | **Event与Context的关系**: | 740 | **Event与Context的关系**: |
| 741 | 741 | ||
| 742 | -- `created-ctx-id`记录Event的创建Context | 742 | +- `created-ctx-id`记录Event的创建Context。 |
| 743 | -- Event可在多个Stream间共享,但必须属于同一Context | 743 | +- Event可在多个Stream间共享,但必须属于同一Context。 |
| 744 | - 跨Context的Event共享需要通过IPC机制(类似Notify) | 744 | - 跨Context的Event共享需要通过IPC机制(类似Notify) |
| 745 | 745 | ||
| 746 | **Event状态管理**: | 746 | **Event状态管理**: |
| 747 | 747 | ||
| 748 | -- `status`字段表示Event当前状态:NotRecorded/Recorded/Completed | 748 | +- `status`字段表示Event当前状态:NotRecorded/Recorded/Completed。 |
| 749 | - `event-flag`用于控制Event的行为(如是否自动重置) | 749 | - `event-flag`用于控制Event的行为(如是否自动重置) |
| 750 | -- `created-time`用于性能统计 | 750 | +- `created-time`用于性能统计。 |
| 751 | 751 | ||
| 752 | ##### 配套接口映射表(Event资源管理) | 752 | ##### 配套接口映射表(Event资源管理) |
| 753 | 753 | ||
| @@ -756,7 +756,7 @@ erDiagram | |||
| 756 | | Event.event-id | `rtCreateEvent`, `rtCreateEventWithFlag`,`rtDestroyEvent`,`rtGetEventId` | | 756 | | Event.event-id | `rtCreateEvent`, `rtCreateEventWithFlag`,`rtDestroyEvent`,`rtGetEventId` | |
| 757 | | Event.status | `rtRecordEvent`,`rtQueryEventStatus` | | 757 | | Event.status | `rtRecordEvent`,`rtQueryEventStatus` | |
| 758 | 758 | ||
| 759 | -#### Event 流程控制 | 759 | +#### 3.3.4 Event 流程控制 |
| 760 | 760 | ||
| 761 | ```mermaid | 761 | ```mermaid |
| 762 | erDiagram | 762 | erDiagram |
| @@ -834,19 +834,19 @@ erDiagram | |||
| 834 | 834 | ||
| 835 | **Event任务类型分类**: | 835 | **Event任务类型分类**: |
| 836 | 836 | ||
| 837 | -- `EventRecordTask`:将Event状态设置为Recorded/Completed | 837 | +- `EventRecordTask`:将Event状态设置为Recorded/Completed。 |
| 838 | -- `EventWaitTask`:等待Event状态变为Completed | 838 | +- `EventWaitTask`:等待Event状态变为Completed。 |
| 839 | - `EventSyncTask`:同步等待Event完成(阻塞调用) | 839 | - `EventSyncTask`:同步等待Event完成(阻塞调用) |
| 840 | -- `EventRICaptureTask`:RI Capture模式下的特殊记录任务 | 840 | +- `EventRICaptureTask`:RI Capture模式下的特殊记录任务。 |
| 841 | -- `EventTimeTask`:记录时间戳相关任务 | 841 | +- `EventTimeTask`:记录时间戳相关任务。 |
| 842 | -- `EventTraceTask`:用于性能追踪的任务记录 | 842 | +- `EventTraceTask`:用于性能追踪的任务记录。 |
| 843 | 843 | ||
| 844 | **Event任务继承关系**: | 844 | **Event任务继承关系**: |
| 845 | 845 | ||
| 846 | -- `EventTask`是基类,包含task-id、event-id、execute-time、finish-time | 846 | +- `EventTask`是基类,包含task-id、event-id、execute-time、finish-time。 |
| 847 | -- `EventSyncTask`继承EventTask,增加op-timeout-s超时参数 | 847 | +- `EventSyncTask`继承EventTask,增加op-timeout-s超时参数。 |
| 848 | -- `EventRecordTask`和`EventWaitTask`继承EventSyncTask | 848 | +- `EventRecordTask`和`EventWaitTask`继承EventSyncTask。 |
| 849 | -- `EventRICaptureTask`、`EventTimeTask`、`EventTraceTask`直接继承EventTask | 849 | +- `EventRICaptureTask`、`EventTimeTask`、`EventTraceTask`直接继承EventTask。 |
| 850 | 850 | ||
| 851 | **Event执行流程**: | 851 | **Event执行流程**: |
| 852 | 852 | ||
| @@ -858,9 +858,9 @@ StreamB: EventWaitTask(event-id=1) -> KernelTask2 | |||
| 858 | 858 | ||
| 859 | **任务追踪关系**: | 859 | **任务追踪关系**: |
| 860 | 860 | ||
| 861 | -- `first-capture-task-id`记录首次Capture的Task ID | 861 | +- `first-capture-task-id`记录首次Capture的Task ID。 |
| 862 | -- `EventTraceTask.start-task-id`关联追踪的开始任务 | 862 | +- `EventTraceTask.start-task-id`关联追踪的开始任务。 |
| 863 | -- EventRecordTask通过EventWaitTask映射实现跨Stream同步 | 863 | +- EventRecordTask通过EventWaitTask映射实现跨Stream同步。 |
| 864 | 864 | ||
| 865 | ##### 配套接口映射表(Event 流程控制) | 865 | ##### 配套接口映射表(Event 流程控制) |
| 866 | 866 | ||
| @@ -872,16 +872,16 @@ StreamB: EventWaitTask(event-id=1) -> KernelTask2 | |||
| 872 | | EventTimeTask | `rtResetEvent`, `rtRecordEvent` | | 872 | | EventTimeTask | `rtResetEvent`, `rtRecordEvent` | |
| 873 | | EventTraceTask | `rtResetEvent`, `rtRecordEvent` | | 873 | | EventTraceTask | `rtResetEvent`, `rtRecordEvent` | |
| 874 | 874 | ||
| 875 | -## 通信域建模 | 875 | +## 4. 通信域建模 |
| 876 | 876 | ||
| 877 | 跨机通信涉及到多通信域混合任务编排。 | 877 | 跨机通信涉及到多通信域混合任务编排。 |
| 878 | 通信域的本质是HCCL在框架层通过Rdma_Agent提供的建链能力,维护的一张多卡的网络拓扑,保存在host进程中。 | 878 | 通信域的本质是HCCL在框架层通过Rdma_Agent提供的建链能力,维护的一张多卡的网络拓扑,保存在host进程中。 |
| 879 | 879 | ||
| 880 | -### 通信域核心概念 | 880 | +### 4.1 通信域核心概念 |
| 881 | 881 | ||
| 882 | 通信域(Communicator)是HCCL集合通信的基本抽象,每个通信域定义了一组参与通信的Rank及其拓扑关系。 | 882 | 通信域(Communicator)是HCCL集合通信的基本抽象,每个通信域定义了一组参与通信的Rank及其拓扑关系。 |
| 883 | 883 | ||
| 884 | -#### 通信域基础定义 | 884 | +#### 4.1.1 通信域基础定义 |
| 885 | 885 | ||
| 886 | ```mermaid | 886 | ```mermaid |
| 887 | erDiagram | 887 | erDiagram |
| @@ -907,7 +907,7 @@ erDiagram | |||
| 907 | Communicator ||--o{ Communicator : "派生(MPI_Comm_split)" | 907 | Communicator ||--o{ Communicator : "派生(MPI_Comm_split)" |
| 908 | ``` | 908 | ``` |
| 909 | 909 | ||
| 910 | -#### 控制面:Socket通信 | 910 | +#### 4.1.2 控制面:Socket通信 |
| 911 | 911 | ||
| 912 | ```mermaid | 912 | ```mermaid |
| 913 | erDiagram | 913 | erDiagram |
| @@ -949,7 +949,7 @@ erDiagram | |||
| 949 | RaSocket ||--o{ RaEpoll : "被监控" | 949 | RaSocket ||--o{ RaEpoll : "被监控" |
| 950 | ``` | 950 | ``` |
| 951 | 951 | ||
| 952 | -#### 数据面:RDMA通信 | 952 | +#### 4.1.3 数据面:RDMA通信 |
| 953 | 953 | ||
| 954 | ```mermaid | 954 | ```mermaid |
| 955 | erDiagram | 955 | erDiagram |
| @@ -1046,7 +1046,7 @@ erDiagram | |||
| 1046 | RaNdaQP ||--|| RaNdaCQ : "使用" | 1046 | RaNdaQP ||--|| RaNdaCQ : "使用" |
| 1047 | ``` | 1047 | ``` |
| 1048 | 1048 | ||
| 1049 | -#### 数据面:UB统一总线 | 1049 | +#### 4.1.4 数据面:UB统一总线 |
| 1050 | 1050 | ||
| 1051 | ```mermaid | 1051 | ```mermaid |
| 1052 | erDiagram | 1052 | erDiagram |
| @@ -1160,7 +1160,7 @@ erDiagram | |||
| 1160 | } | 1160 | } |
| 1161 | ``` | 1161 | ``` |
| 1162 | 1162 | ||
| 1163 | -#### 异步请求管理 | 1163 | +#### 4.1.5 异步请求管理 |
| 1164 | 1164 | ||
| 1165 | ```mermaid | 1165 | ```mermaid |
| 1166 | erDiagram | 1166 | erDiagram |
| @@ -1173,7 +1173,7 @@ erDiagram | |||
| 1173 | } | 1173 | } |
| 1174 | ``` | 1174 | ``` |
| 1175 | 1175 | ||
| 1176 | -#### 通信域架构总结 | 1176 | +#### 4.1.6 通信域架构总结 |
| 1177 | 1177 | ||
| 1178 | ```text | 1178 | ```text |
| 1179 | ┌─────────────────────────────────────────────────────────────────┐ | 1179 | ┌─────────────────────────────────────────────────────────────────┐ |
| @@ -1204,7 +1204,7 @@ erDiagram | |||
| 1204 | └─────────────────────────────────────────────────────────────────┘ | 1204 | └─────────────────────────────────────────────────────────────────┘ |
| 1205 | ``` | 1205 | ``` |
| 1206 | 1206 | ||
| 1207 | -#### 关键实体对照表 | 1207 | +#### 4.1.7 关键实体对照表 |
| 1208 | 1208 | ||
| 1209 | | 概念 | RDMA模式 | UB模式 | 说明 | | 1209 | | 概念 | RDMA模式 | UB模式 | 说明 | |
| 1210 | | --- | --- | --- | --- | | 1210 | | --- | --- | --- | --- | |
| @@ -1217,7 +1217,7 @@ erDiagram | |||
| 1217 | | 传输路径 | - | RaTp | 物理路径管理 | | 1217 | | 传输路径 | - | RaTp | 物理路径管理 | |
| 1218 | | 安全令牌 | - | RaTokenId | 访问控制 | | 1218 | | 安全令牌 | - | RaTokenId | 访问控制 | |
| 1219 | 1219 | ||
| 1220 | -#### HCCP接口分类 | 1220 | +#### 4.1.8 HCCP接口分类 |
| 1221 | 1221 | ||
| 1222 | ```text | 1222 | ```text |
| 1223 | HCCP Network API | 1223 | HCCP Network API |
| @@ -1421,63 +1421,63 @@ HCCP Network API | |||
| 1421 | 1421 | ||
| 1422 | **通信域层次结构**: | 1422 | **通信域层次结构**: |
| 1423 | 1423 | ||
| 1424 | -- `Communicator`是HCCL集合通信的核心抽象,定义了一组参与通信的Rank | 1424 | +- `Communicator`是HCCL集合通信的核心抽象,定义了一组参与通信的Rank。 |
| 1425 | -- `Rank`是通信域中的参与节点,每个Rank绑定到具体的Device | 1425 | +- `Rank`是通信域中的参与节点,每个Rank绑定到具体的Device。 |
| 1426 | - 父通信域派生子通信域(如`color`属性实现分组) | 1426 | - 父通信域派生子通信域(如`color`属性实现分组) |
| 1427 | 1427 | ||
| 1428 | **控制平面与数据平面分离**: | 1428 | **控制平面与数据平面分离**: |
| 1429 | 1429 | ||
| 1430 | -- **控制平面(Socket)**:用于建链、交换QP信息、控制信令,基于TCP协议 | 1430 | +- **控制平面(Socket)**:用于建链、交换QP信息、控制信令,基于TCP协议。 |
| 1431 | -- **数据平面(RDMA/UB)**:用于高性能数据传输,基于RDMA Verbs或UB协议 | 1431 | +- **数据平面(RDMA/UB)**:用于高性能数据传输,基于RDMA Verbs或UB协议。 |
| 1432 | 1432 | ||
| 1433 | **RDMA资源层次**: | 1433 | **RDMA资源层次**: |
| 1434 | 1434 | ||
| 1435 | -- `RaDevice`是虚拟网卡抽象,一个Device可创建多个RaDevice | 1435 | +- `RaDevice`是虚拟网卡抽象,一个Device可创建多个RaDevice。 |
| 1436 | -- `RaQP`是队列对,包含Send Queue和Receive Queue | 1436 | +- `RaQP`是队列对,包含Send Queue和Receive Queue。 |
| 1437 | -- `RaCQ`是完成队列,用于轮询WR完成状态 | 1437 | +- `RaCQ`是完成队列,用于轮询WR完成状态。 |
| 1438 | -- `RaMR`是内存注册,将虚拟内存映射为RDMA可访问的物理内存 | 1438 | +- `RaMR`是内存注册,将虚拟内存映射为RDMA可访问的物理内存。 |
| 1439 | -- `RaSRQ`是共享接收队列,多个QP可共享同一SRQ以提高资源利用率 | 1439 | +- `RaSRQ`是共享接收队列,多个QP可共享同一SRQ以提高资源利用率。 |
| 1440 | 1440 | ||
| 1441 | **UB资源层次**: | 1441 | **UB资源层次**: |
| 1442 | 1442 | ||
| 1443 | -- `RaContext`是UB统一上下文,替代RaDevice的设备抽象 | 1443 | +- `RaContext`是UB统一上下文,替代RaDevice的设备抽象。 |
| 1444 | - `RaJetty`是QP等价物,支持多种模式(URMA_NORMAL/CCU等) | 1444 | - `RaJetty`是QP等价物,支持多种模式(URMA_NORMAL/CCU等) |
| 1445 | -- `RaJfc`是CQ等价物,用于完成请求管理 | 1445 | +- `RaJfc`是CQ等价物,用于完成请求管理。 |
| 1446 | -- `RaLmem/RaRmem`是本地/远端内存管理,替代RaMR | 1446 | +- `RaLmem/RaRmem`是本地/远端内存管理,替代RaMR。 |
| 1447 | -- `RaTp`是传输路径管理,支持RTP/CTP/UTP三种类型 | 1447 | +- `RaTp`是传输路径管理,支持RTP/CTP/UTP三种类型。 |
| 1448 | -- `RaTokenId`是安全通信令牌,用于跨进程内存访问控制 | 1448 | +- `RaTokenId`是安全通信令牌,用于跨进程内存访问控制。 |
| 1449 | 1449 | ||
| 1450 | **实体关联要点**: | 1450 | **实体关联要点**: |
| 1451 | 1451 | ||
| 1452 | -1. `RaSocketPair`需要两个`RaSocket`(client端和server端)才能建立连接 | 1452 | +1. `RaSocketPair`需要两个`RaSocket`(client端和server端)才能建立连接。 |
| 1453 | -2. `RaQP.state`必须经历RESET→INIT→RTR→RTS的状态转换才能正常通信 | 1453 | +2. `RaQP.state`必须经历RESET→INIT→RTR→RTS的状态转换才能正常通信。 |
| 1454 | -3. `RaMR.lkey`用于本地访问,`RaMR.rkey`用于远端RDMA访问 | 1454 | +3. `RaMR.lkey`用于本地访问,`RaMR.rkey`用于远端RDMA访问。 |
| 1455 | -4. `RaJetty`通过`Bind`操作与对端Jetty建立逻辑连接 | 1455 | +4. `RaJetty`通过`Bind`操作与对端Jetty建立逻辑连接。 |
| 1456 | -5. `RaLmem`必须注册后,`RaRmem`才能从对端导入并访问 | 1456 | +5. `RaLmem`必须注册后,`RaRmem`才能从对端导入并访问。 |
| 1457 | 1457 | ||
| 1458 | **NDA(Network Direct Access)直接访问机制**: | 1458 | **NDA(Network Direct Access)直接访问机制**: |
| 1459 | 1459 | ||
| 1460 | -- `RaNdaQP`和`RaNdaCQ`是网络直接访问的QP/CQ变体 | 1460 | +- `RaNdaQP`和`RaNdaCQ`是网络直接访问的QP/CQ变体。 |
| 1461 | -- NDA模式允许绕过部分协议栈,降低延迟 | 1461 | +- NDA模式允许绕过部分协议栈,降低延迟。 |
| 1462 | -- `RaNdaGetDirectFlag`检查设备是否支持NDA模式 | 1462 | +- `RaNdaGetDirectFlag`检查设备是否支持NDA模式。 |
| 1463 | 1463 | ||
| 1464 | **异步请求管理**: | 1464 | **异步请求管理**: |
| 1465 | 1465 | ||
| 1466 | -- `AsyncRequest`统一管理所有异步操作的请求句柄 | 1466 | +- `AsyncRequest`统一管理所有异步操作的请求句柄。 |
| 1467 | -- 异步操作包括:连接、监听、QP创建/销毁、内存注册等 | 1467 | +- 异步操作包括:连接、监听、QP创建/销毁、内存注册等。 |
| 1468 | -- `RaGetAsyncReqResult`轮询异步操作结果 | 1468 | +- `RaGetAsyncReqResult`轮询异步操作结果。 |
| 1469 | 1469 | ||
| 1470 | **Socket事件机制**: | 1470 | **Socket事件机制**: |
| 1471 | 1471 | ||
| 1472 | -- `RaSocketEvent`是事件等待机制的句柄 | 1472 | +- `RaSocketEvent`是事件等待机制的句柄。 |
| 1473 | -- `RaEpoll`实现类似Linux Epoll的事件监控机制 | 1473 | +- `RaEpoll`实现类似Linux Epoll的事件监控机制。 |
| 1474 | -- 支持添加、修改、删除监控的Socket事件 | 1474 | +- 支持添加、修改、删除监控的Socket事件。 |
| 1475 | 1475 | ||
| 1476 | **网络接口与配置**: | 1476 | **网络接口与配置**: |
| 1477 | 1477 | ||
| 1478 | -- `InterfaceInfo`描述网络接口的IP/MAC/MTU等属性 | 1478 | +- `InterfaceInfo`描述网络接口的IP/MAC/MTU等属性。 |
| 1479 | -- `HccnConfig`存储HCCN网络配置键值对 | 1479 | +- `HccnConfig`存储HCCN网络配置键值对。 |
| 1480 | -- `Snapshot`支持设备状态的保存和恢复 | 1480 | +- `Snapshot`支持设备状态的保存和恢复。 |
| 1481 | 1481 | ||
| 1482 | ##### 控制平面 vs 数据平面 | 1482 | ##### 控制平面 vs 数据平面 |
| 1483 | 1483 | ||
| @@ -1501,11 +1501,11 @@ HCCP Network API | |||
| 1501 | 1501 | ||
| 1502 | ##### 关键点说明 | 1502 | ##### 关键点说明 |
| 1503 | 1503 | ||
| 1504 | -1. **RaContext/RaDevice**: 统一上下文实体,支持 RDMA 和 UB 双模式 | 1504 | +1. **RaContext/RaDevice**: 统一上下文实体,支持 RDMA 和 UB 双模式。 |
| 1505 | -2. **RaJetty/RaJfc/RaQP/RaCQ**: UB 模式下的 QP/CQ 等价物 | 1505 | +2. **RaJetty/RaJfc/RaQP/RaCQ**: UB 模式下的 QP/CQ 等价物。 |
| 1506 | -3. **RaLmem/RaRmem/RaMR**: UB 模式下的本地/远端内存管理 | 1506 | +3. **RaLmem/RaRmem/RaMR**: UB 模式下的本地/远端内存管理。 |
| 1507 | -4. **RaTp**: UB 传输路径管理 | 1507 | +4. **RaTp**: UB 传输路径管理。 |
| 1508 | -5. **RaTokenId**: 安全通信令牌 | 1508 | +5. **RaTokenId**: 安全通信令牌。 |
| 1509 | 1509 | ||
| 1510 | ##### 关键属性补充 | 1510 | ##### 关键属性补充 |
| 1511 | 1511 | ||
| @@ -1576,7 +1576,7 @@ HCCP Network API | |||
| 1576 | | RaSocketEvent.events | `RaEpollCtlAdd`, `RaEpollCtlMod`, `RaEpollCtlDel` 管理 | | 1576 | | RaSocketEvent.events | `RaEpollCtlAdd`, `RaEpollCtlMod`, `RaEpollCtlDel` 管理 | |
| 1577 | | RaDevice.direct-flag | `RaNdaGetDirectFlag` 检查NDA支持 | | 1577 | | RaDevice.direct-flag | `RaNdaGetDirectFlag` 检查NDA支持 | |
| 1578 | 1578 | ||
| 1579 | -## 回调与报告关系建模 | 1579 | +## 5. 回调与报告关系建模 |
| 1580 | 1580 | ||
| 1581 | ```mermaid | 1581 | ```mermaid |
| 1582 | erDiagram | 1582 | erDiagram |
| @@ -1628,11 +1628,11 @@ erDiagram | |||
| 1628 | 1628 | ||
| 1629 | ``` | 1629 | ``` |
| 1630 | 1630 | ||
| 1631 | -### 关键关系说明(回调与报告关系建模) | 1631 | +### 5.1 关键关系说明(回调与报告关系建模) |
| 1632 | 1632 | ||
| 1633 | -==**设备**==执行到 CallbackTask 时,触发 Host Runner 线程执行回调 | 1633 | +==**设备**==执行到 CallbackTask 时,触发 Host Runner 线程执行回调。 |
| 1634 | 1634 | ||
| 1635 | -#### 配套接口映射表(回调与报告关系建模) | 1635 | +#### 5.1.1 配套接口映射表(回调与报告关系建模) |
| 1636 | 1636 | ||
| 1637 | | 实体 | 关键管理接口 | | 1637 | | 实体 | 关键管理接口 | |
| 1638 | | ------------- | --------------------------------------------------------------------- | | 1638 | | ------------- | --------------------------------------------------------------------- | |
| @@ -1640,7 +1640,7 @@ erDiagram | |||
| 1640 | | ReportChannel | `rtSubscribeReport`, `rtUnSubscribeReport` | | 1640 | | ReportChannel | `rtSubscribeReport`, `rtUnSubscribeReport` | |
| 1641 | | Runner | `rtProcessReport` | | 1641 | | Runner | `rtProcessReport` | |
| 1642 | 1642 | ||
| 1643 | -## 基础`设备`模型细粒度底层扩展 | 1643 | +## 6. 基础`设备`模型细粒度底层扩展 |
| 1644 | 1644 | ||
| 1645 | ```mermaid | 1645 | ```mermaid |
| 1646 | erDiagram | 1646 | erDiagram |
| @@ -1701,52 +1701,52 @@ erDiagram | |||
| 1701 | } | 1701 | } |
| 1702 | ``` | 1702 | ``` |
| 1703 | 1703 | ||
| 1704 | -### 关键关系说明(基础`设备`模型细粒度底层扩展) | 1704 | +### 6.1 关键关系说明(基础`设备`模型细粒度底层扩展) |
| 1705 | 1705 | ||
| 1706 | **设备调度器层次结构**: | 1706 | **设备调度器层次结构**: |
| 1707 | 1707 | ||
| 1708 | -- `TaskSchedulerDevice`是设备调度器的抽象,一个Device可包含多个调度器 | 1708 | +- `TaskSchedulerDevice`是设备调度器的抽象,一个Device可包含多个调度器。 |
| 1709 | - 调度器类型包括:`Scalar`(标量处理器)、`CCU`(集合通信单元)、`CPU`(AI CPU) | 1709 | - 调度器类型包括:`Scalar`(标量处理器)、`CCU`(集合通信单元)、`CPU`(AI CPU) |
| 1710 | -- 不同类型调度器负责不同计算任务类型 | 1710 | +- 不同类型调度器负责不同计算任务类型。 |
| 1711 | 1711 | ||
| 1712 | **ComputeDie计算单元**: | 1712 | **ComputeDie计算单元**: |
| 1713 | 1713 | ||
| 1714 | -- `ComputeDie`是计算单元的抽象,继承关系表示计算单元类型 | 1714 | +- `ComputeDie`是计算单元的抽象,继承关系表示计算单元类型。 |
| 1715 | -- `Vector`:向量计算单元,处理向量运算 | 1715 | +- `Vector`:向量计算单元,处理向量运算。 |
| 1716 | -- `Cube`:立方计算单元,处理矩阵运算 | 1716 | +- `Cube`:立方计算单元,处理矩阵运算。 |
| 1717 | -- `HybridComputeDie`:混合计算单元,同时支持Vector和Cube | 1717 | +- `HybridComputeDie`:混合计算单元,同时支持Vector和Cube。 |
| 1718 | 1718 | ||
| 1719 | **CCU内部结构**: | 1719 | **CCU内部结构**: |
| 1720 | 1720 | ||
| 1721 | - `xnNum`:XN节点数量(跨节点通信) | 1721 | - `xnNum`:XN节点数量(跨节点通信) |
| 1722 | - `ckeNum`:CKE引擎数量(Checksum引擎) | 1722 | - `ckeNum`:CKE引擎数量(Checksum引擎) |
| 1723 | - `msNum`:MS模块数量(Memory Scheduler) | 1723 | - `msNum`:MS模块数量(Memory Scheduler) |
| 1724 | -- `channelNum`:通信通道数量 | 1724 | +- `channelNum`:通信通道数量。 |
| 1725 | - `version`:CCU版本(v1/v2,决定功能差异) | 1725 | - `version`:CCU版本(v1/v2,决定功能差异) |
| 1726 | 1726 | ||
| 1727 | **DeviceStatus状态管理**: | 1727 | **DeviceStatus状态管理**: |
| 1728 | 1728 | ||
| 1729 | -- `overflow-status`:溢出状态 | 1729 | +- `overflow-status`:溢出状态。 |
| 1730 | -- `synchronize-strategy`:同步策略配置 | 1730 | +- `synchronize-strategy`:同步策略配置。 |
| 1731 | -- `synchronize-timeout`:同步超时设置 | 1731 | +- `synchronize-timeout`:同步超时设置。 |
| 1732 | -- `capability-mask`:设备能力掩码 | 1732 | +- `capability-mask`:设备能力掩码。 |
| 1733 | -- `run-by-host`:是否运行在Host模式 | 1733 | +- `run-by-host`:是否运行在Host模式。 |
| 1734 | -- `ts-core`:调度器核心数量 | 1734 | +- `ts-core`:调度器核心数量。 |
| 1735 | -- `online-status`:设备在线状态 | 1735 | +- `online-status`:设备在线状态。 |
| 1736 | 1736 | ||
| 1737 | **Scalar调度关系**: | 1737 | **Scalar调度关系**: |
| 1738 | 1738 | ||
| 1739 | -- `Scalar`调度器管理`ComputeDie`的执行 | 1739 | +- `Scalar`调度器管理`ComputeDie`的执行。 |
| 1740 | -- 不同ComputeDie类型对应不同的计算负载 | 1740 | +- 不同ComputeDie类型对应不同的计算负载。 |
| 1741 | 1741 | ||
| 1742 | -#### 配套接口映射表(基础设备模型细粒度底层扩展) | 1742 | +#### 6.1.1 配套接口映射表(基础设备模型细粒度底层扩展) |
| 1743 | 1743 | ||
| 1744 | | 实体 | 关键管理接口 | | 1744 | | 实体 | 关键管理接口 | |
| 1745 | | ------------------- | ---------------------------------- | | 1745 | | ------------------- | ---------------------------------- | |
| 1746 | | TaskSchedulerDevice | `rtGetDeviceInfo`, `rtSetTsDevice` | | 1746 | | TaskSchedulerDevice | `rtGetDeviceInfo`, `rtSetTsDevice` | |
| 1747 | | DeviceStatus | `rtGetRunMode` | | 1747 | | DeviceStatus | `rtGetRunMode` | |
| 1748 | 1748 | ||
| 1749 | -## Kernel 运行时关系建模 | 1749 | +## 7. Kernel 运行时关系建模 |
| 1750 | 1750 | ||
| 1751 | ```mermaid | 1751 | ```mermaid |
| 1752 | erDiagram | 1752 | erDiagram |
| @@ -1791,21 +1791,21 @@ erDiagram | |||
| 1791 | } | 1791 | } |
| 1792 | ``` | 1792 | ``` |
| 1793 | 1793 | ||
| 1794 | -### 关键关系说明(Kernel 运行时关系建模) | 1794 | +### 7.1 关键关系说明(Kernel 运行时关系建模) |
| 1795 | 1795 | ||
| 1796 | **KernelBinary加载流程**: | 1796 | **KernelBinary加载流程**: |
| 1797 | 1797 | ||
| 1798 | -1. `KernelBinary`存储.o/.so等二进制文件路径和创建进程PID | 1798 | +1. `KernelBinary`存储.o/.so等二进制文件路径和创建进程PID。 |
| 1799 | -2. `rtBinaryLoadFromFile`或`rtBinaryLoadFromData`将二进制加载到设备内存 | 1799 | +2. `rtBinaryLoadFromFile`或`rtBinaryLoadFromData`将二进制加载到设备内存。 |
| 1800 | -3. 加载后生成`KernelBinaryHandle`,包含handle-id和kernel-id | 1800 | +3. 加载后生成`KernelBinaryHandle`,包含handle-id和kernel-id。 |
| 1801 | -4. `KernelFuncHandle`表示二进制中的具体函数,包含func-name、kernel-name、地址信息 | 1801 | +4. `KernelFuncHandle`表示二进制中的具体函数,包含func-name、kernel-name、地址信息。 |
| 1802 | 1802 | ||
| 1803 | **Kernel函数调用关系**: | 1803 | **Kernel函数调用关系**: |
| 1804 | 1804 | ||
| 1805 | -- `aic-addr`是AI Core函数地址 | 1805 | +- `aic-addr`是AI Core函数地址。 |
| 1806 | -- `aiv-addr`是AI Vector函数地址 | 1806 | +- `aiv-addr`是AI Vector函数地址。 |
| 1807 | -- `KernelFuncArgsHandle`存储函数参数信息 | 1807 | +- `KernelFuncArgsHandle`存储函数参数信息。 |
| 1808 | -- `KernelFuncArgsParamHandle`记录参数大小和是否占位符 | 1808 | +- `KernelFuncArgsParamHandle`记录参数大小和是否占位符。 |
| 1809 | - `KernelLaunchCfg`配置Kernel启动参数(block/grid等) | 1809 | - `KernelLaunchCfg`配置Kernel启动参数(block/grid等) |
| 1810 | 1810 | ||
| 1811 | **Kernel生命周期**: | 1811 | **Kernel生命周期**: |
| @@ -1818,11 +1818,11 @@ rtCreateBinary -> rtBinaryLoad -> rtBinaryGetFunction | |||
| 1818 | 1818 | ||
| 1819 | **参数管理机制**: | 1819 | **参数管理机制**: |
| 1820 | 1820 | ||
| 1821 | -- args-handle支持device和host两种类型 | 1821 | +- args-handle支持device和host两种类型。 |
| 1822 | - is-place-holder标识参数是否为占位符(延迟绑定) | 1822 | - is-place-holder标识参数是否为占位符(延迟绑定) |
| 1823 | -- param-size记录单个参数大小 | 1823 | +- param-size记录单个参数大小。 |
| 1824 | 1824 | ||
| 1825 | -#### 配套接口映射表(Kernel 运行时关系建模) | 1825 | +#### 7.1.1 配套接口映射表(Kernel 运行时关系建模) |
| 1826 | 1826 | ||
| 1827 | | 实体 | 关键管理接口 | | 1827 | | 实体 | 关键管理接口 | |
| 1828 | | ------------------ | --------------------------------------------------------------------------------------------------------------- | | 1828 | | ------------------ | --------------------------------------------------------------------------------------------------------------- | |
| @@ -1830,4 +1830,4 @@ rtCreateBinary -> rtBinaryLoad -> rtBinaryGetFunction | |||
| 1830 | | KernelBinaryHandle | `rtBinaryLoad`, `rtBinaryUnLoad`,`rtBinaryLoadFromFile`,`rtBinaryLoadFromData` | | 1830 | | KernelBinaryHandle | `rtBinaryLoad`, `rtBinaryUnLoad`,`rtBinaryLoadFromFile`,`rtBinaryLoadFromData` | |
| 1831 | | KernelFuncHandle | `rtBinaryGetFunction`, `rtBinaryGetFunctionByEntry`,`rtGetFunctionAddr`,`rtGetFunctionName`,`rtRegisterCpuFunc` | | 1831 | | KernelFuncHandle | `rtBinaryGetFunction`, `rtBinaryGetFunctionByEntry`,`rtGetFunctionAddr`,`rtGetFunctionName`,`rtRegisterCpuFunc` | |
| 1832 | 1832 | ||
| 1833 | -## 模型加载关系建模 | 1833 | +## 8. 模型加载关系建模 |
| @@ -235,7 +235,7 @@ public: | |||
| 235 | return *static_cast<sim::SqliteTable<T>*>(it->second); | 235 | return *static_cast<sim::SqliteTable<T>*>(it->second); |
| 236 | } | 236 | } |
| 237 | std::string name = std::string("Unregistered_") + typeid(T).name(); | 237 | std::string name = std::string("Unregistered_") + typeid(T).name(); |
| 238 | - fprintf(stderr, "[SimRunnerSqliteDB::GetTable] Lazily registering unregistered type: %s\n", name.c_str()); | 238 | + HCCL_VM_WARN("[SimRunnerSqliteDB::GetTable] Lazily registering unregistered type: {}", name.c_str()); |
| 239 | auto table = std::make_unique<sim::SqliteTable<T>>(m_db.GetDb(), name); | 239 | auto table = std::make_unique<sim::SqliteTable<T>>(m_db.GetDb(), name); |
| 240 | auto* rawPtr = table.get(); | 240 | auto* rawPtr = table.get(); |
| 241 | m_lazyTables.push_back(std::move(table)); | 241 | m_lazyTables.push_back(std::move(table)); |
| @@ -27,6 +27,8 @@ | |||
| 27 | 27 | ||
| 28 | 28 | ||
| 29 | 29 | ||
| 30 | + | ||
| 31 | + | ||
| 30 | namespace sim { | 32 | namespace sim { |
| 31 | inline std::shared_mutex& GetConnectionMutex() { | 33 | inline std::shared_mutex& GetConnectionMutex() { |
| 32 | static std::shared_mutex s_connMtx; | 34 | static std::shared_mutex s_connMtx; |
| @@ -74,7 +76,7 @@ public: | |||
| 74 | sqlite3_stmt* rawStmt; | 76 | sqlite3_stmt* rawStmt; |
| 75 | int rc = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr); | 77 | int rc = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr); |
| 76 | if (rc != SQLITE_OK) { | 78 | if (rc != SQLITE_OK) { |
| 77 | - fprintf(stderr, "[SqliteTable::Add] SQL prepare failed [%s]: %s (rc=%d)\n", | 79 | + HCCL_VM_ERROR("[SqliteTable::Add] SQL prepare failed [{}]: {} (rc={})", |
| 78 | m_tableName.c_str(), sqlite3_errmsg(m_db), rc); | 80 | m_tableName.c_str(), sqlite3_errmsg(m_db), rc); |
| 79 | return 0; | 81 | return 0; |
| 80 | } | 82 | } |
| @@ -98,7 +100,7 @@ public: | |||
| 98 | connLock.unlock(); | 100 | connLock.unlock(); |
| 99 | 101 | ||
| 100 | if (!isRetryable || attempt == kMaxRetries) { | 102 | if (!isRetryable || attempt == kMaxRetries) { |
| 101 | - fprintf(stderr, "[SqliteTable::Add] SQL insert failed [%s]: rc=%d, errc=%d, ext=%d, msg=%s\n", | 103 | + HCCL_VM_ERROR("[SqliteTable::Add] SQL insert failed [{}]: rc={}, errc={}, ext={}, msg={}", |
| 102 | m_tableName.c_str(), stepRc, errc, extErrc, sqlite3_errmsg(m_db)); | 104 | m_tableName.c_str(), stepRc, errc, extErrc, sqlite3_errmsg(m_db)); |
| 103 | return 0; | 105 | return 0; |
| 104 | } | 106 | } |
| @@ -131,7 +133,7 @@ public: | |||
| 131 | sqlite3_stmt* rawStmt; | 133 | sqlite3_stmt* rawStmt; |
| 132 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { | 134 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { |
| 133 | sqlite3_exec(m_db, "ROLLBACK", nullptr, nullptr, nullptr); | 135 | sqlite3_exec(m_db, "ROLLBACK", nullptr, nullptr, nullptr); |
| 134 | - fprintf(stderr, "[SqliteTable::Update] SQL prepare failed [%s]: %s\n", | 136 | + HCCL_VM_ERROR("[SqliteTable::Update] SQL prepare failed [{}]: {}", |
| 135 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 137 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 136 | return false; | 138 | return false; |
| 137 | } | 139 | } |
| @@ -153,7 +155,7 @@ public: | |||
| 153 | connLock.unlock(); | 155 | connLock.unlock(); |
| 154 | 156 | ||
| 155 | if (!isRetryable || attempt == kMaxRetries) { | 157 | if (!isRetryable || attempt == kMaxRetries) { |
| 156 | - fprintf(stderr, "[SqliteTable::Update] SQL step failed [%s]: rc=%d, errc=%d, msg=%s\n", | 158 | + HCCL_VM_ERROR("[SqliteTable::Update] SQL step failed [{}]: rc={}, errc={}, msg={}", |
| 157 | m_tableName.c_str(), result, errc, sqlite3_errmsg(m_db)); | 159 | m_tableName.c_str(), result, errc, sqlite3_errmsg(m_db)); |
| 158 | return false; | 160 | return false; |
| 159 | } | 161 | } |
| @@ -176,7 +178,7 @@ public: | |||
| 176 | 178 | ||
| 177 | sqlite3_stmt* rawStmt; | 179 | sqlite3_stmt* rawStmt; |
| 178 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { | 180 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { |
| 179 | - fprintf(stderr, "[SqliteTable::Delete] SQL prepare failed [%s]: %s\n", | 181 | + HCCL_VM_ERROR("[SqliteTable::Delete] SQL prepare failed [{}]: {}", |
| 180 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 182 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 181 | return false; | 183 | return false; |
| 182 | } | 184 | } |
| @@ -190,7 +192,7 @@ public: | |||
| 190 | 192 | ||
| 191 | int errc = sqlite3_errcode(m_db); | 193 | int errc = sqlite3_errcode(m_db); |
| 192 | if (errc != SQLITE_BUSY && errc != SQLITE_LOCKED) { | 194 | if (errc != SQLITE_BUSY && errc != SQLITE_LOCKED) { |
| 193 | - fprintf(stderr, "[SqliteTable::Delete] SQL step failed [%s]: rc=%d, errc=%d\n", | 195 | + HCCL_VM_ERROR("[SqliteTable::Delete] SQL step failed [{}]: rc={}, errc={}", |
| 194 | m_tableName.c_str(), result, errc); | 196 | m_tableName.c_str(), result, errc); |
| 195 | return false; | 197 | return false; |
| 196 | } | 198 | } |
| @@ -208,7 +210,7 @@ public: | |||
| 208 | 210 | ||
| 209 | std::string sql = "DELETE FROM " + m_tableName; | 211 | std::string sql = "DELETE FROM " + m_tableName; |
| 210 | if (sqlite3_exec(m_db, sql.c_str(), nullptr, nullptr, nullptr) != SQLITE_OK) { | 212 | if (sqlite3_exec(m_db, sql.c_str(), nullptr, nullptr, nullptr) != SQLITE_OK) { |
| 211 | - fprintf(stderr, "[SqliteTable::DeleteAll] SQL delete all failed [%s]: %s\n", | 213 | + HCCL_VM_ERROR("[SqliteTable::DeleteAll] SQL delete all failed [{}]: {}", |
| 212 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 214 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 213 | return false; | 215 | return false; |
| 214 | } | 216 | } |
| @@ -246,7 +248,7 @@ public: | |||
| 246 | 248 | ||
| 247 | sqlite3_stmt* rawStmt; | 249 | sqlite3_stmt* rawStmt; |
| 248 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { | 250 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { |
| 249 | - fprintf(stderr, "[SqliteTable::QueryList] SQL prepare failed [%s]: %s\n", | 251 | + HCCL_VM_ERROR("[SqliteTable::QueryList] SQL prepare failed [{}]: {}", |
| 250 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 252 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 251 | return result; | 253 | return result; |
| 252 | } | 254 | } |
| @@ -283,7 +285,7 @@ public: | |||
| 283 | connLock.lock(); | 285 | connLock.lock(); |
| 284 | } | 286 | } |
| 285 | 287 | ||
| 286 | - fprintf(stderr, "[SqliteTable::QueryList] SQLITE_BUSY after %d retries [%s]\n", | 288 | + HCCL_VM_ERROR("[SqliteTable::QueryList] SQLITE_BUSY after {} retries [{}]", |
| 287 | kMaxRetries, m_tableName.c_str()); | 289 | kMaxRetries, m_tableName.c_str()); |
| 288 | return result; | 290 | return result; |
| 289 | } | 291 | } |
| @@ -295,7 +297,7 @@ public: | |||
| 295 | for (int attempt = 0; attempt <= kMaxRetries; ++attempt) { | 297 | for (int attempt = 0; attempt <= kMaxRetries; ++attempt) { |
| 296 | sqlite3_stmt* rawStmt; | 298 | sqlite3_stmt* rawStmt; |
| 297 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { | 299 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { |
| 298 | - fprintf(stderr, "[SqliteTable::Query] SQL prepare failed [%s]: %s\n", | 300 | + HCCL_VM_ERROR("[SqliteTable::Query] SQL prepare failed [{}]: {}", |
| 299 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 301 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 300 | return {ValueType{}, false}; | 302 | return {ValueType{}, false}; |
| 301 | } | 303 | } |
| @@ -333,7 +335,7 @@ public: | |||
| 333 | connLock.lock(); | 335 | connLock.lock(); |
| 334 | } | 336 | } |
| 335 | 337 | ||
| 336 | - fprintf(stderr, "[SqliteTable::Query] SQLITE_BUSY after %d retries [%s]\n", | 338 | + HCCL_VM_ERROR("[SqliteTable::Query] SQLITE_BUSY after {} retries [{}]", |
| 337 | kMaxRetries, m_tableName.c_str()); | 339 | kMaxRetries, m_tableName.c_str()); |
| 338 | return {ValueType{}, false}; | 340 | return {ValueType{}, false}; |
| 339 | } | 341 | } |
| @@ -349,7 +351,7 @@ private: | |||
| 349 | " (id INTEGER PRIMARY KEY AUTOINCREMENT, data BLOB NOT NULL)"; | 351 | " (id INTEGER PRIMARY KEY AUTOINCREMENT, data BLOB NOT NULL)"; |
| 350 | 352 | ||
| 351 | if (sqlite3_exec(m_db, sql.c_str(), nullptr, nullptr, nullptr) != SQLITE_OK) { | 353 | if (sqlite3_exec(m_db, sql.c_str(), nullptr, nullptr, nullptr) != SQLITE_OK) { |
| 352 | - fprintf(stderr, "[SqliteTable::CreateTableIfNotExists] SQL create table failed [%s]: %s\n", | 354 | + HCCL_VM_ERROR("[SqliteTable::CreateTableIfNotExists] SQL create table failed [{}]: {}", |
| 353 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 355 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 354 | } | 356 | } |
| 355 | } | 357 | } |
| @@ -359,7 +361,7 @@ private: | |||
| 359 | 361 | ||
| 360 | sqlite3_stmt* rawStmt; | 362 | sqlite3_stmt* rawStmt; |
| 361 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { | 363 | if (sqlite3_prepare_v2(m_db, sql.c_str(), -1, &rawStmt, nullptr) != SQLITE_OK) { |
| 362 | - fprintf(stderr, "[SqliteTable::FindInternal] SQL prepare failed [%s]: %s\n", | 364 | + HCCL_VM_ERROR("[SqliteTable::FindInternal] SQL prepare failed [{}]: {}", |
| 363 | m_tableName.c_str(), sqlite3_errmsg(m_db)); | 365 | m_tableName.c_str(), sqlite3_errmsg(m_db)); |
| 364 | return false; | 366 | return false; |
| 365 | } | 367 | } |
| @@ -467,7 +469,7 @@ public: | |||
| 467 | if (sqlite3_open_v2(actualPath.c_str(), &m_db, | 469 | if (sqlite3_open_v2(actualPath.c_str(), &m_db, |
| 468 | SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_URI, | 470 | SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_URI, |
| 469 | nullptr) != SQLITE_OK) { | 471 | nullptr) != SQLITE_OK) { |
| 470 | - fprintf(stderr, "[SqliteDatabase] CRITICAL: Failed to open SQLite database: %s\n", | 472 | + HCCL_VM_ERROR("[SqliteDatabase] CRITICAL: Failed to open SQLite database: {}", |
| 471 | sqlite3_errmsg(m_db)); | 473 | sqlite3_errmsg(m_db)); |
| 472 | } else { | 474 | } else { |
| 473 | sqlite3_exec(m_db, "PRAGMA journal_mode = WAL", nullptr, nullptr, nullptr); | 475 | sqlite3_exec(m_db, "PRAGMA journal_mode = WAL", nullptr, nullptr, nullptr); |
| @@ -22,45 +22,51 @@ | |||
| 22 | 22 | ||
| 23 | extern spdlog::logger* g_logger; | 23 | extern spdlog::logger* g_logger; |
| 24 | 24 | ||
| 25 | + | ||
| 26 | + | ||
| 27 | + | ||
| 28 | + | ||
| 29 | + | ||
| 30 | + | ||
| 25 | 31 | ||
| 26 | do { \ | 32 | do { \ |
| 27 | if (g_logger != nullptr) { \ | 33 | if (g_logger != nullptr) { \ |
| 28 | - SPDLOG_LOGGER_TRACE(g_logger, __VA_ARGS__); \ | 34 | + SPDLOG_LOGGER_TRACE(g_logger, HCCL_VM_TAG __VA_ARGS__); \ |
| 29 | } \ | 35 | } \ |
| 30 | } while(0) | 36 | } while(0) |
| 31 | 37 | ||
| 32 | 38 | ||
| 33 | do { \ | 39 | do { \ |
| 34 | if (g_logger != nullptr) { \ | 40 | if (g_logger != nullptr) { \ |
| 35 | - SPDLOG_LOGGER_DEBUG(g_logger, __VA_ARGS__); \ | 41 | + SPDLOG_LOGGER_DEBUG(g_logger, HCCL_VM_TAG __VA_ARGS__); \ |
| 36 | } \ | 42 | } \ |
| 37 | } while(0) | 43 | } while(0) |
| 38 | 44 | ||
| 39 | 45 | ||
| 40 | do { \ | 46 | do { \ |
| 41 | if (g_logger != nullptr) { \ | 47 | if (g_logger != nullptr) { \ |
| 42 | - SPDLOG_LOGGER_INFO(g_logger, __VA_ARGS__); \ | 48 | + SPDLOG_LOGGER_INFO(g_logger, HCCL_VM_TAG __VA_ARGS__); \ |
| 43 | } \ | 49 | } \ |
| 44 | } while(0) | 50 | } while(0) |
| 45 | 51 | ||
| 46 | 52 | ||
| 47 | do { \ | 53 | do { \ |
| 48 | if (g_logger != nullptr) { \ | 54 | if (g_logger != nullptr) { \ |
| 49 | - SPDLOG_LOGGER_WARN(g_logger, __VA_ARGS__); \ | 55 | + SPDLOG_LOGGER_WARN(g_logger, HCCL_VM_TAG __VA_ARGS__); \ |
| 50 | } \ | 56 | } \ |
| 51 | } while(0) | 57 | } while(0) |
| 52 | 58 | ||
| 53 | 59 | ||
| 54 | do { \ | 60 | do { \ |
| 55 | if (g_logger != nullptr) { \ | 61 | if (g_logger != nullptr) { \ |
| 56 | - SPDLOG_LOGGER_ERROR(g_logger, __VA_ARGS__); \ | 62 | + SPDLOG_LOGGER_ERROR(g_logger, HCCL_VM_TAG __VA_ARGS__); \ |
| 57 | } \ | 63 | } \ |
| 58 | } while(0) | 64 | } while(0) |
| 59 | 65 | ||
| 60 | 66 | ||
| 61 | do { \ | 67 | do { \ |
| 62 | if (g_logger != nullptr) { \ | 68 | if (g_logger != nullptr) { \ |
| 63 | - SPDLOG_LOGGER_CRITICAL(g_logger, __VA_ARGS__); \ | 69 | + SPDLOG_LOGGER_CRITICAL(g_logger, HCCL_VM_TAG __VA_ARGS__); \ |
| 64 | } \ | 70 | } \ |
| 65 | } while(0) | 71 | } while(0) |
| 66 | 72 | ||
| @@ -21,6 +21,8 @@ | |||
| 21 | 21 | ||
| 22 | 22 | ||
| 23 | 23 | ||
| 24 | + | ||
| 25 | + | ||
| 24 | namespace sim { | 26 | namespace sim { |
| 25 | // Sentinel value written to proxy_ready to signal runner to exit. | 27 | // Sentinel value written to proxy_ready to signal runner to exit. |
| 26 | // Normal rounds use values in [1, UINT32_MAX-1]. | 28 | // Normal rounds use values in [1, UINT32_MAX-1]. |
| @@ -135,23 +137,23 @@ private: | |||
| 135 | 137 | ||
| 136 | int fd = ::open(tmpPath.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0644); | 138 | int fd = ::open(tmpPath.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0644); |
| 137 | if (fd < 0) { | 139 | if (fd < 0) { |
| 138 | - std::fprintf(stderr, "[ProcessSyncer] open tmp failed: %s, error: %s\n", | 140 | + HCCL_VM_ERROR("[ProcessSyncer] open tmp failed: {}, error: {}", |
| 139 | - tmpPath.c_str(), std::strerror(errno)); | 141 | + tmpPath.c_str(), std::strerror(errno)); |
| 140 | return; | 142 | return; |
| 141 | } | 143 | } |
| 142 | 144 | ||
| 143 | ssize_t written = ::write(fd, buf, n); | 145 | ssize_t written = ::write(fd, buf, n); |
| 144 | if (written != n) { | 146 | if (written != n) { |
| 145 | - std::fprintf(stderr, "[ProcessSyncer] write failed: %s\n", std::strerror(errno)); | 147 | + HCCL_VM_ERROR("[ProcessSyncer] write failed: {}", std::strerror(errno)); |
| 146 | } | 148 | } |
| 147 | 149 | ||
| 148 | if (::fsync(fd) != 0) { | 150 | if (::fsync(fd) != 0) { |
| 149 | - std::fprintf(stderr, "[ProcessSyncer] fsync failed: %s\n", std::strerror(errno)); | 151 | + HCCL_VM_ERROR("[ProcessSyncer] fsync failed: {}", std::strerror(errno)); |
| 150 | } | 152 | } |
| 151 | ::close(fd); | 153 | ::close(fd); |
| 152 | 154 | ||
| 153 | if (::rename(tmpPath.c_str(), filepath.c_str()) != 0) { | 155 | if (::rename(tmpPath.c_str(), filepath.c_str()) != 0) { |
| 154 | - std::fprintf(stderr, "[ProcessSyncer] rename failed: %s\n", std::strerror(errno)); | 156 | + HCCL_VM_ERROR("[ProcessSyncer] rename failed: {}", std::strerror(errno)); |
| 155 | } | 157 | } |
| 156 | } | 158 | } |
| 157 | 159 | ||
| @@ -2,11 +2,11 @@ | |||
| 2 | 2 | ||
| 3 | [toc] | 3 | [toc] |
| 4 | 4 | ||
| 5 | -## 核心功能介绍 | 5 | +## 1 核心功能介绍 |
| 6 | 6 | ||
| 7 | -### 1.启动模拟环境 | 7 | +### 1.1启动模拟环境 |
| 8 | 8 | ||
| 9 | -#### 查看当前所有表名称 | 9 | +#### 1.1.1 查看当前所有表名称 |
| 10 | 10 | ||
| 11 | ```bash | 11 | ```bash |
| 12 | (hvm)$> hccl-vm table show all | 12 | (hvm)$> hccl-vm table show all |
| @@ -19,7 +19,7 @@ Device | |||
| 19 | (hvm)$> | 19 | (hvm)$> |
| 20 | ``` | 20 | ``` |
| 21 | 21 | ||
| 22 | -#### 查看指定表内容 | 22 | +#### 1.1.2 查看指定表内容 |
| 23 | 23 | ||
| 24 | ```bash | 24 | ```bash |
| 25 | (hvm)$> hccl-vm table show Device | 25 | (hvm)$> hccl-vm table show Device |
| @@ -32,7 +32,7 @@ Device | |||
| 32 | (hvm)$> | 32 | (hvm)$> |
| 33 | ``` | 33 | ``` |
| 34 | 34 | ||
| 35 | -#### 更新指定表指定行指定内容(目前只支持devicesoc_version修改) | 35 | +#### 1.1.3 更新指定表指定行指定内容(目前只支持devicesoc_version修改) |
| 36 | 36 | ||
| 37 | ```bash | 37 | ```bash |
| 38 | (hvm)$> hccl-vm table show Device | 38 | (hvm)$> hccl-vm table show Device |
| @@ -62,3 +62,6 @@ target_link_libraries(cmd PRIVATE | |||
| 62 | if(TARGET third_party_yaml_cpp) | 62 | if(TARGET third_party_yaml_cpp) |
| 63 | add_dependencies(cmd third_party_yaml_cpp) | 63 | add_dependencies(cmd third_party_yaml_cpp) |
| 64 | endif() | 64 | endif() |
| 65 | + | ||
| 66 | +# 日志染色: 模块 tag | ||
| 67 | +target_compile_definitions(cmd PRIVATE HCCL_VM_MODULE="CMD") | ||
| @@ -51,6 +51,7 @@ | |||
| 51 | 51 | ||
| 52 | 52 | ||
| 53 | 53 | ||
| 54 | + | ||
| 54 | 55 | ||
| 55 | 56 | ||
| 56 | 57 | ||
| @@ -93,7 +94,7 @@ static void RemoveMessageQueue(const char *name) | |||
| 93 | { | 94 | { |
| 94 | const std::string mqName = ToPosixMqName(name); | 95 | const std::string mqName = ToPosixMqName(name); |
| 95 | if (mq_unlink(mqName.c_str()) == -1 && errno != ENOENT) { | 96 | if (mq_unlink(mqName.c_str()) == -1 && errno != ENOENT) { |
| 96 | - HCCL_VM_WARN("[HVM] {}", MakeMqErrorMessage("remove", mqName, errno)); | 97 | + HCCL_VM_WARN("{}", MakeMqErrorMessage("remove", mqName, errno)); |
| 97 | } | 98 | } |
| 98 | } | 99 | } |
| 99 | 100 | ||
| @@ -282,27 +283,23 @@ bool TryParseAivRankIdFromTaskFileName(const std::string &fileName, uint32_t &ra | |||
| 282 | 283 | ||
| 283 | HcclVmResult ValidateAivTaskJsonByRank(const std::vector<sim::Rank> &allRank) | 284 | HcclVmResult ValidateAivTaskJsonByRank(const std::vector<sim::Rank> &allRank) |
| 284 | { | 285 | { |
| 285 | - const char *installDirEnv = std::getenv("HCCL_VM_INSTALL_DIR"); | 286 | + const std::string &installDir = InstallPath::GetHcclVmInstallAbsPath(); |
| 286 | - if (installDirEnv == nullptr || installDirEnv[0] == '\0') { | ||
| 287 | - HCCL_VM_ERROR("[HVM] env HCCL_VM_INSTALL_DIR is not set in AIV mode."); | ||
| 288 | - return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | ||
| 289 | - } | ||
| 290 | 287 | ||
| 291 | if (allRank.empty()) { | 288 | if (allRank.empty()) { |
| 292 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; | 289 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; |
| 293 | } | 290 | } |
| 294 | 291 | ||
| 295 | - const fs::path dataDir = fs::path(installDirEnv) / "data"; | 292 | + const fs::path dataDir = fs::path(InstallPath::ResolveToInstallRoot("data")); |
| 296 | std::error_code ec; | 293 | std::error_code ec; |
| 297 | if (!fs::exists(dataDir, ec) || ec || !fs::is_directory(dataDir, ec)) { | 294 | if (!fs::exists(dataDir, ec) || ec || !fs::is_directory(dataDir, ec)) { |
| 298 | - HCCL_VM_ERROR("[HVM] AIV data directory is missing: {}", dataDir.string()); | 295 | + HCCL_VM_ERROR("AIV data directory is missing: {}", dataDir.string()); |
| 299 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 296 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 300 | } | 297 | } |
| 301 | 298 | ||
| 302 | std::unordered_set<uint32_t> existingRanks; | 299 | std::unordered_set<uint32_t> existingRanks; |
| 303 | for (const auto &entry : fs::directory_iterator(dataDir, ec)) { | 300 | for (const auto &entry : fs::directory_iterator(dataDir, ec)) { |
| 304 | if (ec) { | 301 | if (ec) { |
| 305 | - HCCL_VM_ERROR("[HVM] failed to iterate AIV data directory: {}", dataDir.string()); | 302 | + HCCL_VM_ERROR("failed to iterate AIV data directory: {}", dataDir.string()); |
| 306 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 303 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 307 | } | 304 | } |
| 308 | if (!entry.is_regular_file()) { | 305 | if (!entry.is_regular_file()) { |
| @@ -317,7 +314,7 @@ HcclVmResult ValidateAivTaskJsonByRank(const std::vector<sim::Rank> &allRank) | |||
| 317 | 314 | ||
| 318 | for (const auto &rank : allRank) { | 315 | for (const auto &rank : allRank) { |
| 319 | if (existingRanks.find(rank.rank_id) == existingRanks.end()) { | 316 | if (existingRanks.find(rank.rank_id) == existingRanks.end()) { |
| 320 | - HCCL_VM_ERROR("[HVM] missing AIV task json file for rank {}", rank.rank_id); | 317 | + HCCL_VM_ERROR("missing AIV task json file for rank {}", rank.rank_id); |
| 321 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 318 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 322 | } | 319 | } |
| 323 | } | 320 | } |
| @@ -356,7 +353,7 @@ std::string ArgvToString(int argc, char *argv[]) { | |||
| 356 | } | 353 | } |
| 357 | 354 | ||
| 358 | void RemoveFromLDPreload(const std::string& targetValue) { | 355 | void RemoveFromLDPreload(const std::string& targetValue) { |
| 359 | - HCCL_VM_DEBUG("[HVM] clean LD_PRELOAD env: {}", targetValue); | 356 | + HCCL_VM_DEBUG("clean LD_PRELOAD env: {}", targetValue); |
| 360 | const char* curVal = std::getenv("LD_PRELOAD"); | 357 | const char* curVal = std::getenv("LD_PRELOAD"); |
| 361 | 358 | ||
| 362 | if (curVal == nullptr) { | 359 | if (curVal == nullptr) { |
| @@ -397,14 +394,14 @@ void RemoveFromLDPreload(const std::string& targetValue) { | |||
| 397 | 394 | ||
| 398 | std::string FileInModelDir(const std::string& fileName) { | 395 | std::string FileInModelDir(const std::string& fileName) { |
| 399 | if (fileName == "ranktable") { | 396 | if (fileName == "ranktable") { |
| 400 | - std::string clusterInfo = GetBinLocation() + "/" + fileName + ".json"; | 397 | + std::string clusterInfo = InstallPath::ResolveToInstallRoot(fileName + ".json"); |
| 401 | std::ifstream f(clusterInfo.c_str()); | 398 | std::ifstream f(clusterInfo.c_str()); |
| 402 | if (!f.good()) { | 399 | if (!f.good()) { |
| 403 | return "config ranktable, but rank table file not found: " + clusterInfo; | 400 | return "config ranktable, but rank table file not found: " + clusterInfo; |
| 404 | } | 401 | } |
| 405 | return ""; | 402 | return ""; |
| 406 | } | 403 | } |
| 407 | - std::string filePath = GetBinLocation() + "/config/topo_meta/" + fileName + ".yaml"; | 404 | + std::string filePath = InstallPath::ResolveToInstallRoot("config/topo_meta/" + fileName + ".yaml"); |
| 408 | auto fileExistd = [&]()->bool { | 405 | auto fileExistd = [&]()->bool { |
| 409 | std::ifstream f(filePath.c_str()); | 406 | std::ifstream f(filePath.c_str()); |
| 410 | return f.good(); | 407 | return f.good(); |
| @@ -423,33 +420,33 @@ std::string GenerateClusterTopo(const std::string& topoFileName) { | |||
| 423 | topoName.erase(topoName.find(".yaml"), 5); | 420 | topoName.erase(topoName.find(".yaml"), 5); |
| 424 | } | 421 | } |
| 425 | 422 | ||
| 426 | - std::string generateShellPath = GetBinLocation() + "/script/generate_cluster_topo.sh"; | 423 | + std::string generateShellPath = InstallPath::ResolveToInstallRoot("script/generate_cluster_topo.sh"); |
| 427 | if (!fs::exists(generateShellPath)) { | 424 | if (!fs::exists(generateShellPath)) { |
| 428 | - HCCL_VM_ERROR("[HVM] generate_cluster_topo.sh not found: {}", generateShellPath); | 425 | + HCCL_VM_ERROR("generate_cluster_topo.sh not found: {}", generateShellPath); |
| 429 | return "[HVM] generate_cluster_topo.sh not found: " + generateShellPath; | 426 | return "[HVM] generate_cluster_topo.sh not found: " + generateShellPath; |
| 430 | } | 427 | } |
| 431 | - std::string clusterConfigFilePath = GetBinLocation() + "/config/cluster/" + topoName + ".yaml"; | 428 | + std::string clusterConfigFilePath = InstallPath::ResolveToInstallRoot("config/cluster/" + topoName + ".yaml"); |
| 432 | if (!fs::exists(clusterConfigFilePath)) { | 429 | if (!fs::exists(clusterConfigFilePath)) { |
| 433 | - HCCL_VM_ERROR("[HVM] cluster config file not found: {}", clusterConfigFilePath); | 430 | + HCCL_VM_ERROR("cluster config file not found: {}", clusterConfigFilePath); |
| 434 | return "[HVM] cluster config file not found: " + clusterConfigFilePath; | 431 | return "[HVM] cluster config file not found: " + clusterConfigFilePath; |
| 435 | } | 432 | } |
| 436 | // 执行generate_cluster_topo.sh脚本(默认路径),生成集群组网 | 433 | // 执行generate_cluster_topo.sh脚本(默认路径),生成集群组网 |
| 437 | std::string cmd = "bash " + generateShellPath + " " + clusterConfigFilePath; | 434 | std::string cmd = "bash " + generateShellPath + " " + clusterConfigFilePath; |
| 438 | if (std::system(cmd.c_str()) != 0) { | 435 | if (std::system(cmd.c_str()) != 0) { |
| 439 | - HCCL_VM_ERROR("[HVM] generate_cluster_topo.sh failed: {}", cmd); | 436 | + HCCL_VM_ERROR("generate_cluster_topo.sh failed: {}", cmd); |
| 440 | return "[HVM] generate_cluster_topo.sh failed: " + cmd; | 437 | return "[HVM] generate_cluster_topo.sh failed: " + cmd; |
| 441 | } | 438 | } |
| 442 | return ""; | 439 | return ""; |
| 443 | } | 440 | } |
| 444 | 441 | ||
| 445 | void ShowModel() { | 442 | void ShowModel() { |
| 446 | - std::string modelPath = GetBinLocation() + "/cluster_model"; | 443 | + std::string modelPath = InstallPath::ResolveToInstallRoot("cluster_model"); |
| 447 | if (!fs::exists(modelPath)) { | 444 | if (!fs::exists(modelPath)) { |
| 448 | - HCCL_VM_ERROR("[HVM] path not exist -> {}", modelPath); | 445 | + HCCL_VM_ERROR("path not exist -> {}", modelPath); |
| 449 | return; | 446 | return; |
| 450 | } | 447 | } |
| 451 | if (!fs::is_directory(modelPath)) { | 448 | if (!fs::is_directory(modelPath)) { |
| 452 | - HCCL_VM_ERROR("[HVM] path not a dict -> {}", modelPath); | 449 | + HCCL_VM_ERROR("path not a dict -> {}", modelPath); |
| 453 | return; | 450 | return; |
| 454 | } | 451 | } |
| 455 | bool hasFiles = false; | 452 | bool hasFiles = false; |
| @@ -468,7 +465,7 @@ void ShowModel() { | |||
| 468 | } | 465 | } |
| 469 | } | 466 | } |
| 470 | if (!hasFiles) { | 467 | if (!hasFiles) { |
| 471 | - HCCL_VM_WARN("[HVM] there is no model"); | 468 | + HCCL_VM_WARN("there is no model"); |
| 472 | } | 469 | } |
| 473 | return; | 470 | return; |
| 474 | } | 471 | } |
| @@ -483,13 +480,13 @@ HcclVmResult InitHvmEnv(const std::string& configClusterDir, uint32_t level, boo | |||
| 483 | void* commPool = sim::MemoryManager::GetInstance().AllocMemByName( | 480 | void* commPool = sim::MemoryManager::GetInstance().AllocMemByName( |
| 484 | sim::CommPoolPolicy::kPoolName, sim::CommPoolPolicy::kPoolSize); | 481 | sim::CommPoolPolicy::kPoolName, sim::CommPoolPolicy::kPoolSize); |
| 485 | if (commPool == nullptr) { | 482 | if (commPool == nullptr) { |
| 486 | - HCCL_VM_ERROR("[HVM] create HcclCommPool fail"); | 483 | + HCCL_VM_ERROR("create HcclCommPool fail"); |
| 487 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 484 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 488 | } | 485 | } |
| 489 | } | 486 | } |
| 490 | void *shmptr =sim::MemoryManager::GetInstance().AllocMemByName("HcclAicpuData", sizeof(HcclAicpuData)); | 487 | void *shmptr =sim::MemoryManager::GetInstance().AllocMemByName("HcclAicpuData", sizeof(HcclAicpuData)); |
| 491 | if (shmptr == nullptr) { | 488 | if (shmptr == nullptr) { |
| 492 | - HCCL_VM_ERROR("[HVM] Alloc Shared Memory fail "); | 489 | + HCCL_VM_ERROR("Alloc Shared Memory fail "); |
| 493 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 490 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 494 | } | 491 | } |
| 495 | 492 | ||
| @@ -500,7 +497,7 @@ HcclVmResult InitHvmEnv(const std::string& configClusterDir, uint32_t level, boo | |||
| 500 | std::string checkerTag = "@checker"; | 497 | std::string checkerTag = "@checker"; |
| 501 | auto chkInstallRet = InstallUserPlugin(checkerTag); | 498 | auto chkInstallRet = InstallUserPlugin(checkerTag); |
| 502 | if (chkInstallRet != HCCL_SIM_HOST_SUCCESS_CMD) { | 499 | if (chkInstallRet != HCCL_SIM_HOST_SUCCESS_CMD) { |
| 503 | - HCCL_VM_ERROR("[HVM] default plugin install fail, please check your plugin path"); | 500 | + HCCL_VM_ERROR("default plugin install fail, please check your plugin path"); |
| 504 | } | 501 | } |
| 505 | 502 | ||
| 506 | HCCL_VM_INFO("===================================="); | 503 | HCCL_VM_INFO("===================================="); |
| @@ -525,7 +522,7 @@ HcclVmResult InitHvmCommEnv(const TopoMeta& topoMeta, const std::string& configF | |||
| 525 | ret = AscendClusterTopoParser::GetInstance().InitCommunicationDomain(topoMeta, false); | 522 | ret = AscendClusterTopoParser::GetInstance().InitCommunicationDomain(topoMeta, false); |
| 526 | } else { | 523 | } else { |
| 527 | HCCL_VM_INFO("mock-comm cmd config rank table file, config by ranktable.json"); | 524 | HCCL_VM_INFO("mock-comm cmd config rank table file, config by ranktable.json"); |
| 528 | - std::string clusterInfo = GetBinLocation() + "/data/ranktable.json"; | 525 | + std::string clusterInfo = InstallPath::ResolveToInstallRoot("data/ranktable.json"); |
| 529 | std::ifstream f(clusterInfo.c_str()); | 526 | std::ifstream f(clusterInfo.c_str()); |
| 530 | if (!f.good()) { | 527 | if (!f.good()) { |
| 531 | HCCL_VM_ERROR("ranktable.json file not found: {}", clusterInfo); | 528 | HCCL_VM_ERROR("ranktable.json file not found: {}", clusterInfo); |
| @@ -535,9 +532,10 @@ HcclVmResult InitHvmCommEnv(const TopoMeta& topoMeta, const std::string& configF | |||
| 535 | ret = AscendClusterTopoParser::GetInstance().ParseRanktableAndInitCommDomain(clusterInfo); | 532 | ret = AscendClusterTopoParser::GetInstance().ParseRanktableAndInitCommDomain(clusterInfo); |
| 536 | } | 533 | } |
| 537 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 534 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 538 | - HCCL_VM_ERROR("[HVM] InitHvmCommEnv failed"); | 535 | + HCCL_VM_ERROR("InitHvmCommEnv failed"); |
| 539 | return ret; | 536 | return ret; |
| 540 | } | 537 | } |
| 538 | + setenv("RANK_TABLE_FILE", InstallPath::ResolveToInstallRoot("data/ranktable.json").c_str(), 1); | ||
| 541 | 539 | ||
| 542 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; | 540 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; |
| 543 | } | 541 | } |
| @@ -553,7 +551,7 @@ HcclVmResult HcclVmExit() | |||
| 553 | HcclPluginManager &pluginManager = HcclPluginManager::GetInstance(); | 551 | HcclPluginManager &pluginManager = HcclPluginManager::GetInstance(); |
| 554 | auto ret = pluginManager.StopAllPlugins(); | 552 | auto ret = pluginManager.StopAllPlugins(); |
| 555 | 553 | ||
| 556 | - const fs::path backupDir = fs::path(g_binDir) / "data" / MakeDataBackupTimestamp(); | 554 | + const fs::path backupDir = fs::path(InstallPath::ResolveToInstallRoot("data/" + MakeDataBackupTimestamp())); |
| 557 | BackupAivTaskFiles(backupDir); | 555 | BackupAivTaskFiles(backupDir); |
| 558 | 556 | ||
| 559 | HCCL_VM_INFO("start Destroy ALL Resources."); | 557 | HCCL_VM_INFO("start Destroy ALL Resources."); |
| @@ -565,6 +563,7 @@ HcclVmResult HcclVmExit() | |||
| 565 | } | 563 | } |
| 566 | int ret1 = system("sudo rm -fr /dev/shm/* 2>/dev/null"); | 564 | int ret1 = system("sudo rm -fr /dev/shm/* 2>/dev/null"); |
| 567 | ret1 |= system("sudo rm -fr /tmp/hccl_sim.db* 2>/dev/null"); | 565 | ret1 |= system("sudo rm -fr /tmp/hccl_sim.db* 2>/dev/null"); |
| 566 | + | ||
| 568 | return ret; | 567 | return ret; |
| 569 | } | 568 | } |
| 570 | 569 | ||
| @@ -572,7 +571,7 @@ HcclVmResult InstallUserPlugin(std::string argStr) { | |||
| 572 | // 处理插件tag和路径 | 571 | // 处理插件tag和路径 |
| 573 | HcclVmResult ret {HcclVmResult::HCCL_SIM_HOST_ERROR_CMD}; | 572 | HcclVmResult ret {HcclVmResult::HCCL_SIM_HOST_ERROR_CMD}; |
| 574 | if (argStr.empty() || argStr[0] != '@') { | 573 | if (argStr.empty() || argStr[0] != '@') { |
| 575 | - HCCL_VM_ERROR("[HVM] plugin tag should start with '@', invalid tag: {}", argStr); | 574 | + HCCL_VM_ERROR("plugin tag should start with '@', invalid tag: {}", argStr); |
| 576 | return ret; | 575 | return ret; |
| 577 | } | 576 | } |
| 578 | argStr.erase(argStr.begin()); | 577 | argStr.erase(argStr.begin()); |
| @@ -581,13 +580,13 @@ HcclVmResult InstallUserPlugin(std::string argStr) { | |||
| 581 | HcclPluginManager &pluginManager = HcclPluginManager::GetInstance(); | 580 | HcclPluginManager &pluginManager = HcclPluginManager::GetInstance(); |
| 582 | ret = pluginManager.RegisterPlugin(argStr); | 581 | ret = pluginManager.RegisterPlugin(argStr); |
| 583 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 582 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 584 | - HCCL_VM_ERROR("[HVM] Install plugin [{}] failed", argStr); | 583 | + HCCL_VM_ERROR("Install plugin [{}] failed", argStr); |
| 585 | return ret; | 584 | return ret; |
| 586 | } | 585 | } |
| 587 | 586 | ||
| 588 | // 装 Runner 时若仅校验模式开着,复用池仍在、大块仍会引流,可能覆盖 Runner 真实数据,告警提示。 | 587 | // 装 Runner 时若仅校验模式开着,复用池仍在、大块仍会引流,可能覆盖 Runner 真实数据,告警提示。 |
| 589 | if (argStr == "runner" && sim::IsCheckOnlyMode()) { | 588 | if (argStr == "runner" && sim::IsCheckOnlyMode()) { |
| 590 | - HCCL_VM_WARN("[HVM] check-only mode on while runner installed; big-block contents are not guaranteed"); | 589 | + HCCL_VM_WARN("check-only mode on while runner installed; big-block contents are not guaranteed"); |
| 591 | } | 590 | } |
| 592 | 591 | ||
| 593 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; | 592 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; |
| @@ -631,7 +630,7 @@ HcclVmResult UninstallUserPlugin(std::string argStr) { | |||
| 631 | 630 | ||
| 632 | for (int i = 0; i < pluginTags.size(); ++i) { | 631 | for (int i = 0; i < pluginTags.size(); ++i) { |
| 633 | if (rets[i] != HcclVmResult::HCCL_SIM_SUCCESS) { | 632 | if (rets[i] != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 634 | - HCCL_VM_ERROR("[HVM] plugin Uninstall fail : {}", pluginTags[i]); | 633 | + HCCL_VM_ERROR("plugin Uninstall fail : {}", pluginTags[i]); |
| 635 | return rets[i]; | 634 | return rets[i]; |
| 636 | } | 635 | } |
| 637 | } | 636 | } |
| @@ -645,7 +644,7 @@ void ShowUserPlugin() { | |||
| 645 | listPlugins = pluginManager.GetPluginStatus(); | 644 | listPlugins = pluginManager.GetPluginStatus(); |
| 646 | 645 | ||
| 647 | if (listPlugins.empty()) { | 646 | if (listPlugins.empty()) { |
| 648 | - HCCL_VM_INFO("[HVM] no plugin installed in hccl_vm"); | 647 | + HCCL_VM_INFO("no plugin installed in hccl_vm"); |
| 649 | } else { | 648 | } else { |
| 650 | for (auto &plugin : listPlugins) { | 649 | for (auto &plugin : listPlugins) { |
| 651 | HCCL_VM_INFO("{}", plugin); | 650 | HCCL_VM_INFO("{}", plugin); |
| @@ -656,18 +655,18 @@ void ShowUserPlugin() { | |||
| 656 | 655 | ||
| 657 | HcclVmResult SetConsoleLogLevel(int level) { | 656 | HcclVmResult SetConsoleLogLevel(int level) { |
| 658 | (void)level; | 657 | (void)level; |
| 659 | - HCCL_VM_WARN("[HVM] set console log level is disabled because ProxyConfig shared memory is removed"); | 658 | + HCCL_VM_WARN("set console log level is disabled because ProxyConfig shared memory is removed"); |
| 660 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 659 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 661 | } | 660 | } |
| 662 | 661 | ||
| 663 | HcclVmResult SetFileLogLevel(int level) { | 662 | HcclVmResult SetFileLogLevel(int level) { |
| 664 | (void)level; | 663 | (void)level; |
| 665 | - HCCL_VM_WARN("[HVM] set file log level is disabled because ProxyConfig shared memory is removed"); | 664 | + HCCL_VM_WARN("set file log level is disabled because ProxyConfig shared memory is removed"); |
| 666 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 665 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 667 | } | 666 | } |
| 668 | 667 | ||
| 669 | HcclVmResult ShowCurrentLogLevel() { | 668 | HcclVmResult ShowCurrentLogLevel() { |
| 670 | - HCCL_VM_WARN("[HVM] show log level is disabled because ProxyConfig shared memory is removed"); | 669 | + HCCL_VM_WARN("show log level is disabled because ProxyConfig shared memory is removed"); |
| 671 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; | 670 | return HcclVmResult::HCCL_SIM_HOST_ERROR_CMD; |
| 672 | } | 671 | } |
| 673 | 672 | ||
| @@ -683,10 +682,10 @@ HcclVmResult StartHvmCmd() { | |||
| 683 | 682 | ||
| 684 | // Child Process(Bash) | 683 | // Child Process(Bash) |
| 685 | // 劫持库存在性判断 | 684 | // 劫持库存在性判断 |
| 686 | - std::string hcclVmbin = g_binDir + "/bin/hccl-vm"; | 685 | + std::string hcclVmbin = InstallPath::ResolveToInstallRoot("bin/hccl-vm"); |
| 687 | - std::string proxyPath = g_binDir + "/lib/x86_64/libhccl_proxy_level" + std::to_string(g_hcclVmLevel) + ".so"; | 686 | + std::string proxyPath = InstallPath::ResolveToInstallRoot("lib/x86_64/libhccl_proxy_level" + std::to_string(g_hcclVmLevel) + ".so"); |
| 688 | if (!fs::exists(proxyPath)) { | 687 | if (!fs::exists(proxyPath)) { |
| 689 | - HCCL_VM_ERROR("[HVM] [ERROR] proxy hacking .so not found {}, please check your proxy hacking .so:" | 688 | + HCCL_VM_ERROR("proxy hacking .so not found {}, please check your proxy hacking .so:" |
| 690 | "1. Whether the hook library has been successfully built and installed. 2. Whether the simulation level matches the proxy hook library version. Current simulation level: {}, Default simulation level: 2" | 689 | "1. Whether the hook library has been successfully built and installed. 2. Whether the simulation level matches the proxy hook library version. Current simulation level: {}, Default simulation level: 2" |
| 691 | , proxyPath, g_hcclVmLevel); | 690 | , proxyPath, g_hcclVmLevel); |
| 692 | return HCCL_SIM_HOST_ERROR_CMD; | 691 | return HCCL_SIM_HOST_ERROR_CMD; |
| @@ -695,7 +694,7 @@ HcclVmResult StartHvmCmd() { | |||
| 695 | // 管道处理的用途是隔绝进程终端在std::cout中的残留 | 694 | // 管道处理的用途是隔绝进程终端在std::cout中的残留 |
| 696 | int pipefds[2] = {-1, -1}; | 695 | int pipefds[2] = {-1, -1}; |
| 697 | if (pipe(pipefds) == -1) { | 696 | if (pipe(pipefds) == -1) { |
| 698 | - HCCL_VM_ERROR("[HVM] pipe failed"); | 697 | + HCCL_VM_ERROR("pipe failed"); |
| 699 | return HCCL_SIM_HOST_ERROR_CMD; | 698 | return HCCL_SIM_HOST_ERROR_CMD; |
| 700 | } | 699 | } |
| 701 | 700 | ||
| @@ -712,7 +711,7 @@ HcclVmResult StartHvmCmd() { | |||
| 712 | 711 | ||
| 713 | ssize_t written = write(pipefds[1], bashrcHack.c_str(), bashrcHack.size()); | 712 | ssize_t written = write(pipefds[1], bashrcHack.c_str(), bashrcHack.size()); |
| 714 | if (written != static_cast<ssize_t>(bashrcHack.size())) { | 713 | if (written != static_cast<ssize_t>(bashrcHack.size())) { |
| 715 | - HCCL_VM_ERROR("[HVM] Failed to write full script to pipe"); | 714 | + HCCL_VM_ERROR("Failed to write full script to pipe"); |
| 716 | close(pipefds[0]); | 715 | close(pipefds[0]); |
| 717 | close(pipefds[1]); | 716 | close(pipefds[1]); |
| 718 | return HCCL_SIM_HOST_ERROR_CMD; | 717 | return HCCL_SIM_HOST_ERROR_CMD; |
| @@ -733,14 +732,15 @@ HcclVmResult StartHvmCmd() { | |||
| 733 | // Fork Bash | 732 | // Fork Bash |
| 734 | pid_t pid = fork(); | 733 | pid_t pid = fork(); |
| 735 | if (pid == -1) { | 734 | if (pid == -1) { |
| 736 | - auto ret = HcclVmExit(); | 735 | + auto ret = HcclVmExit(); |
| 737 | - perror("fork failed"); | 736 | + HCCL_VM_ERROR("fork failed: {}", std::strerror(errno)); |
| 738 | close(pipefds[0]); | 737 | close(pipefds[0]); |
| 739 | return (ret == HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) ? HcclVmResult::HCCL_SIM_HOST_ERROR_CMD : ret; | 738 | return (ret == HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) ? HcclVmResult::HCCL_SIM_HOST_ERROR_CMD : ret; |
| 740 | } else if (pid == 0) { | 739 | } else if (pid == 0) { |
| 741 | setenv(HVM_BASH_ENV_KEY.c_str(), g_binDir.c_str(), 1); | 740 | setenv(HVM_BASH_ENV_KEY.c_str(), g_binDir.c_str(), 1); |
| 742 | setenv("LD_PRELOAD", proxyPath.c_str(), 1); | 741 | setenv("LD_PRELOAD", proxyPath.c_str(), 1); |
| 743 | - setenv("HCCL_VM_INSTALL_ROOT", g_binDir.c_str(), 1); | 742 | + setenv("HCCL_VM_INSTALL_ROOT", g_binDir.c_str(), 1); |
| 743 | + setenv("RANK_TABLE_FILE", InstallPath::ResolveToInstallRoot("data/ranktable.json").c_str(), 1); | ||
| 744 | execv("/bin/bash", bashArgv); | 744 | execv("/bin/bash", bashArgv); |
| 745 | exit(1); | 745 | exit(1); |
| 746 | } else { | 746 | } else { |
| @@ -749,7 +749,7 @@ HcclVmResult StartHvmCmd() { | |||
| 749 | int status; | 749 | int status; |
| 750 | waitpid(pid, &status, 0); | 750 | waitpid(pid, &status, 0); |
| 751 | auto ret = HcclVmExit(); | 751 | auto ret = HcclVmExit(); |
| 752 | - HCCL_VM_INFO("[HVM] Shell exited. Host shutting down."); | 752 | + HCCL_VM_INFO("Shell exited. Host shutting down."); |
| 753 | } | 753 | } |
| 754 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; | 754 | return HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD; |
| 755 | } | 755 | } |
| @@ -772,9 +772,9 @@ void StartHostClient(int argc, char *argv[]) { | |||
| 772 | unsigned int priority = 0; | 772 | unsigned int priority = 0; |
| 773 | bool hasMessage = mqResp.TimedReceive(&buffer, MAX_MSG_SIZE, &priority, deadline, recvdSize); | 773 | bool hasMessage = mqResp.TimedReceive(&buffer, MAX_MSG_SIZE, &priority, deadline, recvdSize); |
| 774 | if (hasMessage) { | 774 | if (hasMessage) { |
| 775 | - HCCL_VM_DEBUG("[HVM] Client : Command parsing response received: {}", std::string(buffer, recvdSize)); | 775 | + HCCL_VM_DEBUG("Client : Command parsing response received: {}", std::string(buffer, recvdSize)); |
| 776 | } else { | 776 | } else { |
| 777 | - HCCL_VM_WARN("[HVM] Client : Command parsing out of time, Client quit"); | 777 | + HCCL_VM_WARN("Client : Command parsing out of time, Client quit"); |
| 778 | } | 778 | } |
| 779 | } | 779 | } |
| 780 | 780 | ||
| @@ -783,7 +783,7 @@ void ServerListen() { | |||
| 783 | mq_attr attr = MakeMessageQueueAttr(); | 783 | mq_attr attr = MakeMessageQueueAttr(); |
| 784 | PosixMessageQueue mqReq(MQ_REQ_NAME, O_CREAT | O_EXCL | O_RDWR, &attr); | 784 | PosixMessageQueue mqReq(MQ_REQ_NAME, O_CREAT | O_EXCL | O_RDWR, &attr); |
| 785 | PosixMessageQueue mqResp(MQ_RESP_NAME, O_CREAT | O_EXCL | O_RDWR, &attr); | 785 | PosixMessageQueue mqResp(MQ_RESP_NAME, O_CREAT | O_EXCL | O_RDWR, &attr); |
| 786 | - HCCL_VM_INFO("[HVM] HOST Server listening..."); | 786 | + HCCL_VM_INFO("HOST Server listening..."); |
| 787 | 787 | ||
| 788 | while(true) { | 788 | while(true) { |
| 789 | char buffer[MAX_MSG_SIZE]; | 789 | char buffer[MAX_MSG_SIZE]; |
| @@ -792,7 +792,7 @@ void ServerListen() { | |||
| 792 | ssize_t recvdSize = mqReq.Receive(&buffer, MAX_MSG_SIZE, &priority); | 792 | ssize_t recvdSize = mqReq.Receive(&buffer, MAX_MSG_SIZE, &priority); |
| 793 | 793 | ||
| 794 | std::string cmd(buffer, recvdSize); | 794 | std::string cmd(buffer, recvdSize); |
| 795 | - HCCL_VM_DEBUG("[HVM] HOST : Command parsing request received: {}", cmd); | 795 | + HCCL_VM_DEBUG("HOST : Command parsing request received: {}", cmd); |
| 796 | 796 | ||
| 797 | // 执行接收到的命令逻辑 | 797 | // 执行接收到的命令逻辑 |
| 798 | ParseCommand(cmd); | 798 | ParseCommand(cmd); |
| @@ -804,7 +804,7 @@ void ServerListen() { | |||
| 804 | 804 | ||
| 805 | // 监听proxy和runner进程,中转进程状态 | 805 | // 监听proxy和runner进程,中转进程状态 |
| 806 | void RunnerListen() { | 806 | void RunnerListen() { |
| 807 | - HCCL_VM_INFO("[HVM] Runner listening..."); | 807 | + HCCL_VM_INFO("Runner listening..."); |
| 808 | 808 | ||
| 809 | sim::ProcessSyncer syncer; | 809 | sim::ProcessSyncer syncer; |
| 810 | syncer.Init(); | 810 | syncer.Init(); |
| @@ -835,7 +835,7 @@ void RunnerListen() { | |||
| 835 | } | 835 | } |
| 836 | } else if(syncer.getCurrentRound() >= 1){ | 836 | } else if(syncer.getCurrentRound() >= 1){ |
| 837 | HCCL_VM_INFO("skip dumping data for round {:d}", syncer.getCurrentRound()); | 837 | HCCL_VM_INFO("skip dumping data for round {:d}", syncer.getCurrentRound()); |
| 838 | - std::string dataDir = g_binDir + "/data/"; | 838 | + std::string dataDir = InstallPath::ResolveToInstallRoot("data") + "/"; |
| 839 | std::remove((dataDir + "runner_hcclvm_instr_data.bin").c_str()); | 839 | std::remove((dataDir + "runner_hcclvm_instr_data.bin").c_str()); |
| 840 | std::remove((dataDir + "runner_hcclvm_syn_data.bin").c_str()); | 840 | std::remove((dataDir + "runner_hcclvm_syn_data.bin").c_str()); |
| 841 | std::remove((dataDir + "runner_hcclvm_task_data.bin").c_str()); | 841 | std::remove((dataDir + "runner_hcclvm_task_data.bin").c_str()); |
| @@ -932,11 +932,11 @@ void ParseCommand(std::string& cmd) { | |||
| 932 | static void BackupDatabase(const std::string& srcPath) | 932 | static void BackupDatabase(const std::string& srcPath) |
| 933 | { | 933 | { |
| 934 | if (!fs::exists(srcPath)) { | 934 | if (!fs::exists(srcPath)) { |
| 935 | - HCCL_VM_ERROR("[HVM] Source database {} does not exist, skipping backup.", srcPath); | 935 | + HCCL_VM_INFO("Source database {} does not exist, skipping backup.", srcPath); |
| 936 | return; | 936 | return; |
| 937 | } | 937 | } |
| 938 | 938 | ||
| 939 | - fs::path dataDir = fs::path(g_binDir) / "data"; | 939 | + fs::path dataDir = fs::path(InstallPath::ResolveToInstallRoot("data")); |
| 940 | fs::create_directories(dataDir); | 940 | fs::create_directories(dataDir); |
| 941 | 941 | ||
| 942 | auto now = std::chrono::system_clock::now(); | 942 | auto now = std::chrono::system_clock::now(); |
| @@ -950,9 +950,9 @@ static void BackupDatabase(const std::string& srcPath) | |||
| 950 | 950 | ||
| 951 | auto ret = HcclSim::DB::OpDbOps::Instance().Backup(destPath); | 951 | auto ret = HcclSim::DB::OpDbOps::Instance().Backup(destPath); |
| 952 | if (ret != HcclSim::HCCL_SIM_SUCCESS) { | 952 | if (ret != HcclSim::HCCL_SIM_SUCCESS) { |
| 953 | - HCCL_VM_ERROR("[HVM] Failed to backup database to {}", destPath); | 953 | + HCCL_VM_ERROR("Failed to backup database to {}", destPath); |
| 954 | } else { | 954 | } else { |
| 955 | - HCCL_VM_INFO("[HVM] Database backed up to: {}", destPath); | 955 | + HCCL_VM_INFO("Database backed up to: {}", destPath); |
| 956 | } | 956 | } |
| 957 | } | 957 | } |
| 958 | 958 | ||
| @@ -972,7 +972,7 @@ static bool EnsureBackupDir(const fs::path &backupDir) | |||
| 972 | std::error_code ec; | 972 | std::error_code ec; |
| 973 | fs::create_directories(backupDir, ec); | 973 | fs::create_directories(backupDir, ec); |
| 974 | if (ec) { | 974 | if (ec) { |
| 975 | - HCCL_VM_WARN("[HVM] failed to create backup directory {}: {}", backupDir.string(), ec.message()); | 975 | + HCCL_VM_WARN("failed to create backup directory {}: {}", backupDir.string(), ec.message()); |
| 976 | return false; | 976 | return false; |
| 977 | } | 977 | } |
| 978 | return true; | 978 | return true; |
| @@ -980,7 +980,7 @@ static bool EnsureBackupDir(const fs::path &backupDir) | |||
| 980 | 980 | ||
| 981 | static void BackupBinFiles(const fs::path &backupDir) | 981 | static void BackupBinFiles(const fs::path &backupDir) |
| 982 | { | 982 | { |
| 983 | - fs::path dataDir = fs::path(g_binDir) / "data"; | 983 | + fs::path dataDir = fs::path(InstallPath::ResolveToInstallRoot("data")); |
| 984 | std::vector<fs::path> binFiles = { | 984 | std::vector<fs::path> binFiles = { |
| 985 | dataDir / "runner_hcclvm_instr_data.bin", | 985 | dataDir / "runner_hcclvm_instr_data.bin", |
| 986 | dataDir / "runner_hcclvm_syn_data.bin", | 986 | dataDir / "runner_hcclvm_syn_data.bin", |
| @@ -1002,18 +1002,18 @@ static void BackupBinFiles(const fs::path &backupDir) | |||
| 1002 | } | 1002 | } |
| 1003 | fs::rename(f, backupDir / f.filename(), ec); | 1003 | fs::rename(f, backupDir / f.filename(), ec); |
| 1004 | if (ec) { | 1004 | if (ec) { |
| 1005 | - HCCL_VM_WARN("[HVM] failed to move {} to {}: {}", f.string(),(backupDir / f.filename()).string(), ec.message()); | 1005 | + HCCL_VM_WARN("failed to move {} to {}: {}", f.string(),(backupDir / f.filename()).string(), ec.message()); |
| 1006 | } | 1006 | } |
| 1007 | } | 1007 | } |
| 1008 | } | 1008 | } |
| 1009 | 1009 | ||
| 1010 | static void BackupAivTaskFiles(const fs::path &backupDir) | 1010 | static void BackupAivTaskFiles(const fs::path &backupDir) |
| 1011 | { | 1011 | { |
| 1012 | - fs::path dataDir = fs::path(g_binDir) / "data"; | 1012 | + fs::path dataDir = fs::path(InstallPath::ResolveToInstallRoot("data")); |
| 1013 | std::error_code ec; | 1013 | std::error_code ec; |
| 1014 | if (!fs::exists(dataDir, ec) || ec || !fs::is_directory(dataDir, ec)) { | 1014 | if (!fs::exists(dataDir, ec) || ec || !fs::is_directory(dataDir, ec)) { |
| 1015 | if (ec) { | 1015 | if (ec) { |
| 1016 | - HCCL_VM_WARN("[HVM] failed to access data directory {}: {}", dataDir.string(), ec.message()); | 1016 | + HCCL_VM_WARN("failed to access data directory {}: {}", dataDir.string(), ec.message()); |
| 1017 | } | 1017 | } |
| 1018 | return; | 1018 | return; |
| 1019 | } | 1019 | } |
| @@ -1022,12 +1022,12 @@ static void BackupAivTaskFiles(const fs::path &backupDir) | |||
| 1022 | static const std::string aivFilePrefix = "hcclvm_aiv_"; | 1022 | static const std::string aivFilePrefix = "hcclvm_aiv_"; |
| 1023 | fs::directory_iterator iter(dataDir, ec); | 1023 | fs::directory_iterator iter(dataDir, ec); |
| 1024 | if (ec) { | 1024 | if (ec) { |
| 1025 | - HCCL_VM_WARN("[HVM] failed to iterate data directory {}: {}", dataDir.string(), ec.message()); | 1025 | + HCCL_VM_WARN("failed to iterate data directory {}: {}", dataDir.string(), ec.message()); |
| 1026 | return; | 1026 | return; |
| 1027 | } | 1027 | } |
| 1028 | for (const fs::directory_iterator end; iter != end; iter.increment(ec)) { | 1028 | for (const fs::directory_iterator end; iter != end; iter.increment(ec)) { |
| 1029 | if (ec) { | 1029 | if (ec) { |
| 1030 | - HCCL_VM_WARN("[HVM] failed to iterate data directory {}: {}", dataDir.string(), ec.message()); | 1030 | + HCCL_VM_WARN("failed to iterate data directory {}: {}", dataDir.string(), ec.message()); |
| 1031 | return; | 1031 | return; |
| 1032 | } | 1032 | } |
| 1033 | 1033 | ||
| @@ -1055,7 +1055,7 @@ static void BackupAivTaskFiles(const fs::path &backupDir) | |||
| 1055 | ec.clear(); | 1055 | ec.clear(); |
| 1056 | fs::rename(f, backupDir / f.filename(), ec); | 1056 | fs::rename(f, backupDir / f.filename(), ec); |
| 1057 | if (ec) { | 1057 | if (ec) { |
| 1058 | - HCCL_VM_WARN("[HVM] failed to move {} to {}: {}", f.string(), (backupDir / f.filename()).string(), | 1058 | + HCCL_VM_WARN("failed to move {} to {}: {}", f.string(), (backupDir / f.filename()).string(), |
| 1059 | ec.message()); | 1059 | ec.message()); |
| 1060 | } | 1060 | } |
| 1061 | } | 1061 | } |
| @@ -1067,7 +1067,7 @@ static void CleanShmFilesByPrefix(const std::vector<std::string> &prefixes) | |||
| 1067 | std::error_code ec; | 1067 | std::error_code ec; |
| 1068 | if (!fs::exists(shmDir, ec) || ec || !fs::is_directory(shmDir, ec)) { | 1068 | if (!fs::exists(shmDir, ec) || ec || !fs::is_directory(shmDir, ec)) { |
| 1069 | if (ec) { | 1069 | if (ec) { |
| 1070 | - HCCL_VM_WARN("[HVM] failed to access shared memory directory {}: {}", shmDir.string(), ec.message()); | 1070 | + HCCL_VM_WARN("failed to access shared memory directory {}: {}", shmDir.string(), ec.message()); |
| 1071 | } | 1071 | } |
| 1072 | return; | 1072 | return; |
| 1073 | } | 1073 | } |
| @@ -1076,12 +1076,12 @@ static void CleanShmFilesByPrefix(const std::vector<std::string> &prefixes) | |||
| 1076 | uint32_t failedCount = 0; | 1076 | uint32_t failedCount = 0; |
| 1077 | fs::directory_iterator iter(shmDir, ec); | 1077 | fs::directory_iterator iter(shmDir, ec); |
| 1078 | if (ec) { | 1078 | if (ec) { |
| 1079 | - HCCL_VM_WARN("[HVM] failed to iterate shared memory directory {}: {}", shmDir.string(), ec.message()); | 1079 | + HCCL_VM_WARN("failed to iterate shared memory directory {}: {}", shmDir.string(), ec.message()); |
| 1080 | return; | 1080 | return; |
| 1081 | } | 1081 | } |
| 1082 | for (const fs::directory_iterator end; iter != end; iter.increment(ec)) { | 1082 | for (const fs::directory_iterator end; iter != end; iter.increment(ec)) { |
| 1083 | if (ec) { | 1083 | if (ec) { |
| 1084 | - HCCL_VM_WARN("[HVM] failed to iterate shared memory directory {}: {}", shmDir.string(), ec.message()); | 1084 | + HCCL_VM_WARN("failed to iterate shared memory directory {}: {}", shmDir.string(), ec.message()); |
| 1085 | return; | 1085 | return; |
| 1086 | } | 1086 | } |
| 1087 | 1087 | ||
| @@ -1103,21 +1103,21 @@ static void CleanShmFilesByPrefix(const std::vector<std::string> &prefixes) | |||
| 1103 | std::error_code removeEc; | 1103 | std::error_code removeEc; |
| 1104 | if (!fs::remove(filePath, removeEc) || removeEc) { | 1104 | if (!fs::remove(filePath, removeEc) || removeEc) { |
| 1105 | ++failedCount; | 1105 | ++failedCount; |
| 1106 | - HCCL_VM_WARN("[HVM] failed to remove shared memory file {}: {}", filePath.string(), | 1106 | + HCCL_VM_WARN("failed to remove shared memory file {}: {}", filePath.string(), |
| 1107 | removeEc.message()); | 1107 | removeEc.message()); |
| 1108 | continue; | 1108 | continue; |
| 1109 | } | 1109 | } |
| 1110 | ++removedCount; | 1110 | ++removedCount; |
| 1111 | } | 1111 | } |
| 1112 | - HCCL_VM_INFO("[HVM] cleaned {} shared memory files in {}, failed {}", removedCount, shmDir.string(), failedCount); | 1112 | + HCCL_VM_INFO("cleaned {} shared memory files in {}, failed {}", removedCount, shmDir.string(), failedCount); |
| 1113 | } | 1113 | } |
| 1114 | 1114 | ||
| 1115 | HcclVmResult ClearDbTables() | 1115 | HcclVmResult ClearDbTables() |
| 1116 | { | 1116 | { |
| 1117 | - const fs::path backupDir = fs::path(g_binDir) / "data" / MakeDataBackupTimestamp(); | 1117 | + const fs::path backupDir = fs::path(InstallPath::ResolveToInstallRoot("data/" + MakeDataBackupTimestamp())); |
| 1118 | BackupBinFiles(backupDir); | 1118 | BackupBinFiles(backupDir); |
| 1119 | BackupAivTaskFiles(backupDir); | 1119 | BackupAivTaskFiles(backupDir); |
| 1120 | - BackupDatabase(g_binDir + "/data/hccl_vm_data.db"); | 1120 | + BackupDatabase(InstallPath::ResolveToInstallRoot("data/hccl_vm_data.db")); |
| 1121 | 1121 | ||
| 1122 | // 1. 清空 OpDbOps (SQLite) 中的静态表数据 | 1122 | // 1. 清空 OpDbOps (SQLite) 中的静态表数据 |
| 1123 | std::vector<std::string> staticTables = { | 1123 | std::vector<std::string> staticTables = { |
| @@ -1191,8 +1191,8 @@ HcclVmResult HcclVmResetCommDomain() | |||
| 1191 | HcclVmResult CopyFile(const std::string& clusterDir) { | 1191 | HcclVmResult CopyFile(const std::string& clusterDir) { |
| 1192 | // 1. 拼接源文件和目标文件路径 | 1192 | // 1. 拼接源文件和目标文件路径 |
| 1193 | auto srcPath = clusterDir + "/superpod0/server0/topo.json"; | 1193 | auto srcPath = clusterDir + "/superpod0/server0/topo.json"; |
| 1194 | - fs::create_directories(fs::path(GetBinLocation() + "/data")); | 1194 | + fs::create_directories(fs::path(InstallPath::ResolveToInstallRoot("data"))); |
| 1195 | - auto destPath = GetBinLocation() + "/data/topo.json"; | 1195 | + auto destPath = InstallPath::ResolveToInstallRoot("data/topo.json"); |
| 1196 | 1196 | ||
| 1197 | // 显式提示:文件已存在,将覆盖 | 1197 | // 显式提示:文件已存在,将覆盖 |
| 1198 | if (fs::exists(destPath)) { | 1198 | if (fs::exists(destPath)) { |
| @@ -14,11 +14,10 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 17 | 18 | ||
| 18 | 19 | ||
| 19 | 20 | ||
| 20 | -std::string GetBinLocation(); | ||
| 21 | - | ||
| 22 | namespace { | 21 | namespace { |
| 23 | constexpr const char* HVM_MODEL_ENV_KEY = "HCCL_VM_MODEL"; | 22 | constexpr const char* HVM_MODEL_ENV_KEY = "HCCL_VM_MODEL"; |
| 24 | constexpr const char* HVM_POD_NUM_ENV_KEY = "HCCL_VM_POD_NUM"; | 23 | constexpr const char* HVM_POD_NUM_ENV_KEY = "HCCL_VM_POD_NUM"; |
| @@ -31,18 +30,18 @@ bool SetEnvValue(const char* key, const std::string& value) | |||
| 31 | if (setenv(key, value.c_str(), 1) == 0) { | 30 | if (setenv(key, value.c_str(), 1) == 0) { |
| 32 | return true; | 31 | return true; |
| 33 | } | 32 | } |
| 34 | - HCCL_VM_ERROR("[HVM] setenv failed: {}={}, errno={}, err={}", key, value, errno, std::strerror(errno)); | 33 | + HCCL_VM_ERROR("setenv failed: {}={}, errno={}, err={}", key, value, errno, std::strerror(errno)); |
| 35 | return false; | 34 | return false; |
| 36 | } | 35 | } |
| 37 | 36 | ||
| 38 | bool ParseYamlTopoImpl(const std::string& fileName, TopoMeta& topo) | 37 | bool ParseYamlTopoImpl(const std::string& fileName, TopoMeta& topo) |
| 39 | { | 38 | { |
| 40 | try { | 39 | try { |
| 41 | - std::string filePath = GetBinLocation() + "/config/topo_meta/" + fileName + ".yaml"; | 40 | + std::string filePath = InstallPath::ResolveToInstallRoot("config/topo_meta/" + fileName + ".yaml"); |
| 42 | YAML::Node root = YAML::LoadFile(filePath); | 41 | YAML::Node root = YAML::LoadFile(filePath); |
| 43 | 42 | ||
| 44 | if (!root["meta"]) { | 43 | if (!root["meta"]) { |
| 45 | - HCCL_VM_ERROR("[HVM] YAML : 'meta' node not found."); | 44 | + HCCL_VM_ERROR("YAML : 'meta' node not found."); |
| 46 | return false; | 45 | return false; |
| 47 | } | 46 | } |
| 48 | uint32_t podNum = root["meta"]["podNum"].as<uint32_t>(); | 47 | uint32_t podNum = root["meta"]["podNum"].as<uint32_t>(); |
| @@ -51,7 +50,7 @@ bool ParseYamlTopoImpl(const std::string& fileName, TopoMeta& topo) | |||
| 51 | HCCL_VM_DEBUG("PodNum: {}, SerNum: {}, RankNum: {}", podNum, serNum, rankNum); | 50 | HCCL_VM_DEBUG("PodNum: {}, SerNum: {}, RankNum: {}", podNum, serNum, rankNum); |
| 52 | topo.reserve(podNum); | 51 | topo.reserve(podNum); |
| 53 | if (podNum <= 0 || podNum >1024 || serNum <= 0 || serNum >1024 || rankNum <= 0 || rankNum >1024) { | 52 | if (podNum <= 0 || podNum >1024 || serNum <= 0 || serNum >1024 || rankNum <= 0 || rankNum >1024) { |
| 54 | - HCCL_VM_ERROR("[HVM] YAML : 'meta' number not surport, please check your config.yaml."); | 53 | + HCCL_VM_ERROR("YAML : 'meta' number not surport, please check your config.yaml."); |
| 55 | return false; | 54 | return false; |
| 56 | } | 55 | } |
| 57 | if (root["topology"] && root["topology"].IsSequence()) { | 56 | if (root["topology"] && root["topology"].IsSequence()) { |
| @@ -73,7 +72,7 @@ bool ParseYamlTopoImpl(const std::string& fileName, TopoMeta& topo) | |||
| 73 | } | 72 | } |
| 74 | return true; | 73 | return true; |
| 75 | } catch (const YAML::Exception& e) { | 74 | } catch (const YAML::Exception& e) { |
| 76 | - HCCL_VM_ERROR("[HVM] Exception when parsing YAML: {}", e.what()); | 75 | + HCCL_VM_ERROR("Exception when parsing YAML: {}", e.what()); |
| 77 | return false; | 76 | return false; |
| 78 | } | 77 | } |
| 79 | } | 78 | } |
| @@ -28,19 +28,19 @@ void MockCommCommand::Setup(CLI::App& app) { | |||
| 28 | } | 28 | } |
| 29 | 29 | ||
| 30 | void MockCommCommand::Execute() { | 30 | void MockCommCommand::Execute() { |
| 31 | - HCCL_VM_INFO("[HVM] Initializing mock communication domain: Config={}, Level={}", configFileName, g_hcclVmLevel); | 31 | + HCCL_VM_INFO("Initializing mock communication domain: Config={}, Level={}", configFileName, g_hcclVmLevel); |
| 32 | 32 | ||
| 33 | // 首先清理动态数据库表项 | 33 | // 首先清理动态数据库表项 |
| 34 | auto ret = ClearDbTables(); | 34 | auto ret = ClearDbTables(); |
| 35 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 35 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 36 | - HCCL_VM_ERROR("[HVM] Failed to clear database tables."); | 36 | + HCCL_VM_ERROR("Failed to clear database tables."); |
| 37 | return; | 37 | return; |
| 38 | } | 38 | } |
| 39 | 39 | ||
| 40 | // 重置通信域 | 40 | // 重置通信域 |
| 41 | ret = HcclVmResetCommDomain(); | 41 | ret = HcclVmResetCommDomain(); |
| 42 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 42 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 43 | - HCCL_VM_ERROR("[HVM] Failed to reset mock communication domain."); | 43 | + HCCL_VM_ERROR("Failed to reset mock communication domain."); |
| 44 | return; | 44 | return; |
| 45 | } | 45 | } |
| 46 | 46 | ||
| @@ -52,7 +52,7 @@ void MockCommCommand::Execute() { | |||
| 52 | 52 | ||
| 53 | ret = InitHvmCommEnv(topoMeta, configFileName, g_hcclVmLevel); | 53 | ret = InitHvmCommEnv(topoMeta, configFileName, g_hcclVmLevel); |
| 54 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 54 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 55 | - HCCL_VM_ERROR("[HVM] Failed to initialize mock communication environment. Cleaning up environment."); | 55 | + HCCL_VM_ERROR("Failed to initialize mock communication environment. Cleaning up environment."); |
| 56 | return; | 56 | return; |
| 57 | } | 57 | } |
| 58 | return; | 58 | return; |
| @@ -23,13 +23,13 @@ void ResetCommand::Setup(CLI::App& app) { | |||
| 23 | } | 23 | } |
| 24 | 24 | ||
| 25 | void ResetCommand::Execute() { | 25 | void ResetCommand::Execute() { |
| 26 | - HCCL_VM_INFO("[HVM] Resetting: Clearing database tables..."); | 26 | + HCCL_VM_INFO("Resetting: Clearing database tables..."); |
| 27 | auto ret = ClearDbTables(); | 27 | auto ret = ClearDbTables(); |
| 28 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 28 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 29 | - HCCL_VM_ERROR("[HVM] Failed to clear database tables."); | 29 | + HCCL_VM_ERROR("Failed to clear database tables."); |
| 30 | return; | 30 | return; |
| 31 | } | 31 | } |
| 32 | - HCCL_VM_INFO("[HVM] Database tables cleared successfully."); | 32 | + HCCL_VM_INFO("Database tables cleared successfully."); |
| 33 | } | 33 | } |
| 34 | 34 | ||
| 35 | static inline CommandAutoRegister<ResetCommand> g_reset_cmd_reg{}; | 35 | static inline CommandAutoRegister<ResetCommand> g_reset_cmd_reg{}; |
| @@ -12,6 +12,7 @@ | |||
| 12 | 12 | ||
| 13 | 13 | ||
| 14 | 14 | ||
| 15 | + | ||
| 15 | 16 | ||
| 16 | 17 | ||
| 17 | 18 | ||
| @@ -32,40 +33,40 @@ void RunCommand::Setup(CLI::App& app) { | |||
| 32 | 33 | ||
| 33 | void RunCommand::Execute(CLI::App& app) { | 34 | void RunCommand::Execute(CLI::App& app) { |
| 34 | if (g_hcclVmBashFlag) { | 35 | if (g_hcclVmBashFlag) { |
| 35 | - HCCL_VM_WARN("[HVM] hccl-vm is already running. Please do not run examples in one_shot mode within a sub-bash. Exit the sub-bash and try again."); | 36 | + HCCL_VM_WARN("hccl-vm is already running. Please do not run examples in one_shot mode within a sub-bash. Exit the sub-bash and try again."); |
| 36 | return; | 37 | return; |
| 37 | } | 38 | } |
| 38 | CLI::App* tmp_cmd = app.get_subcommand("run"); | 39 | CLI::App* tmp_cmd = app.get_subcommand("run"); |
| 39 | std::vector<std::string> leftargvs = tmp_cmd->remaining(); | 40 | std::vector<std::string> leftargvs = tmp_cmd->remaining(); |
| 40 | if (leftargvs.empty()) { | 41 | if (leftargvs.empty()) { |
| 41 | - HCCL_VM_ERROR("[HVM] In one_shot mode, an example execution command must be provided."); | 42 | + HCCL_VM_ERROR("In one_shot mode, an example execution command must be provided."); |
| 42 | return; | 43 | return; |
| 43 | } | 44 | } |
| 44 | - HCCL_VM_INFO("[HVM] Initializing: Model={}, Level={}", configClusterName, g_hcclVmLevel); | 45 | + HCCL_VM_INFO("Initializing: Model={}, Level={}", configClusterName, g_hcclVmLevel); |
| 45 | - auto clusterDir = GetBinLocation() + "/config/network/cluster/" + configClusterName; | 46 | + auto clusterDir = InstallPath::ResolveToInstallRoot("config/network/cluster/" + configClusterName); |
| 46 | 47 | ||
| 47 | sim::RunModeConfig runMode{}; | 48 | sim::RunModeConfig runMode{}; |
| 48 | runMode.mode = checkOnlyMode ? 1 : 0; | 49 | runMode.mode = checkOnlyMode ? 1 : 0; |
| 49 | RunnerDB::DeleteAll<sim::RunModeConfig>(); | 50 | RunnerDB::DeleteAll<sim::RunModeConfig>(); |
| 50 | RunnerDB::Add<sim::RunModeConfig>(runMode); | 51 | RunnerDB::Add<sim::RunModeConfig>(runMode); |
| 51 | - HCCL_VM_INFO("[HVM] run mode: {}", checkOnlyMode ? "check-only" : "normal"); | 52 | + HCCL_VM_INFO("run mode: {}", checkOnlyMode ? "check-only" : "normal"); |
| 52 | 53 | ||
| 53 | auto ret = InitHvmEnv(clusterDir, g_hcclVmLevel, checkOnlyMode); | 54 | auto ret = InitHvmEnv(clusterDir, g_hcclVmLevel, checkOnlyMode); |
| 54 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 55 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 55 | - HCCL_VM_ERROR("[HVM] Failed to initialize simulation environment. Cleaning up environment."); | 56 | + HCCL_VM_ERROR("Failed to initialize simulation environment. Cleaning up environment."); |
| 56 | auto cleanRet = HcclVmExit(); | 57 | auto cleanRet = HcclVmExit(); |
| 57 | if (cleanRet != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 58 | if (cleanRet != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 58 | - HCCL_VM_ERROR("[HVM] Failed to clean up environment. Please check for residual environment artifacts."); | 59 | + HCCL_VM_ERROR("Failed to clean up environment. Please check for residual environment artifacts."); |
| 59 | } | 60 | } |
| 60 | return; | 61 | return; |
| 61 | } | 62 | } |
| 62 | CstyleCmd syscmd(leftargvs); | 63 | CstyleCmd syscmd(leftargvs); |
| 63 | - HCCL_VM_INFO("[HVM] one_shot mode, executing: {}", syscmd.cmd()); | 64 | + HCCL_VM_INFO("one_shot mode, executing: {}", syscmd.cmd()); |
| 64 | - std::string proxyPath = GetBinLocation() + "/lib/" + GetArchStr() + "/libhccl_proxy_level2.so"; | 65 | + std::string proxyPath = InstallPath::ResolveToInstallRoot("lib/" + GetArchStr() + "/libhccl_proxy_level2.so"); |
| 65 | setenv("LD_PRELOAD", proxyPath.c_str(), 1); | 66 | setenv("LD_PRELOAD", proxyPath.c_str(), 1); |
| 66 | int sysRet = std::system(syscmd.cmd().c_str()); // system() 会阻塞当前进程直到子命令结束 | 67 | int sysRet = std::system(syscmd.cmd().c_str()); // system() 会阻塞当前进程直到子命令结束 |
| 67 | if (sysRet != 0) { | 68 | if (sysRet != 0) { |
| 68 | - HCCL_VM_ERROR("[HVM] Example execution failed: {}", sysRet); | 69 | + HCCL_VM_ERROR("Example execution failed: {}", sysRet); |
| 69 | } | 70 | } |
| 70 | RemoveFromLDPreload(proxyPath); | 71 | RemoveFromLDPreload(proxyPath); |
| 71 | auto cleanRet = HcclVmExit(); | 72 | auto cleanRet = HcclVmExit(); |
| @@ -13,6 +13,7 @@ | |||
| 13 | 13 | ||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | + | ||
| 16 | 17 | ||
| 17 | 18 | ||
| 18 | 19 | ||
| @@ -32,15 +33,15 @@ void StartCommand::Setup(CLI::App& app) { | |||
| 32 | 33 | ||
| 33 | void StartCommand::Execute() { | 34 | void StartCommand::Execute() { |
| 34 | if (g_hcclVmBashFlag) { | 35 | if (g_hcclVmBashFlag) { |
| 35 | - HCCL_VM_WARN("[HVM] hccl-vm has already started. Please do not start it again in a sub-bash."); | 36 | + HCCL_VM_WARN("hccl-vm has already started. Please do not start it again in a sub-bash."); |
| 36 | return; | 37 | return; |
| 37 | } | 38 | } |
| 38 | if (configClusterName.find(".yaml") != std::string::npos) { | 39 | if (configClusterName.find(".yaml") != std::string::npos) { |
| 39 | configClusterName.erase(configClusterName.find(".yaml"), 5); | 40 | configClusterName.erase(configClusterName.find(".yaml"), 5); |
| 40 | } | 41 | } |
| 41 | 42 | ||
| 42 | - HCCL_VM_INFO("[HVM] Initializing: Model={}, Level={}", configClusterName, g_hcclVmLevel); | 43 | + HCCL_VM_INFO("Initializing: Model={}, Level={}", configClusterName, g_hcclVmLevel); |
| 43 | - auto clusterDir = GetBinLocation() + "/config/network/cluster/" + configClusterName; | 44 | + auto clusterDir = InstallPath::ResolveToInstallRoot("config/network/cluster/" + configClusterName); |
| 44 | 45 | ||
| 45 | // 拷贝topo.json文件到执行目录 | 46 | // 拷贝topo.json文件到执行目录 |
| 46 | auto ret = CopyFile(clusterDir); | 47 | auto ret = CopyFile(clusterDir); |
| @@ -53,24 +54,24 @@ void StartCommand::Execute() { | |||
| 53 | runMode.mode = checkOnlyMode ? 1 : 0; | 54 | runMode.mode = checkOnlyMode ? 1 : 0; |
| 54 | RunnerDB::DeleteAll<sim::RunModeConfig>(); | 55 | RunnerDB::DeleteAll<sim::RunModeConfig>(); |
| 55 | RunnerDB::Add<sim::RunModeConfig>(runMode); | 56 | RunnerDB::Add<sim::RunModeConfig>(runMode); |
| 56 | - HCCL_VM_INFO("[HVM] run mode: {}", checkOnlyMode ? "check-only" : "normal"); | 57 | + HCCL_VM_INFO("run mode: {}", checkOnlyMode ? "check-only" : "normal"); |
| 57 | 58 | ||
| 58 | ret = InitHvmEnv(clusterDir, g_hcclVmLevel, checkOnlyMode); | 59 | ret = InitHvmEnv(clusterDir, g_hcclVmLevel, checkOnlyMode); |
| 59 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 60 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 60 | - HCCL_VM_ERROR("[HVM] Failed to initialize simulation environment. Cleaning up environment."); | 61 | + HCCL_VM_ERROR("Failed to initialize simulation environment. Cleaning up environment."); |
| 61 | auto cleanRet = HcclVmExit(); | 62 | auto cleanRet = HcclVmExit(); |
| 62 | if (cleanRet != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 63 | if (cleanRet != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 63 | - HCCL_VM_ERROR("[HVM] Failed to clean up environment. Please check for residual environment artifacts."); | 64 | + HCCL_VM_ERROR("Failed to clean up environment. Please check for residual environment artifacts."); |
| 64 | } | 65 | } |
| 65 | return; | 66 | return; |
| 66 | } | 67 | } |
| 67 | 68 | ||
| 68 | ret = StartHvmCmd(); | 69 | ret = StartHvmCmd(); |
| 69 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 70 | if (ret != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 70 | - HCCL_VM_ERROR("[HVM] Failed to start hvm command."); | 71 | + HCCL_VM_ERROR("Failed to start hvm command."); |
| 71 | auto cleanRet = HcclVmExit(); | 72 | auto cleanRet = HcclVmExit(); |
| 72 | if (cleanRet != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { | 73 | if (cleanRet != HcclVmResult::HCCL_SIM_HOST_SUCCESS_CMD) { |
| 73 | - HCCL_VM_ERROR("[HVM] Failed to clean up environment. Please check for residual environment artifacts."); | 74 | + HCCL_VM_ERROR("Failed to clean up environment. Please check for residual environment artifacts."); |
| 74 | } | 75 | } |
| 75 | return; | 76 | return; |
| 76 | } | 77 | } |
| @@ -138,3 +138,6 @@ install(TARGETS device | |||
| 138 | ) | 138 | ) |
| 139 | 139 | ||
| 140 | 140 | ||
| 141 | + | ||
| 142 | +# 日志染色: 模块 tag | ||
| 143 | +target_compile_definitions(device PRIVATE HCCL_VM_MODULE="DEVICE") | ||
| @@ -74,7 +74,7 @@ void UpdateSqTail(uint32_t sqId, uint32_t newTail) | |||
| 74 | sqTailMap[sqId] = newTail; | 74 | sqTailMap[sqId] = newTail; |
| 75 | } | 75 | } |
| 76 | 76 | ||
| 77 | -void *GetRealPtrByDevPtr(void *devPtr) | 77 | +void *GetRealPtrByDevPtrImpl(void *devPtr, const char *file, int line) |
| 78 | { | 78 | { |
| 79 | uint64_t devAddr = reinterpret_cast<uint64_t>(devPtr); | 79 | uint64_t devAddr = reinterpret_cast<uint64_t>(devPtr); |
| 80 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( | 80 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( |
| @@ -83,20 +83,20 @@ void *GetRealPtrByDevPtr(void *devPtr) | |||
| 83 | (devAddr < (virMem.start_ptr + virMem.size)) && | 83 | (devAddr < (virMem.start_ptr + virMem.size)) && |
| 84 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); | 84 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); |
| 85 | if (!virMemRes.second) { | 85 | if (!virMemRes.second) { |
| 86 | - HCCL_VM_ERROR("[GetRealPtrByDevPtr] cannot find virMemRes by devAddr[{}]", devAddr); | 86 | + HCCL_VM_ERROR("cannot find virMemRes by devAddr[{}], called from {}:{}]", devAddr, file, line); |
| 87 | return nullptr; | 87 | return nullptr; |
| 88 | } | 88 | } |
| 89 | 89 | ||
| 90 | auto phyMemId = virMemRes.first.phy_mem_id; | 90 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 91 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 91 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 92 | if (!phyMemRes.has_value()) { | 92 | if (!phyMemRes.has_value()) { |
| 93 | - HCCL_VM_ERROR("[GetRealPtrByDevPtr] cannot find phyMemRes by phyMemId[{}]", phyMemId); | 93 | + HCCL_VM_ERROR("cannot find phyMemRes by phyMemId[{}], called from {}:{}]", phyMemId, file, line); |
| 94 | return nullptr; | 94 | return nullptr; |
| 95 | }; | 95 | }; |
| 96 | 96 | ||
| 97 | std::string memName(phyMemRes->name); | 97 | std::string memName(phyMemRes->name); |
| 98 | void *realPtr = sim::MemoryManager::GetInstance().AcquireMemByName(memName.c_str()); | 98 | void *realPtr = sim::MemoryManager::GetInstance().AcquireMemByName(memName.c_str()); |
| 99 | - HCCL_VM_INFO("[GetRealPtrByDevPtr] devAddr[{}] realPtr[{}] memName[{}]", devAddr, reinterpret_cast<uint64_t>(realPtr), memName); | 99 | + HCCL_VM_INFO("devAddr[{}] realPtr[{}] memName[{}]", devAddr, reinterpret_cast<uint64_t>(realPtr), memName); |
| 100 | 100 | ||
| 101 | return realPtr; | 101 | return realPtr; |
| 102 | } | 102 | } |
| @@ -109,21 +109,21 @@ uint32_t GetRankIdByDevAddr(uint64_t devAddr) | |||
| 109 | (devAddr < (virMem.start_ptr + virMem.size)) && | 109 | (devAddr < (virMem.start_ptr + virMem.size)) && |
| 110 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); | 110 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); |
| 111 | if (!virMemRes.second) { | 111 | if (!virMemRes.second) { |
| 112 | - HCCL_VM_ERROR("[GetRankIdByDevAddr] cannot find virMemRes by devAddr[{}]", devAddr); | 112 | + HCCL_VM_ERROR("cannot find virMemRes by devAddr[{}]", devAddr); |
| 113 | return 0; | 113 | return 0; |
| 114 | } | 114 | } |
| 115 | 115 | ||
| 116 | auto phyMemId = virMemRes.first.phy_mem_id; | 116 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 117 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 117 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 118 | if (!phyMemRes.has_value()) { | 118 | if (!phyMemRes.has_value()) { |
| 119 | - HCCL_VM_ERROR("[GetRankIdByDevAddr] cannot find phyMemRes by phyMemId[{}]", phyMemId); | 119 | + HCCL_VM_ERROR("cannot find phyMemRes by phyMemId[{}]", phyMemId); |
| 120 | return 0; | 120 | return 0; |
| 121 | }; | 121 | }; |
| 122 | 122 | ||
| 123 | uint32_t device_id = phyMemRes->device_id; | 123 | uint32_t device_id = phyMemRes->device_id; |
| 124 | auto rank = RunnerDB::GetOneByPred<sim::Rank>([device_id](const sim::Rank &rank) { return rank.device_id == device_id; }); | 124 | auto rank = RunnerDB::GetOneByPred<sim::Rank>([device_id](const sim::Rank &rank) { return rank.device_id == device_id; }); |
| 125 | if (!rank.second) { | 125 | if (!rank.second) { |
| 126 | - HCCL_VM_ERROR("[GetRankIdByDevAddr] cannot find rank by device_id[{}]", device_id); | 126 | + HCCL_VM_ERROR("cannot find rank by device_id[{}]", device_id); |
| 127 | return 0; | 127 | return 0; |
| 128 | } | 128 | } |
| 129 | 129 | ||
| @@ -134,7 +134,7 @@ HcclAicpuData *GetHcclAicpuDataShmPtr() | |||
| 134 | { | 134 | { |
| 135 | void *shmptr = sim::MemoryManager::GetInstance().AcquireMemByName("HcclAicpuData"); | 135 | void *shmptr = sim::MemoryManager::GetInstance().AcquireMemByName("HcclAicpuData"); |
| 136 | if (shmptr == nullptr) { | 136 | if (shmptr == nullptr) { |
| 137 | - HCCL_VM_ERROR("[GetHcclAicpuDataShmPtr] acquire HcclAicpuData shm failed."); | 137 | + HCCL_VM_ERROR("acquire HcclAicpuData shm failed."); |
| 138 | return nullptr; | 138 | return nullptr; |
| 139 | } | 139 | } |
| 140 | 140 | ||
| @@ -145,14 +145,14 @@ uint32_t GetRankIdByIpAddr(std::string ipAddr) | |||
| 145 | { | 145 | { |
| 146 | auto ret = RunnerDB::GetOneByPred<sim::EndPoint>([ipAddr](const sim::EndPoint &p) { return strcmp(p.ip_addr, ipAddr.c_str()) == 0; }); | 146 | auto ret = RunnerDB::GetOneByPred<sim::EndPoint>([ipAddr](const sim::EndPoint &p) { return strcmp(p.ip_addr, ipAddr.c_str()) == 0; }); |
| 147 | if (!ret.second) { | 147 | if (!ret.second) { |
| 148 | - HCCL_VM_ERROR("[GetRankIdByDevAddr] cannot find Port by ipAddr[{}]", ipAddr); | 148 | + HCCL_VM_ERROR("cannot find Port by ipAddr[{}]", ipAddr); |
| 149 | return 0; | 149 | return 0; |
| 150 | } | 150 | } |
| 151 | 151 | ||
| 152 | uint32_t device_id = ret.first.device_id; | 152 | uint32_t device_id = ret.first.device_id; |
| 153 | auto rank = RunnerDB::GetOneByPred<sim::Rank>([device_id](const sim::Rank &rank) { return rank.device_id == device_id; }); | 153 | auto rank = RunnerDB::GetOneByPred<sim::Rank>([device_id](const sim::Rank &rank) { return rank.device_id == device_id; }); |
| 154 | if (!rank.second) { | 154 | if (!rank.second) { |
| 155 | - HCCL_VM_ERROR("[GetRankIdByDevAddr] cannot find Rank by device_id[{}]", device_id); | 155 | + HCCL_VM_ERROR("cannot find Rank by device_id[{}]", device_id); |
| 156 | return 0; | 156 | return 0; |
| 157 | } | 157 | } |
| 158 | 158 | ||
| @@ -162,7 +162,7 @@ uint32_t GetRankIdByIpAddr(std::string ipAddr) | |||
| 162 | void UpdataKfcStatus(uint64_t d2hAddr) | 162 | void UpdataKfcStatus(uint64_t d2hAddr) |
| 163 | { | 163 | { |
| 164 | if (d2hAddr == 0) { | 164 | if (d2hAddr == 0) { |
| 165 | - HCCL_VM_ERROR("[UpdataKfcStatus] d2hAddr is 0, cannot update kfc status."); | 165 | + HCCL_VM_ERROR("d2hAddr is 0, cannot update kfc status."); |
| 166 | return; | 166 | return; |
| 167 | } | 167 | } |
| 168 | 168 | ||
| @@ -173,7 +173,7 @@ void UpdataKfcStatus(uint64_t d2hAddr) | |||
| 173 | *reinterpret_cast<uint8_t *>(d2hAddr) = devDoneStatus; // 更新设备状态 | 173 | *reinterpret_cast<uint8_t *>(d2hAddr) = devDoneStatus; // 更新设备状态 |
| 174 | *reinterpret_cast<uint32_t *>(d2hAddr + headCntOffset) = headTailCnt; // 更新headCnt | 174 | *reinterpret_cast<uint32_t *>(d2hAddr + headCntOffset) = headTailCnt; // 更新headCnt |
| 175 | *reinterpret_cast<uint32_t *>(d2hAddr + tailCntOffset) = headTailCnt; // 更新tailCnt | 175 | *reinterpret_cast<uint32_t *>(d2hAddr + tailCntOffset) = headTailCnt; // 更新tailCnt |
| 176 | - HCCL_VM_INFO("[UpdataKfcStatus] update kfc status success."); | 176 | + HCCL_VM_INFO("update kfc status success."); |
| 177 | } | 177 | } |
| 178 | 178 | ||
| 179 | // 信号处理函数 | 179 | // 信号处理函数 |
| @@ -186,7 +186,8 @@ void SignalHandler(int signum) | |||
| 186 | int count = backtrace(stack_pointers, MAX_STACK_FRAMES); | 186 | int count = backtrace(stack_pointers, MAX_STACK_FRAMES); |
| 187 | 187 | ||
| 188 | // 2. 打印基本错误信息 | 188 | // 2. 打印基本错误信息 |
| 189 | - HCCL_VM_INFO("\n========================================"); | 189 | + HCCL_VM_ERROR("device process crash captured."); |
| 190 | + HCCL_VM_INFO("========================================"); | ||
| 190 | HCCL_VM_INFO(" ERROR CRASH DETECTED! Signal: {}", signum); | 191 | HCCL_VM_INFO(" ERROR CRASH DETECTED! Signal: {}", signum); |
| 191 | HCCL_VM_INFO("========================================"); | 192 | HCCL_VM_INFO("========================================"); |
| 192 | 193 | ||
| @@ -199,13 +200,13 @@ void SignalHandler(int signum) | |||
| 199 | if (info.dli_sname != nullptr) { | 200 | if (info.dli_sname != nullptr) { |
| 200 | ptrdiff_t symOff = reinterpret_cast<char*>(stack_pointers[i]) | 201 | ptrdiff_t symOff = reinterpret_cast<char*>(stack_pointers[i]) |
| 201 | - reinterpret_cast<char*>(info.dli_saddr); | 202 | - reinterpret_cast<char*>(info.dli_saddr); |
| 202 | - printf(" [%02d] %s(%s+%#tx) [addr: %#tx]\n", i, | 203 | + HCCL_VM_INFO(" [{:02d}] {}({}+{:#x}) [addr: {:#x}]", i, |
| 203 | info.dli_fname, info.dli_sname, symOff, elfOffset); | 204 | info.dli_fname, info.dli_sname, symOff, elfOffset); |
| 204 | } else { | 205 | } else { |
| 205 | - printf(" [%02d] %s(?""?) [addr: %#tx]\n", i, info.dli_fname, elfOffset); | 206 | + HCCL_VM_INFO(" [{:02d}] {}(?""?) [addr: {:#x}]", i, info.dli_fname, elfOffset); |
| 206 | } | 207 | } |
| 207 | } else { | 208 | } else { |
| 208 | - printf(" [%02d] %p (dladdr failed)\n", i, stack_pointers[i]); | 209 | + HCCL_VM_INFO(" [{:02d}] {} (dladdr failed)", i, stack_pointers[i]); |
| 209 | } | 210 | } |
| 210 | } | 211 | } |
| 211 | 212 | ||
| @@ -31,7 +31,9 @@ HcclSim::HcclVmResult GetPiValByJettyId(uint32_t jettyId, uint32_t *piValue); | |||
| 31 | 31 | ||
| 32 | HcclSim::HcclVmResult UpdatePiValByJettyId(uint32_t jettyId, uint32_t piValue); | 32 | HcclSim::HcclVmResult UpdatePiValByJettyId(uint32_t jettyId, uint32_t piValue); |
| 33 | 33 | ||
| 34 | -void *GetRealPtrByDevPtr(void *devPtr); | 34 | +void *GetRealPtrByDevPtrImpl(void *devPtr, const char *file, int line); |
| 35 | + | ||
| 36 | + | ||
| 35 | 37 | ||
| 36 | uint32_t GetRankIdByDevAddr(uint64_t devAddr); | 38 | uint32_t GetRankIdByDevAddr(uint64_t devAddr); |
| 37 | 39 | ||
| @@ -17,6 +17,7 @@ | |||
| 17 | 17 | ||
| 18 | 18 | ||
| 19 | 19 | ||
| 20 | + | ||
| 20 | 21 | ||
| 21 | 22 | ||
| 22 | 23 | ||
| @@ -38,9 +39,9 @@ uint64_t d2hAddr = 0; | |||
| 38 | 39 | ||
| 39 | void ExecuteAicpuKernel(uint32_t rankId, const std::string &kernelName, uint64_t args) | 40 | void ExecuteAicpuKernel(uint32_t rankId, const std::string &kernelName, uint64_t args) |
| 40 | { | 41 | { |
| 41 | - HCCL_VM_INFO("[device] rankId[{}] kernel[{}] start run...", rankId, kernelName); | 42 | + HCCL_VM_INFO("rankId[{}] kernel[{}] start run...", rankId, kernelName); |
| 42 | if (!InitKernelFuncHandle()) { | 43 | if (!InitKernelFuncHandle()) { |
| 43 | - HCCL_VM_ERROR("[device] rankId[{}] init func handle failed.", rankId); | 44 | + HCCL_VM_ERROR("rankId[{}] init func handle failed.", rankId); |
| 44 | return; | 45 | return; |
| 45 | } | 46 | } |
| 46 | 47 | ||
| @@ -72,7 +73,7 @@ void ExecuteAicpuKernel(uint32_t rankId, const std::string &kernelName, uint64_t | |||
| 72 | uint32_t listNum = res->listNum; | 73 | uint32_t listNum = res->listNum; |
| 73 | uint32_t totalListSize = res->uniqueIdSize; | 74 | uint32_t totalListSize = res->uniqueIdSize; |
| 74 | uint32_t listSize = totalListSize / listNum; | 75 | uint32_t listSize = totalListSize / listNum; |
| 75 | - HCCL_VM_INFO("[device] start packing, listNum:{}, totalListSize:{}, listSize:{}", listNum, totalListSize, listSize); | 76 | + HCCL_VM_INFO("start packing, listNum:{}, totalListSize:{}, listSize:{}", listNum, totalListSize, listSize); |
| 76 | res->channelList = GetRealPtrByDevPtr(res->channelList); | 77 | res->channelList = GetRealPtrByDevPtr(res->channelList); |
| 77 | res->channelSizeAddr = GetRealPtrByDevPtr(res->channelSizeAddr); | 78 | res->channelSizeAddr = GetRealPtrByDevPtr(res->channelSizeAddr); |
| 78 | res->uniqueIdAddr = GetRealPtrByDevPtr(res->uniqueIdAddr); | 79 | res->uniqueIdAddr = GetRealPtrByDevPtr(res->uniqueIdAddr); |
| @@ -82,7 +83,7 @@ void ExecuteAicpuKernel(uint32_t rankId, const std::string &kernelName, uint64_t | |||
| 82 | for (auto i = 0; i < listNum; i++) { | 83 | for (auto i = 0; i < listNum; i++) { |
| 83 | uint32_t startOffset = i * listSize; | 84 | uint32_t startOffset = i * listSize; |
| 84 | UniqueIdV2Header *header = reinterpret_cast<UniqueIdV2Header *>(startAddr + startOffset + UNIQUEID_HEADER_OFFSET); | 85 | UniqueIdV2Header *header = reinterpret_cast<UniqueIdV2Header *>(startAddr + startOffset + UNIQUEID_HEADER_OFFSET); |
| 85 | - HCCL_VM_INFO("[device] uniqueId header type:{} notifyNum:{} bufferNum:{} rmtBufferNum:{} connNum:{}.", header->type, header->notifyNum, header->bufferNum, header->rmtBufferNum, header->connNum); | 86 | + HCCL_VM_INFO("uniqueId header type:{} notifyNum:{} bufferNum:{} rmtBufferNum:{} connNum:{}.", header->type, header->notifyNum, header->bufferNum, header->rmtBufferNum, header->connNum); |
| 86 | // 根据UniqueIdV2Header计算偏移 | 87 | // 根据UniqueIdV2Header计算偏移 |
| 87 | uint32_t offset = COMMON_DATA_SIZE + header->notifyNum * NOTIFY_ID_SIZE + | 88 | uint32_t offset = COMMON_DATA_SIZE + header->notifyNum * NOTIFY_ID_SIZE + |
| 88 | COMMON_DATA_SIZE + header->notifyNum * NOTIFY_BUFFER_SIZE + | 89 | COMMON_DATA_SIZE + header->notifyNum * NOTIFY_BUFFER_SIZE + |
| @@ -101,10 +102,10 @@ void ExecuteAicpuKernel(uint32_t rankId, const std::string &kernelName, uint64_t | |||
| 101 | hcclLaunchAicpuKernelPtr(opParam); | 102 | hcclLaunchAicpuKernelPtr(opParam); |
| 102 | UpdataKfcStatus(d2hAddr); | 103 | UpdataKfcStatus(d2hAddr); |
| 103 | } else { | 104 | } else { |
| 104 | - HCCL_VM_ERROR("[device] rankId[{}] kernel[{}] not support.", rankId, kernelName); | 105 | + HCCL_VM_ERROR("rankId[{}] kernel[{}] not support.", rankId, kernelName); |
| 105 | } | 106 | } |
| 106 | 107 | ||
| 107 | - HCCL_VM_INFO("[device] rankId[{}] kernel[{}] finish run...", rankId, kernelName); | 108 | + HCCL_VM_INFO("rankId[{}] kernel[{}] finish run...", rankId, kernelName); |
| 108 | } | 109 | } |
| 109 | 110 | ||
| 110 | void* LoadLibrary(const std::string &libDir, const std::string &libName) | 111 | void* LoadLibrary(const std::string &libDir, const std::string &libName) |
| @@ -113,11 +114,11 @@ void* LoadLibrary(const std::string &libDir, const std::string &libName) | |||
| 113 | std::string libPath = libDir + libName; | 114 | std::string libPath = libDir + libName; |
| 114 | handle = dlopen(libPath.c_str(), RTLD_NOW | RTLD_GLOBAL); | 115 | handle = dlopen(libPath.c_str(), RTLD_NOW | RTLD_GLOBAL); |
| 115 | if (handle != nullptr) { | 116 | if (handle != nullptr) { |
| 116 | - HCCL_VM_INFO("[LoadLibrary] Load so {} from {}", libName, libPath); | 117 | + HCCL_VM_INFO("Load so {} from {}", libName, libPath); |
| 117 | return handle; | 118 | return handle; |
| 118 | } | 119 | } |
| 119 | 120 | ||
| 120 | - HCCL_VM_ERROR("[LoadLibrary] Failed to load so {} from {}", libName, dlerror()); | 121 | + HCCL_VM_ERROR("Failed to load so {} from {}", libName, dlerror()); |
| 121 | return nullptr; | 122 | return nullptr; |
| 122 | } | 123 | } |
| 123 | 124 | ||
| @@ -126,14 +127,13 @@ bool InitKernelFuncHandle() | |||
| 126 | if (gLibsLoaded) { | 127 | if (gLibsLoaded) { |
| 127 | return true; | 128 | return true; |
| 128 | } | 129 | } |
| 129 | - const char* installDir = getenv("HCCL_VM_INSTALL_DIR"); | 130 | + std::string libDir = InstallPath::ResolveToInstallRoot("lib/" + GetArchStr()) + "/"; |
| 130 | - std::string libDir = installDir ? std::string(installDir) + "/lib/" + GetArchStr() + "/" : "./lib/" + GetArchStr() + "/"; | ||
| 131 | gSlogHandle = LoadLibrary(libDir, "libslog.so"); | 131 | gSlogHandle = LoadLibrary(libDir, "libslog.so"); |
| 132 | gCsecHandle = LoadLibrary(libDir, "libc_sec.so"); | 132 | gCsecHandle = LoadLibrary(libDir, "libc_sec.so"); |
| 133 | gHcclKerHandle = LoadLibrary(libDir, "libscatter_aicpu_kernel.so"); | 133 | gHcclKerHandle = LoadLibrary(libDir, "libscatter_aicpu_kernel.so"); |
| 134 | gHcommHandle = LoadLibrary(libDir, "libccl_kernel.so"); | 134 | gHcommHandle = LoadLibrary(libDir, "libccl_kernel.so"); |
| 135 | if (gHcclKerHandle == nullptr || gHcommHandle == nullptr || gSlogHandle == nullptr || gCsecHandle == nullptr) { | 135 | if (gHcclKerHandle == nullptr || gHcommHandle == nullptr || gSlogHandle == nullptr || gCsecHandle == nullptr) { |
| 136 | - HCCL_VM_ERROR("[InitKernelFuncHandle] Failed to load kernel libs."); | 136 | + HCCL_VM_ERROR("Failed to load kernel libs."); |
| 137 | return false; | 137 | return false; |
| 138 | } | 138 | } |
| 139 | runAicpuIndOpCommInitPtr = reinterpret_cast<uint32_t (*)(void *)>(dlsym(gHcommHandle, "RunAicpuIndOpCommInit")); | 139 | runAicpuIndOpCommInitPtr = reinterpret_cast<uint32_t (*)(void *)>(dlsym(gHcommHandle, "RunAicpuIndOpCommInit")); |
| @@ -143,10 +143,10 @@ bool InitKernelFuncHandle() | |||
| 143 | hcclLaunchAicpuKernelPtr = reinterpret_cast<unsigned int (*)(OpParam *)>(dlsym(gHcclKerHandle, "HcclLaunchAicpuKernel")); | 143 | hcclLaunchAicpuKernelPtr = reinterpret_cast<unsigned int (*)(OpParam *)>(dlsym(gHcclKerHandle, "HcclLaunchAicpuKernel")); |
| 144 | if (runAicpuIndOpCommInitPtr == nullptr || runAicpuIndOpThreadInitPtr == nullptr || runAicpuIndOpChannelInitV2Ptr == nullptr || | 144 | if (runAicpuIndOpCommInitPtr == nullptr || runAicpuIndOpThreadInitPtr == nullptr || runAicpuIndOpChannelInitV2Ptr == nullptr || |
| 145 | runAicpuDfxOpInfoInitV2Ptr == nullptr || hcclLaunchAicpuKernelPtr == nullptr) { | 145 | runAicpuDfxOpInfoInitV2Ptr == nullptr || hcclLaunchAicpuKernelPtr == nullptr) { |
| 146 | - HCCL_VM_ERROR("[InitKernelFuncHandle] Failed to get kernel func handle."); | 146 | + HCCL_VM_ERROR("Failed to get kernel func handle."); |
| 147 | return false; | 147 | return false; |
| 148 | } | 148 | } |
| 149 | - HCCL_VM_INFO("[InitKernelFuncHandle] Load kernel libs and get func handles successfully."); | 149 | + HCCL_VM_INFO("Load kernel libs and get func handles successfully."); |
| 150 | gLibsLoaded = true; | 150 | gLibsLoaded = true; |
| 151 | return true; | 151 | return true; |
| 152 | } | 152 | } |
| @@ -21,7 +21,7 @@ void InsertTaskToCollectionDev(HcclTaskMetaData *task) | |||
| 21 | 21 | ||
| 22 | // 1. 入参合法性检查(避免空指针访问) | 22 | // 1. 入参合法性检查(避免空指针访问) |
| 23 | if (task == nullptr) { | 23 | if (task == nullptr) { |
| 24 | - HCCL_VM_ERROR("[InsertTaskToCollection] 错误:输入任务指针 task 不能为空!"); | 24 | + HCCL_VM_ERROR("错误:输入任务指针 task 不能为空!"); |
| 25 | return; | 25 | return; |
| 26 | } | 26 | } |
| 27 | 27 | ||
| @@ -38,7 +38,7 @@ void InsertTaskToCollectionDev(HcclTaskMetaData *task) | |||
| 38 | 38 | ||
| 39 | auto ret = sim::InsertOpTask(opTaskInfo, true); | 39 | auto ret = sim::InsertOpTask(opTaskInfo, true); |
| 40 | if (ret != 0) { | 40 | if (ret != 0) { |
| 41 | - HCCL_VM_ERROR("[InsertTaskToCollection] 错误:插入任务到数据库失败 - {}", ret); | 41 | + HCCL_VM_ERROR("错误:插入任务到数据库失败 - {}", ret); |
| 42 | return; | 42 | return; |
| 43 | } | 43 | } |
| 44 | return; | 44 | return; |
| @@ -31,11 +31,11 @@ int main(int argc, char *argv[]) | |||
| 31 | 31 | ||
| 32 | RegisterSignalHandler(); | 32 | RegisterSignalHandler(); |
| 33 | setvbuf(stdout, nullptr, _IOLBF, 0); | 33 | setvbuf(stdout, nullptr, _IOLBF, 0); |
| 34 | - HCCL_VM_INFO("[device] main process start."); | 34 | + HCCL_VM_INFO("main process start."); |
| 35 | 35 | ||
| 36 | uint32_t rankId = static_cast<uint32_t>(std::stoi(argv[1])); | 36 | uint32_t rankId = static_cast<uint32_t>(std::stoi(argv[1])); |
| 37 | SetCurRankId(rankId); | 37 | SetCurRankId(rankId); |
| 38 | - HCCL_VM_INFO("[device] rankId[{}] process start...", rankId); | 38 | + HCCL_VM_INFO("rankId[{}] process start...", rankId); |
| 39 | 39 | ||
| 40 | HcclAicpuData *aicpuData = GetHcclAicpuDataShmPtr(); | 40 | HcclAicpuData *aicpuData = GetHcclAicpuDataShmPtr(); |
| 41 | if (aicpuData == nullptr) { | 41 | if (aicpuData == nullptr) { |
| @@ -56,7 +56,7 @@ int main(int argc, char *argv[]) | |||
| 56 | uint32_t opDetailId = 0; | 56 | uint32_t opDetailId = 0; |
| 57 | int ret = sim::QueryNewestOpDeatailIdByPid(getppid(), opDetailId); | 57 | int ret = sim::QueryNewestOpDeatailIdByPid(getppid(), opDetailId); |
| 58 | if (ret != 0) { | 58 | if (ret != 0) { |
| 59 | - HCCL_VM_ERROR("[device] rankId[{}] QueryNewestOpDeatailIdByPid failed.", rankId); | 59 | + HCCL_VM_ERROR("rankId[{}] QueryNewestOpDeatailIdByPid failed.", rankId); |
| 60 | } | 60 | } |
| 61 | sim::g_currOpDetailId = opDetailId; | 61 | sim::g_currOpDetailId = opDetailId; |
| 62 | } | 62 | } |
| @@ -12,6 +12,7 @@ target_sources(hccl_device_proxy PRIVATE | |||
| 12 | aicpu_stub.cc | 12 | aicpu_stub.cc |
| 13 | ascend_hal_stub.cc | 13 | ascend_hal_stub.cc |
| 14 | device_sqe_parse_stub.cc | 14 | device_sqe_parse_stub.cc |
| 15 | + aprofiling_stub.cc | ||
| 15 | ${HCCL_VM_SRC}/utils/sim_log.cc | 16 | ${HCCL_VM_SRC}/utils/sim_log.cc |
| 16 | ${HCCL_VM_SRC}/utils/sim_yaml_config.cc | 17 | ${HCCL_VM_SRC}/utils/sim_yaml_config.cc |
| 17 | ${HCCL_VM_SRC}/utils/sim_common_api.cc | 18 | ${HCCL_VM_SRC}/utils/sim_common_api.cc |
| @@ -0,0 +1,121 @@ | |||
| 1 | +/** | ||
| 2 | + * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | + * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | + * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | + * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | + * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | + * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | + */ | ||
| 10 | + | ||
| 11 | + | ||
| 12 | + | ||
| 13 | + | ||
| 14 | + | ||
| 15 | + | ||
| 16 | + | ||
| 17 | +extern "C" { | ||
| 18 | + | ||
| 19 | + | ||
| 20 | +uint64_t MsprofStr2Id(const char *hashInfo, size_t length) | ||
| 21 | +{ | ||
| 22 | + (void) hashInfo; | ||
| 23 | + (void) length; | ||
| 24 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 25 | + return 1; | ||
| 26 | +} | ||
| 27 | + | ||
| 28 | +int32_t MsprofRegTypeInfo(uint16_t level, uint32_t typeId, const char *typeName) | ||
| 29 | +{ | ||
| 30 | + (void) level; | ||
| 31 | + (void) typeId; | ||
| 32 | + (void) typeName; | ||
| 33 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 34 | + return 0; | ||
| 35 | +} | ||
| 36 | + | ||
| 37 | +uint64_t MsprofSysCycleTime() | ||
| 38 | +{ | ||
| 39 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 40 | + return 1; | ||
| 41 | +} | ||
| 42 | + | ||
| 43 | +int32_t MsprofRegisterCallback(uint32_t moduleId, ProfCommandHandle handle) | ||
| 44 | +{ | ||
| 45 | + (void) moduleId; | ||
| 46 | + (void) handle; | ||
| 47 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 48 | + return 0; | ||
| 49 | +} | ||
| 50 | + | ||
| 51 | +int32_t MsprofReportApi(uint32_t agingFlag, const MsprofApi *api) | ||
| 52 | +{ | ||
| 53 | + (void) agingFlag; | ||
| 54 | + (void) api; | ||
| 55 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 56 | + return 0; | ||
| 57 | +} | ||
| 58 | + | ||
| 59 | +int32_t MsprofReportAdditionalInfo(uint32_t agingFlag, const VOID_PTR data, uint32_t length) | ||
| 60 | +{ | ||
| 61 | + (void) agingFlag; | ||
| 62 | + (void) data; | ||
| 63 | + (void) length; | ||
| 64 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 65 | + return 0; | ||
| 66 | +} | ||
| 67 | + | ||
| 68 | +int32_t MsprofReportCompactInfo(uint32_t agingFlag, const VOID_PTR data, uint32_t length) | ||
| 69 | +{ | ||
| 70 | + (void) agingFlag; | ||
| 71 | + (void) data; | ||
| 72 | + (void) length; | ||
| 73 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 74 | + return 0; | ||
| 75 | +} | ||
| 76 | + | ||
| 77 | +int32_t MsprofReportBatchAdditionalInfo(uint32_t nonPersistantFlag, const VOID_PTR data, uint32_t length) | ||
| 78 | +{ | ||
| 79 | + (void) nonPersistantFlag; | ||
| 80 | + (void) data; | ||
| 81 | + (void) length; | ||
| 82 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 83 | + return 0; | ||
| 84 | +} | ||
| 85 | + | ||
| 86 | +int32_t AdprofReportAdditionalInfo(uint32_t agingFlag, const void *data, uint32_t length) | ||
| 87 | +{ | ||
| 88 | + (void) agingFlag; | ||
| 89 | + (void) data; | ||
| 90 | + (void) length; | ||
| 91 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 92 | + return 0; | ||
| 93 | +} | ||
| 94 | + | ||
| 95 | +int32_t AdprofReportBatchAdditionalInfo(uint32_t nonPersistantFlag, const void *data, uint32_t length) | ||
| 96 | +{ | ||
| 97 | + (void) nonPersistantFlag; | ||
| 98 | + (void) data; | ||
| 99 | + (void) length; | ||
| 100 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 101 | + return 0; | ||
| 102 | +} | ||
| 103 | + | ||
| 104 | +uint64_t AdprofGetHashId(const char *hashInfo, size_t length) | ||
| 105 | +{ | ||
| 106 | + (void) hashInfo; | ||
| 107 | + (void) length; | ||
| 108 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 109 | + return 1; | ||
| 110 | +} | ||
| 111 | + | ||
| 112 | +int32_t AdprofCheckFeatureIsOn(uint64_t feature) | ||
| 113 | +{ | ||
| 114 | + (void) feature; | ||
| 115 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 116 | + return 0; | ||
| 117 | +} | ||
| 118 | + | ||
| 119 | + | ||
| 120 | +} | ||
| 121 | + | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -21,6 +24,7 @@ | |||
| 21 | 24 | ||
| 22 | 25 | ||
| 23 | 26 | ||
| 27 | + | ||
| 24 | using namespace HcclSim; | 28 | using namespace HcclSim; |
| 25 | 29 | ||
| 26 | 30 | ||
| @@ -30,14 +34,14 @@ extern "C" { | |||
| 30 | drvError_t halEschedAttachDevice(unsigned int devId) | 34 | drvError_t halEschedAttachDevice(unsigned int devId) |
| 31 | { | 35 | { |
| 32 | (void) devId; | 36 | (void) devId; |
| 33 | - HCCL_VM_INFO("[STUB]AttachDevice is empty."); | 37 | + HCCL_VM_INFO("AttachDevice is empty."); |
| 34 | return DRV_ERROR_NONE; | 38 | return DRV_ERROR_NONE; |
| 35 | } | 39 | } |
| 36 | 40 | ||
| 37 | drvError_t halEschedDettachDevice(unsigned int devId) | 41 | drvError_t halEschedDettachDevice(unsigned int devId) |
| 38 | { | 42 | { |
| 39 | (void) devId; | 43 | (void) devId; |
| 40 | - HCCL_VM_INFO("[STUB]DettachDevice is empty."); | 44 | + HCCL_VM_INFO("DettachDevice is empty."); |
| 41 | return DRV_ERROR_NONE; | 45 | return DRV_ERROR_NONE; |
| 42 | } | 46 | } |
| 43 | 47 | ||
| @@ -46,7 +50,7 @@ drvError_t halEschedCreateGrpEx(uint32_t devId, struct esched_grp_para *grpPara, | |||
| 46 | (void) devId; | 50 | (void) devId; |
| 47 | (void) grpPara; | 51 | (void) grpPara; |
| 48 | (void) grpId; | 52 | (void) grpId; |
| 49 | - HCCL_VM_INFO("[STUB]CreateGrpEx is empty."); | 53 | + HCCL_VM_INFO("CreateGrpEx is empty."); |
| 50 | return DRV_ERROR_NONE; | 54 | return DRV_ERROR_NONE; |
| 51 | } | 55 | } |
| 52 | 56 | ||
| @@ -57,7 +61,7 @@ drvError_t halEschedSubscribeEvent(unsigned int devId, unsigned int grpId, | |||
| 57 | (void) grpId; | 61 | (void) grpId; |
| 58 | (void) threadId; | 62 | (void) threadId; |
| 59 | (void) eventBitmap; | 63 | (void) eventBitmap; |
| 60 | - HCCL_VM_INFO("[STUB]SubscribeEvent is empty."); | 64 | + HCCL_VM_INFO("SubscribeEvent is empty."); |
| 61 | return DRV_ERROR_NONE; | 65 | return DRV_ERROR_NONE; |
| 62 | } | 66 | } |
| 63 | 67 | ||
| @@ -68,7 +72,7 @@ drvError_t halEschedQueryInfo(unsigned int devId, ESCHED_QUERY_TYPE type, struct | |||
| 68 | (void) type; | 72 | (void) type; |
| 69 | (void) inPut; | 73 | (void) inPut; |
| 70 | (void) outPut; | 74 | (void) outPut; |
| 71 | - HCCL_VM_INFO("[STUB]QueryInfo is empty."); | 75 | + HCCL_VM_INFO("QueryInfo is empty."); |
| 72 | return DRV_ERROR_NONE; | 76 | return DRV_ERROR_NONE; |
| 73 | } | 77 | } |
| 74 | 78 | ||
| @@ -80,7 +84,7 @@ drvError_t halEschedWaitEvent(unsigned int devId, unsigned int grpId, | |||
| 80 | (void) threadId; | 84 | (void) threadId; |
| 81 | (void) timeout; | 85 | (void) timeout; |
| 82 | (void) event; | 86 | (void) event; |
| 83 | - HCCL_VM_INFO("[STUB]WaitEvent is empty."); | 87 | + HCCL_VM_INFO("WaitEvent is empty."); |
| 84 | return DRV_ERROR_NONE; | 88 | return DRV_ERROR_NONE; |
| 85 | } | 89 | } |
| 86 | 90 | ||
| @@ -91,14 +95,14 @@ drvError_t halEschedRegisterAckFunc(unsigned int grpId, EVENT_ID eventId, | |||
| 91 | (void) grpId; | 95 | (void) grpId; |
| 92 | (void) eventId; | 96 | (void) eventId; |
| 93 | (void) ackFunc; | 97 | (void) ackFunc; |
| 94 | - HCCL_VM_INFO("[STUB]RegisterAckFunc is empty."); | 98 | + HCCL_VM_INFO("RegisterAckFunc is empty."); |
| 95 | return DRV_ERROR_NONE; | 99 | return DRV_ERROR_NONE; |
| 96 | } | 100 | } |
| 97 | 101 | ||
| 98 | drvError_t drvGetDevNum(uint32_t *num_dev) | 102 | drvError_t drvGetDevNum(uint32_t *num_dev) |
| 99 | { | 103 | { |
| 100 | *num_dev = 1; | 104 | *num_dev = 1; |
| 101 | - HCCL_VM_INFO("[STUB]drvGetDevNum:{}", *num_dev); | 105 | + HCCL_VM_INFO("drvGetDevNum:{}", *num_dev); |
| 102 | return DRV_ERROR_NONE; | 106 | return DRV_ERROR_NONE; |
| 103 | } | 107 | } |
| 104 | 108 | ||
| @@ -106,11 +110,11 @@ drvError_t halGetChipInfo(unsigned int devId, halChipInfo *chipInfo) | |||
| 106 | { | 110 | { |
| 107 | sim::Device locDevice{}; | 111 | sim::Device locDevice{}; |
| 108 | if (sim::GetDeviceByPhysicalId(devId, locDevice) != ACL_SUCCESS) { | 112 | if (sim::GetDeviceByPhysicalId(devId, locDevice) != ACL_SUCCESS) { |
| 109 | - HCCL_VM_ERROR("[STUB]halGetChipInfo get device by physic id {} failed.", devId); | 113 | + HCCL_VM_ERROR("halGetChipInfo get device by physic id {} failed.", devId); |
| 110 | return DRV_ERROR_INNER_ERR; | 114 | return DRV_ERROR_INNER_ERR; |
| 111 | } | 115 | } |
| 112 | strncpy((char *)chipInfo->name, locDevice.soc_version, MAX_CHIP_NAME - 1); | 116 | strncpy((char *)chipInfo->name, locDevice.soc_version, MAX_CHIP_NAME - 1); |
| 113 | - HCCL_VM_INFO("[STUB]GetChipInfo is empty."); | 117 | + HCCL_VM_INFO("GetChipInfo is empty."); |
| 114 | return DRV_ERROR_NONE; | 118 | return DRV_ERROR_NONE; |
| 115 | } | 119 | } |
| 116 | 120 | ||
| @@ -121,7 +125,7 @@ drvError_t halHostRegister(void *src_ptr, UINT64 size, UINT32 flag, UINT32 devid | |||
| 121 | (void) flag; | 125 | (void) flag; |
| 122 | (void) devid; | 126 | (void) devid; |
| 123 | (void) dst_ptr; | 127 | (void) dst_ptr; |
| 124 | - HCCL_VM_INFO("[STUB]HostRegister is empty."); | 128 | + HCCL_VM_INFO("HostRegister is empty."); |
| 125 | return DRV_ERROR_NONE; | 129 | return DRV_ERROR_NONE; |
| 126 | } | 130 | } |
| 127 | 131 | ||
| @@ -129,7 +133,7 @@ drvError_t halHostUnregister(void *src_ptr, UINT32 devid) | |||
| 129 | { | 133 | { |
| 130 | (void) src_ptr; | 134 | (void) src_ptr; |
| 131 | (void) devid; | 135 | (void) devid; |
| 132 | - HCCL_VM_INFO("[STUB]HostUnregister is empty."); | 136 | + HCCL_VM_INFO("HostUnregister is empty."); |
| 133 | return DRV_ERROR_NONE; | 137 | return DRV_ERROR_NONE; |
| 134 | } | 138 | } |
| 135 | 139 | ||
| @@ -141,14 +145,14 @@ drvError_t halMemCtl(int type, void *param_value, size_t param_value_size, void | |||
| 141 | (void) param_value_size; | 145 | (void) param_value_size; |
| 142 | (void) out_value; | 146 | (void) out_value; |
| 143 | (void) out_size_ret; | 147 | (void) out_size_ret; |
| 144 | - HCCL_VM_INFO("[STUB]MemCtl is empty."); | 148 | + HCCL_VM_INFO("MemCtl is empty."); |
| 145 | return DRV_ERROR_NONE; | 149 | return DRV_ERROR_NONE; |
| 146 | } | 150 | } |
| 147 | 151 | ||
| 148 | drvError_t drvGetPlatformInfo(uint32_t *info) | 152 | drvError_t drvGetPlatformInfo(uint32_t *info) |
| 149 | { | 153 | { |
| 150 | *info = 1; | 154 | *info = 1; |
| 151 | - HCCL_VM_INFO("[STUB]drvGetPlatformInfo:{}", *info); | 155 | + HCCL_VM_INFO("drvGetPlatformInfo:{}", *info); |
| 152 | return DRV_ERROR_NONE; | 156 | return DRV_ERROR_NONE; |
| 153 | } | 157 | } |
| 154 | 158 | ||
| @@ -160,7 +164,7 @@ drvError_t drvQueryProcessHostPid(int pid, unsigned int *chip_id, unsigned int * | |||
| 160 | (void) vfid; | 164 | (void) vfid; |
| 161 | (void) host_pid; | 165 | (void) host_pid; |
| 162 | (void) cp_type; | 166 | (void) cp_type; |
| 163 | - HCCL_VM_INFO("[STUB]QueryProcessHostPid is empty."); | 167 | + HCCL_VM_INFO("QueryProcessHostPid is empty."); |
| 164 | return DRV_ERROR_NONE; | 168 | return DRV_ERROR_NONE; |
| 165 | } | 169 | } |
| 166 | 170 | ||
| @@ -182,21 +186,21 @@ drvError_t halEschedSubmitEvent(unsigned int devId, struct event_summary *event) | |||
| 182 | { | 186 | { |
| 183 | (void) devId; | 187 | (void) devId; |
| 184 | (void) event; | 188 | (void) event; |
| 185 | - HCCL_VM_INFO("[STUB]SubmitEvent is empty."); | 189 | + HCCL_VM_INFO("SubmitEvent is empty."); |
| 186 | return DRV_ERROR_NONE; | 190 | return DRV_ERROR_NONE; |
| 187 | } | 191 | } |
| 188 | 192 | ||
| 189 | drvError_t halBindCgroup(BIND_CGROUP_TYPE bindType) | 193 | drvError_t halBindCgroup(BIND_CGROUP_TYPE bindType) |
| 190 | { | 194 | { |
| 191 | (void) bindType; | 195 | (void) bindType; |
| 192 | - HCCL_VM_INFO("[STUB]BindCgroup is empty."); | 196 | + HCCL_VM_INFO("BindCgroup is empty."); |
| 193 | return DRV_ERROR_NONE; | 197 | return DRV_ERROR_NONE; |
| 194 | } | 198 | } |
| 195 | 199 | ||
| 196 | drvError_t halGetAPIVersion(int *halAPIVersion) | 200 | drvError_t halGetAPIVersion(int *halAPIVersion) |
| 197 | { | 201 | { |
| 198 | (void) halAPIVersion; | 202 | (void) halAPIVersion; |
| 199 | - HCCL_VM_INFO("[STUB]GetAPIVersion is empty."); | 203 | + HCCL_VM_INFO("GetAPIVersion is empty."); |
| 200 | return DRV_ERROR_NONE; | 204 | return DRV_ERROR_NONE; |
| 201 | } | 205 | } |
| 202 | 206 | ||
| @@ -205,7 +209,7 @@ drvError_t halSensorNodeRegister(uint32_t devId, struct halSensorNodeCfg *cfg, u | |||
| 205 | (void) devId; | 209 | (void) devId; |
| 206 | (void) cfg; | 210 | (void) cfg; |
| 207 | (void) handle; | 211 | (void) handle; |
| 208 | - HCCL_VM_INFO("[STUB]RegisterSensorNode is empty."); | 212 | + HCCL_VM_INFO("RegisterSensorNode is empty."); |
| 209 | return DRV_ERROR_NONE; | 213 | return DRV_ERROR_NONE; |
| 210 | } | 214 | } |
| 211 | 215 | ||
| @@ -213,7 +217,7 @@ drvError_t halSensorNodeUnregister(uint32_t devId, uint64_t handle) | |||
| 213 | { | 217 | { |
| 214 | (void) devId; | 218 | (void) devId; |
| 215 | (void) handle; | 219 | (void) handle; |
| 216 | - HCCL_VM_INFO("[STUB]UnregisterSensorNode is empty."); | 220 | + HCCL_VM_INFO("UnregisterSensorNode is empty."); |
| 217 | return DRV_ERROR_NONE; | 221 | return DRV_ERROR_NONE; |
| 218 | } | 222 | } |
| 219 | 223 | ||
| @@ -223,7 +227,7 @@ drvError_t halSensorNodeUpdateState(uint32_t devId, uint64_t handle, int val, ha | |||
| 223 | (void) handle; | 227 | (void) handle; |
| 224 | (void) val; | 228 | (void) val; |
| 225 | (void) assertion; | 229 | (void) assertion; |
| 226 | - HCCL_VM_INFO("[STUB]UpdateSensorNodeState is empty."); | 230 | + HCCL_VM_INFO("UpdateSensorNodeState is empty."); |
| 227 | return DRV_ERROR_NONE; | 231 | return DRV_ERROR_NONE; |
| 228 | } | 232 | } |
| 229 | 233 | ||
| @@ -251,28 +255,28 @@ int halGrpQuery(GroupQueryCmdType cmd, void *inBuff, unsigned int inLen, void *o | |||
| 251 | (void) inLen; | 255 | (void) inLen; |
| 252 | (void) outBuff; | 256 | (void) outBuff; |
| 253 | (void) outLen; | 257 | (void) outLen; |
| 254 | - HCCL_VM_INFO("[STUB]GrpQuery is empty."); | 258 | + HCCL_VM_INFO("GrpQuery is empty."); |
| 255 | return 0; | 259 | return 0; |
| 256 | } | 260 | } |
| 257 | 261 | ||
| 258 | pid_t drvDeviceGetBareTgid(void) | 262 | pid_t drvDeviceGetBareTgid(void) |
| 259 | { | 263 | { |
| 260 | pid_t pid = getpid(); | 264 | pid_t pid = getpid(); |
| 261 | - HCCL_VM_INFO("[STUB]drvDeviceGetBareTgid] pid:%u", pid); | 265 | + HCCL_VM_INFO("drvDeviceGetBareTgid] pid:{}", pid); |
| 262 | return pid; | 266 | return pid; |
| 263 | } | 267 | } |
| 264 | 268 | ||
| 265 | drvError_t halResourceIdCheck(struct drvResIdKey *info) | 269 | drvError_t halResourceIdCheck(struct drvResIdKey *info) |
| 266 | { | 270 | { |
| 267 | (void) info; | 271 | (void) info; |
| 268 | - HCCL_VM_INFO("[STUB]ResourceIdCheck is empty."); | 272 | + HCCL_VM_INFO("ResourceIdCheck is empty."); |
| 269 | return DRV_ERROR_NONE; | 273 | return DRV_ERROR_NONE; |
| 270 | } | 274 | } |
| 271 | 275 | ||
| 272 | drvError_t halSqCqQuery(uint32_t devId, struct halSqCqQueryInfo *info) | 276 | drvError_t halSqCqQuery(uint32_t devId, struct halSqCqQueryInfo *info) |
| 273 | { | 277 | { |
| 274 | (void) devId; | 278 | (void) devId; |
| 275 | - HCCL_VM_INFO("[STUB]SqCqQuery is empty."); | 279 | + HCCL_VM_INFO("SqCqQuery is empty."); |
| 276 | if (info == nullptr) { | 280 | if (info == nullptr) { |
| 277 | return DRV_ERROR_INNER_ERR; | 281 | return DRV_ERROR_INNER_ERR; |
| 278 | } | 282 | } |
| @@ -304,7 +308,7 @@ drvError_t halSqCqQuery(uint32_t devId, struct halSqCqQueryInfo *info) | |||
| 304 | 308 | ||
| 305 | drvError_t halSqCqConfig(uint32_t devId, struct halSqCqConfigInfo *info) | 309 | drvError_t halSqCqConfig(uint32_t devId, struct halSqCqConfigInfo *info) |
| 306 | { | 310 | { |
| 307 | - HCCL_VM_INFO("[STUB] devId[{}] is empty.", devId); | 311 | + HCCL_VM_INFO("devId[{}] is empty.", devId); |
| 308 | if (info->prop == DRV_SQCQ_PROP_SQ_TAIL) { | 312 | if (info->prop == DRV_SQCQ_PROP_SQ_TAIL) { |
| 309 | ParseA5SqeFromSqBuffer(devId, info); | 313 | ParseA5SqeFromSqBuffer(devId, info); |
| 310 | } | 314 | } |
| @@ -315,7 +319,7 @@ drvError_t halShrIdInfoGet(const char *name, struct shrIdGetInfo *info) | |||
| 315 | { | 319 | { |
| 316 | (void) name; | 320 | (void) name; |
| 317 | (void) info; | 321 | (void) info; |
| 318 | - HCCL_VM_INFO("[STUB] is empty."); | 322 | + HCCL_VM_INFO("is empty."); |
| 319 | return DRV_ERROR_NONE; | 323 | return DRV_ERROR_NONE; |
| 320 | } | 324 | } |
| 321 | 325 | ||
| @@ -327,21 +331,21 @@ drvError_t halTsdrvCtl(uint32_t devId, int cmd, void *param, size_t paramSize, v | |||
| 327 | (void) paramSize; | 331 | (void) paramSize; |
| 328 | (void) out; | 332 | (void) out; |
| 329 | (void) outSize; | 333 | (void) outSize; |
| 330 | - HCCL_VM_INFO("[STUB] is empty."); | 334 | + HCCL_VM_INFO("is empty."); |
| 331 | return DRV_ERROR_NONE; | 335 | return DRV_ERROR_NONE; |
| 332 | } | 336 | } |
| 333 | 337 | ||
| 334 | drvError_t halResourceIdRestore(struct drvResIdKey *info) | 338 | drvError_t halResourceIdRestore(struct drvResIdKey *info) |
| 335 | { | 339 | { |
| 336 | (void) info; | 340 | (void) info; |
| 337 | - HCCL_VM_INFO("[STUB] is empty."); | 341 | + HCCL_VM_INFO("is empty."); |
| 338 | return DRV_ERROR_NONE; | 342 | return DRV_ERROR_NONE; |
| 339 | } | 343 | } |
| 340 | 344 | ||
| 341 | drvError_t drvGetLocalDevIDByHostDevID(uint32_t host_dev_id, uint32_t *local_dev_id) | 345 | drvError_t drvGetLocalDevIDByHostDevID(uint32_t host_dev_id, uint32_t *local_dev_id) |
| 342 | { | 346 | { |
| 343 | *local_dev_id = host_dev_id; | 347 | *local_dev_id = host_dev_id; |
| 344 | - HCCL_VM_INFO("[STUB] host_dev_id:{}, local_dev_id:{}", host_dev_id, *local_dev_id); | 348 | + HCCL_VM_INFO("host_dev_id:{}, local_dev_id:{}", host_dev_id, *local_dev_id); |
| 345 | return DRV_ERROR_NONE; | 349 | return DRV_ERROR_NONE; |
| 346 | } | 350 | } |
| 347 | 351 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -21,6 +24,7 @@ | |||
| 21 | 24 | ||
| 22 | 25 | ||
| 23 | 26 | ||
| 27 | + | ||
| 24 | constexpr int SHIFT_BIT32 = 32; | 28 | constexpr int SHIFT_BIT32 = 32; |
| 25 | uint32_t curRankId = 0; | 29 | uint32_t curRankId = 0; |
| 26 | 30 | ||
| @@ -72,7 +76,7 @@ void ParseA5SqeFromSqBuffer(uint32_t devId, struct halSqCqConfigInfo *info) | |||
| 72 | break; | 76 | break; |
| 73 | } | 77 | } |
| 74 | default: { | 78 | default: { |
| 75 | - HCCL_VM_ERROR("[ParseA5SqeFromSqBuffer] not support sqe type[{}].", static_cast<uint32_t>(header->type)); | 79 | + HCCL_VM_ERROR("not support sqe type[{}].", static_cast<uint32_t>(header->type)); |
| 76 | break; | 80 | break; |
| 77 | } | 81 | } |
| 78 | } | 82 | } |
| @@ -170,7 +174,7 @@ uint32_t CalculateCiValue(uint64_t wqeBuffer, uint32_t piValue) | |||
| 170 | ubCommon = reinterpret_cast<UdmaSqeCommon *>(wqeAddr); | 174 | ubCommon = reinterpret_cast<UdmaSqeCommon *>(wqeAddr); |
| 171 | } | 175 | } |
| 172 | 176 | ||
| 173 | - HCCL_VM_ERROR("[CalculateCiValue] calculate ciValue failed."); | 177 | + HCCL_VM_ERROR("calculate ciValue failed."); |
| 174 | return 0; | 178 | return 0; |
| 175 | } | 179 | } |
| 176 | 180 | ||
| @@ -181,24 +185,24 @@ void ParseDavidUDMASqe(uint32_t streamId, void *sqeBuf) | |||
| 181 | 185 | ||
| 182 | HcclAicpuData *aicpuData = GetHcclAicpuDataShmPtr(); | 186 | HcclAicpuData *aicpuData = GetHcclAicpuDataShmPtr(); |
| 183 | if (aicpuData == nullptr) { | 187 | if (aicpuData == nullptr) { |
| 184 | - HCCL_VM_ERROR("[ParseDavidUDMASqe] aicpuData is nullptr."); | 188 | + HCCL_VM_ERROR("aicpuData is nullptr."); |
| 185 | return; | 189 | return; |
| 186 | } | 190 | } |
| 187 | uint64_t wqeAddrDev = aicpuData->common.jettyId2WqeBufMap[jettyId]; | 191 | uint64_t wqeAddrDev = aicpuData->common.jettyId2WqeBufMap[jettyId]; |
| 188 | uint64_t wqeBuffer = reinterpret_cast<uint64_t>(GetRealPtrByDevPtr(reinterpret_cast<void *>(wqeAddrDev))); | 192 | uint64_t wqeBuffer = reinterpret_cast<uint64_t>(GetRealPtrByDevPtr(reinterpret_cast<void *>(wqeAddrDev))); |
| 189 | if (wqeBuffer == 0) { | 193 | if (wqeBuffer == 0) { |
| 190 | - HCCL_VM_ERROR("[ParseDavidUDMASqe] wqeBuffer is nullptr, wqeAddrDev[{}].", wqeAddrDev); | 194 | + HCCL_VM_ERROR("wqeBuffer is nullptr, wqeAddrDev[{}].", wqeAddrDev); |
| 191 | return; | 195 | return; |
| 192 | } | 196 | } |
| 193 | 197 | ||
| 194 | uint32_t piValue = ubSqe->piValue1; | 198 | uint32_t piValue = ubSqe->piValue1; |
| 195 | uint32_t ciValue = CalculateCiValue(wqeBuffer, ubSqe->piValue1); | 199 | uint32_t ciValue = CalculateCiValue(wqeBuffer, ubSqe->piValue1); |
| 196 | if (ciValue >= piValue) { | 200 | if (ciValue >= piValue) { |
| 197 | - HCCL_VM_ERROR("[ParseDavidUDMASqe] jettyId[{}] ciVal[{}] piValue[{}] streamId[{}].", jettyId, ciValue, piValue, streamId); | 201 | + HCCL_VM_ERROR("jettyId[{}] ciVal[{}] piValue[{}] streamId[{}].", jettyId, ciValue, piValue, streamId); |
| 198 | return; | 202 | return; |
| 199 | } | 203 | } |
| 200 | 204 | ||
| 201 | - HCCL_VM_INFO("[ParseDavidUDMASqe] jettyId[{}] ciVal[{}] piValue[{}] streamId[{}].", jettyId, ciValue, piValue, streamId); | 205 | + HCCL_VM_INFO("jettyId[{}] ciVal[{}] piValue[{}] streamId[{}].", jettyId, ciValue, piValue, streamId); |
| 202 | for (auto index = ciValue; index < piValue; index++) { | 206 | for (auto index = ciValue; index < piValue; index++) { |
| 203 | uint64_t wqeAddr = wqeBuffer + index * HCCL_WQE_SIZE; | 207 | uint64_t wqeAddr = wqeBuffer + index * HCCL_WQE_SIZE; |
| 204 | UdmaSqeCommon *ubCommon = reinterpret_cast<UdmaSqeCommon *>(wqeAddr); | 208 | UdmaSqeCommon *ubCommon = reinterpret_cast<UdmaSqeCommon *>(wqeAddr); |
| @@ -217,7 +221,7 @@ void ParseDavidUDMASqe(uint32_t streamId, void *sqeBuf) | |||
| 217 | break; | 221 | break; |
| 218 | } | 222 | } |
| 219 | default: { | 223 | default: { |
| 220 | - HCCL_VM_ERROR("[ParseDavidUDMASqe] not support opcode[{}].", static_cast<uint32_t>(ubCommon->opcode)); | 224 | + HCCL_VM_ERROR("not support opcode[{}].", static_cast<uint32_t>(ubCommon->opcode)); |
| 221 | return; | 225 | return; |
| 222 | } | 226 | } |
| 223 | } | 227 | } |
| @@ -344,7 +348,7 @@ HcclReduceOp ParseReduceTypeDavid(uint8_t result) | |||
| 344 | return DavidReduceOpMap[reduceType]; | 348 | return DavidReduceOpMap[reduceType]; |
| 345 | } | 349 | } |
| 346 | 350 | ||
| 347 | - HCCL_VM_ERROR("[ParseReduceTypeDavid] not support reduceType[{}].", reduceType); | 351 | + HCCL_VM_ERROR("not support reduceType[{}].", reduceType); |
| 348 | return HcclReduceOp::HCCL_REDUCE_RESERVED; | 352 | return HcclReduceOp::HCCL_REDUCE_RESERVED; |
| 349 | } | 353 | } |
| 350 | 354 | ||
| @@ -367,7 +371,7 @@ HcclDataType ParseDataTypeDavid(uint8_t result) | |||
| 367 | return DavidDataTypeMap[dataType]; | 371 | return DavidDataTypeMap[dataType]; |
| 368 | } | 372 | } |
| 369 | 373 | ||
| 370 | - HCCL_VM_ERROR("[ParseDataTypeDavid] not support dataType[{}].", dataType); | 374 | + HCCL_VM_ERROR("not support dataType[{}].", dataType); |
| 371 | return HcclDataType::HCCL_DATA_TYPE_RESERVED; | 375 | return HcclDataType::HCCL_DATA_TYPE_RESERVED; |
| 372 | } | 376 | } |
| 373 | 377 | ||
| @@ -383,7 +387,7 @@ HcclReduceOp ParseUbReduceTypeDavid(uint32_t type) | |||
| 383 | return DavidUbReduceOpMap[type]; | 387 | return DavidUbReduceOpMap[type]; |
| 384 | } | 388 | } |
| 385 | 389 | ||
| 386 | - HCCL_VM_ERROR("[ParseUbReduceTypeDavid] not support type[{}].", type); | 390 | + HCCL_VM_ERROR("not support type[{}].", type); |
| 387 | return HcclReduceOp::HCCL_REDUCE_RESERVED; | 391 | return HcclReduceOp::HCCL_REDUCE_RESERVED; |
| 388 | } | 392 | } |
| 389 | 393 | ||
| @@ -405,7 +409,7 @@ HcclDataType ParseUbDataTypeDavid(uint32_t type) | |||
| 405 | return DavidUbDataTypeMap[type]; | 409 | return DavidUbDataTypeMap[type]; |
| 406 | } | 410 | } |
| 407 | 411 | ||
| 408 | - HCCL_VM_ERROR("[ParseUbDataTypeDavid] not support type[{}].", type); | 412 | + HCCL_VM_ERROR("not support type[{}].", type); |
| 409 | return HcclDataType::HCCL_DATA_TYPE_RESERVED; | 413 | return HcclDataType::HCCL_DATA_TYPE_RESERVED; |
| 410 | } | 414 | } |
| 411 | 415 | ||
| @@ -414,21 +418,21 @@ void PrintTaskMetaData(const HcclTaskMetaData &taskMeta) | |||
| 414 | pid_t pid = getpid(); | 418 | pid_t pid = getpid(); |
| 415 | switch (taskMeta.taskType) { | 419 | switch (taskMeta.taskType) { |
| 416 | case HccLTaskMetaType::MEM_CPY: | 420 | case HccLTaskMetaType::MEM_CPY: |
| 417 | - HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[MEM_CPY], srcOffset[{}] dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}]\n", | 421 | + HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[MEM_CPY], srcOffset[{}] dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}]", |
| 418 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.transMem.srcOffset, taskMeta.taskData.transMem.dstOffset, taskMeta.taskData.transMem.len, | 422 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.transMem.srcOffset, taskMeta.taskData.transMem.dstOffset, taskMeta.taskData.transMem.len, |
| 419 | taskMeta.taskData.transMem.srcRankId, taskMeta.taskData.transMem.dstRankId); | 423 | taskMeta.taskData.transMem.srcRankId, taskMeta.taskData.transMem.dstRankId); |
| 420 | break; | 424 | break; |
| 421 | case HccLTaskMetaType::REDUCE: | 425 | case HccLTaskMetaType::REDUCE: |
| 422 | - HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[REDUCE], srcOffset[{}] dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}], reduceOp[%{}], dataType[%{}]\n", | 426 | + HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[REDUCE], srcOffset[{}] dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}], reduceOp[%{}], dataType[%{}]", |
| 423 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.reduce.srcOffset, taskMeta.taskData.reduce.dstOffset, taskMeta.taskData.reduce.dataCount, | 427 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.reduce.srcOffset, taskMeta.taskData.reduce.dstOffset, taskMeta.taskData.reduce.dataCount, |
| 424 | taskMeta.taskData.reduce.srcRankId, taskMeta.taskData.reduce.dstRankId, taskMeta.taskData.reduce.reduceOp, taskMeta.taskData.reduce.dataType); | 428 | taskMeta.taskData.reduce.srcRankId, taskMeta.taskData.reduce.dstRankId, taskMeta.taskData.reduce.reduceOp, taskMeta.taskData.reduce.dataType); |
| 425 | break; | 429 | break; |
| 426 | case HccLTaskMetaType::NOTIFY_WAIT: | 430 | case HccLTaskMetaType::NOTIFY_WAIT: |
| 427 | - HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[NOTIFY_WAIT], notifyId[{}], srcRankId[{}], dstRankId[{}]\n", | 431 | + HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[NOTIFY_WAIT], notifyId[{}], srcRankId[{}], dstRankId[{}]", |
| 428 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); | 432 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); |
| 429 | break; | 433 | break; |
| 430 | case HccLTaskMetaType::NOTIFY_RECORD: | 434 | case HccLTaskMetaType::NOTIFY_RECORD: |
| 431 | - HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[NOTIFY_RECORD], notifyId[{}], srcRankId[{}], dstRankId[{}]\n", | 435 | + HCCL_VM_INFO("pid{}: rankId[{}] streamId[{}] taskType[NOTIFY_RECORD], notifyId[{}], srcRankId[{}], dstRankId[{}]", |
| 432 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); | 436 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); |
| 433 | break; | 437 | break; |
| 434 | default: | 438 | default: |
| @@ -14,13 +14,44 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 17 | 18 | ||
| 18 | 19 | ||
| 19 | 20 | ||
| 21 | +void ArchiveLogsAndData() | ||
| 22 | +{ | ||
| 23 | + const std::string& installRoot = InstallPath::GetHcclVmInstallAbsPath(); | ||
| 24 | + if (installRoot.empty()) | ||
| 25 | + return; | ||
| 26 | + | ||
| 27 | + std::string archiveScript = installRoot + "/script/archive.sh"; | ||
| 28 | + if (access(archiveScript.c_str(), F_OK) != 0) | ||
| 29 | + return; | ||
| 30 | + | ||
| 31 | + pid_t pid1 = fork(); | ||
| 32 | + if (pid1 == 0) { | ||
| 33 | + pid_t pid2 = fork(); | ||
| 34 | + if (pid2 == 0) { | ||
| 35 | + setsid(); | ||
| 36 | + int devNull = open("/dev/null", O_RDWR); | ||
| 37 | + dup2(devNull, STDIN_FILENO); | ||
| 38 | + dup2(devNull, STDOUT_FILENO); | ||
| 39 | + dup2(devNull, STDERR_FILENO); | ||
| 40 | + if (devNull > STDERR_FILENO) { | ||
| 41 | + close(devNull); | ||
| 42 | + } | ||
| 43 | + sleep(1); | ||
| 44 | + execlp("bash", "bash", archiveScript.c_str(), nullptr); | ||
| 45 | + _exit(1); | ||
| 46 | + } | ||
| 47 | + _exit(0); | ||
| 48 | + } | ||
| 49 | +} | ||
| 50 | + | ||
| 20 | void envInit() | 51 | void envInit() |
| 21 | { | 52 | { |
| 22 | setenv("HCCL_VM_INSTALL_ROOT", GetBinLocation().c_str(), 1); | 53 | setenv("HCCL_VM_INSTALL_ROOT", GetBinLocation().c_str(), 1); |
| 23 | - std::string dbPrefix = GetBinLocation() + "/data/hccl_vm_data.db"; | 54 | + std::string dbPrefix = InstallPath::ResolveToInstallRoot("data/hccl_vm_data.db"); |
| 24 | int ret3 = system(("sudo rm -fr " + dbPrefix + " " + dbPrefix + "-wal " + dbPrefix + "-shm 2>/dev/null").c_str()); | 55 | int ret3 = system(("sudo rm -fr " + dbPrefix + " " + dbPrefix + "-wal " + dbPrefix + "-shm 2>/dev/null").c_str()); |
| 25 | int ret1 = system("sudo rm -fr /dev/shm/* 2>/dev/null"); | 56 | int ret1 = system("sudo rm -fr /dev/shm/* 2>/dev/null"); |
| 26 | int ret2 = system("sudo rm -fr /tmp/hccl_sim.db* 2>/dev/null"); | 57 | int ret2 = system("sudo rm -fr /tmp/hccl_sim.db* 2>/dev/null"); |
| @@ -41,6 +72,7 @@ int main(int argc, char *argv[]) | |||
| 41 | LogConfig config = LoadLogConfig("hccl_vm"); | 72 | LogConfig config = LoadLogConfig("hccl_vm"); |
| 42 | InitLogger(config); | 73 | InitLogger(config); |
| 43 | envInit(); | 74 | envInit(); |
| 75 | + std::atexit(ArchiveLogsAndData); | ||
| 44 | std::string cmd = ArgvToString(argc, argv); | 76 | std::string cmd = ArgvToString(argc, argv); |
| 45 | if (argc == 1) { | 77 | if (argc == 1) { |
| 46 | cmd += " --help"; | 78 | cmd += " --help"; |
| @@ -99,3 +99,6 @@ install(FILES "manifest.json" | |||
| 99 | GROUP_READ GROUP_EXECUTE | 99 | GROUP_READ GROUP_EXECUTE |
| 100 | WORLD_READ WORLD_EXECUTE | 100 | WORLD_READ WORLD_EXECUTE |
| 101 | ) | 101 | ) |
| 102 | + | ||
| 103 | +# 日志染色: 模块 tag | ||
| 104 | +target_compile_definitions(checker PRIVATE HCCL_VM_MODULE="CHECKER") | ||
| @@ -31,6 +31,8 @@ | |||
| 31 | 31 | ||
| 32 | 32 | ||
| 33 | 33 | ||
| 34 | + | ||
| 35 | + | ||
| 34 | 36 | ||
| 35 | using json = nlohmann::json; | 37 | using json = nlohmann::json; |
| 36 | loader::Loader g_loader; | 38 | loader::Loader g_loader; |
| @@ -132,32 +134,52 @@ static const char* HcclReduceOpToString(HcclReduceOp t) { | |||
| 132 | } | 134 | } |
| 133 | } | 135 | } |
| 134 | 136 | ||
| 137 | +static void AppendApplicableRoleFields(std::ostringstream &os, const HcclSim::CheckerParam ¶m) | ||
| 138 | +{ | ||
| 139 | + switch (param.cmdType) { | ||
| 140 | + case HCCL_CMD_SEND: | ||
| 141 | + case HCCL_CMD_RECEIVE: | ||
| 142 | + os << ", sourceRank=" << param.srcRank << ", targetRank=" << param.dstRank; | ||
| 143 | + break; | ||
| 144 | + case HCCL_CMD_BROADCAST: | ||
| 145 | + case HCCL_CMD_REDUCE: | ||
| 146 | + case HCCL_CMD_SCATTER: | ||
| 147 | + os << ", rootRank=" << param.root; | ||
| 148 | + break; | ||
| 149 | + default: | ||
| 150 | + break; | ||
| 151 | + } | ||
| 152 | +} | ||
| 153 | + | ||
| 135 | static HcclResult LoadCheckerDataBase( | 154 | static HcclResult LoadCheckerDataBase( |
| 136 | std::vector<sim::CcuChannelTab>& channels, | 155 | std::vector<sim::CcuChannelTab>& channels, |
| 137 | std::vector<sim::CcuInstrResTab>& instrRes, | 156 | std::vector<sim::CcuInstrResTab>& instrRes, |
| 138 | std::vector<sim::SyncRecordTab>& syncRecords, | 157 | std::vector<sim::SyncRecordTab>& syncRecords, |
| 139 | uint32_t& syncIterMaxNum) | 158 | uint32_t& syncIterMaxNum) |
| 140 | { | 159 | { |
| 141 | - HcclResult ret = g_loader.GetCcuChannelInfo(channels); | 160 | + HcclResult ret = g_loader.GetCcuChannelInfo(channels); |
| 142 | if (ret != HcclResult::HCCL_SUCCESS) { | 161 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 143 | - HCCL_VM_ERROR("[LoadCheckerDataBase] GetCcuChannelInfo failed, ret[{}]", static_cast<u32>(ret)); | 162 | + HCCL_VM_ERROR("{} Failed to load CCU channel information", |
| 163 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR)); | ||
| 144 | return ret; | 164 | return ret; |
| 145 | } | 165 | } |
| 146 | 166 | ||
| 147 | ret = g_loader.GetInstrResInfo(instrRes); | 167 | ret = g_loader.GetInstrResInfo(instrRes); |
| 148 | if (ret != HcclResult::HCCL_SUCCESS) { | 168 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 149 | - HCCL_VM_ERROR("[LoadCheckerDataBase] GetInstrResInfo failed, ret[{}]", static_cast<u32>(ret)); | 169 | + HCCL_VM_ERROR("{} Failed to load CCU instruction resource information", |
| 170 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR)); | ||
| 150 | return ret; | 171 | return ret; |
| 151 | } | 172 | } |
| 152 | 173 | ||
| 153 | ret = g_loader.GetSyncInfo(syncRecords); | 174 | ret = g_loader.GetSyncInfo(syncRecords); |
| 154 | if (ret != HcclResult::HCCL_SUCCESS) { | 175 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 155 | - HCCL_VM_ERROR("[LoadCheckerDataBase] GetSyncInfo failed, ret[{}]", static_cast<u32>(ret)); | 176 | + HCCL_VM_ERROR("{} Failed to load sync records", |
| 177 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR)); | ||
| 156 | return ret; | 178 | return ret; |
| 157 | } | 179 | } |
| 158 | 180 | ||
| 159 | if (syncRecords.empty()) { | 181 | if (syncRecords.empty()) { |
| 160 | - HCCL_VM_WARN("[LoadCheckerDataBase] No sync records found"); | 182 | + HCCL_VM_WARN("No sync records were found"); |
| 161 | return HcclResult::HCCL_E_PARA; | 183 | return HcclResult::HCCL_E_PARA; |
| 162 | } | 184 | } |
| 163 | 185 | ||
| @@ -166,7 +188,8 @@ static HcclResult LoadCheckerDataBase( | |||
| 166 | }); | 188 | }); |
| 167 | 189 | ||
| 168 | syncIterMaxNum = syncRecords.back().syncIter; | 190 | syncIterMaxNum = syncRecords.back().syncIter; |
| 169 | - HCCL_VM_INFO("[LoadCheckerDataBase] channels={}, instrRes={}, syncRecords={}, syncIterMaxNum={}", | 191 | + HCCL_VM_INFO("Checker database loaded, channelCount={}, instrResCount={}, syncRecordCount={}, " |
| 192 | + "syncIterMaxNum={}", | ||
| 170 | channels.size(), instrRes.size(), syncRecords.size(), syncIterMaxNum); | 193 | channels.size(), instrRes.size(), syncRecords.size(), syncIterMaxNum); |
| 171 | return HcclResult::HCCL_SUCCESS; | 194 | return HcclResult::HCCL_SUCCESS; |
| 172 | } | 195 | } |
| @@ -209,11 +232,12 @@ static HcclResult DispatchCheckByCmdType( | |||
| 209 | return CheckAll2AllVC(taskQueues, param.rankSize, | 232 | return CheckAll2AllVC(taskQueues, param.rankSize, |
| 210 | static_cast<HcclDataType>(param.all2AllDataDes.sendType), param.all2AllDataDes.sendCountMatrix); | 233 | static_cast<HcclDataType>(param.all2AllDataDes.sendType), param.all2AllDataDes.sendCountMatrix); |
| 211 | case HcclCMDType::HCCL_CMD_ALLTOALLV: | 234 | case HcclCMDType::HCCL_CMD_ALLTOALLV: |
| 212 | - HCCL_VM_WARN("CheckOpSemantics does not support HCCL_CMD_ALLTOALLV, using HCCL_CMD_ALLTOALLVV instead"); | 235 | + HCCL_VM_WARN("Checker does not support AllToAllV and will use the AllToAllVC validation path"); |
| 213 | return CheckAll2AllVC(taskQueues, param.rankSize, | 236 | return CheckAll2AllVC(taskQueues, param.rankSize, |
| 214 | static_cast<HcclDataType>(param.all2AllDataDes.sendType), param.all2AllDataDes.sendCountMatrix); | 237 | static_cast<HcclDataType>(param.all2AllDataDes.sendType), param.all2AllDataDes.sendCountMatrix); |
| 215 | default: | 238 | default: |
| 216 | - HCCL_VM_ERROR("Unknown HcclCMDType {}", static_cast<u32>(cmdType)); | 239 | + HCCL_VM_ERROR("{} Unsupported collective type, collectiveTypeCode={}", |
| 240 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR), static_cast<u32>(cmdType)); | ||
| 217 | return HcclResult::HCCL_E_NOT_SUPPORT; | 241 | return HcclResult::HCCL_E_NOT_SUPPORT; |
| 218 | } | 242 | } |
| 219 | } | 243 | } |
| @@ -227,20 +251,21 @@ static HcclResult LoadOpDataForOneRank( | |||
| 227 | { | 251 | { |
| 228 | HcclResult ret = storage.LoadHcclVmSynthesisData(rankId, op.memInfo, channels); | 252 | HcclResult ret = storage.LoadHcclVmSynthesisData(rankId, op.memInfo, channels); |
| 229 | if (ret != HcclResult::HCCL_SUCCESS) { | 253 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 230 | - HCCL_VM_ERROR("[LoadOpDataForOneRank] LoadHcclVmSynthesisData rankId={} failed, ret={}", | 254 | + HCCL_VM_ERROR("{} Failed to load synthesized memory information for this rank, rankId={}", |
| 231 | - rankId, static_cast<u32>(ret)); | 255 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR), rankId); |
| 232 | return ret; | 256 | return ret; |
| 233 | } | 257 | } |
| 234 | 258 | ||
| 235 | ret = storage.LoadHcclVmInstrData(instrRes); | 259 | ret = storage.LoadHcclVmInstrData(instrRes); |
| 236 | if (ret != HcclResult::HCCL_SUCCESS) { | 260 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 237 | - HCCL_VM_ERROR("[LoadOpDataForOneRank] LoadHcclVmInstrData rankId={} failed, ret={}", | 261 | + HCCL_VM_ERROR("{} Failed to load instruction data for this rank, rankId={}", |
| 238 | - rankId, static_cast<u32>(ret)); | 262 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR), rankId); |
| 239 | return ret; | 263 | return ret; |
| 240 | } | 264 | } |
| 241 | 265 | ||
| 242 | if (op.detail.opDetail.size() < sizeof(::OpDetails)) { | 266 | if (op.detail.opDetail.size() < sizeof(::OpDetails)) { |
| 243 | - HCCL_VM_ERROR("[LoadOpDataForOneRank] opDetail BLOB too small, rankId={}", rankId); | 267 | + HCCL_VM_ERROR("{} Op detail payload is too small to parse, rankId={}", |
| 268 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR), rankId); | ||
| 244 | return HcclResult::HCCL_E_PARA; | 269 | return HcclResult::HCCL_E_PARA; |
| 245 | } | 270 | } |
| 246 | 271 | ||
| @@ -248,8 +273,8 @@ static HcclResult LoadOpDataForOneRank( | |||
| 248 | std::memcpy(&opDetails, op.detail.opDetail.data(), sizeof(::OpDetails)); | 273 | std::memcpy(&opDetails, op.detail.opDetail.data(), sizeof(::OpDetails)); |
| 249 | ret = storage.Trans2CheckerParam(op.detail, opDetails); | 274 | ret = storage.Trans2CheckerParam(op.detail, opDetails); |
| 250 | if (ret != HcclResult::HCCL_SUCCESS) { | 275 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 251 | - HCCL_VM_ERROR("[LoadOpDataForOneRank] Trans2CheckerParam rankId={} failed, ret={}", | 276 | + HCCL_VM_ERROR("{} Failed to convert this rank into checker input parameters, rankId={}", |
| 252 | - rankId, static_cast<u32>(ret)); | 277 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR), rankId); |
| 253 | return ret; | 278 | return ret; |
| 254 | } | 279 | } |
| 255 | return HcclResult::HCCL_SUCCESS; | 280 | return HcclResult::HCCL_SUCCESS; |
| @@ -272,17 +297,19 @@ static HcclResult ProcessOneOpGroup( | |||
| 272 | bool enableOldChecker = settingManager.IsOldCheckerEnabled(); | 297 | bool enableOldChecker = settingManager.IsOldCheckerEnabled(); |
| 273 | bool isAivOp = false; | 298 | bool isAivOp = false; |
| 274 | 299 | ||
| 275 | - HCCL_VM_INFO("[RunChecker] opgroup size {}", opGroup.size()); | 300 | + HCCL_VM_INFO("Start checking one op group, opGroupSize={}", opGroup.size()); |
| 276 | std::vector<std::vector<sim::OpTaskTab>> allTasks; | 301 | std::vector<std::vector<sim::OpTaskTab>> allTasks; |
| 277 | for (auto& entry : opGroup) { | 302 | for (auto& entry : opGroup) { |
| 278 | uint32_t rankId = entry.first; | 303 | uint32_t rankId = entry.first; |
| 279 | - HCCL_VM_INFO("[ProcessOneOpGroup] rankId={}", rankId); | 304 | + HCCL_VM_INFO("Load one rank from this op group, rankId={}", rankId); |
| 280 | sim::CompositeOpDetail& op = entry.second; | 305 | sim::CompositeOpDetail& op = entry.second; |
| 281 | isAivOp = isAivOp || IsAivOpExpansionMode(op.detail.opExpansionMode); | 306 | isAivOp = isAivOp || IsAivOpExpansionMode(op.detail.opExpansionMode); |
| 282 | allTasks.push_back(op.tasks); | 307 | allTasks.push_back(op.tasks); |
| 283 | - HcclResult r = LoadOpDataForOneRank(storage, channels, instrRes, rankId, op); | 308 | + HcclResult ret = LoadOpDataForOneRank(storage, channels, instrRes, rankId, op); |
| 284 | - if (r != HcclResult::HCCL_SUCCESS) { | 309 | + if (ret != HcclResult::HCCL_SUCCESS) { |
| 285 | - return r; | 310 | + HCCL_VM_ERROR("{} Failed to load one rank from this op group, opIndex={}, rankId={}", |
| 311 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR), opIdx, rankId); | ||
| 312 | + return ret; | ||
| 286 | } | 313 | } |
| 287 | if (dumpManager.IsEnabled()) { | 314 | if (dumpManager.IsEnabled()) { |
| 288 | HcclSim::DumpRunManifest::GetInstance().SetOpParam(BuildOpParamSummaryJson(storage.GetCheckerParam())); | 315 | HcclSim::DumpRunManifest::GetInstance().SetOpParam(BuildOpParamSummaryJson(storage.GetCheckerParam())); |
| @@ -290,7 +317,10 @@ static HcclResult ProcessOneOpGroup( | |||
| 290 | } | 317 | } |
| 291 | 318 | ||
| 292 | if (!enableNewChecker && !enableOldChecker) { | 319 | if (!enableNewChecker && !enableOldChecker) { |
| 293 | - HCCL_VM_WARN("[ProcessOneOpGroup] both new checker and old checker are disabled, skip op[{}].", opIdx); | 320 | + HCCL_VM_ERROR("{} This op is skipped because both the new checker and the old checker are disabled, " |
| 321 | + "opIndex={}, newCheckerEnabled={}, oldCheckerEnabled={}", | ||
| 322 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::SETTING_WARNING), opIdx, enableNewChecker, | ||
| 323 | + enableOldChecker); | ||
| 294 | return HcclResult::HCCL_SUCCESS; | 324 | return HcclResult::HCCL_SUCCESS; |
| 295 | } | 325 | } |
| 296 | 326 | ||
| @@ -299,18 +329,21 @@ static HcclResult ProcessOneOpGroup( | |||
| 299 | 329 | ||
| 300 | auto ret = storage.LoadHcclVmTaskMetaData(allTasks); | 330 | auto ret = storage.LoadHcclVmTaskMetaData(allTasks); |
| 301 | if (ret != HcclResult::HCCL_SUCCESS) { | 331 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 302 | - HCCL_VM_ERROR("[ProcessOneOpGroup] LoadHcclVmTaskMetaData failed, ret={}", | 332 | + HCCL_VM_ERROR("{} Failed to load V3 task metadata for this op group", |
| 303 | - static_cast<u32>(ret)); | 333 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::CHECKER_RUNTIME_ERROR)); |
| 304 | return ret; | 334 | return ret; |
| 305 | } | 335 | } |
| 306 | isAivOp = isAivOp || HasAivGraphTask(allTasks); | 336 | isAivOp = isAivOp || HasAivGraphTask(allTasks); |
| 307 | if (isAivOp) { | 337 | if (isAivOp) { |
| 308 | if (!enableNewChecker) { | 338 | if (!enableNewChecker) { |
| 309 | - HCCL_VM_ERROR("[ProcessOneOpGroup] AIV checker requires V3 checker, but new checker is disabled."); | 339 | + HCCL_VM_ERROR("{} This AIV op requires the V3 checker, but V3 is disabled by configuration, " |
| 340 | + "opIndex={}, action=abort, newCheckerEnabled={}", | ||
| 341 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::SETTING_WARNING), opIdx, enableNewChecker); | ||
| 310 | return HcclResult::HCCL_E_NOT_SUPPORT; | 342 | return HcclResult::HCCL_E_NOT_SUPPORT; |
| 311 | } | 343 | } |
| 312 | if (enableOldChecker) { | 344 | if (enableOldChecker) { |
| 313 | - HCCL_VM_INFO("[ProcessOneOpGroup] AIV op detected, skip legacy checker and use V3-only."); | 345 | + HCCL_VM_WARN("AIV op detected, the old checker is skipped and only CheckerV3 will run, " |
| 346 | + "opIndex={}, oldCheckerEnabled={}", opIdx, enableOldChecker); | ||
| 314 | enableOldChecker = false; | 347 | enableOldChecker = false; |
| 315 | } | 348 | } |
| 316 | enableNewChecker = true; | 349 | enableNewChecker = true; |
| @@ -318,41 +351,47 @@ static HcclResult ProcessOneOpGroup( | |||
| 318 | 351 | ||
| 319 | { | 352 | { |
| 320 | auto checkerParamBrief = storage.GetCheckerParam(); | 353 | auto checkerParamBrief = storage.GetCheckerParam(); |
| 321 | - HCCL_VM_INFO("[Checker][OpInfo] op[{}] op = {}, rankSize = {}, dataType = {}, dataCount = {}, reduceType = {}, " | 354 | + std::ostringstream summary; |
| 322 | - "srcRank = {}, dstRank = {}, root = {}, opGroupSize={}, isAivOp={}", | 355 | + summary << "[Main] Op summary, opIndex=" << opIdx |
| 323 | - opIdx, HcclCmdTypeToString(checkerParamBrief.cmdType), checkerParamBrief.rankSize, | 356 | + << ", collectiveType=" << HcclSim::HcclCmdTypeToString(checkerParamBrief.cmdType) |
| 324 | - HcclDataTypeToString(checkerParamBrief.dataType), checkerParamBrief.dataCount, | 357 | + << ", rankCount=" << checkerParamBrief.rankSize |
| 325 | - HcclReduceOpToString(checkerParamBrief.reduceType), | 358 | + << ", dataType=" << HcclSim::HcclDataTypeToString(checkerParamBrief.dataType) |
| 326 | - checkerParamBrief.srcRank, checkerParamBrief.dstRank, checkerParamBrief.root, | 359 | + << ", elementCount=" << checkerParamBrief.dataCount |
| 327 | - opGroup.size(), isAivOp); | 360 | + << ", reduceType=" << HcclReduceOpToString(checkerParamBrief.reduceType); |
| 361 | + AppendApplicableRoleFields(summary, checkerParamBrief); | ||
| 362 | + summary << ", opGroupSize=" << opGroup.size() | ||
| 363 | + << ", containsAivTask=" << isAivOp; | ||
| 364 | + HCCL_VM_INFO("{}", summary.str()); | ||
| 328 | } | 365 | } |
| 329 | 366 | ||
| 330 | HcclResult newCheckerRet = HcclResult::HCCL_SUCCESS; | 367 | HcclResult newCheckerRet = HcclResult::HCCL_SUCCESS; |
| 331 | if (enableNewChecker) { | 368 | if (enableNewChecker) { |
| 369 | + HCCL_VM_INFO("----------[Start CheckerV3]----------"); | ||
| 332 | newCheckerRet = HcclSim::GenAndCheckGraphV3(); | 370 | newCheckerRet = HcclSim::GenAndCheckGraphV3(); |
| 333 | - HCCL_VM_WARN("----------[V3 Checker Finished]----------"); | 371 | + HCCL_VM_INFO("----------[CheckerV3 Finished]----------"); |
| 334 | - HCCL_VM_INFO("[ProcessOneOpGroup] new checker finished, ret={}", static_cast<u32>(newCheckerRet)); | 372 | + HCCL_VM_INFO("CheckerV3 finished for this op, opIndex={}", opIdx); |
| 335 | } else { | 373 | } else { |
| 336 | - HCCL_VM_INFO("[ProcessOneOpGroup] new checker disabled by manifest setting."); | 374 | + HCCL_VM_INFO("CheckerV3 is disabled by configuration"); |
| 337 | } | 375 | } |
| 338 | 376 | ||
| 339 | HcclResult oldCheckerRet = HcclResult::HCCL_SUCCESS; | 377 | HcclResult oldCheckerRet = HcclResult::HCCL_SUCCESS; |
| 340 | if (enableOldChecker) { | 378 | if (enableOldChecker) { |
| 341 | - HCCL_VM_WARN("----------[Start Old Checker]----------"); | 379 | + HCCL_VM_INFO("----------[Start Old Checker]----------"); |
| 380 | + HCCL_VM_INFO("Start running the old checker, opIndex={}, dataId={}", opIdx, | ||
| 381 | + storage.GetDataId()); | ||
| 342 | HcclSim::AllRankTaskQueues taskQueues; | 382 | HcclSim::AllRankTaskQueues taskQueues; |
| 343 | HcclSim::ConvertTaskQueue(taskQueues); | 383 | HcclSim::ConvertTaskQueue(taskQueues); |
| 344 | auto checkerParam = storage.GetCheckerParam(); | 384 | auto checkerParam = storage.GetCheckerParam(); |
| 345 | - HCCL_VM_INFO("[ProcessOneOpGroup] dispatch legacy checker, rankCount={}, cmdType={}", | ||
| 346 | - taskQueues.size(), static_cast<u32>(checkerParam.cmdType)); | ||
| 347 | oldCheckerRet = DispatchCheckByCmdType(taskQueues, checkerParam); | 385 | oldCheckerRet = DispatchCheckByCmdType(taskQueues, checkerParam); |
| 348 | - HCCL_VM_INFO("[ProcessOneOpGroup] old checker finished, ret={}", static_cast<u32>(oldCheckerRet)); | 386 | + HCCL_VM_INFO("----------[Old Checker Finished]----------"); |
| 387 | + HCCL_VM_INFO("Old checker finished for this op, opIndex={}", opIdx); | ||
| 349 | } else { | 388 | } else { |
| 350 | - HCCL_VM_INFO("[ProcessOneOpGroup] old checker disabled by manifest setting."); | 389 | + HCCL_VM_INFO("Old checker is disabled by configuration"); |
| 351 | } | 390 | } |
| 352 | 391 | ||
| 353 | if (!isAivOp && enableNewChecker && enableOldChecker && newCheckerRet != oldCheckerRet) { | 392 | if (!isAivOp && enableNewChecker && enableOldChecker && newCheckerRet != oldCheckerRet) { |
| 354 | - HCCL_VM_WARN("[ProcessOneOpGroup] new checker and old checker results differ, newCheckerRet={}, oldCheckerRet={}", | 393 | + HCCL_VM_WARN("CheckerV3 result differs from old checker result, opIndex={}, checkerV3Ret={}, " |
| 355 | - static_cast<u32>(newCheckerRet), static_cast<u32>(oldCheckerRet)); | 394 | + "oldCheckerRet={}", opIdx, static_cast<u32>(newCheckerRet), static_cast<u32>(oldCheckerRet)); |
| 356 | } | 395 | } |
| 357 | if (enableNewChecker) { | 396 | if (enableNewChecker) { |
| 358 | return newCheckerRet; | 397 | return newCheckerRet; |
| @@ -420,21 +459,22 @@ void RunChecker(const std::string& data_id) { | |||
| 420 | storage.SetDataId(data_id); | 459 | storage.SetDataId(data_id); |
| 421 | const HcclResult settingRefreshRet = HcclSim::SettingManager::GetInstance().Refresh(); | 460 | const HcclResult settingRefreshRet = HcclSim::SettingManager::GetInstance().Refresh(); |
| 422 | if (settingRefreshRet != HcclResult::HCCL_SUCCESS) { | 461 | if (settingRefreshRet != HcclResult::HCCL_SUCCESS) { |
| 423 | - HCCL_VM_WARN("[RunChecker] failed to refresh manifest settings, keep previous settings, ret = {}", | 462 | + HCCL_VM_WARN("Failed to refresh manifest settings, the previous checker settings will be kept"); |
| 424 | - static_cast<u32>(settingRefreshRet)); | ||
| 425 | } | 463 | } |
| 426 | HcclSim::DumpManager &dumpManager = HcclSim::DumpManager::GetInstance(); | 464 | HcclSim::DumpManager &dumpManager = HcclSim::DumpManager::GetInstance(); |
| 427 | dumpManager.Reset(); | 465 | dumpManager.Reset(); |
| 428 | HcclResult dumpInitRet = dumpManager.Initialize(data_id); | 466 | HcclResult dumpInitRet = dumpManager.Initialize(data_id); |
| 429 | if (dumpInitRet != HcclResult::HCCL_SUCCESS) { | 467 | if (dumpInitRet != HcclResult::HCCL_SUCCESS) { |
| 430 | - HCCL_VM_ERROR("[RunChecker] DumpManager initialize failed, ret = {}", static_cast<u32>(dumpInitRet)); | 468 | + HCCL_VM_ERROR("{} Failed to initialize the old checker dump manager, old checker output files " |
| 469 | + "cannot be written, dataId={}", HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::DUMP_FAILED), data_id); | ||
| 431 | return; | 470 | return; |
| 432 | } | 471 | } |
| 433 | HcclSim::DumpV3Manager &dumpV3Manager = HcclSim::DumpV3Manager::GetInstance(); | 472 | HcclSim::DumpV3Manager &dumpV3Manager = HcclSim::DumpV3Manager::GetInstance(); |
| 434 | dumpV3Manager.Reset(); | 473 | dumpV3Manager.Reset(); |
| 435 | dumpInitRet = dumpV3Manager.Initialize(data_id); | 474 | dumpInitRet = dumpV3Manager.Initialize(data_id); |
| 436 | if (dumpInitRet != HcclResult::HCCL_SUCCESS) { | 475 | if (dumpInitRet != HcclResult::HCCL_SUCCESS) { |
| 437 | - HCCL_VM_ERROR("[RunChecker] DumpV3Manager initialize failed, ret = {}", static_cast<u32>(dumpInitRet)); | 476 | + HCCL_VM_ERROR("{} Failed to initialize the V3 dump manager, checker output files cannot be written, " |
| 477 | + "dataId={}", HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::DUMP_FAILED), data_id); | ||
| 438 | return; | 478 | return; |
| 439 | } | 479 | } |
| 440 | HcclSim::DumpRunManifest::GetInstance().Reset(data_id); | 480 | HcclSim::DumpRunManifest::GetInstance().Reset(data_id); |
| @@ -446,7 +486,7 @@ void RunChecker(const std::string& data_id) { | |||
| 446 | 486 | ||
| 447 | HcclResult loadRet = g_loader.LoadOpTaskFile(); | 487 | HcclResult loadRet = g_loader.LoadOpTaskFile(); |
| 448 | if (loadRet != HcclResult::HCCL_SUCCESS) { | 488 | if (loadRet != HcclResult::HCCL_SUCCESS) { |
| 449 | - HCCL_VM_ERROR("[RunChecker] LoadOpTaskFile failed, ret = {}", static_cast<u32>(loadRet)); | 489 | + HCCL_VM_ERROR("Failed to load the op task file"); |
| 450 | return; | 490 | return; |
| 451 | } | 491 | } |
| 452 | 492 | ||
| @@ -459,35 +499,36 @@ void RunChecker(const std::string& data_id) { | |||
| 459 | return; | 499 | return; |
| 460 | } | 500 | } |
| 461 | 501 | ||
| 462 | - HCCL_VM_INFO("[RunChecker] start check semantic: syncRecords size = {}", syncRecords.size()); | 502 | + HCCL_VM_INFO("Start checker run, syncRecordCount={}", syncRecords.size()); |
| 463 | uint32_t opIdx = 0; | 503 | uint32_t opIdx = 0; |
| 464 | do { | 504 | do { |
| 465 | for (uint32_t syncIter = 0; syncIter <= syncIterMaxNum; syncIter++) { | 505 | for (uint32_t syncIter = 0; syncIter <= syncIterMaxNum; syncIter++) { |
| 466 | std::map<uint32_t, std::vector<sim::CompositeOpDetail>> compositeDataMap; | 506 | std::map<uint32_t, std::vector<sim::CompositeOpDetail>> compositeDataMap; |
| 467 | g_loader.LoadCompositeOpDetailBySyncIter(syncIter, compositeDataMap); | 507 | g_loader.LoadCompositeOpDetailBySyncIter(syncIter, compositeDataMap); |
| 468 | auto opGroups = TransposeCompositeOpMap(compositeDataMap); | 508 | auto opGroups = TransposeCompositeOpMap(compositeDataMap); |
| 469 | - HCCL_VM_INFO("[RunChecker] syncIter={}, opGroups={}", syncIter, opGroups.size()); | 509 | + HCCL_VM_INFO("Start one sync iteration, syncIter={}, opGroupCount={}", syncIter, opGroups.size()); |
| 470 | for (auto& opGroup : opGroups) { | 510 | for (auto& opGroup : opGroups) { |
| 471 | - HCCL_VM_INFO("[RunChecker] opgroup size {}", opGroup.size()); | 511 | + HCCL_VM_INFO("Check one op group in this sync iteration, opGroupSize={}", opGroup.size()); |
| 472 | ret = ProcessOneOpGroup(storage, channels, instrRes, opIdx, opGroup); | 512 | ret = ProcessOneOpGroup(storage, channels, instrRes, opIdx, opGroup); |
| 473 | if (dumpManager.IsEnabled()) { | 513 | if (dumpManager.IsEnabled()) { |
| 474 | HcclSim::DumpRunManifest::GetInstance().SetCheckResult(ret); | 514 | HcclSim::DumpRunManifest::GetInstance().SetCheckResult(ret); |
| 475 | const HcclResult flushRet = HcclSim::ValidationIssueRecorder::GetInstance().Flush(); | 515 | const HcclResult flushRet = HcclSim::ValidationIssueRecorder::GetInstance().Flush(); |
| 476 | if (flushRet != HcclResult::HCCL_SUCCESS) { | 516 | if (flushRet != HcclResult::HCCL_SUCCESS) { |
| 477 | - HCCL_VM_WARN("[RunChecker] failed to flush validation issue dump for op[{}], ret={}", | 517 | + HCCL_VM_WARN("{} Failed to flush the validation issue dump, dataId={}, opIndex={}, " |
| 478 | - opIdx, static_cast<u32>(flushRet)); | 518 | + "dumpType=validation_issues", HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::DUMP_FAILED), |
| 519 | + data_id, opIdx); | ||
| 479 | } | 520 | } |
| 480 | const HcclResult manifestRet = HcclSim::DumpRunManifest::GetInstance().Flush(); | 521 | const HcclResult manifestRet = HcclSim::DumpRunManifest::GetInstance().Flush(); |
| 481 | if (manifestRet != HcclResult::HCCL_SUCCESS) { | 522 | if (manifestRet != HcclResult::HCCL_SUCCESS) { |
| 482 | - HCCL_VM_WARN("[RunChecker] failed to flush dump manifest for op[{}], ret={}", | 523 | + HCCL_VM_WARN("{} Failed to flush the dump manifest, dataId={}, opIndex={}", |
| 483 | - opIdx, static_cast<u32>(manifestRet)); | 524 | + HcclSim::MakeErrorCodeText(HcclSim::ErrorCode::DUMP_FAILED), data_id, opIdx); |
| 484 | } | 525 | } |
| 485 | } | 526 | } |
| 486 | if (ret != HcclResult::HCCL_SUCCESS) { | 527 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 487 | - HCCL_VM_ERROR("[RunChecker] op[{}] failed with error code {}", opIdx, static_cast<u32>(ret)); | 528 | + HCCL_VM_ERROR("op[{}] Checker failed", opIdx); |
| 488 | break; | 529 | break; |
| 489 | } | 530 | } |
| 490 | - HCCL_VM_INFO("[RunChecker] op[{}] Checker Success.", opIdx); | 531 | + HCCL_VM_INFO("op[{}] Checker Success", opIdx); |
| 491 | opIdx++; | 532 | opIdx++; |
| 492 | } | 533 | } |
| 493 | } | 534 | } |
| @@ -518,30 +559,31 @@ void JoinCheckerWorker() | |||
| 518 | void ProcessCommand(const std::string& line) { | 559 | void ProcessCommand(const std::string& line) { |
| 519 | try { | 560 | try { |
| 520 | auto j = json::parse(line); | 561 | auto j = json::parse(line); |
| 521 | - const HcclResult settingRefreshRet = HcclSim::SettingManager::GetInstance().Refresh(); | ||
| 522 | - if (settingRefreshRet != HcclResult::HCCL_SUCCESS) { | ||
| 523 | - HCCL_VM_WARN("[Checker] failed to refresh manifest settings, keep previous settings, ret = {}", | ||
| 524 | - static_cast<u32>(settingRefreshRet)); | ||
| 525 | - } | ||
| 526 | std::string action = j.value("action", ""); | 562 | std::string action = j.value("action", ""); |
| 527 | auto payload = j.value("payload", json::object()); | 563 | auto payload = j.value("payload", json::object()); |
| 528 | 564 | ||
| 529 | if (action == "status") { | 565 | if (action == "status") { |
| 530 | std::string status = payload.value("status", ""); | 566 | std::string status = payload.value("status", ""); |
| 531 | - HCCL_VM_INFO("[Checker] Status signal received. Status: {}", status); | 567 | + HCCL_VM_INFO("Received checker status signal, status={}", status); |
| 532 | if (status != "finish") { | 568 | if (status != "finish") { |
| 533 | return; | 569 | return; |
| 534 | } | 570 | } |
| 571 | + // 运行前刷新设置,确保最新的配置生效 | ||
| 572 | + const HcclResult settingRefreshRet = HcclSim::SettingManager::GetInstance().Refresh(); | ||
| 573 | + if (settingRefreshRet != HcclResult::HCCL_SUCCESS) { | ||
| 574 | + HCCL_VM_WARN("Failed to refresh manifest settings, use the previous settings"); | ||
| 575 | + } | ||
| 576 | + | ||
| 535 | std::string data_id = payload.value("data_id", ""); | 577 | std::string data_id = payload.value("data_id", ""); |
| 536 | StartCheckerWorker(data_id); | 578 | StartCheckerWorker(data_id); |
| 537 | } | 579 | } |
| 538 | else if (action == "stop") { | 580 | else if (action == "stop") { |
| 539 | - HCCL_VM_INFO("[Checker] Stop signal received. Initiating shutdown..."); | 581 | + HCCL_VM_INFO("Received checker stop signal, shutdown..."); |
| 540 | g_keep_running.store(false); // 仅仅修改标志位 | 582 | g_keep_running.store(false); // 仅仅修改标志位 |
| 541 | } | 583 | } |
| 542 | // 其他 action... | 584 | // 其他 action... |
| 543 | } catch (const std::exception& e) { | 585 | } catch (const std::exception& e) { |
| 544 | - std::cerr << "[Error] Command processing failed: " << e.what() << std::endl; | 586 | + HCCL_VM_ERROR("Command processing failed: {}", e.what()); |
| 545 | } | 587 | } |
| 546 | } | 588 | } |
| 547 | 589 | ||
| @@ -549,7 +591,7 @@ int main() { | |||
| 549 | LogConfig config = LoadLogConfig("checker"); | 591 | LogConfig config = LoadLogConfig("checker"); |
| 550 | InitLogger(config); | 592 | InitLogger(config); |
| 551 | 593 | ||
| 552 | - HCCL_VM_INFO("[Checker] Plugin process active. Listening for commands..."); | 594 | + HCCL_VM_INFO("Plugin process active. Listening for commands..."); |
| 553 | 595 | ||
| 554 | // 主循环检查标志位 | 596 | // 主循环检查标志位 |
| 555 | std::string line; | 597 | std::string line; |
| @@ -562,6 +604,6 @@ int main() { | |||
| 562 | } | 604 | } |
| 563 | 605 | ||
| 564 | JoinCheckerWorker(); | 606 | JoinCheckerWorker(); |
| 565 | - HCCL_VM_INFO("[Checker] shutdown."); | 607 | + HCCL_VM_INFO("shutdown."); |
| 566 | return 0; // 整个进程唯一的正常出口 | 608 | return 0; // 整个进程唯一的正常出口 |
| 567 | } | 609 | } |
| @@ -33,6 +33,7 @@ | |||
| 33 | 33 | ||
| 34 | 34 | ||
| 35 | 35 | ||
| 36 | + | ||
| 36 | 37 | ||
| 37 | 38 | ||
| 38 | 39 | ||
| @@ -51,16 +52,15 @@ | |||
| 51 | 52 | ||
| 52 | 53 | ||
| 53 | 54 | ||
| 55 | + | ||
| 54 | 56 | ||
| 55 | using namespace std; | 57 | using namespace std; |
| 56 | 58 | ||
| 57 | namespace HcclSim { | 59 | namespace HcclSim { |
| 58 | namespace { | 60 | namespace { |
| 59 | using V3TaskNode = TaskGraphGeneratorV3::TaskNode; | 61 | using V3TaskNode = TaskGraphGeneratorV3::TaskNode; |
| 60 | -using OldNodeStreamPosMap = std::map<TaskNodePtr, uint32_t>; | ||
| 61 | using V3NodeStreamPosMap = std::map<const V3TaskNode *, uint32_t>; | 62 | using V3NodeStreamPosMap = std::map<const V3TaskNode *, uint32_t>; |
| 62 | using V3StageClock = std::chrono::steady_clock; | 63 | using V3StageClock = std::chrono::steady_clock; |
| 63 | -constexpr size_t MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT = 20; | ||
| 64 | constexpr uint32_t INVALID_GRAPH_STREAM_POS = std::numeric_limits<uint32_t>::max(); | 64 | constexpr uint32_t INVALID_GRAPH_STREAM_POS = std::numeric_limits<uint32_t>::max(); |
| 65 | constexpr uint64_t V3_NS_PER_MS = 1000000ULL; | 65 | constexpr uint64_t V3_NS_PER_MS = 1000000ULL; |
| 66 | 66 | ||
| @@ -80,11 +80,11 @@ public: | |||
| 80 | { | 80 | { |
| 81 | const uint64_t elapsedNs = ElapsedV3Ns(start_, V3StageClock::now()); | 81 | const uint64_t elapsedNs = ElapsedV3Ns(start_, V3StageClock::now()); |
| 82 | if (extraInfo_.empty()) { | 82 | if (extraInfo_.empty()) { |
| 83 | - HCCL_VM_WARN("[TaskGraphGeneratorV3][StageTime] {} {}, costMs={}", stage_, status_, | 83 | + HCCL_VM_INFO("CheckerV3 stage finished, stage={}, status={}, costMs={}", stage_, status_, |
| 84 | elapsedNs / V3_NS_PER_MS); | 84 | elapsedNs / V3_NS_PER_MS); |
| 85 | return; | 85 | return; |
| 86 | } | 86 | } |
| 87 | - HCCL_VM_WARN("[TaskGraphGeneratorV3][StageTime] {} {}, costMs={}, {}", stage_, status_, | 87 | + HCCL_VM_INFO("CheckerV3 stage finished, stage={}, status={}, costMs={}, {}", stage_, status_, |
| 88 | elapsedNs / V3_NS_PER_MS, extraInfo_); | 88 | elapsedNs / V3_NS_PER_MS, extraInfo_); |
| 89 | } | 89 | } |
| 90 | 90 | ||
| @@ -118,15 +118,6 @@ struct V3StreamKey { | |||
| 118 | } | 118 | } |
| 119 | }; | 119 | }; |
| 120 | 120 | ||
| 121 | -bool IsOldGraphCompareTransparent(TaskNodePtr node) | ||
| 122 | -{ | ||
| 123 | - if (node == nullptr || node->task == nullptr) { | ||
| 124 | - return false; | ||
| 125 | - } | ||
| 126 | - const TaskTypeStub taskType = node->task->GetType(); | ||
| 127 | - return taskType == TaskTypeStub::GRAPH_SEPARATE || taskType == TaskTypeStub::SUB_GRAPH_END; | ||
| 128 | -} | ||
| 129 | - | ||
| 130 | bool IsV3Boundary(const V3TaskNode *node, TaskGraphGeneratorV3::BoundaryType boundaryType) | 121 | bool IsV3Boundary(const V3TaskNode *node, TaskGraphGeneratorV3::BoundaryType boundaryType) |
| 131 | { | 122 | { |
| 132 | if (node == nullptr) { | 123 | if (node == nullptr) { |
| @@ -143,7 +134,7 @@ bool IsV3Boundary(const V3TaskNode *node, TaskGraphGeneratorV3::BoundaryType bou | |||
| 143 | return false; | 134 | return false; |
| 144 | } | 135 | } |
| 145 | 136 | ||
| 146 | -bool IsV3GraphCompareTransparent(const V3TaskNode *node) | 137 | +bool IsV3StreamPosHiddenNode(const V3TaskNode *node) |
| 147 | { | 138 | { |
| 148 | if (node == nullptr) { | 139 | if (node == nullptr) { |
| 149 | return false; | 140 | return false; |
| @@ -203,32 +194,6 @@ bool IsAivDagLogNode(const V3TaskNode *node) | |||
| 203 | IsV3Boundary(node, TaskGraphGeneratorV3::BoundaryType::AIV_SUB_GRAPH); | 194 | IsV3Boundary(node, TaskGraphGeneratorV3::BoundaryType::AIV_SUB_GRAPH); |
| 204 | } | 195 | } |
| 205 | 196 | ||
| 206 | -std::vector<TaskNodePtr> BuildOldGraphCompareNodes(const std::vector<TaskNodePtr> &oldBfsNodes) | ||
| 207 | -{ | ||
| 208 | - std::vector<TaskNodePtr> result; | ||
| 209 | - result.reserve(oldBfsNodes.size()); | ||
| 210 | - for (TaskNodePtr node : oldBfsNodes) { | ||
| 211 | - if (IsOldGraphCompareTransparent(node)) { | ||
| 212 | - continue; | ||
| 213 | - } | ||
| 214 | - result.push_back(node); | ||
| 215 | - } | ||
| 216 | - return result; | ||
| 217 | -} | ||
| 218 | - | ||
| 219 | -std::vector<const V3TaskNode *> BuildV3GraphCompareNodes(const std::vector<const V3TaskNode *> &newBfsNodes) | ||
| 220 | -{ | ||
| 221 | - std::vector<const V3TaskNode *> result; | ||
| 222 | - result.reserve(newBfsNodes.size()); | ||
| 223 | - for (const V3TaskNode *node : newBfsNodes) { | ||
| 224 | - if (IsV3GraphCompareTransparent(node)) { | ||
| 225 | - continue; | ||
| 226 | - } | ||
| 227 | - result.push_back(node); | ||
| 228 | - } | ||
| 229 | - return result; | ||
| 230 | -} | ||
| 231 | - | ||
| 232 | std::string EscapeGraphvizLabel(const std::string &label) | 197 | std::string EscapeGraphvizLabel(const std::string &label) |
| 233 | { | 198 | { |
| 234 | std::string escaped; | 199 | std::string escaped; |
| @@ -263,7 +228,7 @@ std::string FormatGraphvizLabelText(const std::string &text) | |||
| 263 | 228 | ||
| 264 | std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> GenGraphV3FromTaskMeta() | 229 | std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> GenGraphV3FromTaskMeta() |
| 265 | { | 230 | { |
| 266 | - V3StageTimer timer("GenGraphV3FromTaskMeta"); | 231 | + V3StageTimer timer("GenGraph"); |
| 267 | StorageManager &storage = StorageManager::GetInstance(); | 232 | StorageManager &storage = StorageManager::GetInstance(); |
| 268 | TaskGraphGeneratorV3::TaskMetaTranslatorV3 taskMetaTranslatorV3; | 233 | TaskGraphGeneratorV3::TaskMetaTranslatorV3 taskMetaTranslatorV3; |
| 269 | // V3 主流程入口: | 234 | // V3 主流程入口: |
| @@ -272,8 +237,8 @@ std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> GenGraphV3FromTaskMe | |||
| 272 | // 3. 再生成 V3 图并在日志里记录成图/CCU 展开耗时。 | 237 | // 3. 再生成 V3 图并在日志里记录成图/CCU 展开耗时。 |
| 273 | HcclResult ret = taskMetaTranslatorV3.Translate(storage); | 238 | HcclResult ret = taskMetaTranslatorV3.Translate(storage); |
| 274 | if (ret != HCCL_SUCCESS) { | 239 | if (ret != HCCL_SUCCESS) { |
| 275 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Translate task meta failed, skip V3 graph generation, ret={}", | 240 | + HCCL_VM_WARN("Failed to translate one task into the V3 internal format, V3 graph generation is " |
| 276 | - static_cast<uint32_t>(ret)); | 241 | + "stopped, ret={}", static_cast<uint32_t>(ret)); |
| 277 | return nullptr; | 242 | return nullptr; |
| 278 | } | 243 | } |
| 279 | 244 | ||
| @@ -281,21 +246,21 @@ std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> GenGraphV3FromTaskMe | |||
| 281 | auto translatedTaskQueues = taskMetaTranslatorV3.TakeTaskQueues(); | 246 | auto translatedTaskQueues = taskMetaTranslatorV3.TakeTaskQueues(); |
| 282 | ret = TaskGraphGeneratorV3::CheckSlaveTaskQueue(translatedNodes, translatedTaskQueues); | 247 | ret = TaskGraphGeneratorV3::CheckSlaveTaskQueue(translatedNodes, translatedTaskQueues); |
| 283 | if (ret != HCCL_SUCCESS) { | 248 | if (ret != HCCL_SUCCESS) { |
| 284 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Check slave task queue failed, skip V3 graph generation, ret={}", | 249 | + HCCL_VM_WARN("Slave-stream check failed, V3 graph generation is stopped, ret={}", |
| 285 | static_cast<uint32_t>(ret)); | 250 | static_cast<uint32_t>(ret)); |
| 286 | return nullptr; | 251 | return nullptr; |
| 287 | } | 252 | } |
| 288 | std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> graphGeneratorV3( | 253 | std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> graphGeneratorV3( |
| 289 | new (std::nothrow) TaskGraphGeneratorV3::TaskGraphGeneratorV3); | 254 | new (std::nothrow) TaskGraphGeneratorV3::TaskGraphGeneratorV3); |
| 290 | if (graphGeneratorV3 == nullptr) { | 255 | if (graphGeneratorV3 == nullptr) { |
| 291 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Create graph generator failed."); | 256 | + HCCL_VM_WARN("Failed to create the V3 graph generator object"); |
| 292 | return nullptr; | 257 | return nullptr; |
| 293 | } | 258 | } |
| 294 | 259 | ||
| 295 | ret = graphGeneratorV3->GenGraph(std::move(translatedNodes), std::move(translatedTaskQueues)); | 260 | ret = graphGeneratorV3->GenGraph(std::move(translatedNodes), std::move(translatedTaskQueues)); |
| 296 | if (ret != HCCL_SUCCESS) { | 261 | if (ret != HCCL_SUCCESS) { |
| 297 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Generate graph from translated task nodes failed, ret={}", | 262 | + HCCL_VM_WARN("Failed to build the graph from translated tasks, ret={}, rankCount={}, nodeCount={}", |
| 298 | - static_cast<uint32_t>(ret)); | 263 | + static_cast<uint32_t>(ret), graphGeneratorV3->GetTaskQueues().size(), graphGeneratorV3->GetNodes().size()); |
| 299 | return nullptr; | 264 | return nullptr; |
| 300 | } | 265 | } |
| 301 | const auto &ccuExpandStats = graphGeneratorV3->GetCcuExpandStats(); | 266 | const auto &ccuExpandStats = graphGeneratorV3->GetCcuExpandStats(); |
| @@ -312,35 +277,6 @@ std::unique_ptr<TaskGraphGeneratorV3::TaskGraphGeneratorV3> GenGraphV3FromTaskMe | |||
| 312 | return graphGeneratorV3; | 277 | return graphGeneratorV3; |
| 313 | } | 278 | } |
| 314 | 279 | ||
| 315 | -std::vector<TaskNodePtr> BfsOldGraph(TaskNodePtr start) | ||
| 316 | -{ | ||
| 317 | - std::vector<TaskNodePtr> result; | ||
| 318 | - if (start == nullptr) { | ||
| 319 | - return result; | ||
| 320 | - } | ||
| 321 | - | ||
| 322 | - std::queue<TaskNodePtr> nodeQue; | ||
| 323 | - std::set<TaskNodePtr> visited; | ||
| 324 | - nodeQue.push(start); | ||
| 325 | - visited.insert(start); | ||
| 326 | - | ||
| 327 | - while (!nodeQue.empty()) { | ||
| 328 | - TaskNodePtr currNode = nodeQue.front(); | ||
| 329 | - nodeQue.pop(); | ||
| 330 | - result.push_back(currNode); | ||
| 331 | - | ||
| 332 | - for (TaskNodePtr child : currNode->children) { | ||
| 333 | - if (child == nullptr || visited.count(child) != 0) { | ||
| 334 | - continue; | ||
| 335 | - } | ||
| 336 | - visited.insert(child); | ||
| 337 | - nodeQue.push(child); | ||
| 338 | - } | ||
| 339 | - } | ||
| 340 | - | ||
| 341 | - return result; | ||
| 342 | -} | ||
| 343 | - | ||
| 344 | std::vector<const V3TaskNode *> BfsGraphV3(const V3TaskNode *start) | 280 | std::vector<const V3TaskNode *> BfsGraphV3(const V3TaskNode *start) |
| 345 | { | 281 | { |
| 346 | std::vector<const V3TaskNode *> result; | 282 | std::vector<const V3TaskNode *> result; |
| @@ -370,48 +306,12 @@ std::vector<const V3TaskNode *> BfsGraphV3(const V3TaskNode *start) | |||
| 370 | return result; | 306 | return result; |
| 371 | } | 307 | } |
| 372 | 308 | ||
| 373 | -OldNodeStreamPosMap BuildOldStreamPosMap(const std::vector<TaskNodePtr> &bfsNodes) | ||
| 374 | -{ | ||
| 375 | - std::map<V3StreamKey, std::vector<TaskNodePtr>> nodesByStream; | ||
| 376 | - for (TaskNodePtr node : bfsNodes) { | ||
| 377 | - if (node == nullptr || node->task == nullptr || IsOldGraphCompareTransparent(node)) { | ||
| 378 | - continue; | ||
| 379 | - } | ||
| 380 | - V3StreamKey key; | ||
| 381 | - key.rankId = node->rankIdx; | ||
| 382 | - key.streamId = node->queIdx; | ||
| 383 | - nodesByStream[key].push_back(node); | ||
| 384 | - } | ||
| 385 | - | ||
| 386 | - OldNodeStreamPosMap streamPosByNode; | ||
| 387 | - for (auto &streamItem : nodesByStream) { | ||
| 388 | - auto &streamNodes = streamItem.second; | ||
| 389 | - std::sort(streamNodes.begin(), streamNodes.end(), [](TaskNodePtr lhs, TaskNodePtr rhs) { | ||
| 390 | - if (lhs == rhs) { | ||
| 391 | - return false; | ||
| 392 | - } | ||
| 393 | - if (lhs == nullptr || rhs == nullptr) { | ||
| 394 | - return lhs < rhs; | ||
| 395 | - } | ||
| 396 | - if (lhs->pos != rhs->pos) { | ||
| 397 | - return lhs->pos < rhs->pos; | ||
| 398 | - } | ||
| 399 | - return lhs < rhs; | ||
| 400 | - }); | ||
| 401 | - | ||
| 402 | - for (size_t pos = 0; pos < streamNodes.size(); ++pos) { | ||
| 403 | - streamPosByNode[streamNodes[pos]] = static_cast<uint32_t>(pos); | ||
| 404 | - } | ||
| 405 | - } | ||
| 406 | - return streamPosByNode; | ||
| 407 | -} | ||
| 408 | - | ||
| 409 | V3NodeStreamPosMap BuildV3StreamPosMap(const std::vector<const V3TaskNode *> &bfsNodes) | 309 | V3NodeStreamPosMap BuildV3StreamPosMap(const std::vector<const V3TaskNode *> &bfsNodes) |
| 410 | { | 310 | { |
| 411 | std::map<V3StreamKey, std::vector<const V3TaskNode *>> nodesByStream; | 311 | std::map<V3StreamKey, std::vector<const V3TaskNode *>> nodesByStream; |
| 412 | for (const V3TaskNode *node : bfsNodes) { | 312 | for (const V3TaskNode *node : bfsNodes) { |
| 413 | if (node == nullptr || node->GetType() == TaskGraphGeneratorV3::TaskType::START || | 313 | if (node == nullptr || node->GetType() == TaskGraphGeneratorV3::TaskType::START || |
| 414 | - IsV3GraphCompareTransparent(node)) { | 314 | + IsV3StreamPosHiddenNode(node)) { |
| 415 | continue; | 315 | continue; |
| 416 | } | 316 | } |
| 417 | const auto &loc = node->GetPosition(); | 317 | const auto &loc = node->GetPosition(); |
| @@ -481,67 +381,6 @@ const char *TaskTypeKeyName(TaskGraphGeneratorV3::TaskType type) | |||
| 481 | } | 381 | } |
| 482 | } | 382 | } |
| 483 | 383 | ||
| 484 | -TaskGraphGeneratorV3::MemType ConvertOldMemTypeToV3(BufferType bufferType) | ||
| 485 | -{ | ||
| 486 | - switch (bufferType) { | ||
| 487 | - case ::INPUT: | ||
| 488 | - return TaskGraphGeneratorV3::MemType::INPUT; | ||
| 489 | - case ::OUTPUT: | ||
| 490 | - return TaskGraphGeneratorV3::MemType::OUTPUT; | ||
| 491 | - case ::CCL: | ||
| 492 | - case ::SCRATCH: | ||
| 493 | - return TaskGraphGeneratorV3::MemType::CCL; | ||
| 494 | - case ::MS: | ||
| 495 | - return TaskGraphGeneratorV3::MemType::MS_CCU; | ||
| 496 | - case ::INPUT_AIV: | ||
| 497 | - case ::OUTPUT_AIV: | ||
| 498 | - case ::AIV_COMMINFO: | ||
| 499 | - case ::USERBUF_AIV: | ||
| 500 | - return TaskGraphGeneratorV3::MemType::UB_AIV; | ||
| 501 | - default: | ||
| 502 | - return TaskGraphGeneratorV3::MemType::INVALID; | ||
| 503 | - } | ||
| 504 | -} | ||
| 505 | - | ||
| 506 | -TaskGraphGeneratorV3::ProtocolType ConvertOldProtocolToV3(LinkProtoStub protocol) | ||
| 507 | -{ | ||
| 508 | - if (protocol == LinkProtoStub::RDMA) { | ||
| 509 | - return TaskGraphGeneratorV3::ProtocolType::RDMA; | ||
| 510 | - } | ||
| 511 | - if (protocol == LinkProtoStub::SDMA) { | ||
| 512 | - return TaskGraphGeneratorV3::ProtocolType::SDMA; | ||
| 513 | - } | ||
| 514 | - if (protocol == LinkProtoStub::CCU) { | ||
| 515 | - return TaskGraphGeneratorV3::ProtocolType::CCU; | ||
| 516 | - } | ||
| 517 | - return TaskGraphGeneratorV3::ProtocolType::INVALID; | ||
| 518 | -} | ||
| 519 | - | ||
| 520 | -TaskGraphGeneratorV3::TaskType ConvertOldTaskTypeToV3(TaskTypeStub oldType) | ||
| 521 | -{ | ||
| 522 | - switch (oldType) { | ||
| 523 | - case TaskTypeStub::LOCAL_COPY: | ||
| 524 | - case TaskTypeStub::READ: | ||
| 525 | - case TaskTypeStub::WRITE: | ||
| 526 | - return TaskGraphGeneratorV3::TaskType::TRANS_MEM; | ||
| 527 | - case TaskTypeStub::LOCAL_REDUCE: | ||
| 528 | - case TaskTypeStub::LOCAL_BATCH_REDUCE: | ||
| 529 | - case TaskTypeStub::READ_REDUCE: | ||
| 530 | - case TaskTypeStub::WRITE_REDUCE: | ||
| 531 | - return TaskGraphGeneratorV3::TaskType::REDUCE; | ||
| 532 | - case TaskTypeStub::LOCAL_POST_TO: | ||
| 533 | - case TaskTypeStub::POST: | ||
| 534 | - return TaskGraphGeneratorV3::TaskType::RECORD; | ||
| 535 | - case TaskTypeStub::LOCAL_WAIT_FROM: | ||
| 536 | - case TaskTypeStub::WAIT: | ||
| 537 | - return TaskGraphGeneratorV3::TaskType::WAIT; | ||
| 538 | - case TaskTypeStub::CCU_GRAPH: | ||
| 539 | - return TaskGraphGeneratorV3::TaskType::CCU_GRAPH; | ||
| 540 | - default: | ||
| 541 | - return TaskGraphGeneratorV3::TaskType::INVALID; | ||
| 542 | - } | ||
| 543 | -} | ||
| 544 | - | ||
| 545 | std::string MakePositionKey(uint32_t rankId, uint32_t streamId, uint32_t streamPos) | 384 | std::string MakePositionKey(uint32_t rankId, uint32_t streamId, uint32_t streamPos) |
| 546 | { | 385 | { |
| 547 | std::ostringstream os; | 386 | std::ostringstream os; |
| @@ -564,11 +403,6 @@ std::string MakeMemSliceKey(RankId rankId, TaskGraphGeneratorV3::MemType memType | |||
| 564 | return os.str(); | 403 | return os.str(); |
| 565 | } | 404 | } |
| 566 | 405 | ||
| 567 | -std::string MakeMemSliceKey(RankId rankId, const DataSlice &slice) | ||
| 568 | -{ | ||
| 569 | - return MakeMemSliceKey(rankId, ConvertOldMemTypeToV3(slice.GetType()), slice.GetOffset(), slice.GetSize()); | ||
| 570 | -} | ||
| 571 | - | ||
| 572 | std::string MakeMemSliceKey(const TaskGraphGeneratorV3::MemSlice &slice) | 406 | std::string MakeMemSliceKey(const TaskGraphGeneratorV3::MemSlice &slice) |
| 573 | { | 407 | { |
| 574 | return MakeMemSliceKey(slice.rankId, slice.memType, slice.offset, slice.len); | 408 | return MakeMemSliceKey(slice.rankId, slice.memType, slice.offset, slice.len); |
| @@ -591,162 +425,6 @@ std::string MakeNotifyKey(RankId recordRankId, RankId waitRankId, uint64_t notif | |||
| 591 | return MakeNotifyKey(notify); | 425 | return MakeNotifyKey(notify); |
| 592 | } | 426 | } |
| 593 | 427 | ||
| 594 | -bool MakeOldNodeSemanticKey(TaskNodePtr oldNode, const OldNodeStreamPosMap *streamPosByNode, std::string &key) | ||
| 595 | -{ | ||
| 596 | - if (oldNode == nullptr) { | ||
| 597 | - return false; | ||
| 598 | - } | ||
| 599 | - | ||
| 600 | - if (oldNode->task == nullptr) { | ||
| 601 | - key = "type=START|boundary=MAIN_GRAPH"; | ||
| 602 | - return true; | ||
| 603 | - } | ||
| 604 | - | ||
| 605 | - const TaskGraphGeneratorV3::TaskType newType = ConvertOldTaskTypeToV3(oldNode->task->GetType()); | ||
| 606 | - if (newType == TaskGraphGeneratorV3::TaskType::INVALID) { | ||
| 607 | - return false; | ||
| 608 | - } | ||
| 609 | - | ||
| 610 | - uint32_t streamPos = oldNode->pos; | ||
| 611 | - if (streamPosByNode != nullptr) { | ||
| 612 | - const auto posIter = streamPosByNode->find(oldNode); | ||
| 613 | - streamPos = (posIter == streamPosByNode->end()) ? INVALID_GRAPH_STREAM_POS : posIter->second; | ||
| 614 | - } | ||
| 615 | - | ||
| 616 | - std::ostringstream os; | ||
| 617 | - os << "type=" << TaskTypeKeyName(newType) << "|" << MakePositionKey(oldNode->rankIdx, oldNode->queIdx, | ||
| 618 | - streamPos); | ||
| 619 | - | ||
| 620 | - switch (oldNode->task->GetType()) { | ||
| 621 | - case TaskTypeStub::LOCAL_COPY: { | ||
| 622 | - auto *copy = dynamic_cast<TaskStubLocalCopy *>(oldNode->task); | ||
| 623 | - if (copy == nullptr) { | ||
| 624 | - return false; | ||
| 625 | - } | ||
| 626 | - os << "|src=" << MakeMemSliceKey(oldNode->rankIdx, copy->GetSrcSlice()) | ||
| 627 | - << "|dst=" << MakeMemSliceKey(oldNode->rankIdx, copy->GetDstSlice()) | ||
| 628 | - << "|" << MakeProtocolKey(TaskGraphGeneratorV3::ProtocolType::SDMA); | ||
| 629 | - break; | ||
| 630 | - } | ||
| 631 | - case TaskTypeStub::READ: { | ||
| 632 | - auto *read = dynamic_cast<TaskStubRead *>(oldNode->task); | ||
| 633 | - if (read == nullptr) { | ||
| 634 | - return false; | ||
| 635 | - } | ||
| 636 | - os << "|src=" << MakeMemSliceKey(read->GetRemoteRank(), read->GetRemoteSlice()) | ||
| 637 | - << "|dst=" << MakeMemSliceKey(oldNode->rankIdx, read->GetLocalSlice()) | ||
| 638 | - << "|" << MakeProtocolKey(ConvertOldProtocolToV3(read->GetLinkType())); | ||
| 639 | - break; | ||
| 640 | - } | ||
| 641 | - case TaskTypeStub::WRITE: { | ||
| 642 | - auto *write = dynamic_cast<TaskStubWrite *>(oldNode->task); | ||
| 643 | - if (write == nullptr) { | ||
| 644 | - return false; | ||
| 645 | - } | ||
| 646 | - os << "|src=" << MakeMemSliceKey(oldNode->rankIdx, write->GetLocalSlice()) | ||
| 647 | - << "|dst=" << MakeMemSliceKey(write->GetRemoteRank(), write->GetRemoteSlice()) | ||
| 648 | - << "|" << MakeProtocolKey(ConvertOldProtocolToV3(write->GetLinkType())); | ||
| 649 | - break; | ||
| 650 | - } | ||
| 651 | - case TaskTypeStub::LOCAL_REDUCE: { | ||
| 652 | - auto *reduce = dynamic_cast<TaskStubLocalReduce *>(oldNode->task); | ||
| 653 | - if (reduce == nullptr) { | ||
| 654 | - return false; | ||
| 655 | - } | ||
| 656 | - os << "|src=" << MakeMemSliceKey(oldNode->rankIdx, reduce->GetSrcSlice()) | ||
| 657 | - << "|dst=" << MakeMemSliceKey(oldNode->rankIdx, reduce->GetDstSlice()) | ||
| 658 | - << "|dataType=" << static_cast<uint32_t>(reduce->GetDataType()) | ||
| 659 | - << "|reduceOp=" << static_cast<uint32_t>(reduce->GetReduceOp()) | ||
| 660 | - << "|" << MakeProtocolKey(TaskGraphGeneratorV3::ProtocolType::SDMA); | ||
| 661 | - break; | ||
| 662 | - } | ||
| 663 | - case TaskTypeStub::LOCAL_BATCH_REDUCE: { | ||
| 664 | - auto *reduce = dynamic_cast<TaskStubLocalBatchReduce *>(oldNode->task); | ||
| 665 | - if (reduce == nullptr || reduce->GetSrcSlices().empty()) { | ||
| 666 | - return false; | ||
| 667 | - } | ||
| 668 | - os << "|src=" << MakeMemSliceKey(oldNode->rankIdx, reduce->GetSrcSlice(0)) | ||
| 669 | - << "|dst=" << MakeMemSliceKey(oldNode->rankIdx, reduce->GetDstSlice()) | ||
| 670 | - << "|dataType=" << static_cast<uint32_t>(reduce->GetDataType()) | ||
| 671 | - << "|reduceOp=" << static_cast<uint32_t>(reduce->GetReduceOp()) | ||
| 672 | - << "|" << MakeProtocolKey(TaskGraphGeneratorV3::ProtocolType::SDMA); | ||
| 673 | - break; | ||
| 674 | - } | ||
| 675 | - case TaskTypeStub::READ_REDUCE: { | ||
| 676 | - auto *reduce = dynamic_cast<TaskStubReadReduce *>(oldNode->task); | ||
| 677 | - if (reduce == nullptr) { | ||
| 678 | - return false; | ||
| 679 | - } | ||
| 680 | - os << "|src=" << MakeMemSliceKey(reduce->GetRemoteRank(), reduce->GetRemoteSlice()) | ||
| 681 | - << "|dst=" << MakeMemSliceKey(oldNode->rankIdx, reduce->GetLocalSlice()) | ||
| 682 | - << "|dataType=" << static_cast<uint32_t>(reduce->GetDataType()) | ||
| 683 | - << "|reduceOp=" << static_cast<uint32_t>(reduce->GetReduceOp()) | ||
| 684 | - << "|" << MakeProtocolKey(ConvertOldProtocolToV3(reduce->GetLinkType())); | ||
| 685 | - break; | ||
| 686 | - } | ||
| 687 | - case TaskTypeStub::WRITE_REDUCE: { | ||
| 688 | - auto *reduce = dynamic_cast<TaskStubWriteReduce *>(oldNode->task); | ||
| 689 | - if (reduce == nullptr) { | ||
| 690 | - return false; | ||
| 691 | - } | ||
| 692 | - os << "|src=" << MakeMemSliceKey(oldNode->rankIdx, reduce->GetLocalSlice()) | ||
| 693 | - << "|dst=" << MakeMemSliceKey(reduce->GetRemoteRank(), reduce->GetRemoteSlice()) | ||
| 694 | - << "|dataType=" << static_cast<uint32_t>(reduce->GetDataType()) | ||
| 695 | - << "|reduceOp=" << static_cast<uint32_t>(reduce->GetReduceOp()) | ||
| 696 | - << "|" << MakeProtocolKey(ConvertOldProtocolToV3(reduce->GetLinkType())); | ||
| 697 | - break; | ||
| 698 | - } | ||
| 699 | - case TaskTypeStub::LOCAL_POST_TO: { | ||
| 700 | - auto *post = dynamic_cast<TaskStubLocalPostTo *>(oldNode->task); | ||
| 701 | - if (post == nullptr) { | ||
| 702 | - return false; | ||
| 703 | - } | ||
| 704 | - os << "|notify=" << MakeNotifyKey(oldNode->rankIdx, oldNode->rankIdx, post->GetNotifyId()) | ||
| 705 | - << "|" << MakeProtocolKey(TaskGraphGeneratorV3::ProtocolType::INVALID); | ||
| 706 | - break; | ||
| 707 | - } | ||
| 708 | - case TaskTypeStub::LOCAL_WAIT_FROM: { | ||
| 709 | - auto *wait = dynamic_cast<TaskStubLocalWaitFrom *>(oldNode->task); | ||
| 710 | - if (wait == nullptr) { | ||
| 711 | - return false; | ||
| 712 | - } | ||
| 713 | - os << "|notify=" << MakeNotifyKey(oldNode->rankIdx, oldNode->rankIdx, wait->GetNotifyId()) | ||
| 714 | - << "|" << MakeProtocolKey(TaskGraphGeneratorV3::ProtocolType::INVALID); | ||
| 715 | - break; | ||
| 716 | - } | ||
| 717 | - case TaskTypeStub::POST: { | ||
| 718 | - auto *post = dynamic_cast<TaskStubPost *>(oldNode->task); | ||
| 719 | - if (post == nullptr) { | ||
| 720 | - return false; | ||
| 721 | - } | ||
| 722 | - os << "|notify=" << MakeNotifyKey(oldNode->rankIdx, post->GetRemoteRank(), post->GetNotifyId()) | ||
| 723 | - << "|" << MakeProtocolKey(ConvertOldProtocolToV3(post->GetLinkType())); | ||
| 724 | - break; | ||
| 725 | - } | ||
| 726 | - case TaskTypeStub::WAIT: { | ||
| 727 | - auto *wait = dynamic_cast<TaskStubWait *>(oldNode->task); | ||
| 728 | - if (wait == nullptr) { | ||
| 729 | - return false; | ||
| 730 | - } | ||
| 731 | - os << "|notify=" << MakeNotifyKey(wait->GetRemoteRank(), oldNode->rankIdx, wait->GetNotifyId()) | ||
| 732 | - << "|" << MakeProtocolKey(ConvertOldProtocolToV3(wait->GetLinkType())); | ||
| 733 | - break; | ||
| 734 | - } | ||
| 735 | - case TaskTypeStub::CCU_GRAPH: | ||
| 736 | - break; | ||
| 737 | - default: | ||
| 738 | - return false; | ||
| 739 | - } | ||
| 740 | - | ||
| 741 | - key = os.str(); | ||
| 742 | - return true; | ||
| 743 | -} | ||
| 744 | - | ||
| 745 | -bool MakeOldNodeSemanticKey(TaskNodePtr oldNode, std::string &key) | ||
| 746 | -{ | ||
| 747 | - return MakeOldNodeSemanticKey(oldNode, nullptr, key); | ||
| 748 | -} | ||
| 749 | - | ||
| 750 | bool MakeV3NodeSemanticKey(const V3TaskNode *newNode, const V3NodeStreamPosMap &streamPosByNode, | 428 | bool MakeV3NodeSemanticKey(const V3TaskNode *newNode, const V3NodeStreamPosMap &streamPosByNode, |
| 751 | std::string &key) | 429 | std::string &key) |
| 752 | { | 430 | { |
| @@ -884,27 +562,6 @@ bool MakeV3NodeSemanticKey(const V3TaskNode *newNode, const V3NodeStreamPosMap & | |||
| 884 | return true; | 562 | return true; |
| 885 | } | 563 | } |
| 886 | 564 | ||
| 887 | -std::string MakeOldGraphvizNodeLabel(TaskNodePtr oldNode, size_t bfsIndex) | ||
| 888 | -{ | ||
| 889 | - std::ostringstream os; | ||
| 890 | - os << "old_" << bfsIndex << "\nbfs=" << bfsIndex; | ||
| 891 | - if (oldNode == nullptr) { | ||
| 892 | - os << "\n<null old node>"; | ||
| 893 | - return os.str(); | ||
| 894 | - } | ||
| 895 | - | ||
| 896 | - os << "\nrank=" << oldNode->rankIdx << ", stream=" << oldNode->queIdx << ", index=" << oldNode->pos; | ||
| 897 | - std::string semanticKey; | ||
| 898 | - if (MakeOldNodeSemanticKey(oldNode, semanticKey)) { | ||
| 899 | - os << "\n" << FormatGraphvizLabelText(semanticKey); | ||
| 900 | - } else if (oldNode->task != nullptr) { | ||
| 901 | - os << "\n" << FormatGraphvizLabelText(oldNode->task->Describe()); | ||
| 902 | - } else { | ||
| 903 | - os << "\n<invalid old node>"; | ||
| 904 | - } | ||
| 905 | - return os.str(); | ||
| 906 | -} | ||
| 907 | - | ||
| 908 | std::string MakeV3GraphvizNodeLabel(const V3TaskNode *node, const V3NodeStreamPosMap &streamPosByNode, | 565 | std::string MakeV3GraphvizNodeLabel(const V3TaskNode *node, const V3NodeStreamPosMap &streamPosByNode, |
| 909 | size_t bfsIndex) | 566 | size_t bfsIndex) |
| 910 | { | 567 | { |
| @@ -932,47 +589,6 @@ std::string MakeV3GraphvizNodeLabel(const V3TaskNode *node, const V3NodeStreamPo | |||
| 932 | return os.str(); | 589 | return os.str(); |
| 933 | } | 590 | } |
| 934 | 591 | ||
| 935 | -std::string BuildOldGraphvizDot(const std::vector<TaskNodePtr> &oldBfsNodes) | ||
| 936 | -{ | ||
| 937 | - std::map<TaskNodePtr, size_t> nodeIndexByPtr; | ||
| 938 | - for (size_t i = 0; i < oldBfsNodes.size(); ++i) { | ||
| 939 | - nodeIndexByPtr[oldBfsNodes[i]] = i; | ||
| 940 | - } | ||
| 941 | - | ||
| 942 | - std::set<std::pair<size_t, size_t>> edgeIndexes; | ||
| 943 | - for (TaskNodePtr oldNode : oldBfsNodes) { | ||
| 944 | - const auto parentIter = nodeIndexByPtr.find(oldNode); | ||
| 945 | - if (parentIter == nodeIndexByPtr.end() || oldNode == nullptr) { | ||
| 946 | - continue; | ||
| 947 | - } | ||
| 948 | - | ||
| 949 | - for (TaskNodePtr child : oldNode->children) { | ||
| 950 | - const auto childIter = nodeIndexByPtr.find(child); | ||
| 951 | - if (childIter == nodeIndexByPtr.end()) { | ||
| 952 | - continue; | ||
| 953 | - } | ||
| 954 | - edgeIndexes.insert(std::make_pair(parentIter->second, childIter->second)); | ||
| 955 | - } | ||
| 956 | - } | ||
| 957 | - | ||
| 958 | - std::ostringstream os; | ||
| 959 | - os << "digraph old_task_graph {\n"; | ||
| 960 | - os << " graph [rankdir=LR, label=\"old graph BFS DAG\", labelloc=t, fontsize=20];\n"; | ||
| 961 | - os << " node [shape=box, style=\"rounded,filled\", fillcolor=\"#fff7e6\", fontname=\"Courier\"];\n"; | ||
| 962 | - os << " edge [fontname=\"Courier\"];\n\n"; | ||
| 963 | - for (size_t i = 0; i < oldBfsNodes.size(); ++i) { | ||
| 964 | - os << " old_" << i << " [label=\"" << EscapeGraphvizLabel(MakeOldGraphvizNodeLabel(oldBfsNodes[i], i)) | ||
| 965 | - << "\"];\n"; | ||
| 966 | - } | ||
| 967 | - | ||
| 968 | - os << "\n"; | ||
| 969 | - for (const auto &edge : edgeIndexes) { | ||
| 970 | - os << " old_" << edge.first << " -> old_" << edge.second << ";\n"; | ||
| 971 | - } | ||
| 972 | - os << "}\n"; | ||
| 973 | - return os.str(); | ||
| 974 | -} | ||
| 975 | - | ||
| 976 | std::string BuildV3GraphvizDot(const std::vector<const V3TaskNode *> &newBfsNodes) | 592 | std::string BuildV3GraphvizDot(const std::vector<const V3TaskNode *> &newBfsNodes) |
| 977 | { | 593 | { |
| 978 | const V3NodeStreamPosMap streamPosByNode = BuildV3StreamPosMap(newBfsNodes); | 594 | const V3NodeStreamPosMap streamPosByNode = BuildV3StreamPosMap(newBfsNodes); |
| @@ -1018,8 +634,8 @@ std::string BuildV3GraphvizDot(const std::vector<const V3TaskNode *> &newBfsNode | |||
| 1018 | 634 | ||
| 1019 | void LogGraphvizDot(const std::string &graphName, const std::string &dot) | 635 | void LogGraphvizDot(const std::string &graphName, const std::string &dot) |
| 1020 | { | 636 | { |
| 1021 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][GraphvizDot][{}] GRAPHVIZ_DOT_BEGIN\n{}", graphName, dot); | 637 | + HCCL_VM_INFO("Graphviz DOT dump begin, graphName={}{}", graphName, dot); |
| 1022 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][GraphvizDot][{}] GRAPHVIZ_DOT_END", graphName); | 638 | + HCCL_VM_INFO("Graphviz DOT dump end, graphName={}", graphName); |
| 1023 | } | 639 | } |
| 1024 | 640 | ||
| 1025 | std::string FormatV3NodeIdList(const std::vector<V3TaskNode *> &nodes) | 641 | std::string FormatV3NodeIdList(const std::vector<V3TaskNode *> &nodes) |
| @@ -1044,19 +660,19 @@ std::string FormatV3NodeIdList(const std::vector<V3TaskNode *> &nodes) | |||
| 1044 | std::string MakeAivDagTaskLogLine(const V3TaskNode *node, size_t bfsIndex) | 660 | std::string MakeAivDagTaskLogLine(const V3TaskNode *node, size_t bfsIndex) |
| 1045 | { | 661 | { |
| 1046 | if (node == nullptr) { | 662 | if (node == nullptr) { |
| 1047 | - return "<null v3 node>"; | 663 | + return "node=null"; |
| 1048 | } | 664 | } |
| 1049 | const auto &loc = node->GetPosition(); | 665 | const auto &loc = node->GetPosition(); |
| 1050 | std::ostringstream os; | 666 | std::ostringstream os; |
| 1051 | - os << "bfs=" << bfsIndex | 667 | + os << "bfsIndex=" << bfsIndex |
| 1052 | << ", nodeId=" << node->GetNodeId() | 668 | << ", nodeId=" << node->GetNodeId() |
| 1053 | - << ", type=" << TaskTypeKeyName(node->GetType()) | 669 | + << ", taskType=" << TaskTypeKeyName(node->GetType()) |
| 1054 | - << ", rank=" << loc.rankId | 670 | + << ", rankId=" << loc.rankId |
| 1055 | - << ", stream=" << loc.streamId | 671 | + << ", streamId=" << loc.streamId |
| 1056 | - << ", queue=" << loc.queueId | 672 | + << ", queueId=" << loc.queueId |
| 1057 | - << ", parents=" << FormatV3NodeIdList(node->GetParents()) | 673 | + << ", parentNodeIds=" << FormatV3NodeIdList(node->GetParents()) |
| 1058 | - << ", children=" << FormatV3NodeIdList(node->GetChildren()) | 674 | + << ", childNodeIds=" << FormatV3NodeIdList(node->GetChildren()) |
| 1059 | - << ", desc=" << node->Describe(); | 675 | + << ", detail=" << node->Describe(); |
| 1060 | return os.str(); | 676 | return os.str(); |
| 1061 | } | 677 | } |
| 1062 | 678 | ||
| @@ -1068,10 +684,10 @@ size_t LogAivExpandedDagTasks(const std::vector<const V3TaskNode *> &v3BfsNodes) | |||
| 1068 | if (!IsAivDagLogNode(node)) { | 684 | if (!IsAivDagLogNode(node)) { |
| 1069 | continue; | 685 | continue; |
| 1070 | } | 686 | } |
| 1071 | - HCCL_VM_DEBUG("[TaskGraphGeneratorV3][AivDagTask] {}", MakeAivDagTaskLogLine(node, index)); | 687 | + HCCL_VM_DEBUG("Expanded AIV DAG node detail, {}", MakeAivDagTaskLogLine(node, index)); |
| 1072 | ++aivTaskCount; | 688 | ++aivTaskCount; |
| 1073 | } | 689 | } |
| 1074 | - HCCL_VM_DEBUG("[TaskGraphGeneratorV3][AivDagTask] Dump AIV expanded DAG task detail, taskCount={}", | 690 | + HCCL_VM_DEBUG("Finished dumping expanded AIV DAG node details, taskCount={}", |
| 1075 | aivTaskCount); | 691 | aivTaskCount); |
| 1076 | return aivTaskCount; | 692 | return aivTaskCount; |
| 1077 | } | 693 | } |
| @@ -1079,12 +695,12 @@ size_t LogAivExpandedDagTasks(const std::vector<const V3TaskNode *> &v3BfsNodes) | |||
| 1079 | void LogGraphV3Dag(const V3TaskNode *start) | 695 | void LogGraphV3Dag(const V3TaskNode *start) |
| 1080 | { | 696 | { |
| 1081 | if (start == nullptr) { | 697 | if (start == nullptr) { |
| 1082 | - HCCL_VM_WARN("[TaskGraphGeneratorV3][GraphvizDot][v3] Skip DAG dump, start is null."); | 698 | + HCCL_VM_WARN("Skip DAG dump because the graph start node is null"); |
| 1083 | return; | 699 | return; |
| 1084 | } | 700 | } |
| 1085 | 701 | ||
| 1086 | const auto v3BfsNodes = BfsGraphV3(start); | 702 | const auto v3BfsNodes = BfsGraphV3(start); |
| 1087 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][GraphvizDot][v3] Dump V3 DAG, nodeCount={}", v3BfsNodes.size()); | 703 | + HCCL_VM_INFO("Dumping the V3 DAG as Graphviz DOT text, nodeCount={}", v3BfsNodes.size()); |
| 1088 | LogGraphvizDot("v3", BuildV3GraphvizDot(v3BfsNodes)); | 704 | LogGraphvizDot("v3", BuildV3GraphvizDot(v3BfsNodes)); |
| 1089 | LogAivExpandedDagTasks(v3BfsNodes); | 705 | LogAivExpandedDagTasks(v3BfsNodes); |
| 1090 | } | 706 | } |
| @@ -1092,427 +708,26 @@ void LogGraphV3Dag(const V3TaskNode *start) | |||
| 1092 | void DumpGraphV3DotToFile(const V3TaskNode *start, const std::string &path) | 708 | void DumpGraphV3DotToFile(const V3TaskNode *start, const std::string &path) |
| 1093 | { | 709 | { |
| 1094 | if (start == nullptr) { | 710 | if (start == nullptr) { |
| 1095 | - HCCL_VM_WARN("[TaskGraphGeneratorV3][GraphvizDot][v3] Skip file dump, start is null."); | 711 | + HCCL_VM_WARN("Skip file dump because the graph start node is null, path={}", path); |
| 1096 | return; | 712 | return; |
| 1097 | } | 713 | } |
| 1098 | const auto v3BfsNodes = BfsGraphV3(start); | 714 | const auto v3BfsNodes = BfsGraphV3(start); |
| 1099 | std::ofstream out(path, std::ios::out | std::ios::trunc); | 715 | std::ofstream out(path, std::ios::out | std::ios::trunc); |
| 1100 | if (!out.is_open()) { | 716 | if (!out.is_open()) { |
| 1101 | - HCCL_VM_WARN("[TaskGraphGeneratorV3][GraphvizDot][v3] Failed to open graph file: {}", path); | 717 | + HCCL_VM_WARN("Failed to open the graph output file, path={}", path); |
| 1102 | return; | 718 | return; |
| 1103 | } | 719 | } |
| 1104 | out << BuildV3GraphvizDot(v3BfsNodes); | 720 | out << BuildV3GraphvizDot(v3BfsNodes); |
| 1105 | out.close(); | 721 | out.close(); |
| 1106 | - HCCL_VM_WARN("[TaskGraphGeneratorV3][GraphvizDot][v3] Graph dumped to {}, nodeCount={}", | 722 | + HCCL_VM_INFO("Wrote V3 DAG DOT file, path={}, nodeCount={}", |
| 1107 | path, v3BfsNodes.size()); | 723 | path, v3BfsNodes.size()); |
| 1108 | } | 724 | } |
| 1109 | 725 | ||
| 1110 | -struct SemanticGraphSnapshot { | ||
| 1111 | - std::map<std::string, uint32_t> nodeCounts; | ||
| 1112 | - std::set<std::string> edgeKeys; | ||
| 1113 | - size_t invalidNodeCount{0}; | ||
| 1114 | - size_t duplicateNodeCount{0}; | ||
| 1115 | -}; | ||
| 1116 | - | ||
| 1117 | -void CollectOldVisibleChildren(TaskNodePtr node, std::set<TaskNodePtr> &visibleChildren) | ||
| 1118 | -{ | ||
| 1119 | - if (node == nullptr) { | ||
| 1120 | - return; | ||
| 1121 | - } | ||
| 1122 | - | ||
| 1123 | - std::vector<TaskNodePtr> nodeStack(node->children.begin(), node->children.end()); | ||
| 1124 | - std::set<TaskNodePtr> visited; | ||
| 1125 | - while (!nodeStack.empty()) { | ||
| 1126 | - TaskNodePtr currNode = nodeStack.back(); | ||
| 1127 | - nodeStack.pop_back(); | ||
| 1128 | - if (currNode == nullptr || visited.count(currNode) != 0) { | ||
| 1129 | - continue; | ||
| 1130 | - } | ||
| 1131 | - visited.insert(currNode); | ||
| 1132 | - if (!IsOldGraphCompareTransparent(currNode)) { | ||
| 1133 | - visibleChildren.insert(currNode); | ||
| 1134 | - continue; | ||
| 1135 | - } | ||
| 1136 | - for (TaskNodePtr childNode : currNode->children) { | ||
| 1137 | - nodeStack.push_back(childNode); | ||
| 1138 | - } | ||
| 1139 | - } | ||
| 1140 | -} | ||
| 1141 | - | ||
| 1142 | -void CollectV3VisibleChildren(const V3TaskNode *node, std::set<const V3TaskNode *> &visibleChildren) | ||
| 1143 | -{ | ||
| 1144 | - if (node == nullptr) { | ||
| 1145 | - return; | ||
| 1146 | - } | ||
| 1147 | - | ||
| 1148 | - std::vector<const V3TaskNode *> nodeStack(node->GetChildren().begin(), node->GetChildren().end()); | ||
| 1149 | - std::set<const V3TaskNode *> visited; | ||
| 1150 | - while (!nodeStack.empty()) { | ||
| 1151 | - const V3TaskNode *currNode = nodeStack.back(); | ||
| 1152 | - nodeStack.pop_back(); | ||
| 1153 | - if (currNode == nullptr || visited.count(currNode) != 0) { | ||
| 1154 | - continue; | ||
| 1155 | - } | ||
| 1156 | - visited.insert(currNode); | ||
| 1157 | - if (!IsV3GraphCompareTransparent(currNode)) { | ||
| 1158 | - visibleChildren.insert(currNode); | ||
| 1159 | - continue; | ||
| 1160 | - } | ||
| 1161 | - for (const V3TaskNode *childNode : currNode->GetChildren()) { | ||
| 1162 | - nodeStack.push_back(childNode); | ||
| 1163 | - } | ||
| 1164 | - } | ||
| 1165 | -} | ||
| 1166 | - | ||
| 1167 | -void AddNodeKey(SemanticGraphSnapshot &snapshot, const std::string &nodeKey) | ||
| 1168 | -{ | ||
| 1169 | - uint32_t &count = snapshot.nodeCounts[nodeKey]; | ||
| 1170 | - ++count; | ||
| 1171 | - if (count > 1) { | ||
| 1172 | - ++snapshot.duplicateNodeCount; | ||
| 1173 | - } | ||
| 1174 | -} | ||
| 1175 | - | ||
| 1176 | -SemanticGraphSnapshot BuildOldSemanticGraph(const std::vector<TaskNodePtr> &oldBfsNodes) | ||
| 1177 | -{ | ||
| 1178 | - SemanticGraphSnapshot snapshot; | ||
| 1179 | - const OldNodeStreamPosMap streamPosByNode = BuildOldStreamPosMap(oldBfsNodes); | ||
| 1180 | - std::map<TaskNodePtr, std::string> nodeKeyByPtr; | ||
| 1181 | - | ||
| 1182 | - for (TaskNodePtr oldNode : oldBfsNodes) { | ||
| 1183 | - if (IsOldGraphCompareTransparent(oldNode)) { | ||
| 1184 | - continue; | ||
| 1185 | - } | ||
| 1186 | - std::string nodeKey; | ||
| 1187 | - if (!MakeOldNodeSemanticKey(oldNode, &streamPosByNode, nodeKey)) { | ||
| 1188 | - ++snapshot.invalidNodeCount; | ||
| 1189 | - continue; | ||
| 1190 | - } | ||
| 1191 | - AddNodeKey(snapshot, nodeKey); | ||
| 1192 | - nodeKeyByPtr[oldNode] = nodeKey; | ||
| 1193 | - } | ||
| 1194 | - | ||
| 1195 | - for (TaskNodePtr oldNode : oldBfsNodes) { | ||
| 1196 | - const auto parentIter = nodeKeyByPtr.find(oldNode); | ||
| 1197 | - if (parentIter == nodeKeyByPtr.end()) { | ||
| 1198 | - continue; | ||
| 1199 | - } | ||
| 1200 | - | ||
| 1201 | - std::set<TaskNodePtr> visibleChildren; | ||
| 1202 | - CollectOldVisibleChildren(oldNode, visibleChildren); | ||
| 1203 | - for (TaskNodePtr child : visibleChildren) { | ||
| 1204 | - const auto childIter = nodeKeyByPtr.find(child); | ||
| 1205 | - if (childIter == nodeKeyByPtr.end()) { | ||
| 1206 | - continue; | ||
| 1207 | - } | ||
| 1208 | - snapshot.edgeKeys.insert(parentIter->second + " -> " + childIter->second); | ||
| 1209 | - } | ||
| 1210 | - } | ||
| 1211 | - | ||
| 1212 | - return snapshot; | ||
| 1213 | -} | ||
| 1214 | - | ||
| 1215 | -SemanticGraphSnapshot BuildV3SemanticGraph(const std::vector<const V3TaskNode *> &newBfsNodes) | ||
| 1216 | -{ | ||
| 1217 | - SemanticGraphSnapshot snapshot; | ||
| 1218 | - const V3NodeStreamPosMap streamPosByNode = BuildV3StreamPosMap(newBfsNodes); | ||
| 1219 | - std::map<const V3TaskNode *, std::string> nodeKeyByPtr; | ||
| 1220 | - | ||
| 1221 | - for (const V3TaskNode *node : newBfsNodes) { | ||
| 1222 | - if (IsV3GraphCompareTransparent(node)) { | ||
| 1223 | - continue; | ||
| 1224 | - } | ||
| 1225 | - std::string nodeKey; | ||
| 1226 | - if (!MakeV3NodeSemanticKey(node, streamPosByNode, nodeKey)) { | ||
| 1227 | - ++snapshot.invalidNodeCount; | ||
| 1228 | - continue; | ||
| 1229 | - } | ||
| 1230 | - AddNodeKey(snapshot, nodeKey); | ||
| 1231 | - nodeKeyByPtr[node] = nodeKey; | ||
| 1232 | - } | ||
| 1233 | - | ||
| 1234 | - for (const V3TaskNode *node : newBfsNodes) { | ||
| 1235 | - const auto parentIter = nodeKeyByPtr.find(node); | ||
| 1236 | - if (parentIter == nodeKeyByPtr.end() || node == nullptr) { | ||
| 1237 | - continue; | ||
| 1238 | - } | ||
| 1239 | - | ||
| 1240 | - std::set<const V3TaskNode *> visibleChildren; | ||
| 1241 | - CollectV3VisibleChildren(node, visibleChildren); | ||
| 1242 | - for (const V3TaskNode *childNode : visibleChildren) { | ||
| 1243 | - const auto childIter = nodeKeyByPtr.find(childNode); | ||
| 1244 | - if (childIter == nodeKeyByPtr.end()) { | ||
| 1245 | - continue; | ||
| 1246 | - } | ||
| 1247 | - snapshot.edgeKeys.insert(parentIter->second + " -> " + childIter->second); | ||
| 1248 | - } | ||
| 1249 | - } | ||
| 1250 | - | ||
| 1251 | - return snapshot; | ||
| 1252 | -} | ||
| 1253 | - | ||
| 1254 | -size_t LogBfsOrderDiff(const std::vector<TaskNodePtr> &oldBfsNodes, | ||
| 1255 | - const std::vector<const V3TaskNode *> &newBfsNodes) | ||
| 1256 | -{ | ||
| 1257 | - size_t mismatchCount = 0; | ||
| 1258 | - size_t detailCount = 0; | ||
| 1259 | - const OldNodeStreamPosMap oldStreamPosByNode = BuildOldStreamPosMap(oldBfsNodes); | ||
| 1260 | - const V3NodeStreamPosMap streamPosByNode = BuildV3StreamPosMap(newBfsNodes); | ||
| 1261 | - const size_t commonCount = std::min(oldBfsNodes.size(), newBfsNodes.size()); | ||
| 1262 | - | ||
| 1263 | - for (size_t i = 0; i < commonCount; ++i) { | ||
| 1264 | - std::string oldKey; | ||
| 1265 | - if (!MakeOldNodeSemanticKey(oldBfsNodes[i], &oldStreamPosByNode, oldKey)) { | ||
| 1266 | - oldKey = "<invalid old bfs node>"; | ||
| 1267 | - } | ||
| 1268 | - | ||
| 1269 | - std::string newKey; | ||
| 1270 | - if (!MakeV3NodeSemanticKey(newBfsNodes[i], streamPosByNode, newKey)) { | ||
| 1271 | - newKey = "<invalid v3 bfs node>"; | ||
| 1272 | - } | ||
| 1273 | - | ||
| 1274 | - if (oldKey == newKey) { | ||
| 1275 | - continue; | ||
| 1276 | - } | ||
| 1277 | - | ||
| 1278 | - ++mismatchCount; | ||
| 1279 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1280 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph BFS order mismatch, bfsIndex={}, oldNode={}, v3Node={}", | ||
| 1281 | - i, oldKey, newKey); | ||
| 1282 | - ++detailCount; | ||
| 1283 | - } | ||
| 1284 | - } | ||
| 1285 | - | ||
| 1286 | - for (size_t i = commonCount; i < oldBfsNodes.size(); ++i) { | ||
| 1287 | - ++mismatchCount; | ||
| 1288 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1289 | - std::string oldKey; | ||
| 1290 | - if (!MakeOldNodeSemanticKey(oldBfsNodes[i], &oldStreamPosByNode, oldKey)) { | ||
| 1291 | - oldKey = "<invalid old bfs node>"; | ||
| 1292 | - } | ||
| 1293 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph BFS order mismatch, bfsIndex={}, oldNode={}, v3Node={}", | ||
| 1294 | - i, oldKey, "<missing v3 bfs node>"); | ||
| 1295 | - ++detailCount; | ||
| 1296 | - } | ||
| 1297 | - } | ||
| 1298 | - | ||
| 1299 | - for (size_t i = commonCount; i < newBfsNodes.size(); ++i) { | ||
| 1300 | - ++mismatchCount; | ||
| 1301 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1302 | - std::string newKey; | ||
| 1303 | - if (!MakeV3NodeSemanticKey(newBfsNodes[i], streamPosByNode, newKey)) { | ||
| 1304 | - newKey = "<invalid v3 bfs node>"; | ||
| 1305 | - } | ||
| 1306 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph BFS order mismatch, bfsIndex={}, oldNode={}, v3Node={}", | ||
| 1307 | - i, "<missing old bfs node>", newKey); | ||
| 1308 | - ++detailCount; | ||
| 1309 | - } | ||
| 1310 | - } | ||
| 1311 | - | ||
| 1312 | - if (detailCount >= MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1313 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Too many BFS order mismatches, stop detail logging."); | ||
| 1314 | - } | ||
| 1315 | - | ||
| 1316 | - if (mismatchCount == 0) { | ||
| 1317 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Graph BFS order matched, count={}", oldBfsNodes.size()); | ||
| 1318 | - } else { | ||
| 1319 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph BFS order mismatch, mismatchCount={}, oldCount={}, v3Count={}", | ||
| 1320 | - mismatchCount, oldBfsNodes.size(), newBfsNodes.size()); | ||
| 1321 | - } | ||
| 1322 | - return mismatchCount; | ||
| 1323 | -} | ||
| 1324 | - | ||
| 1325 | -size_t LogNodeCountDiff(const SemanticGraphSnapshot &oldSnapshot, const SemanticGraphSnapshot &newSnapshot) | ||
| 1326 | -{ | ||
| 1327 | - size_t mismatchCount = 0; | ||
| 1328 | - size_t detailCount = 0; | ||
| 1329 | - | ||
| 1330 | - for (const auto &oldEntry : oldSnapshot.nodeCounts) { | ||
| 1331 | - const auto newIter = newSnapshot.nodeCounts.find(oldEntry.first); | ||
| 1332 | - const uint32_t newCount = (newIter == newSnapshot.nodeCounts.end()) ? 0 : newIter->second; | ||
| 1333 | - if (newCount >= oldEntry.second) { | ||
| 1334 | - continue; | ||
| 1335 | - } | ||
| 1336 | - | ||
| 1337 | - const uint32_t missingCount = oldEntry.second - newCount; | ||
| 1338 | - mismatchCount += missingCount; | ||
| 1339 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1340 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] V3 graph missing semantic node, count={}, key={}", | ||
| 1341 | - missingCount, oldEntry.first); | ||
| 1342 | - ++detailCount; | ||
| 1343 | - } | ||
| 1344 | - } | ||
| 1345 | - | ||
| 1346 | - for (const auto &newEntry : newSnapshot.nodeCounts) { | ||
| 1347 | - const auto oldIter = oldSnapshot.nodeCounts.find(newEntry.first); | ||
| 1348 | - const uint32_t oldCount = (oldIter == oldSnapshot.nodeCounts.end()) ? 0 : oldIter->second; | ||
| 1349 | - if (oldCount >= newEntry.second) { | ||
| 1350 | - continue; | ||
| 1351 | - } | ||
| 1352 | - | ||
| 1353 | - const uint32_t extraCount = newEntry.second - oldCount; | ||
| 1354 | - mismatchCount += extraCount; | ||
| 1355 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1356 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] V3 graph extra semantic node, count={}, key={}", | ||
| 1357 | - extraCount, newEntry.first); | ||
| 1358 | - ++detailCount; | ||
| 1359 | - } | ||
| 1360 | - } | ||
| 1361 | - | ||
| 1362 | - if (detailCount >= MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1363 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Too many semantic node mismatches, stop detail logging."); | ||
| 1364 | - } | ||
| 1365 | - return mismatchCount; | ||
| 1366 | -} | ||
| 1367 | - | ||
| 1368 | -bool IsCcuGraphSemanticKey(const std::string &key) | ||
| 1369 | -{ | ||
| 1370 | - return key.find("type=CCU_GRAPH") != std::string::npos; | ||
| 1371 | -} | ||
| 1372 | - | ||
| 1373 | -bool HasCcuGraphNode(const SemanticGraphSnapshot &snapshot) | ||
| 1374 | -{ | ||
| 1375 | - for (const auto &nodeEntry : snapshot.nodeCounts) { | ||
| 1376 | - if (IsCcuGraphSemanticKey(nodeEntry.first)) { | ||
| 1377 | - return true; | ||
| 1378 | - } | ||
| 1379 | - } | ||
| 1380 | - return false; | ||
| 1381 | -} | ||
| 1382 | - | ||
| 1383 | -bool IsCcuBoundaryEdge(const std::string &edgeKey) | ||
| 1384 | -{ | ||
| 1385 | - return IsCcuGraphSemanticKey(edgeKey); | ||
| 1386 | -} | ||
| 1387 | - | ||
| 1388 | -size_t LogEdgeDiff(const std::set<std::string> &oldEdgeKeys, const std::set<std::string> &newEdgeKeys, | ||
| 1389 | - bool ignoreCcuBoundaryEdges = false) | ||
| 1390 | -{ | ||
| 1391 | - size_t mismatchCount = 0; | ||
| 1392 | - size_t detailCount = 0; | ||
| 1393 | - | ||
| 1394 | - for (const auto &edgeKey : oldEdgeKeys) { | ||
| 1395 | - if (ignoreCcuBoundaryEdges && IsCcuBoundaryEdge(edgeKey)) { | ||
| 1396 | - continue; | ||
| 1397 | - } | ||
| 1398 | - if (newEdgeKeys.count(edgeKey) != 0) { | ||
| 1399 | - continue; | ||
| 1400 | - } | ||
| 1401 | - | ||
| 1402 | - ++mismatchCount; | ||
| 1403 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1404 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] V3 graph missing edge, edge={}", edgeKey); | ||
| 1405 | - ++detailCount; | ||
| 1406 | - } | ||
| 1407 | - } | ||
| 1408 | - | ||
| 1409 | - for (const auto &edgeKey : newEdgeKeys) { | ||
| 1410 | - if (ignoreCcuBoundaryEdges && IsCcuBoundaryEdge(edgeKey)) { | ||
| 1411 | - continue; | ||
| 1412 | - } | ||
| 1413 | - if (oldEdgeKeys.count(edgeKey) != 0) { | ||
| 1414 | - continue; | ||
| 1415 | - } | ||
| 1416 | - | ||
| 1417 | - ++mismatchCount; | ||
| 1418 | - if (detailCount < MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1419 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] V3 graph extra edge, edge={}", edgeKey); | ||
| 1420 | - ++detailCount; | ||
| 1421 | - } | ||
| 1422 | - } | ||
| 1423 | - | ||
| 1424 | - if (detailCount >= MAX_GRAPH_COMPARE_DETAIL_LOG_COUNT) { | ||
| 1425 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Too many semantic edge mismatches, stop detail logging."); | ||
| 1426 | - } | ||
| 1427 | - return mismatchCount; | ||
| 1428 | -} | ||
| 1429 | - | ||
| 1430 | -void CompareGraphV3BfsOrder(const std::vector<TaskNodePtr> &oldBfsNodes, | ||
| 1431 | - const std::vector<const V3TaskNode *> &newBfsNodes) | ||
| 1432 | -{ | ||
| 1433 | - if (oldBfsNodes.size() != newBfsNodes.size()) { | ||
| 1434 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph BFS node count mismatch, old={}, v3={}", | ||
| 1435 | - oldBfsNodes.size(), newBfsNodes.size()); | ||
| 1436 | - } else { | ||
| 1437 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Graph BFS node count matched, count={}", oldBfsNodes.size()); | ||
| 1438 | - } | ||
| 1439 | - | ||
| 1440 | - const size_t bfsOrderMismatchCount = LogBfsOrderDiff(oldBfsNodes, newBfsNodes); | ||
| 1441 | - if (bfsOrderMismatchCount == 0 && oldBfsNodes.size() == newBfsNodes.size()) { | ||
| 1442 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Graph BFS compare matched, orderMatched=true, nodeCount={}", | ||
| 1443 | - oldBfsNodes.size()); | ||
| 1444 | - } else { | ||
| 1445 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph BFS compare mismatch, orderMismatch={}, oldNodeCount={}, " | ||
| 1446 | - "v3NodeCount={}", bfsOrderMismatchCount, oldBfsNodes.size(), newBfsNodes.size()); | ||
| 1447 | - } | ||
| 1448 | -} | ||
| 1449 | - | ||
| 1450 | -void CompareGraphV3Semantic(const std::vector<TaskNodePtr> &oldBfsNodes, | ||
| 1451 | - const std::vector<const V3TaskNode *> &newBfsNodes) | ||
| 1452 | -{ | ||
| 1453 | - const SemanticGraphSnapshot oldSnapshot = BuildOldSemanticGraph(oldBfsNodes); | ||
| 1454 | - const SemanticGraphSnapshot newSnapshot = BuildV3SemanticGraph(newBfsNodes); | ||
| 1455 | - const bool hasCcuGraph = HasCcuGraphNode(oldSnapshot) || HasCcuGraphNode(newSnapshot); | ||
| 1456 | - const size_t nodeMismatchCount = LogNodeCountDiff(oldSnapshot, newSnapshot); | ||
| 1457 | - const size_t edgeMismatchCount = LogEdgeDiff(oldSnapshot.edgeKeys, newSnapshot.edgeKeys, hasCcuGraph); | ||
| 1458 | - if (hasCcuGraph) { | ||
| 1459 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Graph semantic compare uses CCU-compatible view: " | ||
| 1460 | - "V3 CCU expanded nodes and old CCU graph separators are ignored; CCU boundary edges are skipped."); | ||
| 1461 | - } | ||
| 1462 | - | ||
| 1463 | - if (oldSnapshot.invalidNodeCount != 0 || newSnapshot.invalidNodeCount != 0 || | ||
| 1464 | - oldSnapshot.duplicateNodeCount != 0 || newSnapshot.duplicateNodeCount != 0) { | ||
| 1465 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph semantic compare snapshot abnormal, oldInvalid={}, " | ||
| 1466 | - "v3Invalid={}, oldDuplicate={}, v3Duplicate={}", oldSnapshot.invalidNodeCount, | ||
| 1467 | - newSnapshot.invalidNodeCount, oldSnapshot.duplicateNodeCount, newSnapshot.duplicateNodeCount); | ||
| 1468 | - } | ||
| 1469 | - | ||
| 1470 | - if (nodeMismatchCount == 0 && edgeMismatchCount == 0 && oldSnapshot.invalidNodeCount == 0 && | ||
| 1471 | - newSnapshot.invalidNodeCount == 0) { | ||
| 1472 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Graph semantic compare matched, nodeKinds={}, edgeCount={}", | ||
| 1473 | - oldSnapshot.nodeCounts.size(), oldSnapshot.edgeKeys.size()); | ||
| 1474 | - } else { | ||
| 1475 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Graph semantic compare mismatch, nodeMismatch={}, edgeMismatch={}, " | ||
| 1476 | - "oldInvalid={}, v3Invalid={}, oldNodeKinds={}, v3NodeKinds={}, oldEdges={}, v3Edges={}", | ||
| 1477 | - nodeMismatchCount, edgeMismatchCount, oldSnapshot.invalidNodeCount, newSnapshot.invalidNodeCount, | ||
| 1478 | - oldSnapshot.nodeCounts.size(), newSnapshot.nodeCounts.size(), oldSnapshot.edgeKeys.size(), | ||
| 1479 | - newSnapshot.edgeKeys.size()); | ||
| 1480 | - } | ||
| 1481 | -} | ||
| 1482 | - | ||
| 1483 | -void CompareGraphV3WithOldGraph(TaskNodePtr oldStart, const V3TaskNode *newStart) | ||
| 1484 | -{ | ||
| 1485 | - if (oldStart == nullptr || newStart == nullptr) { | ||
| 1486 | - HCCL_VM_WARN("[TaskGraphGeneratorV3] Skip graph compare, oldStart or newStart is null."); | ||
| 1487 | - return; | ||
| 1488 | - } | ||
| 1489 | - | ||
| 1490 | - const auto oldBfsNodes = BfsOldGraph(oldStart); | ||
| 1491 | - const auto newBfsNodes = BfsGraphV3(newStart); | ||
| 1492 | - const auto oldCompareNodes = BuildOldGraphCompareNodes(oldBfsNodes); | ||
| 1493 | - const auto newCompareNodes = BuildV3GraphCompareNodes(newBfsNodes); | ||
| 1494 | - // compare 统一使用 BFS 收集节点,随后再过滤掉透明节点(如队列占位节点、CCU 边界节点), | ||
| 1495 | - // 这样既能保持视图稳定,也能让 old/v3 两侧在同一语义层面上比较。 | ||
| 1496 | - const bool useTransparentCompareView = oldCompareNodes.size() != oldBfsNodes.size() || | ||
| 1497 | - newCompareNodes.size() != newBfsNodes.size(); | ||
| 1498 | - if (useTransparentCompareView) { | ||
| 1499 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Graph compare view filtered nodes, oldRaw={}, oldVisible={}, " | ||
| 1500 | - "v3Raw={}, v3Visible={}", oldBfsNodes.size(), oldCompareNodes.size(), newBfsNodes.size(), | ||
| 1501 | - newCompareNodes.size()); | ||
| 1502 | - } | ||
| 1503 | - if (useTransparentCompareView) { | ||
| 1504 | - HCCL_VM_INFO("[TaskGraphGeneratorV3] Skip BFS order compare in transparent graph compare view."); | ||
| 1505 | - } else { | ||
| 1506 | - CompareGraphV3BfsOrder(oldCompareNodes, newCompareNodes); | ||
| 1507 | - } | ||
| 1508 | - CompareGraphV3Semantic(oldCompareNodes, newCompareNodes); | ||
| 1509 | -} | ||
| 1510 | - | ||
| 1511 | HcclResult CheckGraphV3TaskMem(const V3TaskNode *start) | 726 | HcclResult CheckGraphV3TaskMem(const V3TaskNode *start) |
| 1512 | { | 727 | { |
| 1513 | - V3StageTimer timer("CheckGraphV3TaskMem"); | 728 | + V3StageTimer timer("SingleTaskCheck"); |
| 1514 | if (start == nullptr) { | 729 | if (start == nullptr) { |
| 1515 | - HCCL_VM_WARN("[TaskGraphSingleTaskCheckV3] Skip V3 task memory check, start is null."); | 730 | + HCCL_VM_WARN("Skip single-task memory check because the graph start node is null"); |
| 1516 | timer.SetStatus("skipped"); | 731 | timer.SetStatus("skipped"); |
| 1517 | return HCCL_SUCCESS; | 732 | return HCCL_SUCCESS; |
| 1518 | } | 733 | } |
| @@ -1535,9 +750,9 @@ HcclResult CheckGraphV3TaskMem(const V3TaskNode *start) | |||
| 1535 | 750 | ||
| 1536 | HcclResult CheckGraphV3MemConflict(const V3TaskNode *start) | 751 | HcclResult CheckGraphV3MemConflict(const V3TaskNode *start) |
| 1537 | { | 752 | { |
| 1538 | - V3StageTimer timer("CheckGraphV3MemConflict"); | 753 | + V3StageTimer timer("MemConflict"); |
| 1539 | if (start == nullptr) { | 754 | if (start == nullptr) { |
| 1540 | - HCCL_VM_WARN("[TaskGraphMemConflictV3] Skip V3 memory conflict check, start is null."); | 755 | + HCCL_VM_WARN("Skip memory-conflict check because the graph start node is null"); |
| 1541 | timer.SetStatus("skipped"); | 756 | timer.SetStatus("skipped"); |
| 1542 | return HCCL_SUCCESS; | 757 | return HCCL_SUCCESS; |
| 1543 | } | 758 | } |
| @@ -1561,9 +776,9 @@ HcclResult CheckGraphV3MemConflict(const V3TaskNode *start) | |||
| 1561 | 776 | ||
| 1562 | HcclResult CheckGraphV3Semantic(const V3TaskNode *start) | 777 | HcclResult CheckGraphV3Semantic(const V3TaskNode *start) |
| 1563 | { | 778 | { |
| 1564 | - V3StageTimer timer("CheckGraphV3Semantic"); | 779 | + V3StageTimer timer("SemanticCheck"); |
| 1565 | if (start == nullptr) { | 780 | if (start == nullptr) { |
| 1566 | - HCCL_VM_WARN("[TaskGraphSemanticCheckV3] Skip V3 semantic check, start is null."); | 781 | + HCCL_VM_WARN("Skip semantic check because the graph start node is null"); |
| 1567 | timer.SetStatus("skipped"); | 782 | timer.SetStatus("skipped"); |
| 1568 | return HCCL_SUCCESS; | 783 | return HCCL_SUCCESS; |
| 1569 | } | 784 | } |
| @@ -1613,7 +828,8 @@ HcclResult GenAndCheckGraphV3() | |||
| 1613 | } | 828 | } |
| 1614 | const HcclResult dumpRet = DumpV3Dag(*graphGeneratorV3); | 829 | const HcclResult dumpRet = DumpV3Dag(*graphGeneratorV3); |
| 1615 | if (dumpRet != HCCL_SUCCESS) { | 830 | if (dumpRet != HCCL_SUCCESS) { |
| 1616 | - HCCL_VM_WARN("[GenAndCheckGraphV3] failed to dump V3 dag, ret={}", static_cast<uint32_t>(dumpRet)); | 831 | + HCCL_VM_WARN("Failed to dump the V3 DAG, ret={}", |
| 832 | + static_cast<uint32_t>(dumpRet)); | ||
| 1617 | } | 833 | } |
| 1618 | const V3TaskNode *dummyStartV3 = graphGeneratorV3->GetMainStartNode(); | 834 | const V3TaskNode *dummyStartV3 = graphGeneratorV3->GetMainStartNode(); |
| 1619 | // 打印dag task任务 | 835 | // 打印dag task任务 |
| @@ -1654,7 +870,8 @@ HcclResult Checker::GenAndCheckGraph(AllRankTaskQueues &allRankTaskQueues, TaskC | |||
| 1654 | HCCL_VM_INFO("-------------------------------------------------------"); | 870 | HCCL_VM_INFO("-------------------------------------------------------"); |
| 1655 | for (auto& task : taskQueue[i]) { | 871 | for (auto& task : taskQueue[i]) { |
| 1656 | std::string tempStr = task->Describe(); | 872 | std::string tempStr = task->Describe(); |
| 1657 | - HCCL_VM_INFO("rankIdx:{:d}, taskIdx:{:d}, {:s}, pointer: {:p}", rankIdx, taskIdx, tempStr, static_cast<void*>(task.get())); | 873 | + HCCL_VM_INFO("rankIdx:{:d}, taskIdx:{:d}, {:s}, pointer: {:p}", rankIdx, taskIdx, tempStr, |
| 874 | + static_cast<void*>(task.get())); | ||
| 1658 | taskIdx++; | 875 | taskIdx++; |
| 1659 | } | 876 | } |
| 1660 | } | 877 | } |
| @@ -1662,18 +879,18 @@ HcclResult Checker::GenAndCheckGraph(AllRankTaskQueues &allRankTaskQueues, TaskC | |||
| 1662 | { | 879 | { |
| 1663 | const HcclResult dumpRet = DumpInputTaskQueues(allRankTaskQueues); | 880 | const HcclResult dumpRet = DumpInputTaskQueues(allRankTaskQueues); |
| 1664 | if (dumpRet != HcclResult::HCCL_SUCCESS) { | 881 | if (dumpRet != HcclResult::HCCL_SUCCESS) { |
| 1665 | - HCCL_VM_WARN("[Checker::GenAndCheckGraph] failed to dump input task queues, ret[%u].", | 882 | + HCCL_VM_WARN("Failed to dump the old checker input task queues, ret={}", |
| 1666 | static_cast<u32>(dumpRet)); | 883 | static_cast<u32>(dumpRet)); |
| 1667 | } | 884 | } |
| 1668 | } | 885 | } |
| 1669 | 886 | ||
| 1670 | // 1. 检查从流 | 887 | // 1. 检查从流 |
| 1671 | - HCCL_INFO("1. 检查从流"); | 888 | + HCCL_VM_INFO("1. 检查从流"); |
| 1672 | SingleTaskCheck taskChecker; | 889 | SingleTaskCheck taskChecker; |
| 1673 | CHK_RET(taskChecker.CheckSlaveTaskQueue(allRankTaskQueues)); | 890 | CHK_RET(taskChecker.CheckSlaveTaskQueue(allRankTaskQueues)); |
| 1674 | 891 | ||
| 1675 | // 2. 成图 | 892 | // 2. 成图 |
| 1676 | - HCCL_INFO("2. 成图"); | 893 | + HCCL_VM_INFO("2. 成图"); |
| 1677 | TaskNode dummyStart = TaskNode(nullptr, -1, 0, 0); | 894 | TaskNode dummyStart = TaskNode(nullptr, -1, 0, 0); |
| 1678 | TaskNode dummyStartCopy = TaskNode(nullptr, -1, 0, 0); | 895 | TaskNode dummyStartCopy = TaskNode(nullptr, -1, 0, 0); |
| 1679 | TaskGraphGenerator graphGenerator; | 896 | TaskGraphGenerator graphGenerator; |
| @@ -1681,13 +898,13 @@ HcclResult Checker::GenAndCheckGraph(AllRankTaskQueues &allRankTaskQueues, TaskC | |||
| 1681 | 898 | ||
| 1682 | // 3. Task内存校验 | 899 | // 3. Task内存校验 |
| 1683 | // 是否可以复用 taskChecker | 900 | // 是否可以复用 taskChecker |
| 1684 | - HCCL_INFO("3. Task内存校验"); | 901 | + HCCL_VM_INFO("3. Task内存校验"); |
| 1685 | CHK_RET(taskChecker.CheckTaskMem(&dummyStart)); | 902 | CHK_RET(taskChecker.CheckTaskMem(&dummyStart)); |
| 1686 | - HCCL_INFO("3. Task内存校验 END: %d", closeRankMemCheck_); | 903 | + HCCL_VM_INFO("3. Task内存校验 END: {}", closeRankMemCheck_); |
| 1687 | 904 | ||
| 1688 | if (!closeRankMemCheck_) { | 905 | if (!closeRankMemCheck_) { |
| 1689 | // 4. 图复制 | 906 | // 4. 图复制 |
| 1690 | - HCCL_INFO("4. 图复制"); | 907 | + HCCL_VM_INFO("4. 图复制"); |
| 1691 | if (dummyStart.hasCcuTask) { | 908 | if (dummyStart.hasCcuTask) { |
| 1692 | CHK_RET(CopyCcuTaskGraph(&dummyStart, &dummyStartCopy, rankNum)); | 909 | CHK_RET(CopyCcuTaskGraph(&dummyStart, &dummyStartCopy, rankNum)); |
| 1693 | dummyStartCopy.hasCcuTask = true; | 910 | dummyStartCopy.hasCcuTask = true; |
| @@ -1696,7 +913,7 @@ HcclResult Checker::GenAndCheckGraph(AllRankTaskQueues &allRankTaskQueues, TaskC | |||
| 1696 | } | 913 | } |
| 1697 | 914 | ||
| 1698 | // 5. 图改造 | 915 | // 5. 图改造 |
| 1699 | - HCCL_INFO("5. 图改造"); | 916 | + HCCL_VM_INFO("5. 图改造"); |
| 1700 | GraphRevampBilateralSemantics graphRevamp; | 917 | GraphRevampBilateralSemantics graphRevamp; |
| 1701 | 918 | ||
| 1702 | // ccu图改造 | 919 | // ccu图改造 |
| @@ -1720,17 +937,17 @@ HcclResult Checker::GenAndCheckGraph(AllRankTaskQueues &allRankTaskQueues, TaskC | |||
| 1720 | CHK_RET(graphRevamp.Revamp(&dummyStartCopy)); | 937 | CHK_RET(graphRevamp.Revamp(&dummyStartCopy)); |
| 1721 | 938 | ||
| 1722 | // 6. Rank内存校验 | 939 | // 6. Rank内存校验 |
| 1723 | - HCCL_INFO("6. Rank内存校验"); | 940 | + HCCL_VM_INFO("6. Rank内存校验"); |
| 1724 | CheckRankMem checkRankmem(&dummyStartCopy); | 941 | CheckRankMem checkRankmem(&dummyStartCopy); |
| 1725 | CHK_RET(checkRankmem.Execute()); | 942 | CHK_RET(checkRankmem.Execute()); |
| 1726 | } | 943 | } |
| 1727 | 944 | ||
| 1728 | // 7. 语义校验 | 945 | // 7. 语义校验 |
| 1729 | - HCCL_INFO("7. 语义校验"); | 946 | + HCCL_VM_INFO("7. 语义校验"); |
| 1730 | // PrintCcuGraph(&dummyStart); | 947 | // PrintCcuGraph(&dummyStart); |
| 1731 | opSemanticsChecker.SetGraphHead(&dummyStart); | 948 | opSemanticsChecker.SetGraphHead(&dummyStart); |
| 1732 | CHK_RET(opSemanticsChecker.Execute()); | 949 | CHK_RET(opSemanticsChecker.Execute()); |
| 1733 | - HCCL_INFO("8. 语义校验 END"); | 950 | + HCCL_VM_INFO("8. 语义校验 END"); |
| 1734 | 951 | ||
| 1735 | // 成图及校验成功 | 952 | // 成图及校验成功 |
| 1736 | return HCCL_SUCCESS; | 953 | return HCCL_SUCCESS; |
| @@ -73,7 +73,7 @@ void CcuErrorHandler::GetCcuErrorMsg(int32_t deviceId, const ParaCcu &ccuTaskPar | |||
| 73 | const uint16_t missionStatus = missionContext.GetStatus(); | 73 | const uint16_t missionStatus = missionContext.GetStatus(); |
| 74 | if (missionStatus == 0) { | 74 | if (missionStatus == 0) { |
| 75 | HCCL_INFO("[CcuErrorHandler] no err found, mission status is 0, deviceId[%d], dieId[%u], execMissionId[%u]", | 75 | HCCL_INFO("[CcuErrorHandler] no err found, mission status is 0, deviceId[%d], dieId[%u], execMissionId[%u]", |
| 76 | - __func__, deviceId, static_cast<u32>(ccuTaskParam.dieId), static_cast<u32>(ccuTaskParam.execMissionId)); | 76 | + deviceId, static_cast<u32>(ccuTaskParam.dieId), static_cast<u32>(ccuTaskParam.execMissionId)); |
| 77 | return; | 77 | return; |
| 78 | } | 78 | } |
| 79 | 79 | ||
| @@ -495,7 +495,7 @@ HcclResult CheckRankMem::CompareSliceMemoryInfo(FragQueueMemStatus &left, FragQu | |||
| 495 | if (right.count(type) != 0) { | 495 | if (right.count(type) != 0) { |
| 496 | auto ret = CompareBufferTypeMemoryInfo(iter->second, right[type]); | 496 | auto ret = CompareBufferTypeMemoryInfo(iter->second, right[type]); |
| 497 | if (ret != HcclResult::HCCL_SUCCESS) { | 497 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 498 | - HCCL_ERROR("failed to check memory %", type); | 498 | + HCCL_ERROR("failed to check memory type[%d]", static_cast<int>(type)); |
| 499 | return ret; | 499 | return ret; |
| 500 | } | 500 | } |
| 501 | } | 501 | } |
| @@ -901,7 +901,7 @@ HcclResult CheckRankMem::CcuGraphMemCheck() | |||
| 901 | } | 901 | } |
| 902 | numCcuGraph++; | 902 | numCcuGraph++; |
| 903 | } | 903 | } |
| 904 | - HCCL_VM_INFO("[CcuGraphMemCheck] %u CcuGraphMemCheck proc success. ccu graph num: %d", rankId, numCcuGraph); | 904 | + HCCL_VM_INFO("{} CcuGraphMemCheck proc success. ccu graph num: {}", rankId, numCcuGraph); |
| 905 | } | 905 | } |
| 906 | ClearCcuData(); | 906 | ClearCcuData(); |
| 907 | 907 | ||
| @@ -1034,7 +1034,7 @@ void CheckRankMem::GenFragQueueInOneQueueOnly(TaskNode *head) | |||
| 1034 | auto type = child->task->GetType(); | 1034 | auto type = child->task->GetType(); |
| 1035 | if (type == TaskTypeStub::LOCAL_WAIT_FROM || | 1035 | if (type == TaskTypeStub::LOCAL_WAIT_FROM || |
| 1036 | type == TaskTypeStub::LOCAL_WAIT_FROM_SHADOW) { | 1036 | type == TaskTypeStub::LOCAL_WAIT_FROM_SHADOW) { |
| 1037 | - HCCL_VM_DEBUG("[GenFragQueueInOneQueueOnly] ccu multi samequeue node found, node: {} " | 1037 | + HCCL_VM_DEBUG("ccu multi samequeue node found, node: {} " |
| 1038 | "ignore child: {} addr: {}", | 1038 | "ignore child: {} addr: {}", |
| 1039 | curNode->task->Describe(), | 1039 | curNode->task->Describe(), |
| 1040 | child->task->Describe(), | 1040 | child->task->Describe(), |
| @@ -1126,7 +1126,7 @@ HcclResult CheckRankMem::CcuGraphMemCheckProc(RankId rankId, uint32_t queueId, T | |||
| 1126 | CHK_RET(GenFragQueConcurrencyMatrixAndCompare(rankId)); | 1126 | CHK_RET(GenFragQueConcurrencyMatrixAndCompare(rankId)); |
| 1127 | ClearCcuData(); | 1127 | ClearCcuData(); |
| 1128 | 1128 | ||
| 1129 | - HCCL_VM_DEBUG("[CcuGraphMemCheckProc] ccuHeadChildNum= {}", ccuHeadChildNum); | 1129 | + HCCL_VM_DEBUG("ccuHeadChildNum= {}", ccuHeadChildNum); |
| 1130 | return HcclResult::HCCL_SUCCESS; | 1130 | return HcclResult::HCCL_SUCCESS; |
| 1131 | } | 1131 | } |
| 1132 | 1132 | ||
| @@ -17,7 +17,9 @@ | |||
| 17 | 17 | ||
| 18 | 18 | ||
| 19 | 19 | ||
| 20 | -#include "log.h" | 20 | +#include "sim_log.h" |
| 21 | + | ||
| 22 | + | ||
| 21 | 23 | ||
| 22 | namespace HcclSim { | 24 | namespace HcclSim { |
| 23 | void GenSendOffsetForOneRank(All2AllDataDesTagInner &all2AllDataDes, u32 rankSize, RankId targetRank, std::vector<u64> &sendOffsets) | 25 | void GenSendOffsetForOneRank(All2AllDataDesTagInner &all2AllDataDes, u32 rankSize, RankId targetRank, std::vector<u64> &sendOffsets) |
| @@ -42,7 +44,9 @@ HcclResult TaskCheckAll2AllSemantics(std::map<RankId, RankMemorySemantics> &allR | |||
| 42 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 44 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 43 | // 对应的rank不存在需要报错 | 45 | // 对应的rank不存在需要报错 |
| 44 | if (allRankMemSemantics.count(rankId) == 0) { | 46 | if (allRankMemSemantics.count(rankId) == 0) { |
| 45 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 47 | + HCCL_VM_ERROR("{} AllToAll produced no result data for rank {}, but this rank is " |
| 48 | + "expected to receive data from peer ranks, expectedSourceRankCount={}", | ||
| 49 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, rankSize); | ||
| 46 | return HcclResult::HCCL_E_PARA; | 50 | return HcclResult::HCCL_E_PARA; |
| 47 | } | 51 | } |
| 48 | 52 | ||
| @@ -59,39 +63,59 @@ HcclResult TaskCheckAll2AllSemantics(std::map<RankId, RankMemorySemantics> &allR | |||
| 59 | } | 63 | } |
| 60 | 64 | ||
| 61 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 65 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 66 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 62 | if (ele.startAddr != totalSize) { | 67 | if (ele.startAddr != totalSize) { |
| 63 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: " | 68 | + HCCL_VM_ERROR("{} AllToAll result data does not start from the expected address, " |
| 64 | - "exepected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 69 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 65 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 70 | + "\nCurrent result range detail:\n{}", |
| 71 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 72 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 66 | return HcclResult::HCCL_E_PARA; | 73 | return HcclResult::HCCL_E_PARA; |
| 67 | } | 74 | } |
| 68 | 75 | ||
| 69 | if (ele.srcBufs.size() != 1) { | 76 | if (ele.srcBufs.size() != 1) { |
| 70 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, " | 77 | + HCCL_VM_ERROR("{} This AllToAll result range combines multiple sources, but this " |
| 71 | - "while cur buffer semantic is %s", rankId, ele.Describe().c_str()); | 78 | + "operator expects exactly one source, rankId={}, actualSourceCount={}, expectedSourceCount=1, " |
| 79 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 80 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 81 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 72 | return HcclResult::HCCL_E_PARA; | 82 | return HcclResult::HCCL_E_PARA; |
| 73 | } | 83 | } |
| 74 | 84 | ||
| 75 | if (curRankId >= rankSize) { | 85 | if (curRankId >= rankSize) { |
| 76 | - HCCL_ERROR("[rankId:%u]Extra buffer semantic found", rankId); | 86 | + HCCL_VM_ERROR("{} Extra output data exists after the expected AllToAll result " |
| 87 | + "range, rankId={}, extraBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 88 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), rankId, ele.startAddr, rangeEnd, | ||
| 89 | + ele.Describe()); | ||
| 77 | return HcclResult::HCCL_E_PARA; | 90 | return HcclResult::HCCL_E_PARA; |
| 78 | } | 91 | } |
| 79 | 92 | ||
| 80 | if (ele.srcBufs.begin()->rankId != curRankId) { | 93 | if (ele.srcBufs.begin()->rankId != curRankId) { |
| 81 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should come from rank %u, while it come from rank %u, " | 94 | + HCCL_VM_ERROR("{} This AllToAll result range comes from the wrong source rank, " |
| 82 | - "cur buffer semantic is %s", rankId, curRankId, ele.srcBufs.begin()->rankId, ele.Describe().c_str()); | 95 | + "rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" |
| 96 | + "\nCurrent result range detail:\n{}", | ||
| 97 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 98 | + ele.srcBufs.begin()->rankId, curRankId, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 83 | return HcclResult::HCCL_E_PARA; | 99 | return HcclResult::HCCL_E_PARA; |
| 84 | } | 100 | } |
| 85 | 101 | ||
| 86 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 102 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 87 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 103 | + HCCL_VM_ERROR("{} This AllToAll result range comes from a non-INPUT buffer, but it " |
| 88 | - rankId, ele.Describe().c_str()); | 104 | + "should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 105 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 106 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 107 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 108 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 89 | return HcclResult::HCCL_E_PARA; | 109 | return HcclResult::HCCL_E_PARA; |
| 90 | } | 110 | } |
| 91 | 111 | ||
| 92 | if (ele.srcBufs.begin()->srcAddr != sendOffsets[curRankId] + curDataSize) { | 112 | if (ele.srcBufs.begin()->srcAddr != sendOffsets[curRankId] + curDataSize) { |
| 93 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu, while it is %llu, cur buffer semantic is %s", | 113 | + HCCL_VM_ERROR("{} Source address for this AllToAll result range does not match the " |
| 94 | - rankId, sendOffsets[curRankId] + curDataSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 114 | + "expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 115 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 116 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 117 | + ele.srcBufs.begin()->rankId, sendOffsets[curRankId] + curDataSize, | ||
| 118 | + ele.srcBufs.begin()->srcAddr, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 95 | return HcclResult::HCCL_E_PARA; | 119 | return HcclResult::HCCL_E_PARA; |
| 96 | } | 120 | } |
| 97 | 121 | ||
| @@ -106,17 +130,20 @@ HcclResult TaskCheckAll2AllSemantics(std::map<RankId, RankMemorySemantics> &allR | |||
| 106 | curRankId++; | 130 | curRankId++; |
| 107 | } | 131 | } |
| 108 | } else if (curDataSize > recvDataSizeFromCurRank) { | 132 | } else if (curDataSize > recvDataSizeFromCurRank) { |
| 109 | - HCCL_ERROR("[rankId:%u]Accumulated semantic size from rank %u is %llu, greater than expected %llu", | 133 | + HCCL_VM_ERROR("{} Data taken from one source rank is larger than expected in " |
| 110 | - rankId, curRankId, curDataSize, recvDataSizeFromCurRank); | 134 | + "AllToAll, rankId={}, sourceRank={}, actualSize=0x{:x}, expectedSize=0x{:x}, " |
| 135 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 136 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SIZE_ERROR), rankId, curRankId, curDataSize, | ||
| 137 | + recvDataSizeFromCurRank, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 111 | return HcclResult::HCCL_E_PARA; | 138 | return HcclResult::HCCL_E_PARA; |
| 112 | } | 139 | } |
| 113 | totalSize += ele.size; | 140 | totalSize += ele.size; |
| 114 | } | 141 | } |
| 115 | // 如果curRankId等于rankSize,表示已经接受到其他所有rank的数据 | 142 | // 如果curRankId等于rankSize,表示已经接受到其他所有rank的数据 |
| 116 | if (curRankId != rankSize) { | 143 | if (curRankId != rankSize) { |
| 117 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, " | 144 | + HCCL_VM_ERROR("{} AllToAll result data ends before the expected total size is reached, " |
| 118 | - "accumulated semantic size from rank %u is %llu, while rankSize is %u", | 145 | + "rankId={}, checkedSize=0x{:x}, remainingSourceRank={}, remainingSizeFromThatRank=0x{:x}", |
| 119 | - rankId, totalSize, curRankId, curDataSize, rankSize); | 146 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, curRankId, curDataSize); |
| 120 | return HcclResult::HCCL_E_PARA; | 147 | return HcclResult::HCCL_E_PARA; |
| 121 | } | 148 | } |
| 122 | } | 149 | } |
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckAllGatherSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize) | 22 | HcclResult TaskCheckAllGatherSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize) |
| @@ -23,7 +26,9 @@ HcclResult TaskCheckAllGatherSemantics(std::map<RankId, RankMemorySemantics> &al | |||
| 23 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 26 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 24 | // 对应的rank不存在需要报错 | 27 | // 对应的rank不存在需要报错 |
| 25 | if (allRankMemSemantics.count(rankId) == 0) { | 28 | if (allRankMemSemantics.count(rankId) == 0) { |
| 26 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 29 | + HCCL_VM_ERROR("{} AllGather produced no result data for rank {}, but this rank is " |
| 30 | + "expected to receive data from all ranks, expectedSourceRankCount={}, expectedResultSize=0x{:x}", | ||
| 31 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, rankSize, dataSize * rankSize); | ||
| 27 | return HcclResult::HCCL_E_PARA; | 32 | return HcclResult::HCCL_E_PARA; |
| 28 | } | 33 | } |
| 29 | 34 | ||
| @@ -31,31 +36,51 @@ HcclResult TaskCheckAllGatherSemantics(std::map<RankId, RankMemorySemantics> &al | |||
| 31 | RankId curRankId = 0; | 36 | RankId curRankId = 0; |
| 32 | u64 curDataSize = 0; | 37 | u64 curDataSize = 0; |
| 33 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 38 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 39 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 34 | if (ele.startAddr != totalSize) { | 40 | if (ele.startAddr != totalSize) { |
| 35 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 41 | + HCCL_VM_ERROR("{} AllGather result data does not start from the expected address, " |
| 36 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 42 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 43 | + "\nCurrent result range detail:\n{}", | ||
| 44 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 45 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 37 | return HcclResult::HCCL_E_PARA; | 46 | return HcclResult::HCCL_E_PARA; |
| 38 | } | 47 | } |
| 39 | 48 | ||
| 40 | if (ele.srcBufs.size() != 1) { | 49 | if (ele.srcBufs.size() != 1) { |
| 41 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, while cur buffer semantic is %s", rankId, ele.Describe().c_str()); | 50 | + HCCL_VM_ERROR("{} This AllGather result range combines multiple sources, but this " |
| 51 | + "operator expects exactly one source, rankId={}, actualSourceCount={}, expectedSourceCount=1, " | ||
| 52 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 53 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 54 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 42 | return HcclResult::HCCL_E_PARA; | 55 | return HcclResult::HCCL_E_PARA; |
| 43 | } | 56 | } |
| 44 | 57 | ||
| 45 | if (ele.srcBufs.begin()->rankId != curRankId) { | 58 | if (ele.srcBufs.begin()->rankId != curRankId) { |
| 46 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should come from rank %u, while it come from rank %u, cur buffer semantic is %s", rankId, curRankId, ele.srcBufs.begin()->rankId, ele.Describe().c_str()); | 59 | + HCCL_VM_ERROR("{} This AllGather result range comes from the wrong source rank, " |
| 60 | + "rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" | ||
| 61 | + "\nCurrent result range detail:\n{}", | ||
| 62 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 63 | + ele.srcBufs.begin()->rankId, curRankId, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 47 | return HcclResult::HCCL_E_PARA; | 64 | return HcclResult::HCCL_E_PARA; |
| 48 | } | 65 | } |
| 49 | 66 | ||
| 50 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 67 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 51 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 68 | + HCCL_VM_ERROR("{} This AllGather result range comes from a non-INPUT buffer, but it " |
| 52 | - rankId, ele.Describe().c_str()); | 69 | + "should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 70 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 71 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 72 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 73 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 53 | return HcclResult::HCCL_E_PARA; | 74 | return HcclResult::HCCL_E_PARA; |
| 54 | } | 75 | } |
| 55 | 76 | ||
| 56 | if (ele.srcBufs.begin()->srcAddr != curDataSize) { | 77 | if (ele.srcBufs.begin()->srcAddr != curDataSize) { |
| 57 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu, while it is %llu, cur buffer semantic is %s", | 78 | + HCCL_VM_ERROR("{} Source address for this AllGather result range does not match the " |
| 58 | - rankId, curDataSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 79 | + "expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 80 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 81 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 82 | + ele.srcBufs.begin()->rankId, curDataSize, ele.srcBufs.begin()->srcAddr, ele.startAddr, | ||
| 83 | + rangeEnd, ele.Describe()); | ||
| 59 | return HcclResult::HCCL_E_PARA; | 84 | return HcclResult::HCCL_E_PARA; |
| 60 | } | 85 | } |
| 61 | 86 | ||
| @@ -64,15 +89,19 @@ HcclResult TaskCheckAllGatherSemantics(std::map<RankId, RankMemorySemantics> &al | |||
| 64 | curDataSize = 0; | 89 | curDataSize = 0; |
| 65 | curRankId++; | 90 | curRankId++; |
| 66 | } else if (curDataSize > dataSize) { | 91 | } else if (curDataSize > dataSize) { |
| 67 | - HCCL_ERROR("[rankId:%u]Accumulated semantic size from rank %u is %llu, greater than expected %llu", | 92 | + HCCL_VM_ERROR("{} Data taken from one source rank is larger than expected in " |
| 68 | - rankId, curRankId, curDataSize, dataSize); | 93 | + "AllGather, rankId={}, sourceRank={}, actualSize=0x{:x}, expectedSize=0x{:x}, " |
| 94 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 95 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SIZE_ERROR), rankId, curRankId, curDataSize, | ||
| 96 | + dataSize, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 69 | return HcclResult::HCCL_E_PARA; | 97 | return HcclResult::HCCL_E_PARA; |
| 70 | } | 98 | } |
| 71 | totalSize += ele.size; | 99 | totalSize += ele.size; |
| 72 | } | 100 | } |
| 73 | if (totalSize != dataSize * rankSize) { | 101 | if (totalSize != dataSize * rankSize) { |
| 74 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, accumulated semantic size from rank %u is %llu, while expected total size is %llu", | 102 | + HCCL_VM_ERROR("{} AllGather result data ends before the expected total size is " |
| 75 | - rankId, totalSize, curRankId, curDataSize, dataSize * rankSize); | 103 | + "reached, rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", |
| 104 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, dataSize * rankSize); | ||
| 76 | return HcclResult::HCCL_E_PARA; | 105 | return HcclResult::HCCL_E_PARA; |
| 77 | } | 106 | } |
| 78 | } | 107 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/semantics_check/allgather_v_semantics_checker.cc+42-18
| @@ -14,7 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | -#include "log.h" | 17 | +#include "sim_log.h" |
| 18 | + | ||
| 19 | + | ||
| 18 | 20 | ||
| 19 | namespace HcclSim { | 21 | namespace HcclSim { |
| 20 | HcclResult TaskCheckAllGatherVSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, VDataDesTagInner &vDataDes) | 22 | HcclResult TaskCheckAllGatherVSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, VDataDesTagInner &vDataDes) |
| @@ -32,7 +34,9 @@ HcclResult TaskCheckAllGatherVSemantics(std::map<RankId, RankMemorySemantics> &a | |||
| 32 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 34 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 33 | // 对应的rank不存在需要报错 | 35 | // 对应的rank不存在需要报错 |
| 34 | if (allRankMemSemantics.count(rankId) == 0) { | 36 | if (allRankMemSemantics.count(rankId) == 0) { |
| 35 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 37 | + HCCL_VM_ERROR("{} AllGatherV produced no result data for rank {}, but this rank is " |
| 38 | + "expected to receive data from all ranks, expectedSourceRankCount={}, expectedResultSize=0x{:x}", | ||
| 39 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, rankSize, outputSize); | ||
| 36 | return HcclResult::HCCL_E_PARA; | 40 | return HcclResult::HCCL_E_PARA; |
| 37 | } | 41 | } |
| 38 | 42 | ||
| @@ -45,35 +49,52 @@ HcclResult TaskCheckAllGatherVSemantics(std::map<RankId, RankMemorySemantics> &a | |||
| 45 | curRankId++; | 49 | curRankId++; |
| 46 | inputSize = vDataDes.counts[curRankId] * CHECK_SIZE_TABLE[vDataDes.dataType]; | 50 | inputSize = vDataDes.counts[curRankId] * CHECK_SIZE_TABLE[vDataDes.dataType]; |
| 47 | } | 51 | } |
| 52 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 48 | 53 | ||
| 49 | if (ele.startAddr != totalSize) { | 54 | if (ele.startAddr != totalSize) { |
| 50 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: " | 55 | + HCCL_VM_ERROR("{} AllGatherV result data does not start from the expected address, " |
| 51 | - "exepected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 56 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 52 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 57 | + "\nCurrent result range detail:\n{}", |
| 58 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 59 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 53 | return HcclResult::HCCL_E_PARA; | 60 | return HcclResult::HCCL_E_PARA; |
| 54 | } | 61 | } |
| 55 | 62 | ||
| 56 | if (ele.srcBufs.size() != 1) { | 63 | if (ele.srcBufs.size() != 1) { |
| 57 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, " | 64 | + HCCL_VM_ERROR("{} This AllGatherV result range combines multiple sources, but this " |
| 58 | - "while cur buffer semantic is %s", rankId, ele.Describe().c_str()); | 65 | + "operator expects exactly one source, rankId={}, actualSourceCount={}, expectedSourceCount=1, " |
| 66 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 67 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 68 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 59 | return HcclResult::HCCL_E_PARA; | 69 | return HcclResult::HCCL_E_PARA; |
| 60 | } | 70 | } |
| 61 | 71 | ||
| 62 | if (ele.srcBufs.begin()->rankId != curRankId) { | 72 | if (ele.srcBufs.begin()->rankId != curRankId) { |
| 63 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should come from rank %u, while it come from rank %u, " | 73 | + HCCL_VM_ERROR("{} This AllGatherV result range comes from the wrong source rank, " |
| 64 | - "cur buffer semantic is %s", rankId, curRankId, ele.srcBufs.begin()->rankId, ele.Describe().c_str()); | 74 | + "rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" |
| 75 | + "\nCurrent result range detail:\n{}", | ||
| 76 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 77 | + ele.srcBufs.begin()->rankId, curRankId, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 65 | return HcclResult::HCCL_E_PARA; | 78 | return HcclResult::HCCL_E_PARA; |
| 66 | } | 79 | } |
| 67 | 80 | ||
| 68 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 81 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 69 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 82 | + HCCL_VM_ERROR("{} This AllGatherV result range comes from a non-INPUT buffer, but " |
| 70 | - rankId, ele.Describe().c_str()); | 83 | + "it should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 84 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 85 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 86 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 87 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 71 | return HcclResult::HCCL_E_PARA; | 88 | return HcclResult::HCCL_E_PARA; |
| 72 | } | 89 | } |
| 73 | 90 | ||
| 74 | if (ele.srcBufs.begin()->srcAddr != curDataSize) { | 91 | if (ele.srcBufs.begin()->srcAddr != curDataSize) { |
| 75 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu, while it is %llu, cur buffer semantic is %s", | 92 | + HCCL_VM_ERROR("{} Source address for this AllGatherV result range does not match " |
| 76 | - rankId, curDataSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 93 | + "the expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 94 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 95 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 96 | + ele.srcBufs.begin()->rankId, curDataSize, ele.srcBufs.begin()->srcAddr, ele.startAddr, | ||
| 97 | + rangeEnd, ele.Describe()); | ||
| 77 | return HcclResult::HCCL_E_PARA; | 98 | return HcclResult::HCCL_E_PARA; |
| 78 | } | 99 | } |
| 79 | 100 | ||
| @@ -83,16 +104,19 @@ HcclResult TaskCheckAllGatherVSemantics(std::map<RankId, RankMemorySemantics> &a | |||
| 83 | curDataSize = 0; | 104 | curDataSize = 0; |
| 84 | curRankId++; | 105 | curRankId++; |
| 85 | } else if (curDataSize > inputSize) { | 106 | } else if (curDataSize > inputSize) { |
| 86 | - HCCL_ERROR("[rankId:%u]Accumulated semantic size from rank %u is %llu, greater than expected %llu", | 107 | + HCCL_VM_ERROR("{} Data taken from one source rank is larger than expected in " |
| 87 | - rankId, curRankId, curDataSize, inputSize); | 108 | + "AllGatherV, rankId={}, sourceRank={}, actualSize=0x{:x}, expectedSize=0x{:x}, " |
| 109 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 110 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SIZE_ERROR), rankId, curRankId, curDataSize, | ||
| 111 | + inputSize, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 88 | return HcclResult::HCCL_E_PARA; | 112 | return HcclResult::HCCL_E_PARA; |
| 89 | } | 113 | } |
| 90 | totalSize += ele.size; | 114 | totalSize += ele.size; |
| 91 | } | 115 | } |
| 92 | if (totalSize != outputSize) { | 116 | if (totalSize != outputSize) { |
| 93 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, " | 117 | + HCCL_VM_ERROR("{} AllGatherV result data ends before the expected total size is " |
| 94 | - "while outputSize is %llu, rankSize is %u", | 118 | + "reached, rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", |
| 95 | - rankId, totalSize, outputSize, rankSize); | 119 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, outputSize); |
| 96 | return HcclResult::HCCL_E_PARA; | 120 | return HcclResult::HCCL_E_PARA; |
| 97 | } | 121 | } |
| 98 | } | 122 | } |
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckAllReduceSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, | 22 | HcclResult TaskCheckAllReduceSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, |
| @@ -24,54 +27,79 @@ HcclResult TaskCheckAllReduceSemantics(std::map<RankId, RankMemorySemantics> &al | |||
| 24 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 27 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 25 | // 对应的rank不存在需要报错 | 28 | // 对应的rank不存在需要报错 |
| 26 | if (allRankMemSemantics.count(rankId) == 0) { | 29 | if (allRankMemSemantics.count(rankId) == 0) { |
| 27 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 30 | + HCCL_VM_ERROR("{} AllReduce produced no result data for rank {}, but this rank is " |
| 31 | + "expected to have a full reduced result, expectedSourceRankCount={}, expectedResultSize=0x{:x}", | ||
| 32 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, rankSize, dataSize); | ||
| 28 | return HcclResult::HCCL_E_PARA; | 33 | return HcclResult::HCCL_E_PARA; |
| 29 | } | 34 | } |
| 30 | 35 | ||
| 31 | u64 totalSize = 0; | 36 | u64 totalSize = 0; |
| 32 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 37 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 38 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 33 | if (ele.startAddr != totalSize) { | 39 | if (ele.startAddr != totalSize) { |
| 34 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 40 | + HCCL_VM_ERROR("{} AllReduce result data does not start from the expected address, " |
| 35 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 41 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 42 | + "\nCurrent result range detail:\n{}", | ||
| 43 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 44 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 36 | return HcclResult::HCCL_E_PARA; | 45 | return HcclResult::HCCL_E_PARA; |
| 37 | } | 46 | } |
| 38 | 47 | ||
| 39 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { | 48 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { |
| 40 | - HCCL_ERROR("[rankId:%u]cur buffer semantic reduceType %d is unequal to expected reduceType %d, cur buffer semantic is %s", | 49 | + HCCL_VM_ERROR("{} Reduce mode of this AllReduce result range does not match the " |
| 41 | - rankId, ele.reduceType, reduceType, ele.Describe().c_str()); | 50 | + "operator setting, rankId={}, actualReduceType={}, expectedReduceType={}, " |
| 51 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 52 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, | ||
| 53 | + DumpReduceOpToString(ele.reduceType), DumpReduceOpToString(reduceType), | ||
| 54 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 42 | return HcclResult::HCCL_E_PARA; | 55 | return HcclResult::HCCL_E_PARA; |
| 43 | } | 56 | } |
| 44 | 57 | ||
| 45 | if (ele.srcBufs.size() != rankSize) { | 58 | if (ele.srcBufs.size() != rankSize) { |
| 46 | - HCCL_ERROR("[rankId:%u]buffer semantic srcBufs size %u is unequal to rankSize %u, cur buffer semantic is %s", | 59 | + HCCL_VM_ERROR("{} This AllReduce result range does not include the expected number " |
| 47 | - rankId, ele.srcBufs.size(), rankSize, ele.Describe().c_str()); | 60 | + "of source ranks, rankId={}, actualSourceRankCount={}, expectedRankSize={}, " |
| 61 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 62 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 63 | + rankSize, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 48 | return HcclResult::HCCL_E_PARA; | 64 | return HcclResult::HCCL_E_PARA; |
| 49 | } | 65 | } |
| 50 | 66 | ||
| 51 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != (rankSize - 1)) { | 67 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != (rankSize - 1)) { |
| 52 | - HCCL_ERROR("[rankId:%u]cur buffer semantic srcBufs is invalid, cur buffer semantic is %s", | 68 | + HCCL_VM_ERROR("{} Source rank list for this AllReduce result range is not the " |
| 53 | - rankId, ele.Describe().c_str()); | 69 | + "complete expected rank set, rankId={}, firstSourceRank={}, lastSourceRank={}, " |
| 70 | + "expectedFirstSourceRank=0, expectedLastSourceRank={}, outputRange=[0x{:x},0x{:x})" | ||
| 71 | + "\nCurrent result range detail:\n{}", | ||
| 72 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 73 | + ele.srcBufs.begin()->rankId, ele.srcBufs.rbegin()->rankId, rankSize - 1, ele.startAddr, | ||
| 74 | + rangeEnd, ele.Describe()); | ||
| 54 | return HcclResult::HCCL_E_PARA; | 75 | return HcclResult::HCCL_E_PARA; |
| 55 | } | 76 | } |
| 56 | 77 | ||
| 57 | for (auto &srcBuf : ele.srcBufs) { | 78 | for (auto &srcBuf : ele.srcBufs) { |
| 58 | if (srcBuf.bufType != BufferType::INPUT) { | 79 | if (srcBuf.bufType != BufferType::INPUT) { |
| 59 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 80 | + HCCL_VM_ERROR("{} This AllReduce result range comes from a non-INPUT buffer, " |
| 60 | - rankId, ele.Describe().c_str()); | 81 | + "but it should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 82 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 83 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, srcBuf.rankId, | ||
| 84 | + BufferTypeToString(srcBuf.bufType), ele.startAddr, rangeEnd, ele.Describe()); | ||
| 61 | return HcclResult::HCCL_E_PARA; | 85 | return HcclResult::HCCL_E_PARA; |
| 62 | } | 86 | } |
| 63 | 87 | ||
| 64 | if (srcBuf.srcAddr != totalSize) { | 88 | if (srcBuf.srcAddr != totalSize) { |
| 65 | - HCCL_ERROR("[rankId:%u]Expected semantic srcBuf srcAddr is %llu, while cur srcBuf srcAddr is %llu, cur buffer semantic is %s", | 89 | + HCCL_VM_ERROR("{} Source address for this AllReduce result range does not match " |
| 66 | - rankId, totalSize, srcBuf.srcAddr, ele.Describe().c_str()); | 90 | + "the expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, " |
| 91 | + "actualAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 92 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, srcBuf.rankId, totalSize, | ||
| 93 | + srcBuf.srcAddr, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 67 | return HcclResult::HCCL_E_PARA; | 94 | return HcclResult::HCCL_E_PARA; |
| 68 | } | 95 | } |
| 69 | } | 96 | } |
| 70 | totalSize += ele.size; | 97 | totalSize += ele.size; |
| 71 | } | 98 | } |
| 72 | if (totalSize != dataSize) { | 99 | if (totalSize != dataSize) { |
| 73 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, while expected total size is %llu", | 100 | + HCCL_VM_ERROR("{} AllReduce result data ends before the expected total size is reached, " |
| 74 | - rankId, totalSize, dataSize); | 101 | + "rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", |
| 102 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, dataSize); | ||
| 75 | return HcclResult::HCCL_E_PARA; | 103 | return HcclResult::HCCL_E_PARA; |
| 76 | } | 104 | } |
| 77 | } | 105 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/semantics_check/batchsendrecv_semantics_checker.cc+43-13
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckBatchSendRecvSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize) | 22 | HcclResult TaskCheckBatchSendRecvSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize) |
| @@ -23,7 +26,10 @@ HcclResult TaskCheckBatchSendRecvSemantics(std::map<RankId, RankMemorySemantics> | |||
| 23 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 26 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 24 | // 对应的rank不存在需要报错 | 27 | // 对应的rank不存在需要报错 |
| 25 | if (allRankMemSemantics.count(rankId) == 0) { | 28 | if (allRankMemSemantics.count(rankId) == 0) { |
| 26 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 29 | + HCCL_VM_ERROR("{} BatchSendRecv produced no result data for rank {}, but this rank is " |
| 30 | + "expected to receive one segment from each rank, expectedSourceRankCount={}, " | ||
| 31 | + "expectedResultSize=0x{:x}", MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, | ||
| 32 | + rankSize, dataSize * rankSize); | ||
| 27 | return HcclResult::HCCL_E_PARA; | 33 | return HcclResult::HCCL_E_PARA; |
| 28 | } | 34 | } |
| 29 | 35 | ||
| @@ -31,31 +37,51 @@ HcclResult TaskCheckBatchSendRecvSemantics(std::map<RankId, RankMemorySemantics> | |||
| 31 | RankId curRankId = 0; | 37 | RankId curRankId = 0; |
| 32 | u64 curDataSize = 0; | 38 | u64 curDataSize = 0; |
| 33 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 39 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 40 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 34 | if (ele.startAddr != totalSize) { | 41 | if (ele.startAddr != totalSize) { |
| 35 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 42 | + HCCL_VM_ERROR("{} BatchSendRecv result data does not start from the expected " |
| 36 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 43 | + "address, rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, " |
| 44 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 45 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 46 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 37 | return HcclResult::HCCL_E_PARA; | 47 | return HcclResult::HCCL_E_PARA; |
| 38 | } | 48 | } |
| 39 | 49 | ||
| 40 | if (ele.srcBufs.size() != 1) { | 50 | if (ele.srcBufs.size() != 1) { |
| 41 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, while cur buffer semantic is %s", rankId, ele.Describe().c_str()); | 51 | + HCCL_VM_ERROR("{} This BatchSendRecv result range combines multiple sources, but " |
| 52 | + "this operator expects exactly one source, rankId={}, actualSourceCount={}, " | ||
| 53 | + "expectedSourceCount=1, outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 54 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 55 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 42 | return HcclResult::HCCL_E_PARA; | 56 | return HcclResult::HCCL_E_PARA; |
| 43 | } | 57 | } |
| 44 | 58 | ||
| 45 | if (ele.srcBufs.begin()->rankId != curRankId) { | 59 | if (ele.srcBufs.begin()->rankId != curRankId) { |
| 46 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should come from rank %u, while it come from rank %u, cur buffer semantic is %s", rankId, curRankId, ele.srcBufs.begin()->rankId, ele.Describe().c_str()); | 60 | + HCCL_VM_ERROR("{} This BatchSendRecv result range comes from the wrong source " |
| 61 | + "rank, rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" | ||
| 62 | + "\nCurrent result range detail:\n{}", | ||
| 63 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 64 | + ele.srcBufs.begin()->rankId, curRankId, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 47 | return HcclResult::HCCL_E_PARA; | 65 | return HcclResult::HCCL_E_PARA; |
| 48 | } | 66 | } |
| 49 | 67 | ||
| 50 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 68 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 51 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 69 | + HCCL_VM_ERROR("{} This BatchSendRecv result range comes from a non-INPUT buffer, " |
| 52 | - rankId, ele.Describe().c_str()); | 70 | + "but it should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 71 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 72 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 73 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 74 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 53 | return HcclResult::HCCL_E_PARA; | 75 | return HcclResult::HCCL_E_PARA; |
| 54 | } | 76 | } |
| 55 | 77 | ||
| 56 | if (ele.srcBufs.begin()->srcAddr != dataSize * rankId + curDataSize) { | 78 | if (ele.srcBufs.begin()->srcAddr != dataSize * rankId + curDataSize) { |
| 57 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu, while it is %llu, cur buffer semantic is %s", | 79 | + HCCL_VM_ERROR("{} Source address for this BatchSendRecv result range does not " |
| 58 | - rankId, dataSize * rankId + curDataSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 80 | + "match the expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, " |
| 81 | + "actualAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 82 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 83 | + ele.srcBufs.begin()->rankId, dataSize * rankId + curDataSize, ele.srcBufs.begin()->srcAddr, | ||
| 84 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 59 | return HcclResult::HCCL_E_PARA; | 85 | return HcclResult::HCCL_E_PARA; |
| 60 | } | 86 | } |
| 61 | curDataSize += ele.size; | 87 | curDataSize += ele.size; |
| @@ -63,16 +89,20 @@ HcclResult TaskCheckBatchSendRecvSemantics(std::map<RankId, RankMemorySemantics> | |||
| 63 | curDataSize = 0; | 89 | curDataSize = 0; |
| 64 | curRankId++; | 90 | curRankId++; |
| 65 | } else if (curDataSize > dataSize) { | 91 | } else if (curDataSize > dataSize) { |
| 66 | - HCCL_ERROR("[rankId:%u]Accumulated semantic size from rank %u is %llu, greater than expected %llu", | 92 | + HCCL_VM_ERROR("{} Data taken from one source rank is larger than expected in " |
| 67 | - rankId, curRankId, curDataSize, dataSize); | 93 | + "BatchSendRecv, rankId={}, sourceRank={}, actualSize=0x{:x}, expectedSize=0x{:x}, " |
| 94 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 95 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SIZE_ERROR), rankId, curRankId, curDataSize, | ||
| 96 | + dataSize, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 68 | return HcclResult::HCCL_E_PARA; | 97 | return HcclResult::HCCL_E_PARA; |
| 69 | } | 98 | } |
| 70 | totalSize += ele.size; | 99 | totalSize += ele.size; |
| 71 | } | 100 | } |
| 72 | // 如果curRankId等于rankSize,表示已经接受到其他所有rank的数据 | 101 | // 如果curRankId等于rankSize,表示已经接受到其他所有rank的数据 |
| 73 | if (curRankId != rankSize) { | 102 | if (curRankId != rankSize) { |
| 74 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, accumulated semantic size from rank %u is %llu, while rankSize is %u", | 103 | + HCCL_VM_ERROR("{} BatchSendRecv result data ends before the expected total size is " |
| 75 | - rankId, totalSize, curRankId, curDataSize, rankSize); | 104 | + "reached, rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", |
| 105 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, dataSize * rankSize); | ||
| 76 | return HcclResult::HCCL_E_PARA; | 106 | return HcclResult::HCCL_E_PARA; |
| 77 | } | 107 | } |
| 78 | } | 108 | } |
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckBroadcastSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, | 22 | HcclResult TaskCheckBroadcastSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, |
| @@ -24,45 +27,68 @@ HcclResult TaskCheckBroadcastSemantics(std::map<RankId, RankMemorySemantics> &al | |||
| 24 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 27 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 25 | // 对应的rank不存在需要报错 | 28 | // 对应的rank不存在需要报错 |
| 26 | if (allRankMemSemantics.count(rankId) == 0) { | 29 | if (allRankMemSemantics.count(rankId) == 0) { |
| 27 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 30 | + HCCL_VM_ERROR("{} Broadcast produced no result data for rank {}, but this rank is " |
| 31 | + "expected to receive the full root input, expectedSourceRank={}, expectedResultSize=0x{:x}", | ||
| 32 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, root, dataSize); | ||
| 28 | return HcclResult::HCCL_E_PARA; | 33 | return HcclResult::HCCL_E_PARA; |
| 29 | } | 34 | } |
| 30 | 35 | ||
| 31 | u64 totalSize = 0; | 36 | u64 totalSize = 0; |
| 32 | for (auto &ele : allRankMemSemantics[rankId][BufferType::INPUT]) { | 37 | for (auto &ele : allRankMemSemantics[rankId][BufferType::INPUT]) { |
| 38 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 33 | if (ele.startAddr != totalSize) { | 39 | if (ele.startAddr != totalSize) { |
| 34 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 40 | + HCCL_VM_ERROR("{} Broadcast result data does not start from the expected address, " |
| 35 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 41 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 42 | + "\nCurrent result range detail:\n{}", | ||
| 43 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 44 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 36 | return HcclResult::HCCL_E_PARA; | 45 | return HcclResult::HCCL_E_PARA; |
| 37 | } | 46 | } |
| 38 | 47 | ||
| 39 | if (ele.srcBufs.size() != 1) { | 48 | if (ele.srcBufs.size() != 1) { |
| 40 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, while cur buffer semantic is %s", rankId, ele.Describe().c_str()); | 49 | + HCCL_VM_ERROR("{} This Broadcast result range combines multiple sources, but this " |
| 50 | + "operator expects exactly one source, rankId={}, actualSourceCount={}, expectedSourceCount=1, " | ||
| 51 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 52 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 53 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 41 | return HcclResult::HCCL_E_PARA; | 54 | return HcclResult::HCCL_E_PARA; |
| 42 | } | 55 | } |
| 43 | 56 | ||
| 44 | if (ele.srcBufs.begin()->rankId != root) { | 57 | if (ele.srcBufs.begin()->rankId != root) { |
| 45 | - HCCL_ERROR("[rankId:%u]Buffer semantic srcBuf rank[%u] is not from root[%u], cur buffer semantic is %s, cur rank is %u", | 58 | + HCCL_VM_ERROR("{} This Broadcast result range comes from the wrong source rank, " |
| 46 | - rankId, ele.srcBufs.begin()->rankId, root, ele.Describe().c_str(), rankId); | 59 | + "rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" |
| 60 | + "\nCurrent result range detail:\n{}", | ||
| 61 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 62 | + ele.srcBufs.begin()->rankId, root, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 47 | return HcclResult::HCCL_E_PARA; | 63 | return HcclResult::HCCL_E_PARA; |
| 48 | } | 64 | } |
| 49 | 65 | ||
| 50 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 66 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 51 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 67 | + HCCL_VM_ERROR("{} This Broadcast result range comes from a non-INPUT buffer, but " |
| 52 | - rankId, ele.Describe().c_str()); | 68 | + "it should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 69 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 70 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 71 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 72 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 53 | return HcclResult::HCCL_E_PARA; | 73 | return HcclResult::HCCL_E_PARA; |
| 54 | } | 74 | } |
| 55 | 75 | ||
| 56 | if (ele.srcBufs.begin()->srcAddr != totalSize) { | 76 | if (ele.srcBufs.begin()->srcAddr != totalSize) { |
| 57 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu, while it is %llu, cur buffer semantic is %s", | 77 | + HCCL_VM_ERROR("{} Source address for this Broadcast result range does not match " |
| 58 | - rankId, totalSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 78 | + "the expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 79 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 80 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 81 | + ele.srcBufs.begin()->rankId, totalSize, ele.srcBufs.begin()->srcAddr, ele.startAddr, | ||
| 82 | + rangeEnd, ele.Describe()); | ||
| 59 | return HcclResult::HCCL_E_PARA; | 83 | return HcclResult::HCCL_E_PARA; |
| 60 | } | 84 | } |
| 61 | 85 | ||
| 62 | totalSize += ele.size; | 86 | totalSize += ele.size; |
| 63 | } | 87 | } |
| 64 | if (totalSize != dataSize) { | 88 | if (totalSize != dataSize) { |
| 65 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, which should be %llu, cur rank is %u", rankId, totalSize, dataSize, rankId); | 89 | + HCCL_VM_ERROR("{} Broadcast result data ends before the expected total size is reached, " |
| 90 | + "rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", | ||
| 91 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, dataSize); | ||
| 66 | return HcclResult::HCCL_E_PARA; | 92 | return HcclResult::HCCL_E_PARA; |
| 67 | } | 93 | } |
| 68 | } | 94 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/semantics_check/reduce_scatter_semantics_checker.cc+43-16
| @@ -15,6 +15,8 @@ | |||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | 17 | ||
| 18 | + | ||
| 19 | + | ||
| 18 | 20 | ||
| 19 | namespace HcclSim { | 21 | namespace HcclSim { |
| 20 | HcclResult TaskCheckReduceScatterSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, | 22 | HcclResult TaskCheckReduceScatterSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, |
| @@ -25,55 +27,80 @@ HcclResult TaskCheckReduceScatterSemantics(std::map<RankId, RankMemorySemantics> | |||
| 25 | for (u32 rankId = 0; rankId < rankSize; rankId++) { | 27 | for (u32 rankId = 0; rankId < rankSize; rankId++) { |
| 26 | // 对应的rank不存在需要报错 | 28 | // 对应的rank不存在需要报错 |
| 27 | if (allRankMemSemantics.count(rankId) == 0) { | 29 | if (allRankMemSemantics.count(rankId) == 0) { |
| 28 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 30 | + HCCL_VM_ERROR("{} ReduceScatter produced no result data for rank {}, but this rank is " |
| 31 | + "expected to have one reduced shard, expectedSourceRankCount={}, expectedResultSize=0x{:x}", | ||
| 32 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, rankSize, dataSize); | ||
| 29 | return HcclResult::HCCL_E_PARA; | 33 | return HcclResult::HCCL_E_PARA; |
| 30 | } | 34 | } |
| 31 | 35 | ||
| 32 | u64 totalSize = 0; | 36 | u64 totalSize = 0; |
| 33 | - HCCL_VM_INFO("[INFO][TaskCheckReduceScatterSemantics] rankSize= {}, rankId= {} {}", | ||
| 34 | - rankSize, rankId, allRankMemSemantics[rankId][BufferType::OUTPUT].size()); | ||
| 35 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 37 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 38 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 36 | if (ele.startAddr != totalSize) { | 39 | if (ele.startAddr != totalSize) { |
| 37 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 40 | + HCCL_VM_ERROR("{} ReduceScatter result data does not start from the expected " |
| 38 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 41 | + "address, rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, " |
| 42 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 43 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 44 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 39 | return HcclResult::HCCL_E_PARA; | 45 | return HcclResult::HCCL_E_PARA; |
| 40 | } | 46 | } |
| 41 | 47 | ||
| 42 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { | 48 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { |
| 43 | - HCCL_ERROR("[rankId:%u]cur buffer semantic reduceType %d is unequal to expected reduceType %d, cur buffer semantic is %s", | 49 | + HCCL_VM_ERROR("{} Reduce mode of this ReduceScatter result range does not match the " |
| 44 | - rankId, ele.reduceType, reduceType, ele.Describe().c_str()); | 50 | + "operator setting, rankId={}, actualReduceType={}, expectedReduceType={}, " |
| 51 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 52 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, | ||
| 53 | + DumpReduceOpToString(ele.reduceType), DumpReduceOpToString(reduceType), | ||
| 54 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 45 | return HcclResult::HCCL_E_PARA; | 55 | return HcclResult::HCCL_E_PARA; |
| 46 | } | 56 | } |
| 47 | 57 | ||
| 48 | if (ele.srcBufs.size() != rankSize) { | 58 | if (ele.srcBufs.size() != rankSize) { |
| 49 | - HCCL_ERROR("[rankId:%u]buffer semantic srcBufs size %u is unequal to rankSize %u, cur buffer semantic is %s", | 59 | + HCCL_VM_ERROR("{} This ReduceScatter result range does not include the expected " |
| 50 | - rankId, ele.srcBufs.size(), rankSize, ele.Describe().c_str()); | 60 | + "number of source ranks, rankId={}, actualSourceRankCount={}, expectedRankSize={}, " |
| 61 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 62 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 63 | + rankSize, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 51 | return HcclResult::HCCL_E_PARA; | 64 | return HcclResult::HCCL_E_PARA; |
| 52 | } | 65 | } |
| 53 | 66 | ||
| 54 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != static_cast<int>(rankSize - 1)) { | 67 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != static_cast<int>(rankSize - 1)) { |
| 55 | - HCCL_ERROR("[rankId:%u]cur buffer semantic srcBufs is invalid, cur buffer semantic is %s", | 68 | + HCCL_VM_ERROR("{} Source rank list for this ReduceScatter result range is not the " |
| 56 | - rankId, ele.Describe().c_str()); | 69 | + "complete expected rank set, rankId={}, firstSourceRank={}, lastSourceRank={}, " |
| 70 | + "expectedFirstSourceRank=0, expectedLastSourceRank={}, outputRange=[0x{:x},0x{:x})" | ||
| 71 | + "\nCurrent result range detail:\n{}", | ||
| 72 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 73 | + ele.srcBufs.begin()->rankId, ele.srcBufs.rbegin()->rankId, rankSize - 1, ele.startAddr, | ||
| 74 | + rangeEnd, ele.Describe()); | ||
| 57 | return HcclResult::HCCL_E_PARA; | 75 | return HcclResult::HCCL_E_PARA; |
| 58 | } | 76 | } |
| 59 | 77 | ||
| 60 | for (auto &srcBuf : ele.srcBufs) { | 78 | for (auto &srcBuf : ele.srcBufs) { |
| 61 | if (srcBuf.bufType != BufferType::INPUT) { | 79 | if (srcBuf.bufType != BufferType::INPUT) { |
| 62 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 80 | + HCCL_VM_ERROR("{} This ReduceScatter result range comes from a non-INPUT " |
| 63 | - rankId, ele.Describe().c_str()); | 81 | + "buffer, but it should come from INPUT, rankId={}, actualSourceRank={}, " |
| 82 | + "actualSourceBufferType={}, actualBufferRange=[0x{:x},0x{:x})" | ||
| 83 | + "\nCurrent result range detail:\n{}", | ||
| 84 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, srcBuf.rankId, | ||
| 85 | + BufferTypeToString(srcBuf.bufType), ele.startAddr, rangeEnd, ele.Describe()); | ||
| 64 | return HcclResult::HCCL_E_PARA; | 86 | return HcclResult::HCCL_E_PARA; |
| 65 | } | 87 | } |
| 66 | 88 | ||
| 67 | if (srcBuf.srcAddr != rankId * dataSize + totalSize) { | 89 | if (srcBuf.srcAddr != rankId * dataSize + totalSize) { |
| 68 | - HCCL_ERROR("[rankId:%u]Expected semantic srcBuf srcAddr is %llu, while cur srcBuf srcAddr is %llu, cur buffer semantic is %s", | 90 | + HCCL_VM_ERROR("{} Source address for this ReduceScatter result range does not " |
| 69 | - rankId, rankId * dataSize + totalSize, srcBuf.srcAddr, ele.Describe().c_str()); | 91 | + "match the expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, " |
| 92 | + "actualAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 93 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, srcBuf.rankId, | ||
| 94 | + rankId * dataSize + totalSize, srcBuf.srcAddr, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 70 | return HcclResult::HCCL_E_PARA; | 95 | return HcclResult::HCCL_E_PARA; |
| 71 | } | 96 | } |
| 72 | } | 97 | } |
| 73 | totalSize += ele.size; | 98 | totalSize += ele.size; |
| 74 | } | 99 | } |
| 75 | if (totalSize != dataSize) { | 100 | if (totalSize != dataSize) { |
| 76 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, while expected total size is %llu", rankId, totalSize, dataSize); | 101 | + HCCL_VM_ERROR("{} ReduceScatter result data ends before the expected total size is " |
| 102 | + "reached, rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", | ||
| 103 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, dataSize); | ||
| 77 | return HcclResult::HCCL_E_PARA; | 104 | return HcclResult::HCCL_E_PARA; |
| 78 | } | 105 | } |
| 79 | } | 106 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/semantics_check/reduce_scatter_v_semantics_checker.cc+45-21
| @@ -15,7 +15,9 @@ | |||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | 17 | ||
| 18 | -#include "log.h" | 18 | +#include "sim_log.h" |
| 19 | + | ||
| 20 | + | ||
| 19 | 21 | ||
| 20 | namespace HcclSim { | 22 | namespace HcclSim { |
| 21 | HcclResult TaskCheckReduceScatterVSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, | 23 | HcclResult TaskCheckReduceScatterVSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, |
| @@ -27,51 +29,72 @@ HcclResult TaskCheckReduceScatterVSemantics(std::map<RankId, RankMemorySemantics | |||
| 27 | for (u32 rankId = 0; rankId < rankSize; rankId++) { | 29 | for (u32 rankId = 0; rankId < rankSize; rankId++) { |
| 28 | // 对应的rank不存在需要报错 | 30 | // 对应的rank不存在需要报错 |
| 29 | if (allRankMemSemantics.count(rankId) == 0) { | 31 | if (allRankMemSemantics.count(rankId) == 0) { |
| 30 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 32 | + HCCL_VM_ERROR("{} ReduceScatterV produced no result data for rank {}, but this rank is " |
| 33 | + "expected to have one reduced shard, expectedSourceRankCount={}, expectedResultSize=0x{:x}", | ||
| 34 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, rankSize, | ||
| 35 | + vDataDes.counts[rankId] * CHECK_SIZE_TABLE[vDataDes.dataType]); | ||
| 31 | return HcclResult::HCCL_E_PARA; | 36 | return HcclResult::HCCL_E_PARA; |
| 32 | } | 37 | } |
| 33 | 38 | ||
| 34 | u64 totalSize = 0; | 39 | u64 totalSize = 0; |
| 35 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 40 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 41 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 36 | if (ele.startAddr != totalSize) { | 42 | if (ele.startAddr != totalSize) { |
| 37 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: exepected startAddr is %llu, " | 43 | + HCCL_VM_ERROR("{} ReduceScatterV result data does not start from the expected " |
| 38 | - "while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 44 | + "address, rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, " |
| 39 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 45 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", |
| 46 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 47 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 40 | return HcclResult::HCCL_E_PARA; | 48 | return HcclResult::HCCL_E_PARA; |
| 41 | } | 49 | } |
| 42 | 50 | ||
| 43 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { | 51 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { |
| 44 | - HCCL_ERROR("[rankId:%u]cur buffer semantic reduceType %d is unequal to expected reduceType %d, " | 52 | + HCCL_VM_ERROR("{} Reduce mode of this ReduceScatterV result range does not match " |
| 45 | - "cur buffer semantic is %s", | 53 | + "the operator setting, rankId={}, actualReduceType={}, expectedReduceType={}, " |
| 46 | - rankId, ele.reduceType, reduceType, ele.Describe().c_str()); | 54 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", |
| 55 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, | ||
| 56 | + DumpReduceOpToString(ele.reduceType), DumpReduceOpToString(reduceType), | ||
| 57 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 47 | return HcclResult::HCCL_E_PARA; | 58 | return HcclResult::HCCL_E_PARA; |
| 48 | } | 59 | } |
| 49 | 60 | ||
| 50 | if (ele.srcBufs.size() != rankSize) { | 61 | if (ele.srcBufs.size() != rankSize) { |
| 51 | - HCCL_ERROR("[rankId:%u]buffer semantic srcBufs size %u is unequal to rankSize %u, " | 62 | + HCCL_VM_ERROR("{} This ReduceScatterV result range does not include the expected " |
| 52 | - "cur buffer semantic is %s", | 63 | + "number of source ranks, rankId={}, actualSourceRankCount={}, expectedRankSize={}, " |
| 53 | - rankId, ele.srcBufs.size(), rankSize, ele.Describe().c_str()); | 64 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", |
| 65 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 66 | + rankSize, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 54 | return HcclResult::HCCL_E_PARA; | 67 | return HcclResult::HCCL_E_PARA; |
| 55 | } | 68 | } |
| 56 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != static_cast<int>(rankSize - 1)) { | 69 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != static_cast<int>(rankSize - 1)) { |
| 57 | - HCCL_ERROR("[rankId:%u]cur buffer semantic srcBufs is invalid, cur buffer semantic is %s", | 70 | + HCCL_VM_ERROR("{} Source rank list for this ReduceScatterV result range is not the " |
| 58 | - rankId, ele.Describe().c_str()); | 71 | + "complete expected rank set, rankId={}, firstSourceRank={}, lastSourceRank={}, " |
| 72 | + "expectedFirstSourceRank=0, expectedLastSourceRank={}, outputRange=[0x{:x},0x{:x})" | ||
| 73 | + "\nCurrent result range detail:\n{}", | ||
| 74 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 75 | + ele.srcBufs.begin()->rankId, ele.srcBufs.rbegin()->rankId, rankSize - 1, ele.startAddr, | ||
| 76 | + rangeEnd, ele.Describe()); | ||
| 59 | return HcclResult::HCCL_E_PARA; | 77 | return HcclResult::HCCL_E_PARA; |
| 60 | } | 78 | } |
| 61 | 79 | ||
| 62 | for (auto &srcBuf : ele.srcBufs) { | 80 | for (auto &srcBuf : ele.srcBufs) { |
| 63 | if (srcBuf.bufType != BufferType::INPUT) { | 81 | if (srcBuf.bufType != BufferType::INPUT) { |
| 64 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, " | 82 | + HCCL_VM_ERROR("{} This ReduceScatterV result range comes from a non-INPUT " |
| 65 | - "cur buffer semantic is %s", | 83 | + "buffer, but it should come from INPUT, rankId={}, actualSourceRank={}, " |
| 66 | - rankId, ele.Describe().c_str()); | 84 | + "actualSourceBufferType={}, actualBufferRange=[0x{:x},0x{:x})" |
| 85 | + "\nCurrent result range detail:\n{}", | ||
| 86 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, srcBuf.rankId, | ||
| 87 | + BufferTypeToString(srcBuf.bufType), ele.startAddr, rangeEnd, ele.Describe()); | ||
| 67 | HcclResult::HCCL_E_PARA; | 88 | HcclResult::HCCL_E_PARA; |
| 68 | } | 89 | } |
| 69 | 90 | ||
| 70 | const u64 outputOffset = vDataDes.displs[rankId] * CHECK_SIZE_TABLE[vDataDes.dataType]; | 91 | const u64 outputOffset = vDataDes.displs[rankId] * CHECK_SIZE_TABLE[vDataDes.dataType]; |
| 71 | if (srcBuf.srcAddr != outputOffset + totalSize) { | 92 | if (srcBuf.srcAddr != outputOffset + totalSize) { |
| 72 | - HCCL_ERROR("[rankId:%u]Exepected semantic srcBuf srcAddr is %llu, " | 93 | + HCCL_VM_ERROR("{} Source address for this ReduceScatterV result range does not " |
| 73 | - "while cur srcBuf srcAddr is %llu, cur buffer semantic is %s", | 94 | + "match the expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, " |
| 74 | - rankId, outputOffset + totalSize, srcBuf.srcAddr, ele.Describe().c_str()); | 95 | + "actualAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", |
| 96 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, srcBuf.rankId, | ||
| 97 | + outputOffset + totalSize, srcBuf.srcAddr, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 75 | return HcclResult::HCCL_E_PARA; | 98 | return HcclResult::HCCL_E_PARA; |
| 76 | } | 99 | } |
| 77 | } | 100 | } |
| @@ -80,8 +103,9 @@ HcclResult TaskCheckReduceScatterVSemantics(std::map<RankId, RankMemorySemantics | |||
| 80 | outputSize = vDataDes.counts[rankId] * CHECK_SIZE_TABLE[vDataDes.dataType]; | 103 | outputSize = vDataDes.counts[rankId] * CHECK_SIZE_TABLE[vDataDes.dataType]; |
| 81 | 104 | ||
| 82 | if (totalSize != outputSize) { | 105 | if (totalSize != outputSize) { |
| 83 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, " | 106 | + HCCL_VM_ERROR("{} ReduceScatterV result data ends before the expected total size is " |
| 84 | - "while expected output size is %lu", rankId, totalSize, outputSize); | 107 | + "reached, rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", |
| 108 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, outputSize); | ||
| 85 | return HcclResult::HCCL_E_PARA; | 109 | return HcclResult::HCCL_E_PARA; |
| 86 | } | 110 | } |
| 87 | } | 111 | } |
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckReduceSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, | 22 | HcclResult TaskCheckReduceSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, |
| @@ -26,53 +29,78 @@ HcclResult TaskCheckReduceSemantics(std::map<RankId, RankMemorySemantics> &allRa | |||
| 26 | 29 | ||
| 27 | // 对应的rank不存在需要报错 | 30 | // 对应的rank不存在需要报错 |
| 28 | if (allRankMemSemantics.count(root) == 0 && dataSize > 0) { | 31 | if (allRankMemSemantics.count(root) == 0 && dataSize > 0) { |
| 29 | - HCCL_ERROR("Missing rank %d mem semantics", root); | 32 | + HCCL_VM_ERROR("{} Reduce produced no result data for root rank {}, but this rank is " |
| 33 | + "expected to hold the reduced result, expectedSourceRankCount={}, expectedResultSize=0x{:x}", | ||
| 34 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), root, rankSize, dataSize); | ||
| 30 | return HcclResult::HCCL_E_PARA; | 35 | return HcclResult::HCCL_E_PARA; |
| 31 | } | 36 | } |
| 32 | 37 | ||
| 33 | u64 totalSize = 0; | 38 | u64 totalSize = 0; |
| 34 | for (auto &ele : allRankMemSemantics[root][BufferType::OUTPUT]) { | 39 | for (auto &ele : allRankMemSemantics[root][BufferType::OUTPUT]) { |
| 40 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 35 | if (ele.startAddr != totalSize) { | 41 | if (ele.startAddr != totalSize) { |
| 36 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 42 | + HCCL_VM_ERROR("{} Reduce result data does not start from the expected address, " |
| 37 | - root, totalSize, ele.startAddr, ele.Describe().c_str()); | 43 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 44 | + "\nCurrent result range detail:\n{}", | ||
| 45 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), root, totalSize, ele.startAddr, | ||
| 46 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 38 | return HcclResult::HCCL_E_PARA; | 47 | return HcclResult::HCCL_E_PARA; |
| 39 | } | 48 | } |
| 40 | 49 | ||
| 41 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { | 50 | if (ele.srcBufs.size() > 1 && ele.reduceType != reduceType) { |
| 42 | - HCCL_ERROR("[rankId:%u]cur buffer semantic reduceType %d is unequal to expected reduceType %d, cur buffer semantic is %s", | 51 | + HCCL_VM_ERROR("{} Reduce mode of this Reduce result range does not match the operator " |
| 43 | - root, ele.reduceType, reduceType, ele.Describe().c_str()); | 52 | + "setting, rankId={}, actualReduceType={}, expectedReduceType={}, outputRange=[0x{:x},0x{:x})" |
| 53 | + "\nCurrent result range detail:\n{}", | ||
| 54 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), root, | ||
| 55 | + DumpReduceOpToString(ele.reduceType), DumpReduceOpToString(reduceType), ele.startAddr, rangeEnd, | ||
| 56 | + ele.Describe()); | ||
| 44 | return HcclResult::HCCL_E_PARA; | 57 | return HcclResult::HCCL_E_PARA; |
| 45 | } | 58 | } |
| 46 | 59 | ||
| 47 | if (ele.srcBufs.size() != rankSize) { | 60 | if (ele.srcBufs.size() != rankSize) { |
| 48 | - HCCL_ERROR("[rankId:%u]buffer semantic srcBufs size %u is unequal to rankSize %u, cur buffer semantic is %s", | 61 | + HCCL_VM_ERROR("{} This Reduce result range does not include the expected number of " |
| 49 | - root, ele.srcBufs.size(), rankSize, ele.Describe().c_str()); | 62 | + "source ranks, rankId={}, actualSourceRankCount={}, expectedRankSize={}, " |
| 63 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 64 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), root, ele.srcBufs.size(), rankSize, | ||
| 65 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 50 | return HcclResult::HCCL_E_PARA; | 66 | return HcclResult::HCCL_E_PARA; |
| 51 | } | 67 | } |
| 52 | 68 | ||
| 53 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != (rankSize - 1)) { | 69 | if (ele.srcBufs.begin()->rankId != 0 or ele.srcBufs.rbegin()->rankId != (rankSize - 1)) { |
| 54 | - HCCL_ERROR("[rankId:%u]cur buffer semantic srcBufs is invalid, cur buffer semantic is %s", | 70 | + HCCL_VM_ERROR("{} Source rank list for this Reduce result range is not the complete " |
| 55 | - root, ele.Describe().c_str()); | 71 | + "expected rank set, rankId={}, firstSourceRank={}, lastSourceRank={}, expectedFirstSourceRank=0, " |
| 72 | + "expectedLastSourceRank={}, outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 73 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), root, | ||
| 74 | + ele.srcBufs.begin()->rankId, ele.srcBufs.rbegin()->rankId, rankSize - 1, ele.startAddr, rangeEnd, | ||
| 75 | + ele.Describe()); | ||
| 56 | return HcclResult::HCCL_E_PARA; | 76 | return HcclResult::HCCL_E_PARA; |
| 57 | } | 77 | } |
| 58 | 78 | ||
| 59 | for (auto &srcBuf : ele.srcBufs) { | 79 | for (auto &srcBuf : ele.srcBufs) { |
| 60 | if (srcBuf.bufType != BufferType::INPUT) { | 80 | if (srcBuf.bufType != BufferType::INPUT) { |
| 61 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 81 | + HCCL_VM_ERROR("{} This Reduce result range comes from a non-INPUT buffer, but it " |
| 62 | - root, ele.Describe().c_str()); | 82 | + "should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 83 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 84 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), root, srcBuf.rankId, | ||
| 85 | + BufferTypeToString(srcBuf.bufType), ele.startAddr, rangeEnd, ele.Describe()); | ||
| 63 | return HcclResult::HCCL_E_PARA; | 86 | return HcclResult::HCCL_E_PARA; |
| 64 | } | 87 | } |
| 65 | 88 | ||
| 66 | if (srcBuf.srcAddr != totalSize) { | 89 | if (srcBuf.srcAddr != totalSize) { |
| 67 | - HCCL_ERROR("[rankId:%u]Expected semantic srcBuf srcAddr is %llu, while cur srcBuf srcAddr is %llu, cur buffer semantic is %s", | 90 | + HCCL_VM_ERROR("{} Source address for this Reduce result range does not match the " |
| 68 | - root, totalSize, srcBuf.srcAddr, ele.Describe().c_str()); | 91 | + "expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 92 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 93 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), root, srcBuf.rankId, totalSize, | ||
| 94 | + srcBuf.srcAddr, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 69 | return HcclResult::HCCL_E_PARA; | 95 | return HcclResult::HCCL_E_PARA; |
| 70 | } | 96 | } |
| 71 | } | 97 | } |
| 72 | totalSize += ele.size; | 98 | totalSize += ele.size; |
| 73 | } | 99 | } |
| 74 | if (totalSize != dataSize) { | 100 | if (totalSize != dataSize) { |
| 75 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, while expected total size is %llu", root, totalSize, dataSize); | 101 | + HCCL_VM_ERROR("{} Reduce result data ends before the expected total size is reached, " |
| 102 | + "rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", | ||
| 103 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), root, totalSize, dataSize); | ||
| 76 | return HcclResult::HCCL_E_PARA; | 104 | return HcclResult::HCCL_E_PARA; |
| 77 | } | 105 | } |
| 78 | 106 | ||
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckScatterSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, | 22 | HcclResult TaskCheckScatterSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, |
| @@ -24,45 +27,67 @@ HcclResult TaskCheckScatterSemantics(std::map<RankId, RankMemorySemantics> &allR | |||
| 24 | for (RankId rankId = 0; rankId < rankSize; rankId++) { | 27 | for (RankId rankId = 0; rankId < rankSize; rankId++) { |
| 25 | // 对应的rank不存在需要报错 | 28 | // 对应的rank不存在需要报错 |
| 26 | if (allRankMemSemantics.count(rankId) == 0) { | 29 | if (allRankMemSemantics.count(rankId) == 0) { |
| 27 | - HCCL_ERROR("Missing rank %d mem semantics", rankId); | 30 | + HCCL_VM_ERROR("{} Scatter produced no result data for rank {}, but this rank is " |
| 31 | + "expected to receive one shard from root rank {}, expectedResultSize=0x{:x}", | ||
| 32 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, root, dataSize); | ||
| 28 | return HcclResult::HCCL_E_PARA; | 33 | return HcclResult::HCCL_E_PARA; |
| 29 | } | 34 | } |
| 30 | 35 | ||
| 31 | u64 totalSize = 0; | 36 | u64 totalSize = 0; |
| 32 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { | 37 | for (auto &ele : allRankMemSemantics[rankId][BufferType::OUTPUT]) { |
| 38 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 33 | if (ele.startAddr != totalSize) { | 39 | if (ele.startAddr != totalSize) { |
| 34 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: exepected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 40 | + HCCL_VM_ERROR("{} Scatter result data does not start from the expected address, " |
| 35 | - rankId, totalSize, ele.startAddr, ele.Describe().c_str()); | 41 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 42 | + "\nCurrent result range detail:\n{}", | ||
| 43 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, ele.startAddr, | ||
| 44 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 36 | return HcclResult::HCCL_E_PARA; | 45 | return HcclResult::HCCL_E_PARA; |
| 37 | } | 46 | } |
| 38 | 47 | ||
| 39 | if (ele.srcBufs.size() != 1) { | 48 | if (ele.srcBufs.size() != 1) { |
| 40 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, while cur buffer semantic is %s", rankId, ele.Describe().c_str()); | 49 | + HCCL_VM_ERROR("{} This Scatter result range combines multiple sources, but this " |
| 50 | + "operator expects exactly one source, rankId={}, actualSourceCount={}, expectedSourceCount=1, " | ||
| 51 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 52 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), rankId, ele.srcBufs.size(), | ||
| 53 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 41 | return HcclResult::HCCL_E_PARA; | 54 | return HcclResult::HCCL_E_PARA; |
| 42 | } | 55 | } |
| 43 | 56 | ||
| 44 | if (ele.srcBufs.begin()->rankId != root) { | 57 | if (ele.srcBufs.begin()->rankId != root) { |
| 45 | - HCCL_ERROR("[rankId:%u]Buffer semantic srcBuf rank[%u] is not from root[%u], cur buffer semantic is %s, cur rank is %u", | 58 | + HCCL_VM_ERROR("{} This Scatter result range comes from the wrong source rank, " |
| 46 | - rankId, ele.srcBufs.begin()->rankId, root, ele.Describe().c_str(), rankId); | 59 | + "rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" |
| 60 | + "\nCurrent result range detail:\n{}", | ||
| 61 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 62 | + ele.srcBufs.begin()->rankId, root, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 47 | return HcclResult::HCCL_E_PARA; | 63 | return HcclResult::HCCL_E_PARA; |
| 48 | } | 64 | } |
| 49 | 65 | ||
| 50 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 66 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 51 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 67 | + HCCL_VM_ERROR("{} This Scatter result range comes from a non-INPUT buffer, but it " |
| 52 | - rankId, ele.Describe().c_str()); | 68 | + "should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 69 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 70 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 71 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 72 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 53 | return HcclResult::HCCL_E_PARA; | 73 | return HcclResult::HCCL_E_PARA; |
| 54 | } | 74 | } |
| 55 | 75 | ||
| 56 | if (ele.srcBufs.begin()->srcAddr != rankId * dataSize + totalSize) { | 76 | if (ele.srcBufs.begin()->srcAddr != rankId * dataSize + totalSize) { |
| 57 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu," | 77 | + HCCL_VM_ERROR("{} Source address for this Scatter result range does not match the " |
| 58 | - "while it is %llu, cur buffer semantic is %s", | 78 | + "expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 59 | - rankId, rankId * dataSize + totalSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 79 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", |
| 80 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), rankId, | ||
| 81 | + ele.srcBufs.begin()->rankId, rankId * dataSize + totalSize, ele.srcBufs.begin()->srcAddr, | ||
| 82 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 60 | return HcclResult::HCCL_E_PARA; | 83 | return HcclResult::HCCL_E_PARA; |
| 61 | } | 84 | } |
| 62 | totalSize += ele.size; | 85 | totalSize += ele.size; |
| 63 | } | 86 | } |
| 64 | if (totalSize != dataSize) { | 87 | if (totalSize != dataSize) { |
| 65 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, which should be %llu, cur rank is %u", rankId, totalSize, dataSize, rankId); | 88 | + HCCL_VM_ERROR("{} Scatter result data ends before the expected total size is reached, " |
| 89 | + "rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", | ||
| 90 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), rankId, totalSize, dataSize); | ||
| 66 | return HcclResult::HCCL_E_PARA; | 91 | return HcclResult::HCCL_E_PARA; |
| 67 | } | 92 | } |
| 68 | } | 93 | } |
| @@ -14,6 +14,9 @@ | |||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | + | ||
| 18 | + | ||
| 19 | + | ||
| 17 | 20 | ||
| 18 | namespace HcclSim { | 21 | namespace HcclSim { |
| 19 | HcclResult TaskCheckSendRecvSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, | 22 | HcclResult TaskCheckSendRecvSemantics(std::map<RankId, RankMemorySemantics> &allRankMemSemantics, u64 dataSize, |
| @@ -23,43 +26,65 @@ HcclResult TaskCheckSendRecvSemantics(std::map<RankId, RankMemorySemantics> &all | |||
| 23 | 26 | ||
| 24 | // 对应的rank不存在需要报错 | 27 | // 对应的rank不存在需要报错 |
| 25 | if (allRankMemSemantics.size() != 2) { | 28 | if (allRankMemSemantics.size() != 2) { |
| 26 | - HCCL_ERROR("sendrecv only support 2 ranks"); | 29 | + HCCL_VM_ERROR("{} Send/Recv final output validation supports exactly 2 ranks, but got " |
| 30 | + "expectedRankSize=2, actualRankSize={}, sourceRank={}, targetRank={}", | ||
| 31 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_CHECK_FAILED), rankSize, srcRank, dstRank); | ||
| 27 | return HcclResult::HCCL_E_PARA; | 32 | return HcclResult::HCCL_E_PARA; |
| 28 | } | 33 | } |
| 29 | 34 | ||
| 30 | u64 totalSize = 0; | 35 | u64 totalSize = 0; |
| 31 | for (auto &ele : allRankMemSemantics[dstRank][BufferType::OUTPUT]) { | 36 | for (auto &ele : allRankMemSemantics[dstRank][BufferType::OUTPUT]) { |
| 37 | + const u64 rangeEnd = ele.startAddr + ele.size; | ||
| 32 | if (ele.startAddr != totalSize) { | 38 | if (ele.startAddr != totalSize) { |
| 33 | - HCCL_ERROR("[rankId:%u]Missing buffer semantic: expected startAddr is %llu, while cur buffer semantic startAddr is %llu, cur buffer semantic is %s", | 39 | + HCCL_VM_ERROR("{} Send/Recv result data does not start from the expected address, " |
| 34 | - dstRank, totalSize, ele.startAddr, ele.Describe().c_str()); | 40 | + "rankId={}, expectedStartAddr=0x{:x}, actualStartAddr=0x{:x}, actualBufferRange=[0x{:x},0x{:x})" |
| 41 | + "\nCurrent result range detail:\n{}", | ||
| 42 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), dstRank, totalSize, ele.startAddr, | ||
| 43 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 35 | return HcclResult::HCCL_E_PARA; | 44 | return HcclResult::HCCL_E_PARA; |
| 36 | } | 45 | } |
| 37 | 46 | ||
| 38 | if (ele.srcBufs.size() != 1) { | 47 | if (ele.srcBufs.size() != 1) { |
| 39 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic should not be reduce, which mean srcBufs size should be 1, while cur buffer semantic is %s", dstRank, ele.Describe().c_str()); | 48 | + HCCL_VM_ERROR("{} This Send/Recv result range combines multiple sources, but this " |
| 49 | + "operator expects exactly one source, rankId={}, actualSourceCount={}, expectedSourceCount=1, " | ||
| 50 | + "outputRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 51 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_REDUCE_ERROR), dstRank, ele.srcBufs.size(), | ||
| 52 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 40 | return HcclResult::HCCL_E_PARA; | 53 | return HcclResult::HCCL_E_PARA; |
| 41 | } | 54 | } |
| 42 | 55 | ||
| 43 | if (ele.srcBufs.begin()->rankId != srcRank) { | 56 | if (ele.srcBufs.begin()->rankId != srcRank) { |
| 44 | - HCCL_ERROR("[rankId:%u]Buffer semantic srcBuf rank[%u] is not from srcRank[%u], cur buffer semantic is %s", | 57 | + HCCL_VM_ERROR("{} This Send/Recv result range comes from the wrong source rank, " |
| 45 | - dstRank, ele.srcBufs.begin()->rankId, srcRank, ele.Describe().c_str()); | 58 | + "rankId={}, actualSourceRank={}, expectedSourceRank={}, actualBufferRange=[0x{:x},0x{:x})" |
| 59 | + "\nCurrent result range detail:\n{}", | ||
| 60 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), dstRank, | ||
| 61 | + ele.srcBufs.begin()->rankId, srcRank, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 46 | return HcclResult::HCCL_E_PARA; | 62 | return HcclResult::HCCL_E_PARA; |
| 47 | } | 63 | } |
| 48 | 64 | ||
| 49 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { | 65 | if (ele.srcBufs.begin()->bufType != BufferType::INPUT) { |
| 50 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs bufType is not INPUT, cur buffer semantic is %s", | 66 | + HCCL_VM_ERROR("{} This Send/Recv result range comes from a non-INPUT buffer, but it " |
| 51 | - dstRank, ele.Describe().c_str()); | 67 | + "should come from INPUT, rankId={}, actualSourceRank={}, actualSourceBufferType={}, " |
| 68 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 69 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), dstRank, | ||
| 70 | + ele.srcBufs.begin()->rankId, BufferTypeToString(ele.srcBufs.begin()->bufType), | ||
| 71 | + ele.startAddr, rangeEnd, ele.Describe()); | ||
| 52 | return HcclResult::HCCL_E_PARA; | 72 | return HcclResult::HCCL_E_PARA; |
| 53 | } | 73 | } |
| 54 | if (ele.srcBufs.begin()->srcAddr != totalSize) { | 74 | if (ele.srcBufs.begin()->srcAddr != totalSize) { |
| 55 | - HCCL_ERROR("[rankId:%u]Cur buffer semantic srcBufs srcAddr should be %llu, while it is %llu, cur buffer semantic is %s", | 75 | + HCCL_VM_ERROR("{} Source address for this Send/Recv result range does not match the " |
| 56 | - dstRank, totalSize, ele.srcBufs.begin()->srcAddr, ele.Describe().c_str()); | 76 | + "expected input address, rankId={}, sourceRank={}, expectedAddr=0x{:x}, actualAddr=0x{:x}, " |
| 77 | + "actualBufferRange=[0x{:x},0x{:x})\nCurrent result range detail:\n{}", | ||
| 78 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_SRC_ERROR), dstRank, ele.srcBufs.begin()->rankId, | ||
| 79 | + totalSize, ele.srcBufs.begin()->srcAddr, ele.startAddr, rangeEnd, ele.Describe()); | ||
| 57 | return HcclResult::HCCL_E_PARA; | 80 | return HcclResult::HCCL_E_PARA; |
| 58 | } | 81 | } |
| 59 | totalSize += ele.size; | 82 | totalSize += ele.size; |
| 60 | } | 83 | } |
| 61 | if (totalSize != dataSize) { | 84 | if (totalSize != dataSize) { |
| 62 | - HCCL_ERROR("[rankId:%u]Missing buffer semantics in tail: already checked total size is %llu, which should be %llu", dstRank, totalSize, dataSize); | 85 | + HCCL_VM_ERROR("{} Send/Recv result data ends before the expected total size is reached, " |
| 86 | + "rankId={}, checkedSize=0x{:x}, expectedSize=0x{:x}", | ||
| 87 | + MakeErrorCodeText(ErrorCode::SEMANTIC_FINAL_MISSING), dstRank, totalSize, dataSize); | ||
| 63 | return HcclResult::HCCL_E_PARA; | 88 | return HcclResult::HCCL_E_PARA; |
| 64 | } | 89 | } |
| 65 | 90 | ||
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator/ccu_all_rank_param_recorder.cc+1-1
| @@ -105,7 +105,7 @@ HcclResult AllRankParamRecorder::CheckAllPostMatch() | |||
| 105 | for (const auto& regPair: diePair.second) { | 105 | for (const auto& regPair: diePair.second) { |
| 106 | uint16_t regId = regPair.first; | 106 | uint16_t regId = regPair.first; |
| 107 | for (auto& post : seenPost[rank][dieId][regId]) { | 107 | for (auto& post : seenPost[rank][dieId][regId]) { |
| 108 | - HCCL_WARNING("unmatched LocalPost/Post: {}", post->task->Describe().c_str()); | 108 | + HCCL_WARNING("unmatched LocalPost/Post: %s", post->task->Describe().c_str()); |
| 109 | } | 109 | } |
| 110 | } | 110 | } |
| 111 | } | 111 | } |
| @@ -118,7 +118,7 @@ void GenWaitNode(TaskStubCcuGraph *curCcuTask, uint32_t queId, uint16_t waitCKEI | |||
| 118 | HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, TaskStubCcuGraph *curCcuTask, uint32_t queId, | 118 | HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, TaskStubCcuGraph *curCcuTask, uint32_t queId, |
| 119 | uint16_t waitCKEId, uint16_t waitCKEMask, bool& isContinue) | 119 | uint16_t waitCKEId, uint16_t waitCKEMask, bool& isContinue) |
| 120 | { | 120 | { |
| 121 | - HCCL_VM_DEBUG("[ProcessWaitMask] Enter...rank {:d}, die{:d}, que{:d}, waitCke{:d}, waitMask{:d}", | 121 | + HCCL_VM_DEBUG("Enter...rank {:d}, die{:d}, que{:d}, waitCke{:d}, waitMask{:d}", |
| 122 | rankId, static_cast<uint32_t>(dieId), queId, waitCKEId, waitCKEMask); | 122 | rankId, static_cast<uint32_t>(dieId), queId, waitCKEId, waitCKEMask); |
| 123 | if (waitCKEMask != 0x0000) { | 123 | if (waitCKEMask != 0x0000) { |
| 124 | uint16_t ckeValue = 0; | 124 | uint16_t ckeValue = 0; |
| @@ -126,7 +126,7 @@ HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, TaskStubCcuGraph *curC | |||
| 126 | // 条件不满足,继续等待 | 126 | // 条件不满足,继续等待 |
| 127 | if ((ckeValue & waitCKEMask) != waitCKEMask) { | 127 | if ((ckeValue & waitCKEMask) != waitCKEMask) { |
| 128 | isContinue = 0; | 128 | isContinue = 0; |
| 129 | - HCCL_VM_DEBUG("[ProcessWaitMask] Does not meet the condition, continue: ckeId: {:d}, ckeValue: {:d}, mask: {:x}", waitCKEId, ckeValue, waitCKEMask); | 129 | + HCCL_VM_DEBUG("Does not meet the condition, continue: ckeId: {:d}, ckeValue: {:d}, mask: {:x}", waitCKEId, ckeValue, waitCKEMask); |
| 130 | return HCCL_SUCCESS; | 130 | return HCCL_SUCCESS; |
| 131 | } else { | 131 | } else { |
| 132 | GenWaitNode(curCcuTask, queId, waitCKEId, waitCKEMask); | 132 | GenWaitNode(curCcuTask, queId, waitCKEId, waitCKEMask); |
| @@ -216,7 +216,7 @@ HcclResult TransformInstrQue(TaskNodePtr node, TaskStubCcuGraph *curCcuTask, uin | |||
| 216 | AddCcuSubGraphEnd(node, curCcuTask); | 216 | AddCcuSubGraphEnd(node, curCcuTask); |
| 217 | } | 217 | } |
| 218 | 218 | ||
| 219 | - HCCL_VM_DEBUG("[TransformInstrQue] end..."); | 219 | + HCCL_VM_DEBUG("end..."); |
| 220 | return HCCL_SUCCESS; | 220 | return HCCL_SUCCESS; |
| 221 | } | 221 | } |
| 222 | 222 | ||
| @@ -360,7 +360,7 @@ HcclResult GenCcuGraph(TaskNode* dummyStart) { | |||
| 360 | AllRankParamRecorder::Global()->InitParam(); | 360 | AllRankParamRecorder::Global()->InitParam(); |
| 361 | HcclSim::StorageManager::GetInstance().InitCcuInfo(AllRankParamRecorder::Global()->devType_, | 361 | HcclSim::StorageManager::GetInstance().InitCcuInfo(AllRankParamRecorder::Global()->devType_, |
| 362 | AllRankParamRecorder::Global()->ccu_resource_base_addr_); | 362 | AllRankParamRecorder::Global()->ccu_resource_base_addr_); |
| 363 | - HCCL_VM_INFO("[GenCcuGraph] dummyStart children size: {}", dummyStart->children.size()); | 363 | + HCCL_VM_INFO("dummyStart children size: {}", dummyStart->children.size()); |
| 364 | 364 | ||
| 365 | bool isPrintCcuGraph = std::getenv("CCU_TASK_PRINT"); | 365 | bool isPrintCcuGraph = std::getenv("CCU_TASK_PRINT"); |
| 366 | completedNodes.insert(dummyStart); | 366 | completedNodes.insert(dummyStart); |
| @@ -370,7 +370,7 @@ HcclResult GenCcuGraph(TaskNode* dummyStart) { | |||
| 370 | candNode.push_back(child); | 370 | candNode.push_back(child); |
| 371 | } | 371 | } |
| 372 | 372 | ||
| 373 | - HCCL_VM_INFO("[GenCcuGraph] start......{}", candNode.size()); | 373 | + HCCL_VM_INFO("start......{}", candNode.size()); |
| 374 | u32 unmatchedCnt = 0; | 374 | u32 unmatchedCnt = 0; |
| 375 | while(!candNode.empty()) { | 375 | while(!candNode.empty()) { |
| 376 | // 先判断是否存在死锁的情况 | 376 | // 先判断是否存在死锁的情况 |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator/ccu_task_transform_instruct_a5.cc+9-9
| @@ -386,7 +386,7 @@ HcclResult TransformTransLocMemToRmtMemInstr(const CcuRep::CcuInstr *instr, Task | |||
| 386 | CHK_RET(StorageManager::GetInstance().GetSlice(rmtAddr, len, dstSlice, &rId)); | 386 | CHK_RET(StorageManager::GetInstance().GetSlice(rmtAddr, len, dstSlice, &rId)); |
| 387 | 387 | ||
| 388 | RankId rmtRankId = g_allRankChannelInfo[rankId][dieId][channelId].dstRank; | 388 | RankId rmtRankId = g_allRankChannelInfo[rankId][dieId][channelId].dstRank; |
| 389 | - HCCL_VM_DEBUG("[TransformTransLocMemToRmtMemInstr] curRank= {}, dstRank= {}", rankId, rmtRankId); | 389 | + HCCL_VM_DEBUG("curRank= {}, dstRank= {}", rankId, rmtRankId); |
| 390 | // 说明transportId写错了 | 390 | // 说明transportId写错了 |
| 391 | if (rId != rmtRankId) { | 391 | if (rId != rmtRankId) { |
| 392 | HCCL_ERROR("remoteRankId calculated by rmtAddr and channelId %u is not equal," | 392 | HCCL_ERROR("remoteRankId calculated by rmtAddr and channelId %u is not equal," |
| @@ -523,12 +523,12 @@ HcclResult TransformSyncCKEInstr(const CcuRep::CcuInstr *instr, TaskStubCcuGraph | |||
| 523 | uint16_t ckeValue = 0; | 523 | uint16_t ckeValue = 0; |
| 524 | uint16_t localCkeValue = 0; | 524 | uint16_t localCkeValue = 0; |
| 525 | CHK_RET(AllRankParamRecorder::Global()->GetCKE(rankId, dieId, locCKEId, localCkeValue)); | 525 | CHK_RET(AllRankParamRecorder::Global()->GetCKE(rankId, dieId, locCKEId, localCkeValue)); |
| 526 | - HCCL_VM_DEBUG("[TransformSyncCKEInstr] Get loc rank: {:d}, die: {:d} CKE: {:d}, value: {:d}, mask: {:x}", rankId, static_cast<uint32_t>(dieId), locCKEId, localCkeValue, locCKEMask); | 526 | + HCCL_VM_DEBUG("Get loc rank: {:d}, die: {:d} CKE: {:d}, value: {:d}, mask: {:x}", rankId, static_cast<uint32_t>(dieId), locCKEId, localCkeValue, locCKEMask); |
| 527 | localCkeValue = localCkeValue & locCKEMask; | 527 | localCkeValue = localCkeValue & locCKEMask; |
| 528 | CHK_RET(AllRankParamRecorder::Global()->GetCKE(rmtRankId, rmtDieId, rmtCKEId, ckeValue)); | 528 | CHK_RET(AllRankParamRecorder::Global()->GetCKE(rmtRankId, rmtDieId, rmtCKEId, ckeValue)); |
| 529 | - HCCL_VM_DEBUG("[TransformSyncCKEInstr] Get rmt rank: {:d}, die: {:d} CKE: {:d}, value: {:d}, mask: {:x}", rmtRankId, static_cast<uint32_t>(rmtDieId), rmtCKEId, ckeValue, locCKEMask); | 529 | + HCCL_VM_DEBUG("Get rmt rank: {:d}, die: {:d} CKE: {:d}, value: {:d}, mask: {:x}", rmtRankId, static_cast<uint32_t>(rmtDieId), rmtCKEId, ckeValue, locCKEMask); |
| 530 | CHK_RET(AllRankParamRecorder::Global()->SetCKE(rmtRankId, rmtDieId, rmtCKEId, ckeValue | localCkeValue)); | 530 | CHK_RET(AllRankParamRecorder::Global()->SetCKE(rmtRankId, rmtDieId, rmtCKEId, ckeValue | localCkeValue)); |
| 531 | - HCCL_VM_DEBUG("[TransformSyncCKEInstr] Set rank: {:d}, die: {:d} CKE: {:d}, value: {:d}, mask: {:x}", rmtRankId, static_cast<uint32_t>(rmtDieId), rmtCKEId, localCkeValue, locCKEMask); | 531 | + HCCL_VM_DEBUG("Set rank: {:d}, die: {:d} CKE: {:d}, value: {:d}, mask: {:x}", rmtRankId, static_cast<uint32_t>(rmtDieId), rmtCKEId, localCkeValue, locCKEMask); |
| 532 | } | 532 | } |
| 533 | 533 | ||
| 534 | CHK_RET(ProcessSetMask(rankId, dieId, curCcuTask, queId, setCKEId, setCKEMask, true)); | 534 | CHK_RET(ProcessSetMask(rankId, dieId, curCcuTask, queId, setCKEId, setCKEMask, true)); |
| @@ -1084,7 +1084,7 @@ HcclResult TransformTransRmtMemToLocMemInstr(const CcuRep::CcuInstr *instr, Task | |||
| 1084 | CHK_RET(StorageManager::GetInstance().GetSlice(localAddr, len, dstSlice)); | 1084 | CHK_RET(StorageManager::GetInstance().GetSlice(localAddr, len, dstSlice)); |
| 1085 | 1085 | ||
| 1086 | RankId rmtRankId = g_allRankChannelInfo[rankId][dieId][channelId].dstRank; | 1086 | RankId rmtRankId = g_allRankChannelInfo[rankId][dieId][channelId].dstRank; |
| 1087 | - HCCL_VM_DEBUG("[TransformTransRmtMemToLocMemInstr] curRank= {}, dstRank= {}", rankId, rmtRankId); | 1087 | + HCCL_VM_DEBUG("curRank= {}, dstRank= {}", rankId, rmtRankId); |
| 1088 | // 说明transportId写错了 | 1088 | // 说明transportId写错了 |
| 1089 | if (rId != rmtRankId) { | 1089 | if (rId != rmtRankId) { |
| 1090 | HCCL_ERROR("remoteRankId calculated by rmtAddr and channelId %u is not equal," | 1090 | HCCL_ERROR("remoteRankId calculated by rmtAddr and channelId %u is not equal," |
| @@ -1150,7 +1150,7 @@ HcclResult TransformLoopGroupInstr(const CcuRep::CcuInstr *instr, TaskStubCcuGra | |||
| 1150 | hcomm::CcuRep::CcuInstrInfo µCodeQue = curCcuTask->instrInfo[queId]; | 1150 | hcomm::CcuRep::CcuInstrInfo µCodeQue = curCcuTask->instrInfo[queId]; |
| 1151 | auto loopGroupIdx = curCcuTask->loopGroupIdx++; | 1151 | auto loopGroupIdx = curCcuTask->loopGroupIdx++; |
| 1152 | uint64_t loopCnt = loopGroupParam.loopGroupXn.loopInsCnt; | 1152 | uint64_t loopCnt = loopGroupParam.loopGroupXn.loopInsCnt; |
| 1153 | - HCCL_VM_DEBUG("[LoopGroup]loop cnt= {}, loop offset= {}, expand cnt= {}", | 1153 | + HCCL_VM_DEBUG("loop cnt= {}, loop offset= {}, expand cnt= {}", |
| 1154 | loopCnt, static_cast<uint64_t>(loopGroupParam.loopGroupXn.expandOffset), static_cast<uint64_t>(loopGroupParam.loopGroupXn.expandCnt)); | 1154 | loopCnt, static_cast<uint64_t>(loopGroupParam.loopGroupXn.expandOffset), static_cast<uint64_t>(loopGroupParam.loopGroupXn.expandCnt)); |
| 1155 | 1155 | ||
| 1156 | for (u32 curLoopIdx = 0; curLoopIdx < loopCnt; curLoopIdx++) { | 1156 | for (u32 curLoopIdx = 0; curLoopIdx < loopCnt; curLoopIdx++) { |
| @@ -1662,15 +1662,15 @@ HcclResult ProcessLoopIns(const CcuRep::CcuInstr *instr, TaskStubCcuGraph *curCc | |||
| 1662 | xm.value = value; | 1662 | xm.value = value; |
| 1663 | loopGroupParam->loopXms.push_back(xm); | 1663 | loopGroupParam->loopXms.push_back(xm); |
| 1664 | 1664 | ||
| 1665 | - HCCL_VM_DEBUG("[ProcessLoopIns] debug...{}, {}", loopGroupIdx, curCcuTask->loopGroupInfo_.size()); | 1665 | + HCCL_VM_DEBUG("debug...{}, {}", loopGroupIdx, curCcuTask->loopGroupInfo_.size()); |
| 1666 | auto loopIdx = curCcuTask->loopIdx[loopGroupIdx]++; | 1666 | auto loopIdx = curCcuTask->loopIdx[loopGroupIdx]++; |
| 1667 | auto loopStart = AddLoopStartTask(queId, loopIdx, loopGroupIdx, curCcuTask); // loop前新增loopStart标记节点 | 1667 | auto loopStart = AddLoopStartTask(queId, loopIdx, loopGroupIdx, curCcuTask); // loop前新增loopStart标记节点 |
| 1668 | 1668 | ||
| 1669 | - HCCL_VM_DEBUG("[LOOP] loop cnt = {}", static_cast<uint64_t>(xm.loopCnt)); | 1669 | + HCCL_VM_DEBUG("loop cnt = {}", static_cast<uint64_t>(xm.loopCnt)); |
| 1670 | hcomm::CcuRep::CcuInstrInfo& microCodeQue = curCcuTask->instrInfo[queId]; | 1670 | hcomm::CcuRep::CcuInstrInfo& microCodeQue = curCcuTask->instrInfo[queId]; |
| 1671 | for (u32 curLoopCnt = 0; curLoopCnt < xm.loopCnt; curLoopCnt++) { | 1671 | for (u32 curLoopCnt = 0; curLoopCnt < xm.loopCnt; curLoopCnt++) { |
| 1672 | loopGroupParam->curLoopCnt = curLoopCnt; | 1672 | loopGroupParam->curLoopCnt = curLoopCnt; |
| 1673 | - HCCL_VM_DEBUG("[LOOP] loop cnt = {}, cur loop= {}", static_cast<uint64_t>(xm.loopCnt), curLoopCnt); | 1673 | + HCCL_VM_DEBUG("loop cnt = {}, cur loop= {}", static_cast<uint64_t>(xm.loopCnt), curLoopCnt); |
| 1674 | for (uint16_t insId = startInstrId; insId <= endInstrId; insId++) { | 1674 | for (uint16_t insId = startInstrId; insId <= endInstrId; insId++) { |
| 1675 | // 获取当前要处理的指令 | 1675 | // 获取当前要处理的指令 |
| 1676 | const CcuRep::CcuInstr* instr = µCodeQue.instrVec[insId - curCcuTask->startInstrIdInQue[queId]]; | 1676 | const CcuRep::CcuInstr* instr = µCodeQue.instrVec[insId - curCcuTask->startInstrIdInQue[queId]]; |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/aiv_graph_generator_v3/aiv_snapshot_json_loader_v3.cc+5-14
| @@ -17,6 +17,7 @@ | |||
| 17 | 17 | ||
| 18 | 18 | ||
| 19 | 19 | ||
| 20 | + | ||
| 20 | 21 | ||
| 21 | 22 | ||
| 22 | namespace HcclSim { | 23 | namespace HcclSim { |
| @@ -41,15 +42,9 @@ void SetError(std::string &errorMessage, const std::string &message) | |||
| 41 | 42 | ||
| 42 | std::string ResolveAivTaskFilePath(RankId rankId, uint64_t launchIndex) | 43 | std::string ResolveAivTaskFilePath(RankId rankId, uint64_t launchIndex) |
| 43 | { | 44 | { |
| 44 | - const char *installDir = std::getenv("HCCL_VM_INSTALL_DIR"); | 45 | + const std::string dataRelPath = std::string("data/") + AIV_RANK_TASK_FILE_PREFIX + std::to_string(rankId) |
| 45 | - if (installDir == nullptr || installDir[0] == '\0') { | 46 | + + AIV_RANK_TASK_FILE_LAUNCH_MARKER + std::to_string(launchIndex) + AIV_RANK_TASK_FILE_SUFFIX; |
| 46 | - return ""; | 47 | + return InstallPath::ResolveToInstallRoot(dataRelPath); |
| 47 | - } | ||
| 48 | - | ||
| 49 | - std::ostringstream os; | ||
| 50 | - os << installDir << "/data/" << AIV_RANK_TASK_FILE_PREFIX << rankId | ||
| 51 | - << AIV_RANK_TASK_FILE_LAUNCH_MARKER << launchIndex << AIV_RANK_TASK_FILE_SUFFIX; | ||
| 52 | - return os.str(); | ||
| 53 | } | 48 | } |
| 54 | 49 | ||
| 55 | bool CheckObjectField(const Json &json, const char *fieldName, std::string &errorMessage) | 50 | bool CheckObjectField(const Json &json, const char *fieldName, std::string &errorMessage) |
| @@ -225,10 +220,6 @@ HcclResult AivSnapshotJsonLoaderV3::LoadByRankAndLaunch(RankId rankId, uint64_t | |||
| 225 | { | 220 | { |
| 226 | snapshot = AivRuntimeTaskSnapshotV3 {}; | 221 | snapshot = AivRuntimeTaskSnapshotV3 {}; |
| 227 | const std::string filePath = ResolveAivTaskFilePath(rankId, launchIndex); | 222 | const std::string filePath = ResolveAivTaskFilePath(rankId, launchIndex); |
| 228 | - if (filePath.empty()) { | ||
| 229 | - SetError(errorMessage, "HCCL_VM_INSTALL_DIR is not set"); | ||
| 230 | - return HCCL_E_NOT_FOUND; | ||
| 231 | - } | ||
| 232 | if (!FileExists(filePath)) { | 223 | if (!FileExists(filePath)) { |
| 233 | SetError(errorMessage, "AIV task json file does not exist: " + filePath); | 224 | SetError(errorMessage, "AIV task json file does not exist: " + filePath); |
| 234 | return HCCL_E_NOT_FOUND; | 225 | return HCCL_E_NOT_FOUND; |
| @@ -263,7 +254,7 @@ HcclResult AivSnapshotJsonLoaderV3::LoadByRankAndLaunch(RankId rankId, uint64_t | |||
| 263 | return HCCL_E_PARA; | 254 | return HCCL_E_PARA; |
| 264 | } | 255 | } |
| 265 | 256 | ||
| 266 | - HCCL_VM_INFO("[AivSnapshotJsonLoaderV3] Loaded AIV snapshot, rankId={}, launchIndex={}, blockCount={}, file={}", | 257 | + HCCL_VM_INFO("Loaded AIV snapshot, rankId={}, launchIndex={}, blockCount={}, file={}", |
| 267 | rankId, launchIndex, snapshot.blocks.size(), filePath); | 258 | rankId, launchIndex, snapshot.blocks.size(), filePath); |
| 268 | return HCCL_SUCCESS; | 259 | return HCCL_SUCCESS; |
| 269 | } | 260 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/aiv_graph_generator_v3/aiv_task_transform_v3.cc+133-48
| @@ -25,6 +25,7 @@ | |||
| 25 | 25 | ||
| 26 | 26 | ||
| 27 | 27 | ||
| 28 | + | ||
| 28 | 29 | ||
| 29 | namespace HcclSim { | 30 | namespace HcclSim { |
| 30 | namespace TaskGraphGeneratorV3 { | 31 | namespace TaskGraphGeneratorV3 { |
| @@ -353,8 +354,9 @@ HcclResult ValidateSnapshot(const AivLaunchContext &ctx) | |||
| 353 | } | 354 | } |
| 354 | const CheckerParam param = ctx.storage->GetCheckerParam(); | 355 | const CheckerParam param = ctx.storage->GetCheckerParam(); |
| 355 | if (ctx.snapshot.rankSize != 0 && param.rankSize != 0 && ctx.snapshot.rankSize != param.rankSize) { | 356 | if (ctx.snapshot.rankSize != 0 && param.rankSize != 0 && ctx.snapshot.rankSize != param.rankSize) { |
| 356 | - HCCL_VM_ERROR("[AivTaskTransformV3] AIV snapshot rankSize mismatch, rank={}, launch={}, " | 357 | + HCCL_VM_ERROR("{} The AIV snapshot was captured for a different rank count than the current " |
| 357 | - "snapshotRankSize={}, checkerRankSize={}, file={}", ctx.placeholder->GetRankId(), | 358 | + "checker input, rankId={}, launchId={}, snapshotRankCount={}, currentRankCount={}, snapshotFile={}", |
| 359 | + MakeErrorCodeText(ErrorCode::GRAPH_SNAPSHOT_MISMATCH), ctx.placeholder->GetRankId(), | ||
| 358 | ctx.placeholder->GetLaunchIdx(), ctx.snapshot.rankSize, param.rankSize, ctx.snapshot.filePath); | 360 | ctx.placeholder->GetLaunchIdx(), ctx.snapshot.rankSize, param.rankSize, ctx.snapshot.filePath); |
| 359 | return HCCL_E_PARA; | 361 | return HCCL_E_PARA; |
| 360 | } | 362 | } |
| @@ -375,9 +377,10 @@ HcclResult UpdateAivBufferSize(uint64_t snapshotSize, const char *fieldName, con | |||
| 375 | return HCCL_SUCCESS; | 377 | return HCCL_SUCCESS; |
| 376 | } | 378 | } |
| 377 | 379 | ||
| 378 | - HCCL_VM_ERROR("[AivTaskTransformV3] AIV {} mismatch, rank={}, launch={}, expected={}, actual={}, file={}", | 380 | + HCCL_VM_ERROR("{} AIV buffer size is inconsistent across snapshots, field={}, rankId={}, " |
| 379 | - fieldName, ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), bufferSize, snapshotSize, | 381 | + "launchId={}, expectedSize={}, actualSize={}, expectedSource=firstNonZeroSnapshot, snapshotFile={}", |
| 380 | - ctx.snapshot.filePath); | 382 | + MakeErrorCodeText(ErrorCode::GRAPH_SNAPSHOT_MISMATCH), fieldName, ctx.placeholder->GetRankId(), |
| 383 | + ctx.placeholder->GetLaunchIdx(), bufferSize, snapshotSize, ctx.snapshot.filePath); | ||
| 381 | return HCCL_E_PARA; | 384 | return HCCL_E_PARA; |
| 382 | } | 385 | } |
| 383 | 386 | ||
| @@ -385,9 +388,11 @@ HcclResult AppendRuntimeTask(AivLaunchContext &ctx, const AivRuntimeTaskV3 &task | |||
| 385 | { | 388 | { |
| 386 | std::unique_ptr<TaskNode> node = TranslateAivRuntimeTask(task, ctx.placeholder->GetLaunchIdx()); | 389 | std::unique_ptr<TaskNode> node = TranslateAivRuntimeTask(task, ctx.placeholder->GetLaunchIdx()); |
| 387 | if (node == nullptr) { | 390 | if (node == nullptr) { |
| 388 | - HCCL_VM_ERROR("[AivTaskTransformV3] Translate AIV task failed, rank={}, launch={}, taskId={}, taskType={}, " | 391 | + HCCL_VM_ERROR("{} One AIV runtime task type is not supported, " |
| 389 | - "file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), task.taskId, | 392 | + "rankId={}, launchId={}, taskId={}, taskType={}, snapshotFile={}", |
| 390 | - static_cast<uint32_t>(task.taskType), ctx.snapshot.filePath); | 393 | + MakeErrorCodeText(ErrorCode::GRAPH_UNSUPPORTED), ctx.placeholder->GetRankId(), |
| 394 | + ctx.placeholder->GetLaunchIdx(), task.taskId, static_cast<uint32_t>(task.taskType), | ||
| 395 | + ctx.snapshot.filePath); | ||
| 391 | return HCCL_E_NOT_SUPPORT; | 396 | return HCCL_E_NOT_SUPPORT; |
| 392 | } | 397 | } |
| 393 | 398 | ||
| @@ -507,8 +512,21 @@ HcclResult AddSetWaitFlagEdges(AivLaunchContext &ctx) | |||
| 507 | while (!readyQueue.empty()) { | 512 | while (!readyQueue.empty()) { |
| 508 | if (unmatchedCnt >= readyQueue.size()) { | 513 | if (unmatchedCnt >= readyQueue.size()) { |
| 509 | TaskNode *node = ctx.graph->GetNode(readyQueue.front()); | 514 | TaskNode *node = ctx.graph->GetNode(readyQueue.front()); |
| 510 | - HCCL_VM_ERROR("[AivTaskTransformV3] SetFlag/WaitFlag matching deadlock, firstUnmatched={}, file={}", | 515 | + size_t seenMatchingSetFlagCount = 0; |
| 511 | - node == nullptr ? "null" : node->Describe(), ctx.snapshot.filePath); | 516 | + if (node != nullptr && node->GetType() == TaskType::AIV_WAIT_FLAG) { |
| 517 | + const auto *waitFlag = dynamic_cast<const TaskAivWaitFlag *>(node); | ||
| 518 | + if (waitFlag != nullptr) { | ||
| 519 | + const SetWaitKey key = MakeSetWaitKey(waitFlag->GetEvent()); | ||
| 520 | + const auto iter = seenSetFlags.find(key); | ||
| 521 | + seenMatchingSetFlagCount = (iter == seenSetFlags.end()) ? 0U : iter->second.size(); | ||
| 522 | + } | ||
| 523 | + } | ||
| 524 | + HCCL_VM_ERROR("{} AIV SetFlag/WaitFlag matching is stuck. Some WaitFlag tasks are still " | ||
| 525 | + "blocked, but no matching SetFlag has become available, firstBlockedWaitFlagNode={}, " | ||
| 526 | + "blockedWaitFlagNodeCount={}, availableSetFlagCountForThisWait={}, snapshotFile={}", | ||
| 527 | + MakeErrorCodeText(ErrorCode::GRAPH_DEADLOCK), node == nullptr ? "node=null" : node->Describe(), | ||
| 528 | + readyQueue.size(), | ||
| 529 | + seenMatchingSetFlagCount, ctx.snapshot.filePath); | ||
| 512 | return HCCL_E_INTERNAL; | 530 | return HCCL_E_INTERNAL; |
| 513 | } | 531 | } |
| 514 | 532 | ||
| @@ -584,8 +602,11 @@ HcclResult AddSetWaitFlagEdges(AivLaunchContext &ctx) | |||
| 584 | for (const auto &entry : seenSetFlags) { | 602 | for (const auto &entry : seenSetFlags) { |
| 585 | if (!entry.second.empty()) { | 603 | if (!entry.second.empty()) { |
| 586 | TaskNode *node = ctx.graph->GetNode(entry.second.front()); | 604 | TaskNode *node = ctx.graph->GetNode(entry.second.front()); |
| 587 | - HCCL_VM_WARN("[AivTaskTransformV3] Unconsumed SetFlag, node={}, file={}", | 605 | + HCCL_VM_WARN("{} Found SetFlag tasks that were never consumed by any WaitFlag task, " |
| 588 | - node == nullptr ? "null" : node->Describe(), ctx.snapshot.filePath); | 606 | + "firstUnconsumedSetFlagNode={}, unconsumedSetFlagCount={}, snapshotFile={}", |
| 607 | + MakeErrorCodeText(ErrorCode::GRAPH_UNMATCHED), node == nullptr ? "node=null" : node->Describe(), | ||
| 608 | + entry.second.size(), | ||
| 609 | + ctx.snapshot.filePath); | ||
| 589 | } | 610 | } |
| 590 | } | 611 | } |
| 591 | return HCCL_SUCCESS; | 612 | return HCCL_SUCCESS; |
| @@ -1535,8 +1556,9 @@ HcclResult MergeCpGM2GMLoops(AivLaunchContext &ctx) | |||
| 1535 | for (const auto &block : ctx.snapshot.blocks) { | 1556 | for (const auto &block : ctx.snapshot.blocks) { |
| 1536 | std::vector<NodeId> topo; | 1557 | std::vector<NodeId> topo; |
| 1537 | if (!CollectCpGmBlockTopo(ctx, block.blockIdx, topo)) { | 1558 | if (!CollectCpGmBlockTopo(ctx, block.blockIdx, topo)) { |
| 1538 | - HCCL_VM_WARN("[AivTaskTransformV3][CpGM2GMMerge] Skip block due to topo failure, rank={}, launch={}, " | 1559 | + HCCL_VM_WARN("Skip CpGM-to-GM merge for one block because the block DAG order could not " |
| 1539 | - "block={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), block.blockIdx, | 1560 | + "be determined, rankId={}, launchId={}, blockId={}, snapshotFile={}", |
| 1561 | + ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), block.blockIdx, | ||
| 1540 | ctx.snapshot.filePath); | 1562 | ctx.snapshot.filePath); |
| 1541 | continue; | 1563 | continue; |
| 1542 | } | 1564 | } |
| @@ -1570,10 +1592,11 @@ HcclResult MergeCpGM2GMLoops(AivLaunchContext &ctx) | |||
| 1570 | } | 1592 | } |
| 1571 | } | 1593 | } |
| 1572 | ctx.dagNodeCountAfterCpGmMerge = CountReachableActiveInternalNodes(ctx); | 1594 | ctx.dagNodeCountAfterCpGmMerge = CountReachableActiveInternalNodes(ctx); |
| 1573 | - HCCL_VM_INFO("[AivTaskTransformV3][CpGM2GMMerge] rank={}, launch={}, taskJsonTotalTaskCount={}, " | 1595 | + HCCL_VM_INFO("Finished CpGM-to-GM merge analysis, rankId={}, launchId={}, " |
| 1574 | - "dagNodeCountBeforeMerge={}, dagNodeCountAfterMerge={}, cpGmLoopMergeCount={}, " | 1596 | + "taskJsonTotalTaskCount={}, dagNodeCountBeforeMerge={}, dagNodeCountAfterMerge={}, " |
| 1575 | - "cpGmMergedIterationCount={}, cpGmMergedOriginalNodeCount={}, cpGmGeneratedNodeCount={}, " | 1597 | + "cpGmLoopMergeCount={}, cpGmMergedIterationCount={}, cpGmMergedOriginalNodeCount={}, " |
| 1576 | - "cpGmInactiveNodeCount={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1598 | + "cpGmGeneratedNodeCount={}, cpGmInactiveNodeCount={}", ctx.placeholder->GetRankId(), |
| 1599 | + ctx.placeholder->GetLaunchIdx(), | ||
| 1577 | ctx.taskJsonTotalTaskCount, ctx.dagNodeCountBeforeCpGmMerge, ctx.dagNodeCountAfterCpGmMerge, | 1600 | ctx.taskJsonTotalTaskCount, ctx.dagNodeCountBeforeCpGmMerge, ctx.dagNodeCountAfterCpGmMerge, |
| 1578 | ctx.cpGmLoopMergeCount, ctx.cpGmMergedIterationCount, ctx.cpGmMergedOriginalNodeCount, | 1601 | ctx.cpGmLoopMergeCount, ctx.cpGmMergedIterationCount, ctx.cpGmMergedOriginalNodeCount, |
| 1579 | ctx.cpGmGeneratedNodeCount, ctx.cpGmInactiveNodeCount); | 1602 | ctx.cpGmGeneratedNodeCount, ctx.cpGmInactiveNodeCount); |
| @@ -1710,23 +1733,26 @@ HcclResult MergePipeBarrierGroups(AivLaunchContext &ctx) | |||
| 1710 | TaskNode *member = ctx.graph->GetNode(memberNodeId); | 1733 | TaskNode *member = ctx.graph->GetNode(memberNodeId); |
| 1711 | const auto *barrier = dynamic_cast<const TaskAivPipeBarrier *>(member); | 1734 | const auto *barrier = dynamic_cast<const TaskAivPipeBarrier *>(member); |
| 1712 | if (barrier == nullptr) { | 1735 | if (barrier == nullptr) { |
| 1713 | - HCCL_VM_ERROR("[AivTaskTransformV3] Invalid PipeBarrier member node, rank={}, launch={}, " | 1736 | + HCCL_VM_ERROR("{} One PipeBarrier group member is not a valid PipeBarrier node, " |
| 1714 | - "nodeId={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1737 | + "rankId={}, launchId={}, nodeId={}, snapshotFile={}", |
| 1715 | - memberNodeId, ctx.snapshot.filePath); | 1738 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), ctx.placeholder->GetRankId(), |
| 1739 | + ctx.placeholder->GetLaunchIdx(), memberNodeId, ctx.snapshot.filePath); | ||
| 1716 | return HCCL_E_INTERNAL; | 1740 | return HCCL_E_INTERNAL; |
| 1717 | } | 1741 | } |
| 1718 | const AivBarrierInfo &memberInfo = barrier->GetInfo(); | 1742 | const AivBarrierInfo &memberInfo = barrier->GetInfo(); |
| 1719 | if (memberInfo.taskLoc.blockId != group.blockId) { | 1743 | if (memberInfo.taskLoc.blockId != group.blockId) { |
| 1720 | - HCCL_VM_ERROR("[AivTaskTransformV3] PipeBarrier group crosses block, rank={}, launch={}, " | 1744 | + HCCL_VM_ERROR("{} One PipeBarrier group mixes tasks from different blocks, rankId={}, " |
| 1721 | - "groupBlock={}, memberBlock={}, memberTaskId={}, file={}", ctx.placeholder->GetRankId(), | 1745 | + "launchId={}, expectedBlockId={}, actualBlockId={}, memberTaskId={}, snapshotFile={}", |
| 1746 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), ctx.placeholder->GetRankId(), | ||
| 1722 | ctx.placeholder->GetLaunchIdx(), group.blockId, memberInfo.taskLoc.blockId, | 1747 | ctx.placeholder->GetLaunchIdx(), group.blockId, memberInfo.taskLoc.blockId, |
| 1723 | memberInfo.taskLoc.taskId, ctx.snapshot.filePath); | 1748 | memberInfo.taskLoc.taskId, ctx.snapshot.filePath); |
| 1724 | return HCCL_E_INTERNAL; | 1749 | return HCCL_E_INTERNAL; |
| 1725 | } | 1750 | } |
| 1726 | if (!memberPipes.insert(memberInfo.taskLoc.pipe).second) { | 1751 | if (!memberPipes.insert(memberInfo.taskLoc.pipe).second) { |
| 1727 | - HCCL_VM_ERROR("[AivTaskTransformV3] Duplicate PipeBarrier member pipe, rank={}, launch={}, " | 1752 | + HCCL_VM_ERROR("{} One PipeBarrier group contains two tasks from the same pipe, " |
| 1728 | - "block={}, pipe={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1753 | + "rankId={}, launchId={}, blockId={}, duplicatedPipeId={}, snapshotFile={}", |
| 1729 | - group.blockId, memberInfo.taskLoc.pipe, ctx.snapshot.filePath); | 1754 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), ctx.placeholder->GetRankId(), |
| 1755 | + ctx.placeholder->GetLaunchIdx(), group.blockId, memberInfo.taskLoc.pipe, ctx.snapshot.filePath); | ||
| 1730 | return HCCL_E_INTERNAL; | 1756 | return HCCL_E_INTERNAL; |
| 1731 | } | 1757 | } |
| 1732 | } | 1758 | } |
| @@ -1735,9 +1761,30 @@ HcclResult MergePipeBarrierGroups(AivLaunchContext &ctx) | |||
| 1735 | if (memberPipes.count(pipe) != 0) { | 1761 | if (memberPipes.count(pipe) != 0) { |
| 1736 | continue; | 1762 | continue; |
| 1737 | } | 1763 | } |
| 1738 | - HCCL_VM_ERROR("[AivTaskTransformV3] Missing PIPE_ALL PipeBarrier member, rank={}, launch={}, " | 1764 | + std::ostringstream existingPipeIds; |
| 1739 | - "block={}, pipe={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1765 | + existingPipeIds << "["; |
| 1740 | - group.blockId, pipe, ctx.snapshot.filePath); | 1766 | + for (size_t i = 0; i < group.memberPipes.size(); ++i) { |
| 1767 | + if (i != 0) { | ||
| 1768 | + existingPipeIds << ","; | ||
| 1769 | + } | ||
| 1770 | + existingPipeIds << group.memberPipes[i]; | ||
| 1771 | + } | ||
| 1772 | + existingPipeIds << "]"; | ||
| 1773 | + std::ostringstream memberTaskIds; | ||
| 1774 | + memberTaskIds << "["; | ||
| 1775 | + for (size_t i = 0; i < group.memberTaskIds.size(); ++i) { | ||
| 1776 | + if (i != 0) { | ||
| 1777 | + memberTaskIds << ","; | ||
| 1778 | + } | ||
| 1779 | + memberTaskIds << group.memberTaskIds[i]; | ||
| 1780 | + } | ||
| 1781 | + memberTaskIds << "]"; | ||
| 1782 | + HCCL_VM_ERROR("{} One PIPE_ALL PipeBarrier is incomplete because one pipe is missing, " | ||
| 1783 | + "rankId={}, launchId={}, blockId={}, missingPipeId={}, existingPipeIds={}, memberTaskIds={}, " | ||
| 1784 | + "snapshotFile={}", | ||
| 1785 | + MakeErrorCodeText(ErrorCode::GRAPH_MEMBER_MISSING), ctx.placeholder->GetRankId(), | ||
| 1786 | + ctx.placeholder->GetLaunchIdx(), group.blockId, pipe, existingPipeIds.str(), memberTaskIds.str(), | ||
| 1787 | + ctx.snapshot.filePath); | ||
| 1741 | return HCCL_E_INTERNAL; | 1788 | return HCCL_E_INTERNAL; |
| 1742 | } | 1789 | } |
| 1743 | } | 1790 | } |
| @@ -1781,9 +1828,12 @@ HcclResult MergeSyncAllGroups(AivLaunchContext &ctx) | |||
| 1781 | continue; | 1828 | continue; |
| 1782 | } | 1829 | } |
| 1783 | if (expectedMemberCount != 0 && group.memberNodeIds.size() != expectedMemberCount) { | 1830 | if (expectedMemberCount != 0 && group.memberNodeIds.size() != expectedMemberCount) { |
| 1784 | - HCCL_VM_ERROR("[AivTaskTransformV3] SyncAll member count mismatch, rank={}, launch={}, syncRound={}, " | 1831 | + HCCL_VM_ERROR("{} One SyncAll group is incomplete because it does not contain one member " |
| 1785 | - "actual={}, expected={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1832 | + "for every expected block/pipe pair, rankId={}, launchId={}, syncRound={}, actualCount={}, " |
| 1786 | - group.syncRound, group.memberNodeIds.size(), expectedMemberCount, ctx.snapshot.filePath); | 1833 | + "expectedCount={}, snapshotFile={}", |
| 1834 | + MakeErrorCodeText(ErrorCode::GRAPH_MEMBER_MISSING), | ||
| 1835 | + ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), group.syncRound, | ||
| 1836 | + group.memberNodeIds.size(), expectedMemberCount, ctx.snapshot.filePath); | ||
| 1787 | return HCCL_E_INTERNAL; | 1837 | return HCCL_E_INTERNAL; |
| 1788 | } | 1838 | } |
| 1789 | std::set<std::pair<uint32_t, uint32_t>> members; | 1839 | std::set<std::pair<uint32_t, uint32_t>> members; |
| @@ -1791,16 +1841,19 @@ HcclResult MergeSyncAllGroups(AivLaunchContext &ctx) | |||
| 1791 | TaskNode *member = ctx.graph->GetNode(memberNodeId); | 1841 | TaskNode *member = ctx.graph->GetNode(memberNodeId); |
| 1792 | const auto *syncAll = dynamic_cast<const TaskAivSyncAll *>(member); | 1842 | const auto *syncAll = dynamic_cast<const TaskAivSyncAll *>(member); |
| 1793 | if (syncAll == nullptr) { | 1843 | if (syncAll == nullptr) { |
| 1794 | - HCCL_VM_ERROR("[AivTaskTransformV3] Invalid SyncAll member node, rank={}, launch={}, syncRound={}, " | 1844 | + HCCL_VM_ERROR("{} One SyncAll group member is not a valid SyncAll node, rankId={}, " |
| 1795 | - "nodeId={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1845 | + "launchId={}, syncRound={}, nodeId={}, snapshotFile={}", |
| 1796 | - group.syncRound, memberNodeId, ctx.snapshot.filePath); | 1846 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), ctx.placeholder->GetRankId(), |
| 1847 | + ctx.placeholder->GetLaunchIdx(), group.syncRound, memberNodeId, ctx.snapshot.filePath); | ||
| 1797 | return HCCL_E_INTERNAL; | 1848 | return HCCL_E_INTERNAL; |
| 1798 | } | 1849 | } |
| 1799 | const TaskPosition &taskLoc = syncAll->GetInfo().taskLoc; | 1850 | const TaskPosition &taskLoc = syncAll->GetInfo().taskLoc; |
| 1800 | if (!members.insert({taskLoc.blockId, taskLoc.pipe}).second) { | 1851 | if (!members.insert({taskLoc.blockId, taskLoc.pipe}).second) { |
| 1801 | - HCCL_VM_ERROR("[AivTaskTransformV3] Duplicate SyncAll member, rank={}, launch={}, syncRound={}, " | 1852 | + HCCL_VM_ERROR("{} One SyncAll group contains duplicate members for the same " |
| 1802 | - "block={}, pipe={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1853 | + "block/pipe pair, rankId={}, launchId={}, syncRound={}, blockId={}, pipeId={}, snapshotFile={}", |
| 1803 | - group.syncRound, taskLoc.blockId, taskLoc.pipe, ctx.snapshot.filePath); | 1854 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), ctx.placeholder->GetRankId(), |
| 1855 | + ctx.placeholder->GetLaunchIdx(), group.syncRound, taskLoc.blockId, taskLoc.pipe, | ||
| 1856 | + ctx.snapshot.filePath); | ||
| 1804 | return HCCL_E_INTERNAL; | 1857 | return HCCL_E_INTERNAL; |
| 1805 | } | 1858 | } |
| 1806 | } | 1859 | } |
| @@ -1809,9 +1862,12 @@ HcclResult MergeSyncAllGroups(AivLaunchContext &ctx) | |||
| 1809 | if (members.count({block.blockIdx, pipe}) != 0) { | 1862 | if (members.count({block.blockIdx, pipe}) != 0) { |
| 1810 | continue; | 1863 | continue; |
| 1811 | } | 1864 | } |
| 1812 | - HCCL_VM_ERROR("[AivTaskTransformV3] Missing SyncAll member, rank={}, launch={}, syncRound={}, " | 1865 | + HCCL_VM_ERROR("{} One SyncAll group is incomplete because a block/pipe member is " |
| 1813 | - "block={}, pipe={}, file={}", ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), | 1866 | + "missing, rankId={}, launchId={}, syncRound={}, missingBlockId={}, missingPipeId={}, " |
| 1814 | - group.syncRound, block.blockIdx, pipe, ctx.snapshot.filePath); | 1867 | + "snapshotFile={}", |
| 1868 | + MakeErrorCodeText(ErrorCode::GRAPH_MEMBER_MISSING), | ||
| 1869 | + ctx.placeholder->GetRankId(), ctx.placeholder->GetLaunchIdx(), group.syncRound, | ||
| 1870 | + block.blockIdx, pipe, ctx.snapshot.filePath); | ||
| 1815 | return HCCL_E_INTERNAL; | 1871 | return HCCL_E_INTERNAL; |
| 1816 | } | 1872 | } |
| 1817 | } | 1873 | } |
| @@ -1955,7 +2011,8 @@ HcclResult ExpandOneAivGraph(TaskGraphGeneratorV3 *graph, StorageManager *storag | |||
| 1955 | HcclResult ret = loader.LoadByRankAndLaunch(aivGraph->GetRankId(), aivGraph->GetLaunchIdx(), ctx.snapshot, | 2011 | HcclResult ret = loader.LoadByRankAndLaunch(aivGraph->GetRankId(), aivGraph->GetLaunchIdx(), ctx.snapshot, |
| 1956 | errorMessage); | 2012 | errorMessage); |
| 1957 | if (ret != HCCL_SUCCESS) { | 2013 | if (ret != HCCL_SUCCESS) { |
| 1958 | - HCCL_VM_ERROR("[AivTaskTransformV3] Load AIV snapshot failed, rank={}, launch={}, ret={}, err={}", | 2014 | + HCCL_VM_ERROR("{} Failed to load the AIV runtime snapshot file, rankId={}, launchId={}, " |
| 2015 | + "ret={}, loaderMessage={}", MakeErrorCodeText(ErrorCode::GRAPH_RESOURCE_NOT_FOUND), | ||
| 1959 | aivGraph->GetRankId(), aivGraph->GetLaunchIdx(), static_cast<uint32_t>(ret), errorMessage); | 2016 | aivGraph->GetRankId(), aivGraph->GetLaunchIdx(), static_cast<uint32_t>(ret), errorMessage); |
| 1960 | return ret; | 2017 | return ret; |
| 1961 | } | 2018 | } |
| @@ -2079,7 +2136,8 @@ HcclResult EnqueueChildrenForFlagMatch(const TaskGraphGeneratorV3 *graph, NodeId | |||
| 2079 | return HCCL_SUCCESS; | 2136 | return HCCL_SUCCESS; |
| 2080 | } | 2137 | } |
| 2081 | 2138 | ||
| 2082 | -HcclResult MatchSendRecvFlagEdges(TaskGraphGeneratorV3 *graph, size_t &matchedEdgeCount) | 2139 | +HcclResult MatchSendRecvFlagEdges(TaskGraphGeneratorV3 *graph, const std::string &snapshotFile, |
| 2140 | + size_t &matchedEdgeCount) | ||
| 2083 | { | 2141 | { |
| 2084 | matchedEdgeCount = 0; | 2142 | matchedEdgeCount = 0; |
| 2085 | if (graph == nullptr || graph->GetMainStartNode() == nullptr) { | 2143 | if (graph == nullptr || graph->GetMainStartNode() == nullptr) { |
| @@ -2104,8 +2162,33 @@ HcclResult MatchSendRecvFlagEdges(TaskGraphGeneratorV3 *graph, size_t &matchedEd | |||
| 2104 | while (!readyQueue.empty()) { | 2162 | while (!readyQueue.empty()) { |
| 2105 | if (unmatchedCnt >= readyQueue.size()) { | 2163 | if (unmatchedCnt >= readyQueue.size()) { |
| 2106 | const TaskNode *node = graph->GetNode(readyQueue.front()); | 2164 | const TaskNode *node = graph->GetNode(readyQueue.front()); |
| 2107 | - HCCL_VM_ERROR("[AivTaskTransformV3] Send/RecvFlag matching deadlock, firstUnmatched={}", | 2165 | + std::string flagCellStateText = "{}"; |
| 2108 | - node == nullptr ? "null" : node->Describe()); | 2166 | + if (node != nullptr && node->GetType() == TaskType::AIV_RECV_FLAG) { |
| 2167 | + const auto *recvFlag = dynamic_cast<const TaskAivRecvFlag *>(node); | ||
| 2168 | + if (recvFlag != nullptr) { | ||
| 2169 | + const auto stateIter = flagStates.find(MakeFlagCellKey(recvFlag->GetFlag())); | ||
| 2170 | + if (stateIter != flagStates.end()) { | ||
| 2171 | + std::ostringstream os; | ||
| 2172 | + os << "{currentValue=" << stateIter->second.currentValue | ||
| 2173 | + << ", producerCount=" << stateIter->second.producerNodes.size() | ||
| 2174 | + << ", producerNodeIds=["; | ||
| 2175 | + for (size_t i = 0; i < stateIter->second.producerNodes.size(); ++i) { | ||
| 2176 | + if (i != 0) { | ||
| 2177 | + os << ","; | ||
| 2178 | + } | ||
| 2179 | + os << stateIter->second.producerNodes[i]; | ||
| 2180 | + } | ||
| 2181 | + os << "]}"; | ||
| 2182 | + flagCellStateText = os.str(); | ||
| 2183 | + } | ||
| 2184 | + } | ||
| 2185 | + } | ||
| 2186 | + HCCL_VM_ERROR("{} AIV SendFlag/RecvFlag matching is stuck. Some RecvFlag tasks are still " | ||
| 2187 | + "blocked, but no matching SendFlag value has become available, firstBlockedRecvFlagNode={}, " | ||
| 2188 | + "blockedRecvFlagNodeCount={}, currentFlagCellState={}, snapshotFile={}", | ||
| 2189 | + MakeErrorCodeText(ErrorCode::GRAPH_DEADLOCK), node == nullptr ? "node=null" : node->Describe(), | ||
| 2190 | + readyQueue.size(), | ||
| 2191 | + flagCellStateText, snapshotFile); | ||
| 2109 | return HCCL_E_INTERNAL; | 2192 | return HCCL_E_INTERNAL; |
| 2110 | } | 2193 | } |
| 2111 | 2194 | ||
| @@ -2134,8 +2217,8 @@ HcclResult MatchSendRecvFlagEdges(TaskGraphGeneratorV3 *graph, size_t &matchedEd | |||
| 2134 | } | 2217 | } |
| 2135 | FlagCellState &state = flagStates[MakeFlagCellKey(sendFlag->GetFlag())]; | 2218 | FlagCellState &state = flagStates[MakeFlagCellKey(sendFlag->GetFlag())]; |
| 2136 | if (state.currentValue == sendFlag->GetFlag().value && !state.producerNodes.empty()) { | 2219 | if (state.currentValue == sendFlag->GetFlag().value && !state.producerNodes.empty()) { |
| 2137 | - HCCL_VM_WARN("[AivTaskTransformV3] Repeated SendFlag value on same flag cell, node={}", | 2220 | + HCCL_VM_WARN("The same flag cell received the same SendFlag value again before any " |
| 2138 | - node->Describe()); | 2221 | + "new RecvFlag consumed it, node={}", node == nullptr ? "node=null" : node->Describe()); |
| 2139 | } | 2222 | } |
| 2140 | state.currentValue = sendFlag->GetFlag().value; | 2223 | state.currentValue = sendFlag->GetFlag().value; |
| 2141 | state.producerNodes.clear(); | 2224 | state.producerNodes.clear(); |
| @@ -2188,7 +2271,9 @@ HcclResult ExpandAivGraphsV3(const AivGraphsGenerateInputV3 &input, AivGraphGene | |||
| 2188 | } | 2271 | } |
| 2189 | } | 2272 | } |
| 2190 | 2273 | ||
| 2191 | - HcclResult ret = MatchSendRecvFlagEdges(input.graph, output.sendRecvEdgeCount); | 2274 | + const std::string snapshotFileHint = |
| 2275 | + input.aivGraphs.empty() ? "aiv_snapshot_unavailable" : "multiple_aiv_snapshots"; | ||
| 2276 | + HcclResult ret = MatchSendRecvFlagEdges(input.graph, snapshotFileHint, output.sendRecvEdgeCount); | ||
| 2192 | output.expandNs = ElapsedNs(start, AivExpandClock::now()); | 2277 | output.expandNs = ElapsedNs(start, AivExpandClock::now()); |
| 2193 | return ret; | 2278 | return ret; |
| 2194 | } | 2279 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_all_rank_param_recorder_v3.cc+23-16
| @@ -10,6 +10,8 @@ | |||
| 10 | 10 | ||
| 11 | 11 | ||
| 12 | 12 | ||
| 13 | + | ||
| 14 | + | ||
| 13 | 15 | ||
| 14 | namespace HcclSim { | 16 | namespace HcclSim { |
| 15 | namespace TaskGraphGeneratorV3 { | 17 | namespace TaskGraphGeneratorV3 { |
| @@ -46,22 +48,22 @@ void AllRankParamRecorder::RegisterPostNode(TaskNode *node, const CcuPostNodeMet | |||
| 46 | postNodeMeta[node] = meta; | 48 | postNodeMeta[node] = meta; |
| 47 | } | 49 | } |
| 48 | 50 | ||
| 49 | -uint32_t AllRankParamRecorder::GetPostTopicId(const TaskNode *node) const | 51 | +uint32_t AllRankParamRecorder::GetPostRemainingCkeMask(const TaskNode *node) const |
| 50 | { | 52 | { |
| 51 | auto it = postNodeMeta.find(node); | 53 | auto it = postNodeMeta.find(node); |
| 52 | if (it == postNodeMeta.end()) { | 54 | if (it == postNodeMeta.end()) { |
| 53 | return 0; | 55 | return 0; |
| 54 | } | 56 | } |
| 55 | - return it->second.topicId; | 57 | + return it->second.remainingCkeMask; |
| 56 | } | 58 | } |
| 57 | 59 | ||
| 58 | -void AllRankParamRecorder::SetPostTopicId(TaskNode *node, uint32_t topicId) | 60 | +void AllRankParamRecorder::SetPostRemainingCkeMask(TaskNode *node, uint32_t remainingCkeMask) |
| 59 | { | 61 | { |
| 60 | auto it = postNodeMeta.find(node); | 62 | auto it = postNodeMeta.find(node); |
| 61 | if (it == postNodeMeta.end()) { | 63 | if (it == postNodeMeta.end()) { |
| 62 | return; | 64 | return; |
| 63 | } | 65 | } |
| 64 | - it->second.topicId = static_cast<uint16_t>(topicId); | 66 | + it->second.remainingCkeMask = static_cast<uint16_t>(remainingCkeMask); |
| 65 | } | 67 | } |
| 66 | 68 | ||
| 67 | const CcuPostNodeMetaV3 *AllRankParamRecorder::GetPostNodeMeta(const TaskNode *node) const | 69 | const CcuPostNodeMetaV3 *AllRankParamRecorder::GetPostNodeMeta(const TaskNode *node) const |
| @@ -101,7 +103,7 @@ HcclResult AllRankParamRecorder::GetXn(uint32_t rankId, uint32_t dieId, uint16_t | |||
| 101 | } | 103 | } |
| 102 | } | 104 | } |
| 103 | } | 105 | } |
| 104 | - HCCL_ERROR("[GetXn]curXn is not be initialized, rankId[%u], dieId[%u], xnId[%hu]", rankId, dieId, xnId); | 106 | + xnValue = 0; |
| 105 | return HCCL_E_PARA; | 107 | return HCCL_E_PARA; |
| 106 | } | 108 | } |
| 107 | 109 | ||
| @@ -115,7 +117,7 @@ HcclResult AllRankParamRecorder::GetGSA(uint32_t rankId, uint32_t dieId, uint16 | |||
| 115 | } | 117 | } |
| 116 | } | 118 | } |
| 117 | } | 119 | } |
| 118 | - HCCL_ERROR("[GetGSA]curGSA is not be initialized, rankId[%u], dieId[%u], gsaId[%hu]", rankId, dieId, gsaId); | 120 | + gsaValue = 0; |
| 119 | return HCCL_E_PARA; | 121 | return HCCL_E_PARA; |
| 120 | } | 122 | } |
| 121 | 123 | ||
| @@ -184,14 +186,21 @@ HcclResult AllRankParamRecorder::CheckAllPostMatch() | |||
| 184 | for (const auto *post : regPair.second) { | 186 | for (const auto *post : regPair.second) { |
| 185 | const auto *meta = GetPostNodeMeta(post); | 187 | const auto *meta = GetPostNodeMeta(post); |
| 186 | if (meta != nullptr) { | 188 | if (meta != nullptr) { |
| 187 | - HCCL_WARNING("unmatched LocalPost/Post: node={:p}, waitRank={}, dieId={}, ckeId={}, " | 189 | + HCCL_VM_WARN("{} Found CCU post/local-post tasks that were never consumed by " |
| 188 | - "recordRank={}, topicId=0x{:x}, isLocal={}", static_cast<const void *>(post), | 190 | + "any Wait task, firstUnconsumedPostNode={}, unconsumedPostCount={}, waitRankId={}, " |
| 189 | - meta->waitRankId, meta->dieId, meta->ckeId, meta->recordRankId, meta->topicId, | 191 | + "dieId={}, " |
| 190 | - meta->isLocal); | 192 | + "ckeId={}, recordRankId={}, remainingCkeMask=0x{:x}, isLocal={}", |
| 193 | + MakeErrorCodeText(ErrorCode::GRAPH_UNMATCHED).c_str(), | ||
| 194 | + post == nullptr ? "node=null" : post->Describe().c_str(), regPair.second.size(), | ||
| 195 | + meta->waitRankId, meta->dieId, meta->ckeId, | ||
| 196 | + meta->recordRankId, meta->remainingCkeMask, meta->isLocal); | ||
| 191 | continue; | 197 | continue; |
| 192 | } | 198 | } |
| 193 | - HCCL_WARNING("unmatched LocalPost/Post: node={:p}, waitRank={}, dieId={}, ckeId={}", | 199 | + HCCL_VM_WARN("{} Found CCU post/local-post tasks that were never consumed by any " |
| 194 | - static_cast<const void *>(post), rank, dieId, regId); | 200 | + "Wait task, firstUnconsumedPostNode={}, unconsumedPostCount={}, waitRankId={}, dieId={}, " |
| 201 | + "ckeId={}", MakeErrorCodeText(ErrorCode::GRAPH_UNMATCHED).c_str(), | ||
| 202 | + post == nullptr ? "node=null" : post->Describe().c_str(), regPair.second.size(), rank, | ||
| 203 | + dieId, regId); | ||
| 195 | } | 204 | } |
| 196 | } | 205 | } |
| 197 | } | 206 | } |
| @@ -202,9 +211,7 @@ HcclResult AllRankParamRecorder::CheckAllPostMatch() | |||
| 202 | 211 | ||
| 203 | HcclResult AllRankParamRecorder::SetHBM(uint32_t rankId, uint32_t dieId, uint64_t hbmAddr, | 212 | HcclResult AllRankParamRecorder::SetHBM(uint32_t rankId, uint32_t dieId, uint64_t hbmAddr, |
| 204 | const std::vector<uint64_t>& data) { | 213 | const std::vector<uint64_t>& data) { |
| 205 | - // 合法性校验 | 214 | + if (data.size() % 8 != 0) { |
| 206 | - if (data.size() % 8 != 0){ | ||
| 207 | - HCCL_ERROR("hbm data size is not 8 bytes aligned"); | ||
| 208 | return HCCL_E_PARA; | 215 | return HCCL_E_PARA; |
| 209 | } | 216 | } |
| 210 | curHBM[rankId][dieId][hbmAddr] = data; | 217 | curHBM[rankId][dieId][hbmAddr] = data; |
| @@ -220,7 +227,7 @@ HcclResult AllRankParamRecorder::GetHBM(uint32_t rankId, uint32_t dieId, uint64_ | |||
| 220 | } | 227 | } |
| 221 | } | 228 | } |
| 222 | } | 229 | } |
| 223 | - HCCL_ERROR("[GetGSA]curGSA is not be initialized, rankId[%u], dieId[%u], hbmAddr[%llu]", rankId, dieId, hbmAddr); | 230 | + data.clear(); |
| 224 | return HCCL_E_PARA; | 231 | return HCCL_E_PARA; |
| 225 | } | 232 | } |
| 226 | 233 | ||
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_all_rank_param_recorder_v3.h+3-3
| @@ -28,7 +28,7 @@ struct CcuPostNodeMetaV3 { | |||
| 28 | RankId waitRankId{INVALID_RANK_ID}; | 28 | RankId waitRankId{INVALID_RANK_ID}; |
| 29 | uint32_t dieId{INVALID_DIE_ID}; | 29 | uint32_t dieId{INVALID_DIE_ID}; |
| 30 | uint16_t ckeId{INVALID_CCU_CKE}; | 30 | uint16_t ckeId{INVALID_CCU_CKE}; |
| 31 | - uint16_t topicId{0}; | 31 | + uint16_t remainingCkeMask{0}; |
| 32 | bool isLocal{false}; | 32 | bool isLocal{false}; |
| 33 | }; | 33 | }; |
| 34 | 34 | ||
| @@ -39,8 +39,8 @@ public: | |||
| 39 | void Reset(); | 39 | void Reset(); |
| 40 | HcclResult CheckAllPostMatch(); | 40 | HcclResult CheckAllPostMatch(); |
| 41 | void RegisterPostNode(TaskNode *node, const CcuPostNodeMetaV3 &meta); | 41 | void RegisterPostNode(TaskNode *node, const CcuPostNodeMetaV3 &meta); |
| 42 | - uint32_t GetPostTopicId(const TaskNode *node) const; | 42 | + uint32_t GetPostRemainingCkeMask(const TaskNode *node) const; |
| 43 | - void SetPostTopicId(TaskNode *node, uint32_t topicId); | 43 | + void SetPostRemainingCkeMask(TaskNode *node, uint32_t remainingCkeMask); |
| 44 | const CcuPostNodeMetaV3 *GetPostNodeMeta(const TaskNode *node) const; | 44 | const CcuPostNodeMetaV3 *GetPostNodeMeta(const TaskNode *node) const; |
| 45 | 45 | ||
| 46 | HcclResult SetXn(uint32_t rankId, uint32_t dieId, uint16_t xnId, uint64_t xnValue); | 46 | HcclResult SetXn(uint32_t rankId, uint32_t dieId, uint16_t xnId, uint64_t xnValue); |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_loop_merge_v3.cc+83-20
| @@ -17,6 +17,7 @@ | |||
| 17 | 17 | ||
| 18 | 18 | ||
| 19 | 19 | ||
| 20 | + | ||
| 20 | 21 | ||
| 21 | namespace HcclSim { | 22 | namespace HcclSim { |
| 22 | namespace TaskGraphGeneratorV3 { | 23 | namespace TaskGraphGeneratorV3 { |
| @@ -256,7 +257,8 @@ bool CcuLoopInstrV3::MergeMemSlices(std::vector<MemSlice> &slices, bool allowOve | |||
| 256 | const uint64_t lastEnd = last.offset + last.len; | 257 | const uint64_t lastEnd = last.offset + last.len; |
| 257 | const uint64_t curEnd = cur.offset + cur.len; | 258 | const uint64_t curEnd = cur.offset + cur.len; |
| 258 | if (lastEnd > cur.offset && !allowOverlap) { | 259 | if (lastEnd > cur.offset && !allowOverlap) { |
| 259 | - HCCL_VM_INFO("[ccu_loop_merge_V3] MemSlice overlap between loop expands, instrId={}", instrId); | 260 | + HCCL_VM_WARN("Skip merged-loop optimization because memory slices from different " |
| 261 | + "loop expansions overlap, loopInstrId={}", instrId); | ||
| 260 | return false; | 262 | return false; |
| 261 | } | 263 | } |
| 262 | // 相邻或重叠的切片合并为一个,取并集区间 | 264 | // 相邻或重叠的切片合并为一个,取并集区间 |
| @@ -478,14 +480,15 @@ bool CcuLoopV3::CheckResourceConflict() const | |||
| 478 | for (const auto &loopInstrs : loopExpands) { | 480 | for (const auto &loopInstrs : loopExpands) { |
| 479 | for (uint16_t ckeId : loopInstrs.GetUsedCKE()) { | 481 | for (uint16_t ckeId : loopInstrs.GetUsedCKE()) { |
| 480 | if (!allCKEs.insert(ckeId).second) { | 482 | if (!allCKEs.insert(ckeId).second) { |
| 481 | - HCCL_VM_INFO("[ccu_loop_merge_V3] loop expand cke conflict, loopInstrId={}, ckeId={}", instrId, | 483 | + HCCL_VM_WARN("Skip merged-loop optimization because different loop expansions reuse " |
| 482 | - ckeId); | 484 | + "the same CKE resource, loopInstrId={}, ckeId={}", instrId, ckeId); |
| 483 | return false; | 485 | return false; |
| 484 | } | 486 | } |
| 485 | } | 487 | } |
| 486 | for (uint16_t msId : loopInstrs.GetUsedMS()) { | 488 | for (uint16_t msId : loopInstrs.GetUsedMS()) { |
| 487 | if (!allMSs.insert(msId).second) { | 489 | if (!allMSs.insert(msId).second) { |
| 488 | - HCCL_VM_INFO("[ccu_loop_merge_V3] loop expand ms conflict, loopInstrId={}, msId={}", instrId, msId); | 490 | + HCCL_VM_WARN("Skip merged-loop optimization because different loop expansions reuse " |
| 491 | + "the same MS resource, loopInstrId={}, msId={}", instrId, msId); | ||
| 489 | return false; | 492 | return false; |
| 490 | } | 493 | } |
| 491 | } | 494 | } |
| @@ -497,7 +500,8 @@ bool CcuLoopV3::LoopIterationMerge() | |||
| 497 | { | 500 | { |
| 498 | for (auto &loopInstrs : loopExpands) { | 501 | for (auto &loopInstrs : loopExpands) { |
| 499 | if (!loopInstrs.MemMerge(true)) { | 502 | if (!loopInstrs.MemMerge(true)) { |
| 500 | - HCCL_VM_INFO("[ccu_loop_merge_V3] loop iteration merge failed, loopInstrId={}", instrId); | 503 | + HCCL_VM_WARN("Skip merged-loop optimization because memory slices inside one loop " |
| 504 | + "iteration cannot be merged, loopInstrId={}", instrId); | ||
| 501 | return false; | 505 | return false; |
| 502 | } | 506 | } |
| 503 | } | 507 | } |
| @@ -514,7 +518,8 @@ bool CcuLoopV3::LoopExpandMerge() | |||
| 514 | const size_t instrCnt = loopExpands.front().instrs.size(); | 518 | const size_t instrCnt = loopExpands.front().instrs.size(); |
| 515 | for (const auto &expand : loopExpands) { | 519 | for (const auto &expand : loopExpands) { |
| 516 | if (expand.instrs.size() != instrCnt) { | 520 | if (expand.instrs.size() != instrCnt) { |
| 517 | - HCCL_VM_INFO("[ccu_loop_merge_V3] loop expand instr count mismatch, loopInstrId={}", instrId); | 521 | + HCCL_VM_WARN("Skip merged-loop optimization because loop expansions contain different " |
| 522 | + "numbers of instructions, loopInstrId={}", instrId); | ||
| 518 | return false; | 523 | return false; |
| 519 | } | 524 | } |
| 520 | } | 525 | } |
| @@ -534,14 +539,15 @@ bool CcuLoopV3::LoopExpandMerge() | |||
| 534 | } | 539 | } |
| 535 | auto mergedInstr = loopExpands.front().instrs[instrIdx]->InstrMerge(parallelInstrs); | 540 | auto mergedInstr = loopExpands.front().instrs[instrIdx]->InstrMerge(parallelInstrs); |
| 536 | if (mergedInstr == nullptr) { | 541 | if (mergedInstr == nullptr) { |
| 537 | - HCCL_VM_INFO("[ccu_loop_merge_V3] loop expand instr merge failed, loopInstrId={}, instrIdx={}", instrId, | 542 | + HCCL_VM_WARN("Skip merged-loop optimization because one instruction position cannot be " |
| 538 | - instrIdx); | 543 | + "merged across loop expansions, loopInstrId={}, instructionIndex={}", instrId, instrIdx); |
| 539 | return false; | 544 | return false; |
| 540 | } | 545 | } |
| 541 | merged.instrs.push_back(std::move(mergedInstr)); | 546 | merged.instrs.push_back(std::move(mergedInstr)); |
| 542 | } | 547 | } |
| 543 | if (!merged.MemMerge(false)) { | 548 | if (!merged.MemMerge(false)) { |
| 544 | - HCCL_VM_INFO("[ccu_loop_merge_V3] loop expand mem merge failed, loopInstrId={}", instrId); | 549 | + HCCL_VM_WARN("Skip merged-loop optimization because merged instruction memory ranges still " |
| 550 | + "overlap, loopInstrId={}", instrId); | ||
| 545 | return false; | 551 | return false; |
| 546 | } | 552 | } |
| 547 | 553 | ||
| @@ -555,14 +561,26 @@ HcclResult EmitMergedLoopInstrsV3(CcuGraphStateV3 *curCcuTask, uint32_t queId, | |||
| 555 | const CcuLoopInstrsV3 &mergedInstrs, bool &isContinue) | 561 | const CcuLoopInstrsV3 &mergedInstrs, bool &isContinue) |
| 556 | { | 562 | { |
| 557 | if (curCcuTask == nullptr) { | 563 | if (curCcuTask == nullptr) { |
| 564 | + HCCL_VM_ERROR("{} Failed to emit merged loop instructions because loop graph state is null, " | ||
| 565 | + "queueId={}", MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), queId); | ||
| 558 | return HCCL_E_PTR; | 566 | return HCCL_E_PTR; |
| 559 | } | 567 | } |
| 560 | const RankId rankId = curCcuTask->GetRankId(); | 568 | const RankId rankId = curCcuTask->GetRankId(); |
| 561 | for (const auto &instr : mergedInstrs.instrs) { | 569 | for (const auto &instr : mergedInstrs.instrs) { |
| 562 | if (instr == nullptr) { | 570 | if (instr == nullptr) { |
| 571 | + HCCL_VM_ERROR("{} Failed to emit one merged loop instruction because the merged instruction " | ||
| 572 | + "entry is null, rankId={}, queueId={}", MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), | ||
| 573 | + static_cast<uint32_t>(rankId), queId); | ||
| 563 | return HCCL_E_PTR; | 574 | return HCCL_E_PTR; |
| 564 | } | 575 | } |
| 565 | - CHK_RET(ProcessWaitOps(curCcuTask, queId, instr->waitOps, isContinue)); | 576 | + HcclResult ret = ProcessWaitOps(curCcuTask, queId, instr->waitOps, isContinue); |
| 577 | + if (ret != HCCL_SUCCESS) { | ||
| 578 | + HCCL_VM_ERROR("{} Failed to process wait conditions before emitting one merged loop " | ||
| 579 | + "instruction, rankId={}, queueId={}, mergedLoopInstr={}", | ||
| 580 | + MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), static_cast<uint32_t>(rankId), queId, | ||
| 581 | + instr->Describe()); | ||
| 582 | + return ret; | ||
| 583 | + } | ||
| 566 | if (!isContinue) { | 584 | if (!isContinue) { |
| 567 | return HCCL_SUCCESS; | 585 | return HCCL_SUCCESS; |
| 568 | } | 586 | } |
| @@ -570,27 +588,72 @@ HcclResult EmitMergedLoopInstrsV3(CcuGraphStateV3 *curCcuTask, uint32_t queId, | |||
| 570 | if (auto trans = std::dynamic_pointer_cast<CcuLoopTransMemV3>(instr)) { | 588 | if (auto trans = std::dynamic_pointer_cast<CcuLoopTransMemV3>(instr)) { |
| 571 | CcuNodeRoleV3 role = CcuNodeRoleV3::UNKNOWN; | 589 | CcuNodeRoleV3 role = CcuNodeRoleV3::UNKNOWN; |
| 572 | RankId peerRank = INVALID_RANK_ID; | 590 | RankId peerRank = INVALID_RANK_ID; |
| 573 | - CHK_RET(ResolveTransRole(rankId, trans->srcs, trans->dsts, role, peerRank)); | 591 | + ret = ResolveTransRole(rankId, trans->srcs, trans->dsts, role, peerRank); |
| 574 | - CHK_RET(AddBatchTransMem(rankId, queId, curCcuTask, trans->srcs, trans->dsts, trans->mergedSrcs, | 592 | + if (ret != HCCL_SUCCESS) { |
| 575 | - trans->mergedDsts, role, peerRank)); | 593 | + HCCL_VM_ERROR("{} Failed to determine the transfer direction for one merged loop " |
| 594 | + "instruction, rankId={}, queueId={}, mergedLoopInstr={}", | ||
| 595 | + MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), | ||
| 596 | + static_cast<uint32_t>(rankId), queId, trans->Describe()); | ||
| 597 | + return ret; | ||
| 598 | + } | ||
| 599 | + ret = AddBatchTransMem(rankId, queId, curCcuTask, trans->srcs, trans->dsts, trans->mergedSrcs, | ||
| 600 | + trans->mergedDsts, role, peerRank); | ||
| 601 | + if (ret != HCCL_SUCCESS) { | ||
| 602 | + HCCL_VM_ERROR("{} Failed to emit one merged loop transfer task, rankId={}, queueId={}, " | ||
| 603 | + "mergedLoopInstr={}", MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), | ||
| 604 | + static_cast<uint32_t>(rankId), queId, trans->Describe()); | ||
| 605 | + return ret; | ||
| 606 | + } | ||
| 576 | } else if (auto reduce = std::dynamic_pointer_cast<CcuLoopReduceV3>(instr)) { | 607 | } else if (auto reduce = std::dynamic_pointer_cast<CcuLoopReduceV3>(instr)) { |
| 577 | CcuNodeRoleV3 role = CcuNodeRoleV3::UNKNOWN; | 608 | CcuNodeRoleV3 role = CcuNodeRoleV3::UNKNOWN; |
| 578 | RankId peerRank = INVALID_RANK_ID; | 609 | RankId peerRank = INVALID_RANK_ID; |
| 579 | - CHK_RET(ResolveReduceRole(rankId, reduce->srcs, reduce->dsts, role, peerRank)); | 610 | + ret = ResolveReduceRole(rankId, reduce->srcs, reduce->dsts, role, peerRank); |
| 580 | - CHK_RET(AddBatchReduce(rankId, queId, curCcuTask, reduce->srcs, reduce->dsts, reduce->mergedSrcs, | 611 | + if (ret != HCCL_SUCCESS) { |
| 581 | - reduce->mergedDsts, reduce->dataType, reduce->reduceOp, role, peerRank)); | 612 | + HCCL_VM_ERROR("{} Failed to determine the reduce direction for one merged loop " |
| 613 | + "instruction, rankId={}, queueId={}, mergedLoopInstr={}", | ||
| 614 | + MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), | ||
| 615 | + static_cast<uint32_t>(rankId), queId, reduce->Describe()); | ||
| 616 | + return ret; | ||
| 617 | + } | ||
| 618 | + ret = AddBatchReduce(rankId, queId, curCcuTask, reduce->srcs, reduce->dsts, reduce->mergedSrcs, | ||
| 619 | + reduce->mergedDsts, reduce->dataType, reduce->reduceOp, role, peerRank); | ||
| 620 | + if (ret != HCCL_SUCCESS) { | ||
| 621 | + HCCL_VM_ERROR("{} Failed to emit one merged loop reduce task, rankId={}, queueId={}, " | ||
| 622 | + "mergedLoopInstr={}", MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), | ||
| 623 | + static_cast<uint32_t>(rankId), queId, reduce->Describe()); | ||
| 624 | + return ret; | ||
| 625 | + } | ||
| 582 | } else if (auto clear = std::dynamic_pointer_cast<CcuLoopClearCkeV3>(instr)) { | 626 | } else if (auto clear = std::dynamic_pointer_cast<CcuLoopClearCkeV3>(instr)) { |
| 583 | if (clear->clearMask != 0) { | 627 | if (clear->clearMask != 0) { |
| 584 | - HCCL_VM_INFO("[ccu_loop_merge_V3] clear cke mask is not supported in merged loop, instrId={}", | 628 | + HCCL_VM_WARN("{} Skip merged-loop optimization because ClearCKE uses a non-zero " |
| 585 | - clear->instrId); | 629 | + "mask, instrId={}, clearMask=0x{:x}", MakeErrorCodeText(ErrorCode::GRAPH_UNSUPPORTED), |
| 630 | + clear->instrId, clear->clearMask); | ||
| 586 | return HCCL_E_NOT_SUPPORT; | 631 | return HCCL_E_NOT_SUPPORT; |
| 587 | } | 632 | } |
| 588 | } else { | 633 | } else { |
| 634 | + HCCL_VM_ERROR("{} Failed to emit one merged loop instruction because its merged instruction " | ||
| 635 | + "type is not supported, rankId={}, queueId={}, mergedLoopInstr={}", | ||
| 636 | + MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), static_cast<uint32_t>(rankId), queId, | ||
| 637 | + instr->Describe()); | ||
| 589 | return HCCL_E_NOT_SUPPORT; | 638 | return HCCL_E_NOT_SUPPORT; |
| 590 | } | 639 | } |
| 591 | 640 | ||
| 592 | - CHK_RET(ProcessSetOps(curCcuTask, queId, instr->setOps)); | 641 | + ret = ProcessSetOps(curCcuTask, queId, instr->setOps); |
| 593 | - CHK_RET(ClearWaitOps(curCcuTask, queId, instr->waitOps)); | 642 | + if (ret != HCCL_SUCCESS) { |
| 643 | + HCCL_VM_ERROR("{} Failed to apply set masks after emitting one merged loop instruction, " | ||
| 644 | + "rankId={}, queueId={}, mergedLoopInstr={}", | ||
| 645 | + MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), static_cast<uint32_t>(rankId), queId, | ||
| 646 | + instr->Describe()); | ||
| 647 | + return ret; | ||
| 648 | + } | ||
| 649 | + ret = ClearWaitOps(curCcuTask, queId, instr->waitOps); | ||
| 650 | + if (ret != HCCL_SUCCESS) { | ||
| 651 | + HCCL_VM_ERROR("{} Failed to clear consumed wait masks after emitting one merged loop " | ||
| 652 | + "instruction, rankId={}, queueId={}, mergedLoopInstr={}", | ||
| 653 | + MakeErrorCodeText(ErrorCode::GRAPH_LOOP_MERGE_ERROR).c_str(), static_cast<uint32_t>(rankId), queId, | ||
| 654 | + instr->Describe()); | ||
| 655 | + return ret; | ||
| 656 | + } | ||
| 594 | } | 657 | } |
| 595 | return HCCL_SUCCESS; | 658 | return HCCL_SUCCESS; |
| 596 | } | 659 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_task_common_v3.h+6-5
| @@ -59,7 +59,7 @@ struct CcuNodeMetaV3 { | |||
| 59 | RankId peerRank{INVALID_RANK_ID}; | 59 | RankId peerRank{INVALID_RANK_ID}; |
| 60 | uint32_t dieId{INVALID_DIE_ID}; | 60 | uint32_t dieId{INVALID_DIE_ID}; |
| 61 | uint16_t ckeId{INVALID_CCU_CKE}; | 61 | uint16_t ckeId{INVALID_CCU_CKE}; |
| 62 | - uint16_t topicId{0}; | 62 | + uint16_t remainingCkeMask{0}; |
| 63 | bool invalidPost{false}; | 63 | bool invalidPost{false}; |
| 64 | }; | 64 | }; |
| 65 | 65 | ||
| @@ -98,20 +98,21 @@ public: | |||
| 98 | HcclResult InitInstrInfo(StorageManager &storage); | 98 | HcclResult InitInstrInfo(StorageManager &storage); |
| 99 | HcclResult CreateHeadNode(); | 99 | HcclResult CreateHeadNode(); |
| 100 | HcclResult AppendGeneratedNode(std::unique_ptr<TaskNode> node, RankId rankId, uint32_t queId, | 100 | HcclResult AppendGeneratedNode(std::unique_ptr<TaskNode> node, RankId rankId, uint32_t queId, |
| 101 | - CcuNodeRoleV3 role, TaskNode *&outNode, RankId peerRank = INVALID_RANK_ID, uint16_t topicId = 0, | 101 | + CcuNodeRoleV3 role, TaskNode *&outNode, RankId peerRank = INVALID_RANK_ID, uint16_t remainingCkeMask = 0, |
| 102 | uint32_t dieId = INVALID_DIE_ID, uint16_t ckeId = INVALID_CCU_CKE, bool invalidPost = false); | 102 | uint32_t dieId = INVALID_DIE_ID, uint16_t ckeId = INVALID_CCU_CKE, bool invalidPost = false); |
| 103 | HcclResult AppendGeneratedNode(std::unique_ptr<TaskNode> node, const TaskPosition &position, | 103 | HcclResult AppendGeneratedNode(std::unique_ptr<TaskNode> node, const TaskPosition &position, |
| 104 | - CcuNodeRoleV3 role, TaskNode *&outNode, RankId peerRank = INVALID_RANK_ID, uint16_t topicId = 0, | 104 | + CcuNodeRoleV3 role, TaskNode *&outNode, RankId peerRank = INVALID_RANK_ID, uint16_t remainingCkeMask = 0, |
| 105 | uint32_t dieId = INVALID_DIE_ID, uint16_t ckeId = INVALID_CCU_CKE, bool invalidPost = false); | 105 | uint32_t dieId = INVALID_DIE_ID, uint16_t ckeId = INVALID_CCU_CKE, bool invalidPost = false); |
| 106 | 106 | ||
| 107 | void SetNodeMeta(TaskNode *node, const CcuNodeMetaV3 &meta); | 107 | void SetNodeMeta(TaskNode *node, const CcuNodeMetaV3 &meta); |
| 108 | const CcuNodeMetaV3 *GetNodeMeta(const TaskNode *node) const; | 108 | const CcuNodeMetaV3 *GetNodeMeta(const TaskNode *node) const; |
| 109 | CcuNodeRoleV3 GetNodeRole(const TaskNode *node) const; | 109 | CcuNodeRoleV3 GetNodeRole(const TaskNode *node) const; |
| 110 | RankId GetNodePeerRank(const TaskNode *node) const; | 110 | RankId GetNodePeerRank(const TaskNode *node) const; |
| 111 | - uint16_t GetNodeTopicId(const TaskNode *node) const; | 111 | + uint16_t GetNodeRemainingCkeMask(const TaskNode *node) const; |
| 112 | - void SetNodeTopicId(TaskNode *node, uint16_t topicId); | 112 | + void SetNodeRemainingCkeMask(TaskNode *node, uint16_t remainingCkeMask); |
| 113 | void SetNodePeerRank(TaskNode *node, RankId peerRank); | 113 | void SetNodePeerRank(TaskNode *node, RankId peerRank); |
| 114 | std::string Describe() const; | 114 | std::string Describe() const; |
| 115 | + std::string DescribeNextInstructionByQueue() const; | ||
| 115 | 116 | ||
| 116 | void GetSqe(uint32_t queId, uint16_t sqeArgsId, uint64_t &argVal); | 117 | void GetSqe(uint32_t queId, uint16_t sqeArgsId, uint64_t &argVal); |
| 117 | void GetDieId(uint32_t queId, uint32_t &dieId) const; | 118 | void GetDieId(uint32_t queId, uint32_t &dieId) const; |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_task_transform_instruct_a5_v3.cc+373-182
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_task_transform_v3.cc+95-35
| @@ -25,22 +25,23 @@ | |||
| 25 | 25 | ||
| 26 | 26 | ||
| 27 | 27 | ||
| 28 | + | ||
| 28 | 29 | ||
| 29 | namespace HcclSim { | 30 | namespace HcclSim { |
| 30 | namespace TaskGraphGeneratorV3 { | 31 | namespace TaskGraphGeneratorV3 { |
| 31 | 32 | ||
| 32 | std::unique_ptr<InstructMapBase> g_instrMap; | 33 | std::unique_ptr<InstructMapBase> g_instrMap; |
| 33 | 34 | ||
| 34 | -// 作用:读取 post 节点当前挂着的 topicId。 | 35 | +// 作用:读取 post 节点当前还未被 wait 消费掉的 CKE mask。 |
| 35 | -u32 GetTopicId(TaskNode *post) | 36 | +u32 GetPostRemainingCkeMask(TaskNode *post) |
| 36 | { | 37 | { |
| 37 | - return AllRankParamRecorder::Global()->GetPostTopicId(post); | 38 | + return AllRankParamRecorder::Global()->GetPostRemainingCkeMask(post); |
| 38 | } | 39 | } |
| 39 | 40 | ||
| 40 | -// 作用:更新 post 节点剩余未消费的 topicId。 | 41 | +// 作用:更新 post 节点当前还未被 wait 消费掉的 CKE mask。 |
| 41 | -void SetTopicId(TaskNode *post, u32 topicId) | 42 | +void SetPostRemainingCkeMask(TaskNode *post, u32 remainingCkeMask) |
| 42 | { | 43 | { |
| 43 | - AllRankParamRecorder::Global()->SetPostTopicId(post, topicId); | 44 | + AllRankParamRecorder::Global()->SetPostRemainingCkeMask(post, remainingCkeMask); |
| 44 | } | 45 | } |
| 45 | 46 | ||
| 46 | // 作用:按 wait mask 生成本地/远端 wait 节点,并补齐 post 到 wait 的依赖边。 | 47 | // 作用:按 wait mask 生成本地/远端 wait 节点,并补齐 post 到 wait 的依赖边。 |
| @@ -55,13 +56,13 @@ void GenWaitNode(CcuGraphStateV3 *curCcuTask, uint32_t queId, uint16_t waitCKEId | |||
| 55 | TaskNode *localWaitNode = nullptr; | 56 | TaskNode *localWaitNode = nullptr; |
| 56 | TaskNode *remoteWaitNode = nullptr; | 57 | TaskNode *remoteWaitNode = nullptr; |
| 57 | 58 | ||
| 58 | - // 只有目标 CKE bit 全部就绪后才真正生成 wait 节点;此时会把待消费的 topic bit | 59 | + // 只有目标 CKE bit 全部就绪后才真正生成 wait 节点;此时会把待消费的 CKE mask bit |
| 59 | // 反查回它实际依赖的那些 post 节点。 | 60 | // 反查回它实际依赖的那些 post 节点。 |
| 60 | std::set<TaskNode *> &seenPosts = AllRankParamRecorder::Global()->seenPost[rankId][dieId][waitCKEId]; | 61 | std::set<TaskNode *> &seenPosts = AllRankParamRecorder::Global()->seenPost[rankId][dieId][waitCKEId]; |
| 61 | std::vector<TaskNode *> localPosts; | 62 | std::vector<TaskNode *> localPosts; |
| 62 | for (auto *post : seenPosts) { | 63 | for (auto *post : seenPosts) { |
| 63 | - u32 topicId = GetTopicId(post); | 64 | + u32 remainingCkeMask = GetPostRemainingCkeMask(post); |
| 64 | - uint16_t curPostMask = topicId & waitCKEMask; | 65 | + uint16_t curPostMask = remainingCkeMask & waitCKEMask; |
| 65 | if (curPostMask == 0) { | 66 | if (curPostMask == 0) { |
| 66 | continue; | 67 | continue; |
| 67 | } | 68 | } |
| @@ -88,8 +89,8 @@ void GenWaitNode(CcuGraphStateV3 *curCcuTask, uint32_t queId, uint16_t waitCKEId | |||
| 88 | 89 | ||
| 89 | for (auto it = seenPosts.begin(); it != seenPosts.end();) { | 90 | for (auto it = seenPosts.begin(); it != seenPosts.end();) { |
| 90 | auto *post = *it; | 91 | auto *post = *it; |
| 91 | - u32 topicId = GetTopicId(post); | 92 | + u32 remainingCkeMask = GetPostRemainingCkeMask(post); |
| 92 | - uint16_t curPostMask = topicId & waitCKEMask; | 93 | + uint16_t curPostMask = remainingCkeMask & waitCKEMask; |
| 93 | if (curPostMask == 0) { | 94 | if (curPostMask == 0) { |
| 94 | ++it; | 95 | ++it; |
| 95 | continue; | 96 | continue; |
| @@ -106,9 +107,9 @@ void GenWaitNode(CcuGraphStateV3 *curCcuTask, uint32_t queId, uint16_t waitCKEId | |||
| 106 | } | 107 | } |
| 107 | (void)CollectBilateralWaitInfo(curCcuTask, queId, remoteWaitNode); | 108 | (void)CollectBilateralWaitInfo(curCcuTask, queId, remoteWaitNode); |
| 108 | } | 109 | } |
| 109 | - topicId = topicId & (~waitCKEMask); | 110 | + remainingCkeMask = remainingCkeMask & (~waitCKEMask); |
| 110 | - SetTopicId(post, topicId); | 111 | + SetPostRemainingCkeMask(post, remainingCkeMask); |
| 111 | - if (topicId == 0) { | 112 | + if (remainingCkeMask == 0) { |
| 112 | it = seenPosts.erase(it); | 113 | it = seenPosts.erase(it); |
| 113 | } else { | 114 | } else { |
| 114 | ++it; | 115 | ++it; |
| @@ -120,16 +121,17 @@ void GenWaitNode(CcuGraphStateV3 *curCcuTask, uint32_t queId, uint16_t waitCKEId | |||
| 120 | HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, CcuGraphStateV3 *curCcuTask, uint32_t queId, | 121 | HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, CcuGraphStateV3 *curCcuTask, uint32_t queId, |
| 121 | uint16_t waitCKEId, uint16_t waitCKEMask, bool &isContinue) | 122 | uint16_t waitCKEId, uint16_t waitCKEMask, bool &isContinue) |
| 122 | { | 123 | { |
| 123 | - HCCL_VM_DEBUG("[ProcessWaitMask] Enter...rank {:d}, die{:d}, que{:d}, waitCke{:d}, waitMask{:d}", | 124 | + HCCL_VM_DEBUG("Evaluate CCU wait condition, rankId={}, dieId={}, queueId={}, waitCkeId={}, " |
| 124 | - rankId, static_cast<uint32_t>(dieId), queId, waitCKEId, waitCKEMask); | 125 | + "requiredMask=0x{:04x}", rankId, static_cast<uint32_t>(dieId), queId, waitCKEId, waitCKEMask); |
| 125 | if (waitCKEMask != 0x0000) { | 126 | if (waitCKEMask != 0x0000) { |
| 126 | uint16_t ckeValue = 0; | 127 | uint16_t ckeValue = 0; |
| 127 | CHK_RET(AllRankParamRecorder::Global()->GetCKE(rankId, dieId, waitCKEId, ckeValue)); | 128 | CHK_RET(AllRankParamRecorder::Global()->GetCKE(rankId, dieId, waitCKEId, ckeValue)); |
| 128 | if ((ckeValue & waitCKEMask) != waitCKEMask) { | 129 | if ((ckeValue & waitCKEMask) != waitCKEMask) { |
| 129 | // 当前 queue 在这里暂停,直到生产侧把所需的 wait bit 全部 post 完成。 | 130 | // 当前 queue 在这里暂停,直到生产侧把所需的 wait bit 全部 post 完成。 |
| 130 | isContinue = false; | 131 | isContinue = false; |
| 131 | - HCCL_VM_DEBUG("[ProcessWaitMask] Does not meet the condition, continue: ckeId: {:d}, " | 132 | + HCCL_VM_DEBUG("Pause this CCU queue because the wait condition is not satisfied yet, " |
| 132 | - "ckeValue: {:d}, mask: {:x}", waitCKEId, ckeValue, waitCKEMask); | 133 | + "rankId={}, dieId={}, queueId={}, waitCkeId={}, currentCkeValue=0x{:04x}, requiredMask=0x{:04x}", |
| 134 | + rankId, static_cast<uint32_t>(dieId), queId, waitCKEId, ckeValue, waitCKEMask); | ||
| 133 | return HCCL_SUCCESS; | 135 | return HCCL_SUCCESS; |
| 134 | } | 136 | } |
| 135 | GenWaitNode(curCcuTask, queId, waitCKEId, waitCKEMask); | 137 | GenWaitNode(curCcuTask, queId, waitCKEId, waitCKEMask); |
| @@ -142,11 +144,17 @@ HcclResult TransformInstr(const CcuRep::CcuInstr *instr, CcuGraphStateV3 *curCcu | |||
| 142 | bool &isContinue) | 144 | bool &isContinue) |
| 143 | { | 145 | { |
| 144 | if (g_instrMap == nullptr) { | 146 | if (g_instrMap == nullptr) { |
| 145 | - HCCL_ERROR("g_instrMap error"); | 147 | + HCCL_VM_ERROR("{} Failed to create the CCU instruction translator, rankId={}, queueId={}", |
| 148 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR).c_str(), | ||
| 149 | + curCcuTask == nullptr ? std::string("null") : std::to_string(curCcuTask->GetRankId()), queId); | ||
| 146 | return HCCL_E_INTERNAL; | 150 | return HCCL_E_INTERNAL; |
| 147 | } | 151 | } |
| 148 | if (!g_instrMap->IsSupported(instr->header.header)) { | 152 | if (!g_instrMap->IsSupported(instr->header.header)) { |
| 149 | - HCCL_ERROR("ins type not supported %hu", instr->header.header); | 153 | + HCCL_VM_ERROR("{} This CCU instruction type is not supported by CheckerV3 graph expansion, " |
| 154 | + "rankId={}, queueId={}, instructionHeader=0x{:04x}", | ||
| 155 | + MakeErrorCodeText(ErrorCode::GRAPH_UNSUPPORTED).c_str(), | ||
| 156 | + curCcuTask == nullptr ? std::string("null") : std::to_string(curCcuTask->GetRankId()), queId, | ||
| 157 | + instr->header.header); | ||
| 150 | return HCCL_E_INTERNAL; | 158 | return HCCL_E_INTERNAL; |
| 151 | } | 159 | } |
| 152 | return g_instrMap.get()->Transform(instr, curCcuTask, queId, isContinue, nullptr); | 160 | return g_instrMap.get()->Transform(instr, curCcuTask, queId, isContinue, nullptr); |
| @@ -177,9 +185,19 @@ HcclResult TransformInstrQue(TaskNode *node, CcuGraphStateV3 *curCcuTask, uint32 | |||
| 177 | // 单个 queue 严格按指令顺序推进;跨 queue 的协同只通过生成出来的 wait/post 边表达。 | 185 | // 单个 queue 严格按指令顺序推进;跨 queue 的协同只通过生成出来的 wait/post 边表达。 |
| 178 | while (pos < endInstrId) { | 186 | while (pos < endInstrId) { |
| 179 | u32 prePos = pos; | 187 | u32 prePos = pos; |
| 180 | - HCCL_VM_DEBUG("Current process ccu graph: {}, instruction id={}, header={}", | 188 | + const uint32_t instrId = curCcuTask->startInstrIdInQue[queId] + pos; |
| 181 | - curCcuTask->Describe(), pos, microCodeQue.instrVec[pos].header.header); | 189 | + HCCL_VM_DEBUG("Translating one CCU instruction, graph={}, instructionId={}, " |
| 182 | - CHK_RET(TransformInstr(µCodeQue.instrVec[pos], curCcuTask, queId, isContinue)); | 190 | + "instructionHeader=0x{:04x}", |
| 191 | + curCcuTask->Describe(), instrId, microCodeQue.instrVec[pos].header.header); | ||
| 192 | + HcclResult ret = TransformInstr(µCodeQue.instrVec[pos], curCcuTask, queId, isContinue); | ||
| 193 | + if (ret != HCCL_SUCCESS) { | ||
| 194 | + HCCL_VM_ERROR("{} Failed to translate one CCU instruction, rankId={}, queueId={}, " | ||
| 195 | + "instructionId={}, instructionHeader=0x{:04x}, ret={}, graph={}", | ||
| 196 | + MakeErrorCodeText(ErrorCode::GRAPH_TRANSLATE_FAILED).c_str(), curCcuTask->GetRankId(), queId, | ||
| 197 | + instrId, microCodeQue.instrVec[pos].header.header, static_cast<uint32_t>(ret), | ||
| 198 | + curCcuTask->Describe().c_str()); | ||
| 199 | + return ret; | ||
| 200 | + } | ||
| 183 | if (!isContinue) { | 201 | if (!isContinue) { |
| 184 | return HCCL_SUCCESS; | 202 | return HCCL_SUCCESS; |
| 185 | } | 203 | } |
| @@ -192,7 +210,8 @@ HcclResult TransformInstrQue(TaskNode *node, CcuGraphStateV3 *curCcuTask, uint32 | |||
| 192 | curCcuTask->isGenGraphed = std::all_of(curCcuTask->instrQueStatus.begin(), | 210 | curCcuTask->isGenGraphed = std::all_of(curCcuTask->instrQueStatus.begin(), |
| 193 | curCcuTask->instrQueStatus.end(), [](bool b) { return b; }); | 211 | curCcuTask->instrQueStatus.end(), [](bool b) { return b; }); |
| 194 | 212 | ||
| 195 | - HCCL_VM_DEBUG("[TransformInstrQue] end..."); | 213 | + HCCL_VM_DEBUG("Finished translating one CCU queue, rankId={}, queueId={}", |
| 214 | + curCcuTask->GetRankId(), queId); | ||
| 196 | return HCCL_SUCCESS; | 215 | return HCCL_SUCCESS; |
| 197 | } | 216 | } |
| 198 | 217 | ||
| @@ -566,7 +585,8 @@ HcclResult BuildLoopParallelGroupV3(const std::vector<LoopInfoV3> &loopGroup, | |||
| 566 | group.beforeNodes = GetLoopBeforeNodesV3(loopGroup.front().loopStart, originalQueueId, compressedBoundary); | 585 | group.beforeNodes = GetLoopBeforeNodesV3(loopGroup.front().loopStart, originalQueueId, compressedBoundary); |
| 567 | group.afterNodes = GetSameQueueChildrenV3(loopGroup.back().loopEnd, originalQueueId); | 586 | group.afterNodes = GetSameQueueChildrenV3(loopGroup.back().loopEnd, originalQueueId); |
| 568 | if (group.beforeNodes.empty() || group.afterNodes.empty()) { | 587 | if (group.beforeNodes.empty() || group.afterNodes.empty()) { |
| 569 | - HCCL_ERROR("Loop parallel group boundary is invalid."); | 588 | + HCCL_VM_ERROR("{} Failed to split one CCU loop-parallel group because its entry or exit " |
| 589 | + "boundary could not be determined.", MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID).c_str()); | ||
| 570 | return HCCL_E_INTERNAL; | 590 | return HCCL_E_INTERNAL; |
| 571 | } | 591 | } |
| 572 | 592 | ||
| @@ -1078,7 +1098,11 @@ HcclResult GenerateRawCcuGraphStateV3(TaskNode *node, CcuGraphStateV3 &state) | |||
| 1078 | ++unmatchedCnt; | 1098 | ++unmatchedCnt; |
| 1079 | } | 1099 | } |
| 1080 | if (unmatchedCnt > state.queueNum_) { | 1100 | if (unmatchedCnt > state.queueNum_) { |
| 1081 | - HCCL_ERROR("deadLocking occurs due to mismatch."); | 1101 | + HCCL_VM_ERROR("{} CCU raw graph generation is stuck inside one graph. All queues are " |
| 1102 | + "blocked and no instruction can advance, rankId={}, queueCount={}, nextInstructionByQueue={}, " | ||
| 1103 | + "blockedGraphSummary={}", | ||
| 1104 | + MakeErrorCodeText(ErrorCode::GRAPH_DEADLOCK).c_str(), state.GetRankId(), state.queueNum_, | ||
| 1105 | + state.DescribeNextInstructionByQueue().c_str(), state.Describe().c_str()); | ||
| 1082 | return HCCL_E_INTERNAL; | 1106 | return HCCL_E_INTERNAL; |
| 1083 | } | 1107 | } |
| 1084 | } | 1108 | } |
| @@ -1111,18 +1135,16 @@ void PrintGeneratedCcuGraphsV3(const std::vector<std::unique_ptr<CcuGraphRuntime | |||
| 1111 | return lhs->ccuGraph->GetNodeId() < rhs->ccuGraph->GetNodeId(); | 1135 | return lhs->ccuGraph->GetNodeId() < rhs->ccuGraph->GetNodeId(); |
| 1112 | }); | 1136 | }); |
| 1113 | 1137 | ||
| 1114 | - HCCL_VM_INFO("======================================================="); | 1138 | + HCCL_VM_INFO("Printing all generated CCU graphs before queue parallelization, graphCount={}", |
| 1115 | - HCCL_VM_INFO("[PrintCcuGraphV3] All generated CCU graphs before queue parallelization, graphCount={}", | ||
| 1116 | orderedRuntimes.size()); | 1139 | orderedRuntimes.size()); |
| 1117 | for (const CcuGraphRuntimeV3 *runtime : orderedRuntimes) { | 1140 | for (const CcuGraphRuntimeV3 *runtime : orderedRuntimes) { |
| 1118 | if (runtime == nullptr || runtime->state == nullptr) { | 1141 | if (runtime == nullptr || runtime->state == nullptr) { |
| 1119 | continue; | 1142 | continue; |
| 1120 | } | 1143 | } |
| 1121 | - HCCL_VM_INFO("[PrintCcuGraphV3] ccuGraphNodeId={}, rankId={}", runtime->ccuGraph->GetNodeId(), | 1144 | + HCCL_VM_INFO("CCU graph detail, ccuGraphNodeId={}, rankId={}", runtime->ccuGraph->GetNodeId(), |
| 1122 | runtime->ccuGraph->GetRankId()); | 1145 | runtime->ccuGraph->GetRankId()); |
| 1123 | PrintCcuGraph(runtime->state->ccuHeadTaskNode); | 1146 | PrintCcuGraph(runtime->state->ccuHeadTaskNode); |
| 1124 | } | 1147 | } |
| 1125 | - HCCL_VM_INFO("======================================================="); | ||
| 1126 | } | 1148 | } |
| 1127 | 1149 | ||
| 1128 | // 作用:把已经展开完成的内部 CCU 子图重新接回主图中的占位节点位置。 | 1150 | // 作用:把已经展开完成的内部 CCU 子图重新接回主图中的占位节点位置。 |
| @@ -1205,7 +1227,24 @@ HcclResult GenerateRawCcuGraphsV3(const CcuGraphsGenerateInputV3 &input, | |||
| 1205 | size_t generatedCount = 0; | 1227 | size_t generatedCount = 0; |
| 1206 | while (!candNode.empty()) { | 1228 | while (!candNode.empty()) { |
| 1207 | if (unmatchedCnt >= candNode.size()) { | 1229 | if (unmatchedCnt >= candNode.size()) { |
| 1208 | - HCCL_ERROR("deadLocking occurs due to mismatch."); | 1230 | + TaskNode *firstBlockedGraph = candNode.front(); |
| 1231 | + const CcuGraphRuntimeV3 *blockedRuntime = nullptr; | ||
| 1232 | + auto runtimeIter = runtimeByNodeId.find(firstBlockedGraph == nullptr ? | ||
| 1233 | + INVALID_NODE_ID : firstBlockedGraph->GetNodeId()); | ||
| 1234 | + if (runtimeIter != runtimeByNodeId.end()) { | ||
| 1235 | + blockedRuntime = runtimeIter->second; | ||
| 1236 | + } | ||
| 1237 | + HCCL_VM_ERROR("{} CCU multi-graph raw generation is stuck. Some CCU graphs are still " | ||
| 1238 | + "blocked by dependencies, but no graph can advance, blockedGraphCount={}, rankId={}, queueCount={}, " | ||
| 1239 | + "nextInstructionByQueue={}, firstBlockedGraph={}", | ||
| 1240 | + MakeErrorCodeText(ErrorCode::GRAPH_DEADLOCK).c_str(), candNode.size(), | ||
| 1241 | + blockedRuntime == nullptr || blockedRuntime->state == nullptr ? std::string("null") : | ||
| 1242 | + std::to_string(blockedRuntime->state->GetRankId()), | ||
| 1243 | + blockedRuntime == nullptr || blockedRuntime->state == nullptr ? 0U : | ||
| 1244 | + blockedRuntime->state->queueNum_, | ||
| 1245 | + blockedRuntime == nullptr || blockedRuntime->state == nullptr ? "[]" : | ||
| 1246 | + blockedRuntime->state->DescribeNextInstructionByQueue().c_str(), | ||
| 1247 | + firstBlockedGraph == nullptr ? "node=null" : firstBlockedGraph->Describe().c_str()); | ||
| 1209 | return HCCL_E_INTERNAL; | 1248 | return HCCL_E_INTERNAL; |
| 1210 | } | 1249 | } |
| 1211 | 1250 | ||
| @@ -1273,8 +1312,19 @@ HcclResult GenerateRawCcuGraphsV3(const CcuGraphsGenerateInputV3 &input, | |||
| 1273 | } | 1312 | } |
| 1274 | 1313 | ||
| 1275 | if (generatedCount != input.ccuGraphs.size()) { | 1314 | if (generatedCount != input.ccuGraphs.size()) { |
| 1276 | - HCCL_ERROR("CCU graph global expansion finished with unfinished graph, generated=%zu, total=%zu", | 1315 | + const CcuGraphRuntimeV3 *firstUnfinishedRuntime = nullptr; |
| 1277 | - generatedCount, input.ccuGraphs.size()); | 1316 | + for (const auto &runtime : runtimes) { |
| 1317 | + if (runtime != nullptr && !runtime->generated) { | ||
| 1318 | + firstUnfinishedRuntime = runtime.get(); | ||
| 1319 | + break; | ||
| 1320 | + } | ||
| 1321 | + } | ||
| 1322 | + HCCL_VM_ERROR("{} CCU raw graph generation ended before all CCU graphs were finished, " | ||
| 1323 | + "generatedGraphCount={}, totalGraphCount={}, firstUnfinishedGraph={}", | ||
| 1324 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID).c_str(), | ||
| 1325 | + generatedCount, input.ccuGraphs.size(), firstUnfinishedRuntime == nullptr || | ||
| 1326 | + firstUnfinishedRuntime->ccuGraph == nullptr ? "node=null" : | ||
| 1327 | + firstUnfinishedRuntime->ccuGraph->Describe().c_str()); | ||
| 1278 | return HCCL_E_INTERNAL; | 1328 | return HCCL_E_INTERNAL; |
| 1279 | } | 1329 | } |
| 1280 | return HCCL_SUCCESS; | 1330 | return HCCL_SUCCESS; |
| @@ -1293,8 +1343,18 @@ HcclResult FinalizeGeneratedCcuGraphsV3(TaskGraphGeneratorV3 &graph, | |||
| 1293 | ++finalizedCount; | 1343 | ++finalizedCount; |
| 1294 | } | 1344 | } |
| 1295 | if (finalizedCount != runtimes.size()) { | 1345 | if (finalizedCount != runtimes.size()) { |
| 1296 | - HCCL_ERROR("CCU graph global finalization finished with unfinished graph, finalized=%zu, total=%zu", | 1346 | + const CcuGraphRuntimeV3 *firstUnfinishedRuntime = nullptr; |
| 1297 | - finalizedCount, runtimes.size()); | 1347 | + for (const auto &runtime : runtimes) { |
| 1348 | + if (runtime != nullptr && !runtime->finalized) { | ||
| 1349 | + firstUnfinishedRuntime = runtime.get(); | ||
| 1350 | + break; | ||
| 1351 | + } | ||
| 1352 | + } | ||
| 1353 | + HCCL_VM_ERROR("{} CCU graph finalization ended before all generated graphs were attached back " | ||
| 1354 | + "to the main graph, finalizedGraphCount={}, totalGraphCount={}, firstUnfinishedGraph={}", | ||
| 1355 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID).c_str(), finalizedCount, runtimes.size(), | ||
| 1356 | + firstUnfinishedRuntime == nullptr || firstUnfinishedRuntime->ccuGraph == nullptr ? "node=null" : | ||
| 1357 | + firstUnfinishedRuntime->ccuGraph->Describe().c_str()); | ||
| 1298 | return HCCL_E_INTERNAL; | 1358 | return HCCL_E_INTERNAL; |
| 1299 | } | 1359 | } |
| 1300 | return HCCL_SUCCESS; | 1360 | return HCCL_SUCCESS; |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/ccu_graph_generator_v3/ccu_task_transform_v3.h+2-2
| @@ -56,9 +56,9 @@ HcclResult ExpandCcuGraphsV3(const CcuGraphsGenerateInputV3 &input, CcuGraphGene | |||
| 56 | 56 | ||
| 57 | HcclResult GetHcclDataTypeFromCCUDataType(uint16_t ccuDataType, uint16_t ccuReduceType, DataType& dataType); | 57 | HcclResult GetHcclDataTypeFromCCUDataType(uint16_t ccuDataType, uint16_t ccuReduceType, DataType& dataType); |
| 58 | 58 | ||
| 59 | -uint32_t GetTopicId(TaskNode* post); | 59 | +uint32_t GetPostRemainingCkeMask(TaskNode* post); |
| 60 | 60 | ||
| 61 | -void SetTopicId(TaskNode* post, u32 topicId); | 61 | +void SetPostRemainingCkeMask(TaskNode* post, u32 remainingCkeMask); |
| 62 | 62 | ||
| 63 | HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, CcuGraphStateV3 *curCcuTask, uint32_t queId, | 63 | HcclResult ProcessWaitMask(RankId rankId, uint32_t dieId, CcuGraphStateV3 *curCcuTask, uint32_t queId, |
| 64 | uint16_t waitCKEId, uint16_t waitCKEMask, bool& isContinue); | 64 | uint16_t waitCKEId, uint16_t waitCKEMask, bool& isContinue); |
| @@ -55,6 +55,26 @@ enum class MemType : uint8_t { | |||
| 55 | INVALID = UINT8_MAX, | 55 | INVALID = UINT8_MAX, |
| 56 | }; | 56 | }; |
| 57 | 57 | ||
| 58 | +inline const char *DescribeMemType(MemType memType) | ||
| 59 | +{ | ||
| 60 | + switch (memType) { | ||
| 61 | + case MemType::INPUT: | ||
| 62 | + return "INPUT"; | ||
| 63 | + case MemType::OUTPUT: | ||
| 64 | + return "OUTPUT"; | ||
| 65 | + case MemType::CCL: | ||
| 66 | + return "CCL"; | ||
| 67 | + case MemType::UB_AIV: | ||
| 68 | + return "UB_AIV"; | ||
| 69 | + case MemType::FLAG_AIV: | ||
| 70 | + return "FLAG_AIV"; | ||
| 71 | + case MemType::MS_CCU: | ||
| 72 | + return "MS_CCU"; | ||
| 73 | + default: | ||
| 74 | + return "invalid"; | ||
| 75 | + } | ||
| 76 | +} | ||
| 77 | + | ||
| 58 | enum class BoundaryType : uint8_t { | 78 | enum class BoundaryType : uint8_t { |
| 59 | MAIN_GRAPH = 0, | 79 | MAIN_GRAPH = 0, |
| 60 | CCU_SUB_GRAPH, | 80 | CCU_SUB_GRAPH, |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_graph_generator_v3.cc+62-84
| @@ -20,6 +20,7 @@ | |||
| 20 | 20 | ||
| 21 | 21 | ||
| 22 | 22 | ||
| 23 | + | ||
| 23 | 24 | ||
| 24 | namespace HcclSim { | 25 | namespace HcclSim { |
| 25 | namespace TaskGraphGeneratorV3 { | 26 | namespace TaskGraphGeneratorV3 { |
| @@ -27,46 +28,6 @@ namespace { | |||
| 27 | using SeenLocalRecords = std::vector<NodeId>; | 28 | using SeenLocalRecords = std::vector<NodeId>; |
| 28 | using SeenInterRankRecords = std::map<RankId, std::map<RankId, std::vector<NodeId>>>; | 29 | using SeenInterRankRecords = std::map<RankId, std::map<RankId, std::vector<NodeId>>>; |
| 29 | 30 | ||
| 30 | -const char *TaskTypeName(TaskType type) | ||
| 31 | -{ | ||
| 32 | - switch (type) { | ||
| 33 | - case TaskType::TRANS_MEM: | ||
| 34 | - return "TRANS_MEM"; | ||
| 35 | - case TaskType::BATCH_TRANS_MEM: | ||
| 36 | - return "BATCH_TRANS_MEM"; | ||
| 37 | - case TaskType::REDUCE: | ||
| 38 | - return "REDUCE"; | ||
| 39 | - case TaskType::BATCH_REDUCE: | ||
| 40 | - return "BATCH_REDUCE"; | ||
| 41 | - case TaskType::RECORD: | ||
| 42 | - return "RECORD"; | ||
| 43 | - case TaskType::WAIT: | ||
| 44 | - return "WAIT"; | ||
| 45 | - case TaskType::CCU_GRAPH: | ||
| 46 | - return "CCU_GRAPH"; | ||
| 47 | - case TaskType::AIV_GRAPH: | ||
| 48 | - return "AIV_GRAPH"; | ||
| 49 | - case TaskType::AIV_SET_FLAG: | ||
| 50 | - return "AIV_SET_FLAG"; | ||
| 51 | - case TaskType::AIV_WAIT_FLAG: | ||
| 52 | - return "AIV_WAIT_FLAG"; | ||
| 53 | - case TaskType::AIV_PIPE_BARRIER: | ||
| 54 | - return "AIV_PIPE_BARRIER"; | ||
| 55 | - case TaskType::AIV_SYNC_ALL: | ||
| 56 | - return "AIV_SYNC_ALL"; | ||
| 57 | - case TaskType::AIV_SEND_FLAG: | ||
| 58 | - return "AIV_SEND_FLAG"; | ||
| 59 | - case TaskType::AIV_RECV_FLAG: | ||
| 60 | - return "AIV_RECV_FLAG"; | ||
| 61 | - case TaskType::START: | ||
| 62 | - return "START"; | ||
| 63 | - case TaskType::END: | ||
| 64 | - return "END"; | ||
| 65 | - default: | ||
| 66 | - return "INVALID"; | ||
| 67 | - } | ||
| 68 | -} | ||
| 69 | - | ||
| 70 | std::string NodeIdsToString(const std::vector<TaskNode *> &nodes) | 31 | std::string NodeIdsToString(const std::vector<TaskNode *> &nodes) |
| 71 | { | 32 | { |
| 72 | std::ostringstream os; | 33 | std::ostringstream os; |
| @@ -182,8 +143,11 @@ HcclResult TaskGraphGeneratorV3::RemoveEdge(NodeId parentNodeId, NodeId childNod | |||
| 182 | TaskNode *parentNode = GetNode(parentNodeId); | 143 | TaskNode *parentNode = GetNode(parentNodeId); |
| 183 | TaskNode *childNode = GetNode(childNodeId); | 144 | TaskNode *childNode = GetNode(childNodeId); |
| 184 | if (parentNode == nullptr || childNode == nullptr) { | 145 | if (parentNode == nullptr || childNode == nullptr) { |
| 185 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][RemoveEdge] Invalid edge, parent={}, child={}", | 146 | + HCCL_VM_ERROR("{} Failed to remove one graph edge because the parent or child node does not " |
| 186 | - parentNodeId, childNodeId); | 147 | + "exist, parentNodeId={}, childNodeId={}, parentNode={}, childNode={}", |
| 148 | + MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), parentNodeId, childNodeId, | ||
| 149 | + parentNode == nullptr ? "null" : parentNode->Describe(), | ||
| 150 | + childNode == nullptr ? "null" : childNode->Describe()); | ||
| 187 | return HCCL_E_PARA; | 151 | return HCCL_E_PARA; |
| 188 | } | 152 | } |
| 189 | 153 | ||
| @@ -403,8 +367,8 @@ HcclResult TaskGraphGeneratorV3::GenGraph(std::vector<std::unique_ptr<TaskNode>> | |||
| 403 | AllRankNodeQueues translatedTaskQueues) | 367 | AllRankNodeQueues translatedTaskQueues) |
| 404 | { | 368 | { |
| 405 | Reset(); | 369 | Reset(); |
| 406 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][GenGraph] Start gen graph from translated task nodes, nodeCount={}, " | 370 | + HCCL_VM_INFO("Start building the CheckerV3 graph from translated nodes, rankCount={}, nodeCount={}", |
| 407 | - "rankCount={}", translatedNodes.size(), translatedTaskQueues.size()); | 371 | + translatedTaskQueues.size(), translatedNodes.size()); |
| 408 | nodes_ = std::move(translatedNodes); | 372 | nodes_ = std::move(translatedNodes); |
| 409 | taskQueues_ = std::move(translatedTaskQueues); | 373 | taskQueues_ = std::move(translatedTaskQueues); |
| 410 | hasAiv_ = std::any_of(nodes_.begin(), nodes_.end(), [](const std::unique_ptr<TaskNode> &node) { | 374 | hasAiv_ = std::any_of(nodes_.begin(), nodes_.end(), [](const std::unique_ptr<TaskNode> &node) { |
| @@ -435,8 +399,8 @@ HcclResult TaskGraphGeneratorV3::CreateMainStartNode() | |||
| 435 | 399 | ||
| 436 | HcclResult TaskGraphGeneratorV3::BuildDagEdges() | 400 | HcclResult TaskGraphGeneratorV3::BuildDagEdges() |
| 437 | { | 401 | { |
| 438 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][BuildDagEdges] Start DAG edge build, nodeCount={}, rankCount={}, " | 402 | + HCCL_VM_INFO("Start building graph edges, rankCount={}, nodeCount={}, mainStartNodeId={}", |
| 439 | - "mainStartNodeId={}", nodes_.size(), taskQueues_.size(), mainStartNodeId_); | 403 | + taskQueues_.size(), nodes_.size(), mainStartNodeId_); |
| 440 | 404 | ||
| 441 | HcclResult ret = HCCL_SUCCESS; | 405 | HcclResult ret = HCCL_SUCCESS; |
| 442 | for (const auto &rankEntry : taskQueues_) { | 406 | for (const auto &rankEntry : taskQueues_) { |
| @@ -472,8 +436,8 @@ HcclResult TaskGraphGeneratorV3::BuildDagEdges() | |||
| 472 | } | 436 | } |
| 473 | } | 437 | } |
| 474 | 438 | ||
| 475 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][BuildDagEdges] DAG edge build success, nodeCount={}, edgeCount={}, " | 439 | + HCCL_VM_INFO("Finished building graph edges, nodeCount={}, edgeCount={}, mainStartNodeId={}", |
| 476 | - "mainStartNodeId={}", nodes_.size(), CountEdges(), mainStartNodeId_); | 440 | + nodes_.size(), CountEdges(), mainStartNodeId_); |
| 477 | return HCCL_SUCCESS; | 441 | return HCCL_SUCCESS; |
| 478 | } | 442 | } |
| 479 | 443 | ||
| @@ -492,8 +456,10 @@ HcclResult TaskGraphGeneratorV3::ExpandAivSubGraphs() | |||
| 492 | 456 | ||
| 493 | auto *aivGraph = dynamic_cast<TaskAivGraph *>(node); | 457 | auto *aivGraph = dynamic_cast<TaskAivGraph *>(node); |
| 494 | if (aivGraph == nullptr) { | 458 | if (aivGraph == nullptr) { |
| 495 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][ExpandAivSubGraphs] AIV node cast failed, nodeId={}", | 459 | + HCCL_VM_ERROR("{} One node expected to be an AIV subgraph entry is actually another node " |
| 496 | - node->GetNodeId()); | 460 | + "type, nodeId={}, node={}", MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), |
| 461 | + node == nullptr ? std::string("null") : std::to_string(node->GetNodeId()), | ||
| 462 | + node == nullptr ? "node=null" : node->Describe()); | ||
| 497 | return HCCL_E_INTERNAL; | 463 | return HCCL_E_INTERNAL; |
| 498 | } | 464 | } |
| 499 | aivGraphs.push_back(aivGraph); | 465 | aivGraphs.push_back(aivGraph); |
| @@ -506,8 +472,9 @@ HcclResult TaskGraphGeneratorV3::ExpandAivSubGraphs() | |||
| 506 | AivGraphGenerateOutputV3 result; | 472 | AivGraphGenerateOutputV3 result; |
| 507 | HcclResult ret = ExpandAivGraphsV3(input, result); | 473 | HcclResult ret = ExpandAivGraphsV3(input, result); |
| 508 | if (ret != HCCL_SUCCESS) { | 474 | if (ret != HCCL_SUCCESS) { |
| 509 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][ExpandAivSubGraphs] Expand AIV graphs failed, graphCount={}, ret={}", | 475 | + HCCL_VM_ERROR("{} Failed to expand AIV Graph nodes, " |
| 510 | - aivGraphs.size(), static_cast<uint32_t>(ret)); | 476 | + "aivGraphCount={}, ret={}", MakeErrorCodeText(ErrorCode::GRAPH_TRANSLATE_FAILED), aivGraphs.size(), |
| 477 | + static_cast<uint32_t>(ret)); | ||
| 511 | return ret; | 478 | return ret; |
| 512 | } | 479 | } |
| 513 | 480 | ||
| @@ -527,12 +494,12 @@ HcclResult TaskGraphGeneratorV3::ExpandAivSubGraphs() | |||
| 527 | stats.cpGmInactiveNodeCount = result.cpGmInactiveNodeCount; | 494 | stats.cpGmInactiveNodeCount = result.cpGmInactiveNodeCount; |
| 528 | stats.totalExpandNs = result.expandNs; | 495 | stats.totalExpandNs = result.expandNs; |
| 529 | aivExpandStats_ = stats; | 496 | aivExpandStats_ = stats; |
| 530 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][ExpandAivSubGraphs] AIV subgraph expansion finished, aivGraphCount={}, " | 497 | + HCCL_VM_INFO("Finished expanding AIV Graph nodes, aivGraphCount={}, internalNodeCount={}, " |
| 531 | - "internalNodeCount={}, setWaitEdgeCount={}, pipeBarrierMergeCount={}, syncAllMergeCount={}, " | 498 | + "setWaitEdgeCount={}, pipeBarrierMergeCount={}, syncAllMergeCount={}, sendRecvEdgeCount={}, " |
| 532 | - "sendRecvEdgeCount={}, taskJsonTotalTaskCount={}, dagNodeCountBeforeCpGmMerge={}, " | 499 | + "taskJsonTotalTaskCount={}, dagNodeCountBeforeCpGmMerge={}, dagNodeCountAfterCpGmMerge={}, " |
| 533 | - "dagNodeCountAfterCpGmMerge={}, cpGmLoopMergeCount={}, cpGmMergedIterationCount={}, " | 500 | + "cpGmLoopMergeCount={}, cpGmMergedIterationCount={}, cpGmMergedOriginalNodeCount={}, " |
| 534 | - "cpGmMergedOriginalNodeCount={}, cpGmGeneratedNodeCount={}, cpGmInactiveNodeCount={}, " | 501 | + "cpGmGeneratedNodeCount={}, cpGmInactiveNodeCount={}, ubBufferSize={}, flagBufferSize={}, " |
| 535 | - "ubBufferSize={}, flagBufferSize={}, expandTotalMs={}", stats.graphCount, stats.internalNodeCount, | 502 | + "expandTotalMs={}", stats.graphCount, stats.internalNodeCount, |
| 536 | stats.setWaitEdgeCount, stats.pipeBarrierMergeCount, stats.syncAllMergeCount, stats.sendRecvEdgeCount, | 503 | stats.setWaitEdgeCount, stats.pipeBarrierMergeCount, stats.syncAllMergeCount, stats.sendRecvEdgeCount, |
| 537 | stats.taskJsonTotalTaskCount, stats.dagNodeCountBeforeCpGmMerge, stats.dagNodeCountAfterCpGmMerge, | 504 | stats.taskJsonTotalTaskCount, stats.dagNodeCountBeforeCpGmMerge, stats.dagNodeCountAfterCpGmMerge, |
| 538 | stats.cpGmLoopMergeCount, stats.cpGmMergedIterationCount, stats.cpGmMergedOriginalNodeCount, | 505 | stats.cpGmLoopMergeCount, stats.cpGmMergedIterationCount, stats.cpGmMergedOriginalNodeCount, |
| @@ -556,8 +523,10 @@ HcclResult TaskGraphGeneratorV3::ExpandCcuSubGraphs() | |||
| 556 | 523 | ||
| 557 | auto *ccuGraph = dynamic_cast<TaskCcuGraph *>(node); | 524 | auto *ccuGraph = dynamic_cast<TaskCcuGraph *>(node); |
| 558 | if (ccuGraph == nullptr) { | 525 | if (ccuGraph == nullptr) { |
| 559 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][ExpandCcuSubGraphs] CCU node cast failed, nodeId={}", | 526 | + HCCL_VM_ERROR("{} One node expected to be a CCU subgraph entry is actually another node " |
| 560 | - node->GetNodeId()); | 527 | + "type, nodeId={}, node={}", MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), |
| 528 | + node == nullptr ? std::string("null") : std::to_string(node->GetNodeId()), | ||
| 529 | + node == nullptr ? "node=null" : node->Describe()); | ||
| 561 | return HCCL_E_INTERNAL; | 530 | return HCCL_E_INTERNAL; |
| 562 | } | 531 | } |
| 563 | 532 | ||
| @@ -571,8 +540,9 @@ HcclResult TaskGraphGeneratorV3::ExpandCcuSubGraphs() | |||
| 571 | CcuGraphGenerateOutputV3 result; | 540 | CcuGraphGenerateOutputV3 result; |
| 572 | HcclResult ret = ExpandCcuGraphsV3(input, result); | 541 | HcclResult ret = ExpandCcuGraphsV3(input, result); |
| 573 | if (ret != HCCL_SUCCESS) { | 542 | if (ret != HCCL_SUCCESS) { |
| 574 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][ExpandCcuSubGraphs] Expand CCU graphs failed, graphCount={}, ret={}", | 543 | + HCCL_VM_ERROR("{} Failed to expand CCU Graph nodes, " |
| 575 | - ccuGraphs.size(), static_cast<uint32_t>(ret)); | 544 | + "ccuGraphCount={}, ret={}", MakeErrorCodeText(ErrorCode::GRAPH_TRANSLATE_FAILED), ccuGraphs.size(), |
| 545 | + static_cast<uint32_t>(ret)); | ||
| 576 | return ret; | 546 | return ret; |
| 577 | } | 547 | } |
| 578 | stats.graphCount = ccuGraphs.size(); | 548 | stats.graphCount = ccuGraphs.size(); |
| @@ -581,8 +551,8 @@ HcclResult TaskGraphGeneratorV3::ExpandCcuSubGraphs() | |||
| 581 | stats.totalExpandNs = result.expandNs; | 551 | stats.totalExpandNs = result.expandNs; |
| 582 | 552 | ||
| 583 | ccuExpandStats_ = stats; | 553 | ccuExpandStats_ = stats; |
| 584 | - HCCL_VM_INFO("[TaskGraphGeneratorV3][ExpandCcuSubGraphs] CCU subgraph expansion finished, ccuGraphCount={}, " | 554 | + HCCL_VM_INFO("Finished expanding CCU Graph nodes, ccuGraphCount={}, internalNodeCount={}, " |
| 585 | - "internalNodeCount={}, recordWaitEdgeCount={}, expandTotalMs={}", stats.graphCount, | 555 | + "recordWaitEdgeCount={}, expandTotalMs={}", stats.graphCount, |
| 586 | stats.internalNodeCount, stats.recordWaitEdgeCount, stats.totalExpandNs / 1000000ULL); | 556 | stats.internalNodeCount, stats.recordWaitEdgeCount, stats.totalExpandNs / 1000000ULL); |
| 587 | return HCCL_SUCCESS; | 557 | return HCCL_SUCCESS; |
| 588 | } | 558 | } |
| @@ -648,8 +618,8 @@ HcclResult TaskGraphGeneratorV3::GenGraph4Rank(RankId rankId, const RankNodeQueu | |||
| 648 | ++streamOrderEdgeCount; | 618 | ++streamOrderEdgeCount; |
| 649 | } | 619 | } |
| 650 | } | 620 | } |
| 651 | - HCCL_VM_DEBUG("[TaskGraphGeneratorV3][RankSkeleton] Rank {}, streamCount={}, taskNodeCount={}, " | 621 | + HCCL_VM_DEBUG("Built per-rank skeleton edges, rankId={}, nonEmptyStreamCount={}, taskNodeCount={}, " |
| 652 | - "startEdges={}, streamOrderEdges={}", rankId, nonEmptyStreamCount, taskNodeCount, startEdgeCount, | 622 | + "startEdgeCount={}, streamOrderEdgeCount={}", rankId, nonEmptyStreamCount, taskNodeCount, startEdgeCount, |
| 653 | streamOrderEdgeCount); | 623 | streamOrderEdgeCount); |
| 654 | return HCCL_SUCCESS; | 624 | return HCCL_SUCCESS; |
| 655 | } | 625 | } |
| @@ -660,7 +630,6 @@ HcclResult TaskGraphGeneratorV3::AddLocalNotifyEdges(RankId rankId, const RankNo | |||
| 660 | SeenLocalRecords seenLocalRecords; | 630 | SeenLocalRecords seenLocalRecords; |
| 661 | uint64_t unmatchedCnt = 0; | 631 | uint64_t unmatchedCnt = 0; |
| 662 | size_t matchedEdgeCount = 0; | 632 | size_t matchedEdgeCount = 0; |
| 663 | - size_t requeueCount = 0; | ||
| 664 | 633 | ||
| 665 | if (!rankTaskQueues.empty() && !rankTaskQueues[0].empty()) { | 634 | if (!rankTaskQueues.empty() && !rankTaskQueues[0].empty()) { |
| 666 | rankNodeQue.push_back(rankTaskQueues[0].front()); | 635 | rankNodeQue.push_back(rankTaskQueues[0].front()); |
| @@ -674,8 +643,11 @@ HcclResult TaskGraphGeneratorV3::AddLocalNotifyEdges(RankId rankId, const RankNo | |||
| 674 | while (!rankNodeQue.empty()) { | 643 | while (!rankNodeQue.empty()) { |
| 675 | if (unmatchedCnt >= rankNodeQue.size()) { | 644 | if (unmatchedCnt >= rankNodeQue.size()) { |
| 676 | const TaskNode *node = GetNode(rankNodeQue.front()); | 645 | const TaskNode *node = GetNode(rankNodeQue.front()); |
| 677 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][LocalNotify] Rank {} local notify deadlock, firstUnmatched={}", | 646 | + HCCL_VM_ERROR("{} Local Record/Wait matching is stuck on this rank. Some Wait tasks are " |
| 678 | - rankId, node == nullptr ? "null" : node->Describe()); | 647 | + "still blocked, but no new local Record task can unblock them, rankId={}, firstBlockedWaitNode={}, " |
| 648 | + "blockedWaitNodeCount={}", | ||
| 649 | + MakeErrorCodeText(ErrorCode::GRAPH_DEADLOCK), rankId, | ||
| 650 | + node == nullptr ? "null" : node->Describe(), rankNodeQue.size()); | ||
| 679 | return HCCL_E_INTERNAL; | 651 | return HCCL_E_INTERNAL; |
| 680 | } | 652 | } |
| 681 | 653 | ||
| @@ -723,7 +695,6 @@ HcclResult TaskGraphGeneratorV3::AddLocalNotifyEdges(RankId rankId, const RankNo | |||
| 723 | if (!matched) { | 695 | if (!matched) { |
| 724 | rankNodeQue.push_back(currNodeId); | 696 | rankNodeQue.push_back(currNodeId); |
| 725 | ++unmatchedCnt; | 697 | ++unmatchedCnt; |
| 726 | - ++requeueCount; | ||
| 727 | } | 698 | } |
| 728 | continue; | 699 | continue; |
| 729 | } | 700 | } |
| @@ -737,13 +708,16 @@ HcclResult TaskGraphGeneratorV3::AddLocalNotifyEdges(RankId rankId, const RankNo | |||
| 737 | 708 | ||
| 738 | if (!seenLocalRecords.empty()) { | 709 | if (!seenLocalRecords.empty()) { |
| 739 | const TaskNode *node = GetNode(seenLocalRecords.front()); | 710 | const TaskNode *node = GetNode(seenLocalRecords.front()); |
| 740 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][LocalNotify] Rank {} has unmatched local record, firstUnmatched={}", | 711 | + HCCL_VM_ERROR("{} Found local Record tasks that were never consumed by any local Wait task, " |
| 741 | - rankId, node == nullptr ? "null" : node->Describe()); | 712 | + "rankId={}, firstUnconsumedRecordNode={}, unconsumedRecordCount={}", |
| 713 | + MakeErrorCodeText(ErrorCode::GRAPH_UNMATCHED), rankId, | ||
| 714 | + node == nullptr ? "node=null" : node->Describe(), seenLocalRecords.size()); | ||
| 742 | return HCCL_E_INTERNAL; | 715 | return HCCL_E_INTERNAL; |
| 743 | } | 716 | } |
| 744 | 717 | ||
| 745 | - HCCL_VM_DEBUG("[TaskGraphGeneratorV3][LocalNotify] Rank {} local notify edges generated, matchedEdges={}, " | 718 | + HCCL_VM_DEBUG("Finished matching local Record/Wait edges on one rank, rankId={}, " |
| 746 | - "requeueCount={}", rankId, matchedEdgeCount, requeueCount); | 719 | + "matchedRecordWaitEdgeCount={}", |
| 720 | + rankId, matchedEdgeCount); | ||
| 747 | return HCCL_SUCCESS; | 721 | return HCCL_SUCCESS; |
| 748 | } | 722 | } |
| 749 | 723 | ||
| @@ -755,7 +729,6 @@ HcclResult TaskGraphGeneratorV3::AddInterRankNotifyEdges() | |||
| 755 | std::vector<uint8_t> traverseFlags(nodes_.size(), 0); | 729 | std::vector<uint8_t> traverseFlags(nodes_.size(), 0); |
| 756 | uint64_t unmatchedCnt = 0; | 730 | uint64_t unmatchedCnt = 0; |
| 757 | size_t matchedEdgeCount = 0; | 731 | size_t matchedEdgeCount = 0; |
| 758 | - size_t requeueCount = 0; | ||
| 759 | 732 | ||
| 760 | HcclResult ret = ExecuteNode(mainStartNodeId_, graphNodeQue, execFlags, traverseFlags); | 733 | HcclResult ret = ExecuteNode(mainStartNodeId_, graphNodeQue, execFlags, traverseFlags); |
| 761 | if (ret != HCCL_SUCCESS) { | 734 | if (ret != HCCL_SUCCESS) { |
| @@ -765,8 +738,11 @@ HcclResult TaskGraphGeneratorV3::AddInterRankNotifyEdges() | |||
| 765 | while (!graphNodeQue.empty()) { | 738 | while (!graphNodeQue.empty()) { |
| 766 | if (unmatchedCnt >= graphNodeQue.size()) { | 739 | if (unmatchedCnt >= graphNodeQue.size()) { |
| 767 | const TaskNode *node = GetNode(graphNodeQue.front()); | 740 | const TaskNode *node = GetNode(graphNodeQue.front()); |
| 768 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][InterRankNotify] Inter-rank notify deadlock, firstUnmatched={}", | 741 | + HCCL_VM_ERROR("{} Cross-rank Record/Wait matching is stuck. Some Wait tasks are still " |
| 769 | - node == nullptr ? "null" : node->Describe()); | 742 | + "blocked, but no new cross-rank Record task can unblock them, firstBlockedWaitNode={}, " |
| 743 | + "blockedWaitNodeCount={}", | ||
| 744 | + MakeErrorCodeText(ErrorCode::GRAPH_DEADLOCK), | ||
| 745 | + node == nullptr ? "node=null" : node->Describe(), graphNodeQue.size()); | ||
| 770 | return HCCL_E_INTERNAL; | 746 | return HCCL_E_INTERNAL; |
| 771 | } | 747 | } |
| 772 | 748 | ||
| @@ -780,7 +756,6 @@ HcclResult TaskGraphGeneratorV3::AddInterRankNotifyEdges() | |||
| 780 | if (!IsExecutable(currNodeId, execFlags)) { | 756 | if (!IsExecutable(currNodeId, execFlags)) { |
| 781 | graphNodeQue.push_back(currNodeId); | 757 | graphNodeQue.push_back(currNodeId); |
| 782 | ++unmatchedCnt; | 758 | ++unmatchedCnt; |
| 783 | - ++requeueCount; | ||
| 784 | continue; | 759 | continue; |
| 785 | } | 760 | } |
| 786 | 761 | ||
| @@ -829,7 +804,6 @@ HcclResult TaskGraphGeneratorV3::AddInterRankNotifyEdges() | |||
| 829 | 804 | ||
| 830 | graphNodeQue.push_back(currNodeId); | 805 | graphNodeQue.push_back(currNodeId); |
| 831 | ++unmatchedCnt; | 806 | ++unmatchedCnt; |
| 832 | - ++requeueCount; | ||
| 833 | continue; | 807 | continue; |
| 834 | } | 808 | } |
| 835 | 809 | ||
| @@ -844,16 +818,20 @@ HcclResult TaskGraphGeneratorV3::AddInterRankNotifyEdges() | |||
| 844 | for (const auto &peerEntry : rankEntry.second) { | 818 | for (const auto &peerEntry : rankEntry.second) { |
| 845 | if (!peerEntry.second.empty()) { | 819 | if (!peerEntry.second.empty()) { |
| 846 | const TaskNode *node = GetNode(peerEntry.second.front()); | 820 | const TaskNode *node = GetNode(peerEntry.second.front()); |
| 847 | - HCCL_VM_ERROR("[TaskGraphGeneratorV3][InterRankNotify] Unmatched inter-rank record, recordRank={}, " | 821 | + const AicpuNotify *recordNotify = GetRecordNotify(node); |
| 848 | - "waitRank={}, firstUnmatched={}", rankEntry.first, peerEntry.first, | 822 | + HCCL_VM_ERROR("{} Found cross-rank Record tasks that were never consumed by any matching " |
| 849 | - node == nullptr ? "null" : node->Describe()); | 823 | + "Wait task, recordRankId={}, waitRankId={}, notifyId={}, firstUnconsumedRecordNode={}, " |
| 824 | + "unconsumedRecordCount={}", MakeErrorCodeText(ErrorCode::GRAPH_UNMATCHED), rankEntry.first, | ||
| 825 | + peerEntry.first, recordNotify == nullptr ? std::string("null") : std::to_string(recordNotify->notifyId), | ||
| 826 | + node == nullptr ? "null" : node->Describe(), peerEntry.second.size()); | ||
| 850 | return HCCL_E_INTERNAL; | 827 | return HCCL_E_INTERNAL; |
| 851 | } | 828 | } |
| 852 | } | 829 | } |
| 853 | } | 830 | } |
| 854 | 831 | ||
| 855 | - HCCL_VM_DEBUG("[TaskGraphGeneratorV3][InterRankNotify] Inter-rank notify edges generated, matchedEdges={}, " | 832 | + HCCL_VM_DEBUG("Finished matching cross-rank Record/Wait edges, " |
| 856 | - "requeueCount={}", matchedEdgeCount, requeueCount); | 833 | + "matchedCrossRankRecordWaitEdgeCount={}", |
| 834 | + matchedEdgeCount); | ||
| 857 | return HCCL_SUCCESS; | 835 | return HCCL_SUCCESS; |
| 858 | } | 836 | } |
| 859 | } // namespace TaskGraphGeneratorV3 | 837 | } // namespace TaskGraphGeneratorV3 |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_graph_mem_conflict_v3.cc+46-41
| @@ -24,6 +24,7 @@ | |||
| 24 | 24 | ||
| 25 | 25 | ||
| 26 | 26 | ||
| 27 | + | ||
| 27 | 28 | ||
| 28 | namespace HcclSim { | 29 | namespace HcclSim { |
| 29 | namespace TaskGraphGeneratorV3 { | 30 | namespace TaskGraphGeneratorV3 { |
| @@ -143,26 +144,6 @@ struct ConflictCandidate { | |||
| 143 | 144 | ||
| 144 | using ConflictCandidates = std::vector<ConflictCandidate>; | 145 | using ConflictCandidates = std::vector<ConflictCandidate>; |
| 145 | 146 | ||
| 146 | -const char *MemTypeName(MemType memType) | ||
| 147 | -{ | ||
| 148 | - switch (memType) { | ||
| 149 | - case MemType::INPUT: | ||
| 150 | - return "INPUT"; | ||
| 151 | - case MemType::OUTPUT: | ||
| 152 | - return "OUTPUT"; | ||
| 153 | - case MemType::CCL: | ||
| 154 | - return "CCL"; | ||
| 155 | - case MemType::UB_AIV: | ||
| 156 | - return "UB_AIV"; | ||
| 157 | - case MemType::FLAG_AIV: | ||
| 158 | - return "FLAG_AIV"; | ||
| 159 | - case MemType::MS_CCU: | ||
| 160 | - return "MS_CCU"; | ||
| 161 | - default: | ||
| 162 | - return "INVALID"; | ||
| 163 | - } | ||
| 164 | -} | ||
| 165 | - | ||
| 166 | const char *AccessKindName(MemoryAccessKind kind) | 147 | const char *AccessKindName(MemoryAccessKind kind) |
| 167 | { | 148 | { |
| 168 | return kind == MemoryAccessKind::WRITE ? "WRITE" : "READ"; | 149 | return kind == MemoryAccessKind::WRITE ? "WRITE" : "READ"; |
| @@ -216,7 +197,7 @@ uint64_t ElapsedMemConflictMs(MemConflictClock::time_point start, MemConflictClo | |||
| 216 | 197 | ||
| 217 | void LogMemConflictStageTime(const char *stage, const char *status, MemConflictClock::time_point start) | 198 | void LogMemConflictStageTime(const char *stage, const char *status, MemConflictClock::time_point start) |
| 218 | { | 199 | { |
| 219 | - HCCL_VM_WARN("[TaskGraphMemConflictV3][StageTime] {} {}, costMs={}", stage, status, | 200 | + HCCL_VM_INFO("Stage finished, stage={}, status={}, costMs={}", stage, status, |
| 220 | ElapsedMemConflictMs(start, MemConflictClock::now())); | 201 | ElapsedMemConflictMs(start, MemConflictClock::now())); |
| 221 | } | 202 | } |
| 222 | 203 | ||
| @@ -230,18 +211,29 @@ bool IsDataMoveTaskNode(const TaskNode *node) | |||
| 230 | std::string DescribeKey(const MemoryKey &key) | 211 | std::string DescribeKey(const MemoryKey &key) |
| 231 | { | 212 | { |
| 232 | std::ostringstream os; | 213 | std::ostringstream os; |
| 233 | - os << "{rank=" << key.rankId << ", mem=" << MemTypeName(key.memType) << ", aivUbIdx=" << key.aivUbIdx << "}"; | 214 | + os << "{rankId="; |
| 215 | + if (key.rankId == INVALID_RANK_ID) { | ||
| 216 | + os << "invalid"; | ||
| 217 | + } else { | ||
| 218 | + os << key.rankId; | ||
| 219 | + } | ||
| 220 | + os << ", memoryType=" << DescribeMemType(key.memType); | ||
| 221 | + if (key.memType == MemType::UB_AIV) { | ||
| 222 | + os << ", aivBlockIndex=" << key.aivUbIdx; | ||
| 223 | + } | ||
| 224 | + os << "}"; | ||
| 234 | return os.str(); | 225 | return os.str(); |
| 235 | } | 226 | } |
| 236 | 227 | ||
| 237 | std::string DescribeAccess(const MemoryAccess &access) | 228 | std::string DescribeAccess(const MemoryAccess &access) |
| 238 | { | 229 | { |
| 239 | std::ostringstream os; | 230 | std::ostringstream os; |
| 240 | - os << "{nodeId=" << access.nodeId << ", access=" << AccessKindName(access.kind) | 231 | + os << "{taskNodeId=" << access.nodeId << ", taskAction=" |
| 241 | - << ", range=[0x" << std::hex << access.start << ",0x" << access.end << ")" << std::dec | 232 | + << (access.kind == MemoryAccessKind::WRITE ? "write" : "read") |
| 242 | - << ", key=" << DescribeKey(access.key); | 233 | + << ", memory=" << DescribeKey(access.key) |
| 234 | + << ", byteRange=[0x" << std::hex << access.start << ",0x" << access.end << ")" << std::dec; | ||
| 243 | if (access.node != nullptr) { | 235 | if (access.node != nullptr) { |
| 244 | - os << ", desc=" << access.node->Describe(); | 236 | + os << ", task=" << access.node->Describe(); |
| 245 | } | 237 | } |
| 246 | os << "}"; | 238 | os << "}"; |
| 247 | return os.str(); | 239 | return os.str(); |
| @@ -284,7 +276,8 @@ HcclResult CollectReachableTaskNodes(const TaskNode *start, std::vector<const Ta | |||
| 284 | 276 | ||
| 285 | for (const TaskNode *childNode : node->GetChildren()) { | 277 | for (const TaskNode *childNode : node->GetChildren()) { |
| 286 | if (childNode == nullptr) { | 278 | if (childNode == nullptr) { |
| 287 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Invalid null child, parent={}", node->Describe()); | 279 | + HCCL_VM_ERROR("{} Graph structure is broken because one child node is null, " |
| 280 | + "parent={}", MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), node->Describe()); | ||
| 288 | return HCCL_E_PTR; | 281 | return HCCL_E_PTR; |
| 289 | } | 282 | } |
| 290 | if (visited.count(childNode) != 0) { | 283 | if (visited.count(childNode) != 0) { |
| @@ -331,20 +324,26 @@ HcclResult AddAccess(const MemSlice &slice, MemoryAccessKind kind, const TaskNod | |||
| 331 | return HCCL_E_PTR; | 324 | return HCCL_E_PTR; |
| 332 | } | 325 | } |
| 333 | if (node->GetNodeId() < 0) { | 326 | if (node->GetNodeId() < 0) { |
| 334 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Invalid data task node id, node={}", node->Describe()); | 327 | + HCCL_VM_ERROR("{} Data task node id is invalid, nodeId={}, task={}", |
| 328 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), node->GetNodeId(), node->Describe()); | ||
| 335 | return HCCL_E_PARA; | 329 | return HCCL_E_PARA; |
| 336 | } | 330 | } |
| 337 | if (slice.len == 0) { | 331 | if (slice.len == 0) { |
| 338 | return HCCL_SUCCESS; | 332 | return HCCL_SUCCESS; |
| 339 | } | 333 | } |
| 340 | if (slice.rankId == INVALID_RANK_ID || slice.memType == MemType::INVALID) { | 334 | if (slice.rankId == INVALID_RANK_ID || slice.memType == MemType::INVALID) { |
| 341 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Invalid memory slice, node={}, rank={}, memType={}", | 335 | + HCCL_VM_ERROR("{} One memory slice is missing a valid rank or memory type, " |
| 342 | - node->Describe(), slice.rankId, static_cast<uint32_t>(slice.memType)); | 336 | + "task={}, rankId={}, memType={}, offset=0x{:x}, length=0x{:x}", |
| 337 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 338 | + slice.rankId == INVALID_RANK_ID ? "invalid" : std::to_string(slice.rankId), | ||
| 339 | + DescribeMemType(slice.memType), slice.offset, slice.len); | ||
| 343 | return HCCL_E_PARA; | 340 | return HCCL_E_PARA; |
| 344 | } | 341 | } |
| 345 | if (slice.len > std::numeric_limits<uint64_t>::max() - slice.offset) { | 342 | if (slice.len > std::numeric_limits<uint64_t>::max() - slice.offset) { |
| 346 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Memory slice overflow, node={}, offset=0x{:x}, len=0x{:x}", | 343 | + HCCL_VM_ERROR("{} One memory slice is invalid because offset + length overflowed, " |
| 347 | - node->Describe(), slice.offset, slice.len); | 344 | + "task={}, offset=0x{:x}, length=0x{:x}, endLimit=0x{:x}", |
| 345 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), slice.offset, slice.len, | ||
| 346 | + std::numeric_limits<uint64_t>::max()); | ||
| 348 | return HCCL_E_PARA; | 347 | return HCCL_E_PARA; |
| 349 | } | 348 | } |
| 350 | 349 | ||
| @@ -387,7 +386,8 @@ HcclResult BuildTaskMemoryAccesses(const TaskNode *node, TaskMemoryAccesses &acc | |||
| 387 | } | 386 | } |
| 388 | const std::vector<MemSlice> &srcs = reduce->GetSrcs(); | 387 | const std::vector<MemSlice> &srcs = reduce->GetSrcs(); |
| 389 | if (srcs.empty()) { | 388 | if (srcs.empty()) { |
| 390 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Reduce source list is empty, node={}", node->Describe()); | 389 | + HCCL_VM_ERROR("{} This reduce task has no source data, task={}", |
| 390 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe()); | ||
| 391 | return HCCL_E_PARA; | 391 | return HCCL_E_PARA; |
| 392 | } | 392 | } |
| 393 | for (size_t index = 0; index < srcs.size(); ++index) { | 393 | for (size_t index = 0; index < srcs.size(); ++index) { |
| @@ -429,13 +429,17 @@ HcclResult BuildTaskMemoryAccesses(const TaskNode *node, TaskMemoryAccesses &acc | |||
| 429 | const auto &srcGroups = reduce->GetMergedSrcs(); | 429 | const auto &srcGroups = reduce->GetMergedSrcs(); |
| 430 | const auto &dsts = reduce->GetMergedDsts(); | 430 | const auto &dsts = reduce->GetMergedDsts(); |
| 431 | if (srcGroups.empty() || srcGroups.size() != dsts.size()) { | 431 | if (srcGroups.empty() || srcGroups.size() != dsts.size()) { |
| 432 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Batch reduce group size mismatch, node={}", node->Describe()); | 432 | + HCCL_VM_ERROR("{} Batch reduce has different counts of source groups and target " |
| 433 | + "memory slices, task={}, srcGroupCount={}, dstCount={}", | ||
| 434 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), srcGroups.size(), | ||
| 435 | + dsts.size()); | ||
| 433 | return HCCL_E_PARA; | 436 | return HCCL_E_PARA; |
| 434 | } | 437 | } |
| 435 | for (size_t groupIndex = 0; groupIndex < srcGroups.size(); ++groupIndex) { | 438 | for (size_t groupIndex = 0; groupIndex < srcGroups.size(); ++groupIndex) { |
| 436 | if (srcGroups[groupIndex].empty()) { | 439 | if (srcGroups[groupIndex].empty()) { |
| 437 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Batch reduce source list is empty, node={}", | 440 | + HCCL_VM_ERROR("{} One reduce group has no source data at all, task={}, " |
| 438 | - node->Describe()); | 441 | + "groupIndex={}", MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), |
| 442 | + groupIndex); | ||
| 439 | return HCCL_E_PARA; | 443 | return HCCL_E_PARA; |
| 440 | } | 444 | } |
| 441 | for (const auto &src : srcGroups[groupIndex]) { | 445 | for (const auto &src : srcGroups[groupIndex]) { |
| @@ -632,9 +636,10 @@ HcclResult CheckCollectedConflictCandidates(const ConflictCandidates &candidates | |||
| 632 | ++stats.parallelCandidatePairCount; | 636 | ++stats.parallelCandidatePairCount; |
| 633 | const uint64_t overlapStart = std::max(candidate.current->start, candidate.history->start); | 637 | const uint64_t overlapStart = std::max(candidate.current->start, candidate.history->start); |
| 634 | const uint64_t overlapEnd = std::min(candidate.current->end, candidate.history->end); | 638 | const uint64_t overlapEnd = std::min(candidate.current->end, candidate.history->end); |
| 635 | - HCCL_VM_ERROR("[TaskGraphMemConflictV3] Memory conflict detected, memory={}, overlap=[0x{:x},0x{:x}), " | 639 | + HCCL_VM_ERROR("{} Memory conflict detected, memory={}, overlap=[0x{:x},0x{:x}), " |
| 636 | - "current={}, history={}", DescribeKey(candidate.current->key), overlapStart, overlapEnd, | 640 | + "conflictTask1={}, conflictTask2={}", MakeErrorCodeText(ErrorCode::MEMCONFLICT_DETECTED), |
| 637 | - DescribeAccess(*candidate.current), DescribeAccess(*candidate.history)); | 641 | + DescribeKey(candidate.current->key), overlapStart, overlapEnd, DescribeAccess(*candidate.current), |
| 642 | + DescribeAccess(*candidate.history)); | ||
| 638 | return HCCL_E_MEMORY; | 643 | return HCCL_E_MEMORY; |
| 639 | } | 644 | } |
| 640 | return HCCL_SUCCESS; | 645 | return HCCL_SUCCESS; |
| @@ -687,8 +692,8 @@ HcclResult CollectMemoryAccessBuckets(const std::vector<const TaskNode *> &nodes | |||
| 687 | accessCountByTaskType[TaskTypeName(node->GetType())]++; | 692 | accessCountByTaskType[TaskTypeName(node->GetType())]++; |
| 688 | accessCountByTaskTypeAndKind[std::string(TaskTypeName(node->GetType())) + ":" + | 693 | accessCountByTaskTypeAndKind[std::string(TaskTypeName(node->GetType())) + ":" + |
| 689 | AccessKindName(access.kind)]++; | 694 | AccessKindName(access.kind)]++; |
| 690 | - accessCountByMemType[MemTypeName(access.key.memType)]++; | 695 | + accessCountByMemType[DescribeMemType(access.key.memType)]++; |
| 691 | - accessCountByMemTypeAndKind[std::string(MemTypeName(access.key.memType)) + ":" + | 696 | + accessCountByMemTypeAndKind[std::string(DescribeMemType(access.key.memType)) + ":" + |
| 692 | AccessKindName(access.kind)]++; | 697 | AccessKindName(access.kind)]++; |
| 693 | accessCountByBucket[DescribeKey(access.key)]++; | 698 | accessCountByBucket[DescribeKey(access.key)]++; |
| 694 | } | 699 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_graph_reachability_v3.cc+59-27
| @@ -10,13 +10,16 @@ | |||
| 10 | 10 | ||
| 11 | 11 | ||
| 12 | 12 | ||
| 13 | + | ||
| 13 | 14 | ||
| 14 | 15 | ||
| 15 | 16 | ||
| 16 | 17 | ||
| 18 | + | ||
| 17 | 19 | ||
| 18 | 20 | ||
| 19 | 21 | ||
| 22 | + | ||
| 20 | 23 | ||
| 21 | namespace HcclSim { | 24 | namespace HcclSim { |
| 22 | namespace TaskGraphGeneratorV3 { | 25 | namespace TaskGraphGeneratorV3 { |
| @@ -88,8 +91,8 @@ HcclResult AddReachableTaskNode(const TaskNode *node, ReachableTaskNodes &reacha | |||
| 88 | 91 | ||
| 89 | const NodeId nodeId = node->GetNodeId(); | 92 | const NodeId nodeId = node->GetNodeId(); |
| 90 | if (nodeId < 0) { | 93 | if (nodeId < 0) { |
| 91 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid task node id, nodeId={}, node={}", | 94 | + HCCL_VM_ERROR("{} Task node id is invalid, nodeId={}, node={}", |
| 92 | - nodeId, node->Describe()); | 95 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), nodeId, node->Describe()); |
| 93 | return HCCL_E_PARA; | 96 | return HCCL_E_PARA; |
| 94 | } | 97 | } |
| 95 | 98 | ||
| @@ -99,14 +102,16 @@ HcclResult AddReachableTaskNode(const TaskNode *node, ReachableTaskNodes &reacha | |||
| 99 | const size_t previousIndex = idResult.first->second; | 102 | const size_t previousIndex = idResult.first->second; |
| 100 | const TaskNode *previousNode = (previousIndex < reachable.nodes.size()) ? | 103 | const TaskNode *previousNode = (previousIndex < reachable.nodes.size()) ? |
| 101 | reachable.nodes[previousIndex] : nullptr; | 104 | reachable.nodes[previousIndex] : nullptr; |
| 102 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Duplicate task node id, nodeId={}, lhs={}, rhs={}", | 105 | + HCCL_VM_ERROR("{} Two different nodes share the same task node id, nodeId={}, " |
| 103 | - nodeId, (previousNode == nullptr) ? "null" : previousNode->Describe(), node->Describe()); | 106 | + "firstNode={}, secondNode={}", MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), nodeId, |
| 107 | + (previousNode == nullptr) ? "null" : previousNode->Describe(), node->Describe()); | ||
| 104 | return HCCL_E_PARA; | 108 | return HCCL_E_PARA; |
| 105 | } | 109 | } |
| 106 | 110 | ||
| 107 | const auto nodeResult = reachable.indexByNode.insert(std::make_pair(node, nodeIndex)); | 111 | const auto nodeResult = reachable.indexByNode.insert(std::make_pair(node, nodeIndex)); |
| 108 | if (!nodeResult.second) { | 112 | if (!nodeResult.second) { |
| 109 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Duplicate task node pointer, node={}", node->Describe()); | 113 | + HCCL_VM_ERROR("{} The same task node object was inserted more than once, node={}", |
| 114 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), node->Describe()); | ||
| 110 | return HCCL_E_INTERNAL; | 115 | return HCCL_E_INTERNAL; |
| 111 | } | 116 | } |
| 112 | 117 | ||
| @@ -138,7 +143,9 @@ HcclResult CollectReachableTaskNodes(const TaskNode *start, ReachableTaskNodes & | |||
| 138 | { | 143 | { |
| 139 | reachable = ReachableTaskNodes(); | 144 | reachable = ReachableTaskNodes(); |
| 140 | if (!IsMainGraphStartNode(start)) { | 145 | if (!IsMainGraphStartNode(start)) { |
| 141 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid main start node."); | 146 | + HCCL_VM_ERROR("{} Reachability analysis cannot start because the main start node is " |
| 147 | + "invalid, mainStartNode={}", MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), | ||
| 148 | + start == nullptr ? "null" : start->Describe()); | ||
| 142 | return HCCL_E_PARA; | 149 | return HCCL_E_PARA; |
| 143 | } | 150 | } |
| 144 | 151 | ||
| @@ -158,7 +165,8 @@ HcclResult CollectReachableTaskNodes(const TaskNode *start, ReachableTaskNodes & | |||
| 158 | 165 | ||
| 159 | for (const TaskNode *childNode : node->GetChildren()) { | 166 | for (const TaskNode *childNode : node->GetChildren()) { |
| 160 | if (childNode == nullptr) { | 167 | if (childNode == nullptr) { |
| 161 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid null child, parent={}", node->Describe()); | 168 | + HCCL_VM_ERROR("{} Graph structure is broken because one child node is null, " |
| 169 | + "parent={}", MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), node->Describe()); | ||
| 162 | return HCCL_E_PTR; | 170 | return HCCL_E_PTR; |
| 163 | } | 171 | } |
| 164 | if (visited.count(childNode) != 0) { | 172 | if (visited.count(childNode) != 0) { |
| @@ -178,14 +186,16 @@ HcclResult GetValidChildIndex(const TaskNode *parent, const TaskNode *child, | |||
| 178 | return HCCL_E_PTR; | 186 | return HCCL_E_PTR; |
| 179 | } | 187 | } |
| 180 | if (child == nullptr) { | 188 | if (child == nullptr) { |
| 181 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid null child, parent={}", parent->Describe()); | 189 | + HCCL_VM_ERROR("{} Graph structure is broken because one child node is null, parent={}", |
| 190 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), parent->Describe()); | ||
| 182 | return HCCL_E_PTR; | 191 | return HCCL_E_PTR; |
| 183 | } | 192 | } |
| 184 | 193 | ||
| 185 | const auto iter = reachable.indexByNode.find(child); | 194 | const auto iter = reachable.indexByNode.find(child); |
| 186 | if (iter == reachable.indexByNode.end()) { | 195 | if (iter == reachable.indexByNode.end()) { |
| 187 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid edge, parent={}, child={}", | 196 | + HCCL_VM_ERROR("{} One graph edge is invalid, so reachability analysis cannot continue, " |
| 188 | - parent->Describe(), child->Describe()); | 197 | + "parent={}, child={}", MakeErrorCodeText(ErrorCode::GRAPH_STRUCTURE_INVALID), parent->Describe(), |
| 198 | + child->Describe()); | ||
| 189 | return HCCL_E_PARA; | 199 | return HCCL_E_PARA; |
| 190 | } | 200 | } |
| 191 | 201 | ||
| @@ -268,10 +278,24 @@ HcclResult BuildReachabilityContext(const TaskNode *start, ReachabilityBuildCont | |||
| 268 | } | 278 | } |
| 269 | 279 | ||
| 270 | if (context.taskTopoNodes.size() != nodeCount) { | 280 | if (context.taskTopoNodes.size() != nodeCount) { |
| 271 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] V3 graph is not fully reachable from mainStart or is not a DAG, " | 281 | + const TaskNode *firstUnreachedNode = nullptr; |
| 272 | - "topoSize={}, expectedTopoSize={}, taskTopoSize={}, taskNodeCount={}, mainStartNodeId={}", | 282 | + for (const TaskNode *node : context.reachable.nodes) { |
| 283 | + if (node == nullptr) { | ||
| 284 | + continue; | ||
| 285 | + } | ||
| 286 | + const bool found = std::find(context.taskTopoNodes.begin(), context.taskTopoNodes.end(), node) != | ||
| 287 | + context.taskTopoNodes.end(); | ||
| 288 | + if (!found) { | ||
| 289 | + firstUnreachedNode = node; | ||
| 290 | + break; | ||
| 291 | + } | ||
| 292 | + } | ||
| 293 | + HCCL_VM_ERROR("{} This V3 graph is not a complete DAG from the main start node, " | ||
| 294 | + "topoSize={}, expectedTopoSize={}, reachableTaskCount={}, taskNodeCount={}, mainStartNodeId={}", | ||
| 295 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), | ||
| 273 | context.fullTopoOrder.size(), nodeCount + 1U, context.taskTopoNodes.size(), nodeCount, | 296 | context.fullTopoOrder.size(), nodeCount + 1U, context.taskTopoNodes.size(), nodeCount, |
| 274 | - MAIN_START_NODE_ID); | 297 | + MAIN_START_NODE_ID, |
| 298 | + firstUnreachedNode == nullptr ? "null" : firstUnreachedNode->Describe()); | ||
| 275 | return HCCL_E_INTERNAL; | 299 | return HCCL_E_INTERNAL; |
| 276 | } | 300 | } |
| 277 | 301 | ||
| @@ -295,7 +319,8 @@ HcclResult BuildDenseReachabilityClosure(const ReachabilityBuildContext &context | |||
| 295 | const TaskNode *node = *iter; | 319 | const TaskNode *node = *iter; |
| 296 | const auto nodeIter = context.reachable.indexByNode.find(node); | 320 | const auto nodeIter = context.reachable.indexByNode.find(node); |
| 297 | if (nodeIter == context.reachable.indexByNode.end()) { | 321 | if (nodeIter == context.reachable.indexByNode.end()) { |
| 298 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Missing topo node index, node={}", node->Describe()); | 322 | + HCCL_VM_ERROR("{} This node is missing its topological index, node={}", |
| 323 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), node->Describe()); | ||
| 299 | return HCCL_E_INTERNAL; | 324 | return HCCL_E_INTERNAL; |
| 300 | } | 325 | } |
| 301 | const size_t nodeIndex = nodeIter->second; | 326 | const size_t nodeIndex = nodeIter->second; |
| @@ -323,7 +348,8 @@ HcclResult BuildDenseReachabilityClosure(const ReachabilityBuildContext &context | |||
| 323 | const size_t dataIndex = dataItem.second; | 348 | const size_t dataIndex = dataItem.second; |
| 324 | const auto nodeIter = context.reachable.indexByNode.find(node); | 349 | const auto nodeIter = context.reachable.indexByNode.find(node); |
| 325 | if (nodeIter == context.reachable.indexByNode.end() || dataIndex >= closure.matrix.size()) { | 350 | if (nodeIter == context.reachable.indexByNode.end() || dataIndex >= closure.matrix.size()) { |
| 326 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Missing data move node index, node={}", node->Describe()); | 351 | + HCCL_VM_ERROR("{} This data-move node is missing its reachability index, node={}", |
| 352 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), node->Describe()); | ||
| 327 | return HCCL_E_INTERNAL; | 353 | return HCCL_E_INTERNAL; |
| 328 | } | 354 | } |
| 329 | closure.matrix[dataIndex] = reachableDataByNode[nodeIter->second]; | 355 | closure.matrix[dataIndex] = reachableDataByNode[nodeIter->second]; |
| @@ -361,23 +387,29 @@ HcclResult IsReachable(const ReachabilityClosure &closure, NodeId fromNodeId, No | |||
| 361 | isReachable = false; | 387 | isReachable = false; |
| 362 | const auto fromIter = closure.dataIndexByNodeId.find(fromNodeId); | 388 | const auto fromIter = closure.dataIndexByNodeId.find(fromNodeId); |
| 363 | if (fromIter == closure.dataIndexByNodeId.end()) { | 389 | if (fromIter == closure.dataIndexByNodeId.end()) { |
| 364 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Reachability query only supports data move task nodes, " | 390 | + HCCL_VM_ERROR("{} Task dependency query only supports memory-copy or reduce tasks, " |
| 365 | - "fromNodeId={}, toNodeId={}", fromNodeId, toNodeId); | 391 | + "but the source task is not in that set, sourceTaskId={}, targetTaskId={}, " |
| 392 | + "sourceReachabilityIndex=null", | ||
| 393 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), fromNodeId, toNodeId); | ||
| 366 | return HCCL_E_PARA; | 394 | return HCCL_E_PARA; |
| 367 | } | 395 | } |
| 368 | const auto toIter = closure.dataIndexByNodeId.find(toNodeId); | 396 | const auto toIter = closure.dataIndexByNodeId.find(toNodeId); |
| 369 | if (toIter == closure.dataIndexByNodeId.end()) { | 397 | if (toIter == closure.dataIndexByNodeId.end()) { |
| 370 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Reachability query only supports data move task nodes, " | 398 | + HCCL_VM_ERROR("{} Task dependency query only supports memory-copy or reduce tasks, " |
| 371 | - "fromNodeId={}, toNodeId={}", fromNodeId, toNodeId); | 399 | + "but the target task is not in that set, sourceTaskId={}, targetTaskId={}, " |
| 400 | + "sourceReachabilityIndex={}, targetReachabilityIndex=null", | ||
| 401 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), fromNodeId, toNodeId, fromIter->second); | ||
| 372 | return HCCL_E_PARA; | 402 | return HCCL_E_PARA; |
| 373 | } | 403 | } |
| 374 | 404 | ||
| 375 | const size_t fromIndex = fromIter->second; | 405 | const size_t fromIndex = fromIter->second; |
| 376 | const size_t toIndex = toIter->second; | 406 | const size_t toIndex = toIter->second; |
| 377 | if (fromIndex >= closure.matrix.size() || toIndex / BITS_PER_WORD >= closure.matrix[fromIndex].size()) { | 407 | if (fromIndex >= closure.matrix.size() || toIndex / BITS_PER_WORD >= closure.matrix[fromIndex].size()) { |
| 378 | - HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid data move reachability index, fromNodeId={}, " | 408 | + HCCL_VM_ERROR("{} Reachability index is invalid for this data-move task pair, " |
| 379 | - "toNodeId={}, fromIndex={}, toIndex={}, matrixSize={}", | 409 | + "sourceTaskId={}, targetTaskId={}, sourceIndex={}, targetIndex={}, matrixRowCount={}, " |
| 380 | - fromNodeId, toNodeId, fromIndex, toIndex, closure.matrix.size()); | 410 | + "matrixWordCountInSourceRow={}", |
| 411 | + MakeErrorCodeText(ErrorCode::MEMCONFLICT_DAG_INVALID), fromNodeId, toNodeId, fromIndex, toIndex, | ||
| 412 | + closure.matrix.size(), fromIndex < closure.matrix.size() ? closure.matrix[fromIndex].size() : 0U); | ||
| 381 | return HCCL_E_INTERNAL; | 413 | return HCCL_E_INTERNAL; |
| 382 | } | 414 | } |
| 383 | isReachable = TestBit(closure.matrix[fromIndex], toIndex); | 415 | isReachable = TestBit(closure.matrix[fromIndex], toIndex); |
| @@ -390,13 +422,13 @@ HcclResult IsReachable(const ReachabilityClosure &closure, NodeId fromNodeId, No | |||
| 390 | // isReachable = false; | 422 | // isReachable = false; |
| 391 | // const auto fromIter = closure.dataIndexByNodeId.find(fromNodeId); | 423 | // const auto fromIter = closure.dataIndexByNodeId.find(fromNodeId); |
| 392 | // if (fromIter == closure.dataIndexByNodeId.end()) { | 424 | // if (fromIter == closure.dataIndexByNodeId.end()) { |
| 393 | -// HCCL_VM_ERROR("[TaskGraphReachabilityV3] Roaring reachability query only supports data move task nodes, " | 425 | +// HCCL_VM_ERROR("Roaring reachability query only supports data move task nodes, " |
| 394 | // "fromNodeId={}, toNodeId={}", fromNodeId, toNodeId); | 426 | // "fromNodeId={}, toNodeId={}", fromNodeId, toNodeId); |
| 395 | // return HCCL_E_PARA; | 427 | // return HCCL_E_PARA; |
| 396 | // } | 428 | // } |
| 397 | // const auto toIter = closure.dataIndexByNodeId.find(toNodeId); | 429 | // const auto toIter = closure.dataIndexByNodeId.find(toNodeId); |
| 398 | // if (toIter == closure.dataIndexByNodeId.end()) { | 430 | // if (toIter == closure.dataIndexByNodeId.end()) { |
| 399 | -// HCCL_VM_ERROR("[TaskGraphReachabilityV3] Roaring reachability query only supports data move task nodes, " | 431 | +// HCCL_VM_ERROR("Roaring reachability query only supports data move task nodes, " |
| 400 | // "fromNodeId={}, toNodeId={}", fromNodeId, toNodeId); | 432 | // "fromNodeId={}, toNodeId={}", fromNodeId, toNodeId); |
| 401 | // return HCCL_E_PARA; | 433 | // return HCCL_E_PARA; |
| 402 | // } | 434 | // } |
| @@ -404,7 +436,7 @@ HcclResult IsReachable(const ReachabilityClosure &closure, NodeId fromNodeId, No | |||
| 404 | // const size_t fromIndex = fromIter->second; | 436 | // const size_t fromIndex = fromIter->second; |
| 405 | // const size_t toIndex = toIter->second; | 437 | // const size_t toIndex = toIter->second; |
| 406 | // if (fromIndex >= closure.matrix.size()) { | 438 | // if (fromIndex >= closure.matrix.size()) { |
| 407 | -// HCCL_VM_ERROR("[TaskGraphReachabilityV3] Invalid roaring data move reachability index, fromNodeId={}, " | 439 | +// HCCL_VM_ERROR("Invalid roaring data move reachability index, fromNodeId={}, " |
| 408 | // "toNodeId={}, fromIndex={}, toIndex={}, matrixSize={}", | 440 | // "toNodeId={}, fromIndex={}, toIndex={}, matrixSize={}", |
| 409 | // fromNodeId, toNodeId, fromIndex, toIndex, closure.matrix.size()); | 441 | // fromNodeId, toNodeId, fromIndex, toIndex, closure.matrix.size()); |
| 410 | // return HCCL_E_INTERNAL; | 442 | // return HCCL_E_INTERNAL; |
| @@ -472,8 +504,8 @@ HcclResult CheckDataMoveTaskReachability(const TaskNode *start, DataTaskReachabi | |||
| 472 | } | 504 | } |
| 473 | } | 505 | } |
| 474 | 506 | ||
| 475 | - HCCL_VM_INFO("[TaskGraphReachabilityV3] V3 data move reachability generated, nodeCount={}, edgeCount={}, " | 507 | + HCCL_VM_INFO("Task dependency summary generated, nodeCount={}, edgeCount={}, " |
| 476 | - "dataTaskNodeCount={}, dataTaskPairs={}, reachablePairs={}, unreachablePairs={}", | 508 | + "memoryTaskCount={}, taskPairsChecked={}, orderedTaskPairs={}, parallelTaskPairs={}", |
| 477 | localStats.nodeCount, localStats.edgeCount, localStats.dataTaskNodeCount, | 509 | localStats.nodeCount, localStats.edgeCount, localStats.dataTaskNodeCount, |
| 478 | localStats.dataTaskPairCount, localStats.reachablePairCount, localStats.unreachablePairCount); | 510 | localStats.dataTaskPairCount, localStats.reachablePairCount, localStats.unreachablePairCount); |
| 479 | 511 | ||
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_graph_semantic_check_v3.cc+96-49
| @@ -34,6 +34,8 @@ | |||
| 34 | 34 | ||
| 35 | 35 | ||
| 36 | 36 | ||
| 37 | + | ||
| 38 | + | ||
| 37 | 39 | ||
| 38 | 40 | ||
| 39 | namespace HcclSim { | 41 | namespace HcclSim { |
| @@ -46,6 +48,20 @@ enum class SliceOpV3 : uint8_t { | |||
| 46 | REDUCE, | 48 | REDUCE, |
| 47 | }; | 49 | }; |
| 48 | 50 | ||
| 51 | +std::string DescribeMemSliceForSemantic(const MemSlice &slice) | ||
| 52 | +{ | ||
| 53 | + std::ostringstream os; | ||
| 54 | + os << "{rankId="; | ||
| 55 | + if (slice.rankId == INVALID_RANK_ID) { | ||
| 56 | + os << "invalid"; | ||
| 57 | + } else { | ||
| 58 | + os << slice.rankId; | ||
| 59 | + } | ||
| 60 | + os << ", memoryType=" << DescribeMemType(slice.memType) | ||
| 61 | + << ", offset=0x" << std::hex << slice.offset << ", length=0x" << slice.len << std::dec << "}"; | ||
| 62 | + return os.str(); | ||
| 63 | +} | ||
| 64 | + | ||
| 49 | struct SliceOpPairV3 { | 65 | struct SliceOpPairV3 { |
| 50 | RankId srcRank{INVALID_RANK_ID}; | 66 | RankId srcRank{INVALID_RANK_ID}; |
| 51 | RankId dstRank{INVALID_RANK_ID}; | 67 | RankId dstRank{INVALID_RANK_ID}; |
| @@ -60,15 +76,20 @@ struct SliceOpPairV3 { | |||
| 60 | std::string Describe() const | 76 | std::string Describe() const |
| 61 | { | 77 | { |
| 62 | std::ostringstream os; | 78 | std::ostringstream os; |
| 63 | - os << "{srcRank=" << srcRank << ", dstRank=" << dstRank | 79 | + os << "{operation=" << (op == SliceOpV3::REDUCE ? "reduce" : "overwrite") |
| 64 | - << ", srcMemType=" << (int)(src.memType) << ", dstMemType=" << (int)(dst.memType) | 80 | + << ", sourceMemorySlice=" << DescribeMemSliceForSemantic(src) |
| 65 | - << ", srcOffset=0x" << std::hex << src.offset | 81 | + << ", targetMemorySlice=" << DescribeMemSliceForSemantic(dst) |
| 66 | - << ", dstOffset=0x" << dst.offset << std::dec << ", len=" << src.len | 82 | + << ", launchIdx=" << position.launchIdx |
| 67 | - << ", launch=" << position.launchIdx << ", block=" << position.blockId | 83 | + << ", blockId=" << position.blockId |
| 68 | - << ", pipe=" << position.pipe << ", taskId=" << position.taskId | 84 | + << ", pipeId=" << position.pipe |
| 69 | - << ", index=" << index << ", op=" << static_cast<uint32_t>(op) | 85 | + << ", taskId=" << position.taskId; |
| 70 | - << ", useBatchIndex=" << useBatchIndex | 86 | + if (useBatchIndex) { |
| 71 | - << ", reduceType=" << static_cast<uint32_t>(reduceType) << "}"; | 87 | + os << ", batchGroup=" << index; |
| 88 | + } | ||
| 89 | + if (op == SliceOpV3::REDUCE) { | ||
| 90 | + os << ", reduceType=" << DumpReduceOpToString(reduceType); | ||
| 91 | + } | ||
| 92 | + os << "}"; | ||
| 72 | return os.str(); | 93 | return os.str(); |
| 73 | } | 94 | } |
| 74 | }; | 95 | }; |
| @@ -269,7 +290,11 @@ HcclResult InitState(SemanticState &state) | |||
| 269 | { | 290 | { |
| 270 | state.param = StorageManager::GetInstance().GetCheckerParam(); | 291 | state.param = StorageManager::GetInstance().GetCheckerParam(); |
| 271 | if (state.param.rankSize == 0) { | 292 | if (state.param.rankSize == 0) { |
| 272 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] rankSize is zero"); | 293 | + HCCL_VM_ERROR("{} Semantic check initialization failed because the rank count is 0, " |
| 294 | + "collectiveType={}, dataType={}, elementCount={}, reduceType={}", | ||
| 295 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), HcclCmdTypeToString(state.param.cmdType), | ||
| 296 | + HcclDataTypeToString(state.param.dataType), state.param.dataCount, | ||
| 297 | + DumpReduceOpToString(state.param.reduceType)); | ||
| 273 | return HCCL_E_PARA; | 298 | return HCCL_E_PARA; |
| 274 | } | 299 | } |
| 275 | CalcDataSize(state.param.cmdType, state.param.dataCount, state.param.dataType, state.dataSize); | 300 | CalcDataSize(state.param.cmdType, state.param.dataCount, state.param.dataType, state.dataSize); |
| @@ -500,8 +525,9 @@ HcclResult CheckBufSemantics(const std::vector<const BufferSemantic *> &bufSeman | |||
| 500 | } | 525 | } |
| 501 | if (bufSemantics.empty()) { | 526 | if (bufSemantics.empty()) { |
| 502 | if (!ignoreError) { | 527 | if (!ignoreError) { |
| 503 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Empty buffer semantics, start=0x{:x}, size=0x{:x}", | 528 | + HCCL_VM_ERROR("{} No source/output information was found for the target memory range, " |
| 504 | - startAddr, size); | 529 | + "startAddr=0x{:x}, size=0x{:x}", MakeErrorCodeText(ErrorCode::SEMANTIC_BUFFER_EMPTY), startAddr, |
| 530 | + size); | ||
| 505 | } | 531 | } |
| 506 | return HCCL_E_PARA; | 532 | return HCCL_E_PARA; |
| 507 | } | 533 | } |
| @@ -510,7 +536,8 @@ HcclResult CheckBufSemantics(const std::vector<const BufferSemantic *> &bufSeman | |||
| 510 | const BufferSemantic *prev = bufSemantics.front(); | 536 | const BufferSemantic *prev = bufSemantics.front(); |
| 511 | if (prev->startAddr > startAddr) { | 537 | if (prev->startAddr > startAddr) { |
| 512 | if (!ignoreError) { | 538 | if (!ignoreError) { |
| 513 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Head gap, expect start=0x{:x}, actual start=0x{:x}", | 539 | + HCCL_VM_ERROR("{} Output data does not start from the expected address; the beginning is " |
| 540 | + "missing, expectedStart=0x{:x}, actualStart=0x{:x}", MakeErrorCodeText(ErrorCode::SEMANTIC_GAP), | ||
| 514 | startAddr, prev->startAddr); | 541 | startAddr, prev->startAddr); |
| 515 | } | 542 | } |
| 516 | return HCCL_E_PARA; | 543 | return HCCL_E_PARA; |
| @@ -525,7 +552,8 @@ HcclResult CheckBufSemantics(const std::vector<const BufferSemantic *> &bufSeman | |||
| 525 | const BufferSemantic *cur = bufSemantics[index]; | 552 | const BufferSemantic *cur = bufSemantics[index]; |
| 526 | if (cur->startAddr != prev->startAddr + prev->size) { | 553 | if (cur->startAddr != prev->startAddr + prev->size) { |
| 527 | if (!ignoreError) { | 554 | if (!ignoreError) { |
| 528 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Middle gap, prev end=0x{:x}, cur start=0x{:x}", | 555 | + HCCL_VM_ERROR("{} Output data is broken in the middle; one piece ends at 0x{:x} " |
| 556 | + "but the next starts at 0x{:x}", MakeErrorCodeText(ErrorCode::SEMANTIC_GAP), | ||
| 529 | prev->startAddr + prev->size, cur->startAddr); | 557 | prev->startAddr + prev->size, cur->startAddr); |
| 530 | } | 558 | } |
| 531 | return HCCL_E_PARA; | 559 | return HCCL_E_PARA; |
| @@ -540,8 +568,9 @@ HcclResult CheckBufSemantics(const std::vector<const BufferSemantic *> &bufSeman | |||
| 540 | 568 | ||
| 541 | if (totalSize != size) { | 569 | if (totalSize != size) { |
| 542 | if (!ignoreError) { | 570 | if (!ignoreError) { |
| 543 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Tail gap, expect end=0x{:x}, actual end=0x{:x}", | 571 | + HCCL_VM_ERROR("{} Output data ends too early; the tail is missing, expectedEnd=0x{:x}, " |
| 544 | - startAddr + size, startAddr + totalSize); | 572 | + "actualEnd=0x{:x}", MakeErrorCodeText(ErrorCode::SEMANTIC_GAP), startAddr + size, |
| 573 | + startAddr + totalSize); | ||
| 545 | } | 574 | } |
| 546 | return HCCL_E_PARA; | 575 | return HCCL_E_PARA; |
| 547 | } | 576 | } |
| @@ -594,14 +623,18 @@ HcclResult AddReduceSourcesToDstSegments(const SliceOpPairV3 &pair, const Buffer | |||
| 594 | } | 623 | } |
| 595 | if (dstSemantic->srcBufs.size() == 1) { | 624 | if (dstSemantic->srcBufs.size() == 1) { |
| 596 | if (dstSemantic->isReduce) { | 625 | if (dstSemantic->isReduce) { |
| 597 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Invalid dst reduce semantic, pair={}", pair.Describe()); | 626 | + HCCL_VM_ERROR("{} Target output range is already marked as a reduce result before " |
| 627 | + "enough source data is merged, dataMapping={}", | ||
| 628 | + MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), pair.Describe()); | ||
| 598 | return HCCL_E_PARA; | 629 | return HCCL_E_PARA; |
| 599 | } | 630 | } |
| 600 | dstSemantic->isReduce = true; | 631 | dstSemantic->isReduce = true; |
| 601 | dstSemantic->reduceType = pair.reduceType; | 632 | dstSemantic->reduceType = pair.reduceType; |
| 602 | } | 633 | } |
| 603 | if (srcSemantic.srcBufs.size() > 1 && dstSemantic->reduceType != srcSemantic.reduceType) { | 634 | if (srcSemantic.srcBufs.size() > 1 && dstSemantic->reduceType != srcSemantic.reduceType) { |
| 604 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Reduce type mismatch, pair={}", pair.Describe()); | 635 | + HCCL_VM_ERROR("{} Reduce result type is inconsistent while merging one source data " |
| 636 | + "range, dataMapping={}", MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), | ||
| 637 | + pair.Describe()); | ||
| 605 | return HCCL_E_PARA; | 638 | return HCCL_E_PARA; |
| 606 | } | 639 | } |
| 607 | 640 | ||
| @@ -610,12 +643,10 @@ HcclResult AddReduceSourcesToDstSegments(const SliceOpPairV3 &pair, const Buffer | |||
| 610 | const size_t before = dstSemantic->srcBufs.size(); | 643 | const size_t before = dstSemantic->srcBufs.size(); |
| 611 | dstSemantic->srcBufs.insert(srcBuf); | 644 | dstSemantic->srcBufs.insert(srcBuf); |
| 612 | if (before == dstSemantic->srcBufs.size()) { | 645 | if (before == dstSemantic->srcBufs.size()) { |
| 613 | - | 646 | + HCCL_VM_ERROR("{} The same source data is added twice to one output range during " |
| 614 | - | 647 | + "reduce, dataMapping={}, sourceOffsetInThisRange=0x{:x}, outputRange=[0x{:x},0x{:x})", |
| 615 | - | 648 | + MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), pair.Describe(), srcOffset, |
| 616 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Duplicate reduce source, pair={}, srcOffset=0x{:x}, " | 649 | + dstSemantic->startAddr, GetSegmentEndAddr(*dstSemantic)); |
| 617 | - "dstSegmentStart=0x{:x}, dstSegmentEnd=0x{:x}", | ||
| 618 | - pair.Describe(), srcOffset, dstSemantic->startAddr, GetSegmentEndAddr(*dstSemantic)); | ||
| 619 | return HCCL_E_PARA; | 650 | return HCCL_E_PARA; |
| 620 | } | 651 | } |
| 621 | } | 652 | } |
| @@ -648,17 +679,19 @@ HcclResult ApplyReduceSemantic(const SliceOpPairV3 &pair, SemanticState &state, | |||
| 648 | } | 679 | } |
| 649 | HcclResult ret = CheckBufSemantics(affectedViews, dstStartAddr, dstEndAddr - dstStartAddr, false); | 680 | HcclResult ret = CheckBufSemantics(affectedViews, dstStartAddr, dstEndAddr - dstStartAddr, false); |
| 650 | if (ret != HCCL_SUCCESS) { | 681 | if (ret != HCCL_SUCCESS) { |
| 651 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Destination reduce semantic incomplete, pair={}, dstRange=[0x{:x},0x{:x}), " | 682 | + HCCL_VM_ERROR("{} Target output range is only partially filled before reduce continues, " |
| 652 | - "dstSegmentCnt={}", | 683 | + "dataMapping={}, outputRange=[0x{:x},0x{:x}), pieceCount={}", |
| 653 | - pair.Describe(), dstStartAddr, dstEndAddr, affectedDstSemantics.size()); | 684 | + MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), pair.Describe(), dstStartAddr, dstEndAddr, |
| 685 | + affectedDstSemantics.size()); | ||
| 654 | return ret; | 686 | return ret; |
| 655 | } | 687 | } |
| 656 | 688 | ||
| 657 | ret = AddReduceSourcesToDstSegments(pair, srcSemantic, affectedDstSemantics, srcStartAddr); | 689 | ret = AddReduceSourcesToDstSegments(pair, srcSemantic, affectedDstSemantics, srcStartAddr); |
| 658 | if (ret != HCCL_SUCCESS) { | 690 | if (ret != HCCL_SUCCESS) { |
| 659 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Add reduce sources to dst segments failed, pair={}, " | 691 | + HCCL_VM_ERROR("{} Failed to merge one source data range into the target output range during " |
| 660 | - "srcRange=[0x{:x},0x{:x}), dstRange=[0x{:x},0x{:x}), ret={}", | 692 | + "reduce, dataMapping={}, srcRange=[0x{:x},0x{:x}), dstRange=[0x{:x},0x{:x}), ret={}", |
| 661 | - pair.Describe(), srcStartAddr, srcEndAddr, dstStartAddr, dstEndAddr, static_cast<int32_t>(ret)); | 693 | + MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), pair.Describe(), srcStartAddr, srcEndAddr, |
| 694 | + dstStartAddr, dstEndAddr, static_cast<int32_t>(ret)); | ||
| 662 | return ret; | 695 | return ret; |
| 663 | } | 696 | } |
| 664 | 697 | ||
| @@ -693,11 +726,13 @@ HcclResult LoadSliceOpPairSrcSemantics(const SliceOpPairV3 &pair, const Semantic | |||
| 693 | return HCCL_SUCCESS; | 726 | return HCCL_SUCCESS; |
| 694 | } | 727 | } |
| 695 | if (!IsValidMemorySlice(pair.src) || !IsValidMemorySlice(pair.dst)) { | 728 | if (!IsValidMemorySlice(pair.src) || !IsValidMemorySlice(pair.dst)) { |
| 696 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Invalid memory slice, pair={}", pair.Describe()); | 729 | + HCCL_VM_ERROR("{} One source-to-target mapping uses an invalid memory slice, " |
| 730 | + "dataMapping={}", MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), pair.Describe()); | ||
| 697 | return HCCL_E_PARA; | 731 | return HCCL_E_PARA; |
| 698 | } | 732 | } |
| 699 | if (!IsSupportedSemanticMemType(pair.src.memType) || !IsSupportedSemanticMemType(pair.dst.memType)) { | 733 | if (!IsSupportedSemanticMemType(pair.src.memType) || !IsSupportedSemanticMemType(pair.dst.memType)) { |
| 700 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Unsupported memory type, pair={}", pair.Describe()); | 734 | + HCCL_VM_ERROR("{} This data mapping uses a memory type that semantic check does not support, " |
| 735 | + "dataMapping={}", MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), pair.Describe()); | ||
| 701 | return HCCL_E_NOT_SUPPORT; | 736 | return HCCL_E_NOT_SUPPORT; |
| 702 | } | 737 | } |
| 703 | 738 | ||
| @@ -709,14 +744,14 @@ HcclResult LoadSliceOpPairSrcSemantics(const SliceOpPairV3 &pair, const Semantic | |||
| 709 | HcclResult ret = CheckBufSemantics(srcViews, pair.src.offset, pair.src.len, true); | 744 | HcclResult ret = CheckBufSemantics(srcViews, pair.src.offset, pair.src.len, true); |
| 710 | if (ret != HCCL_SUCCESS) { | 745 | if (ret != HCCL_SUCCESS) { |
| 711 | if (pair.op == SliceOpV3::REDUCE) { | 746 | if (pair.op == SliceOpV3::REDUCE) { |
| 712 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Incomplete reduce source semantics, pair={}", | 747 | + HCCL_VM_ERROR("{} Source data needed by this reduce is missing, dataMapping={}", |
| 713 | - pair.Describe()); | 748 | + MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), pair.Describe()); |
| 714 | return ret; | 749 | return ret; |
| 715 | } | 750 | } |
| 716 | // 老语义实现对 override 的不完整源语义是放行但告警; | 751 | // 老语义实现对 override 的不完整源语义是放行但告警; |
| 717 | // 这里保持同样语义,至少把“并非完整 memcpy 语义”的风险显式打出来。 | 752 | // 这里保持同样语义,至少把“并非完整 memcpy 语义”的风险显式打出来。 |
| 718 | - HCCL_VM_WARN("[TaskGraphSemanticCheckV3] Incomplete override source semantics, pair={}", | 753 | + HCCL_VM_WARN("{} Source data needed by this overwrite is missing, dataMapping={}", |
| 719 | - pair.Describe()); | 754 | + MakeErrorCodeText(ErrorCode::SEMANTIC_REDUCE_ERROR), pair.Describe()); |
| 720 | } | 755 | } |
| 721 | 756 | ||
| 722 | srcSemantics.reserve(srcViews.size()); | 757 | srcSemantics.reserve(srcViews.size()); |
| @@ -741,7 +776,6 @@ HcclResult ApplyPreparedSliceOpPair(const PreparedSliceOpPairV3 &prepared, Seman | |||
| 741 | } | 776 | } |
| 742 | HcclResult ret = ApplySrcSemanticsToDst(prepared.pair, state, srcViews); | 777 | HcclResult ret = ApplySrcSemanticsToDst(prepared.pair, state, srcViews); |
| 743 | if (ret != HCCL_SUCCESS) { | 778 | if (ret != HCCL_SUCCESS) { |
| 744 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Apply src semantics failed, pair={}", prepared.pair.Describe()); | ||
| 745 | return ret; | 779 | return ret; |
| 746 | } | 780 | } |
| 747 | return HCCL_SUCCESS; | 781 | return HCCL_SUCCESS; |
| @@ -844,8 +878,6 @@ HcclResult SimulateTask(const TaskNode *node, SemanticState &state) | |||
| 844 | prepared.pair = pair; | 878 | prepared.pair = pair; |
| 845 | HcclResult ret = LoadSliceOpPairSrcSemantics(pair, state, prepared.srcSemantics); | 879 | HcclResult ret = LoadSliceOpPairSrcSemantics(pair, state, prepared.srcSemantics); |
| 846 | if (ret != HCCL_SUCCESS) { | 880 | if (ret != HCCL_SUCCESS) { |
| 847 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Load slice op pair src semantics failed, node={}, pair={}", | ||
| 848 | - node->Describe(), pair.Describe()); | ||
| 849 | return ret; | 881 | return ret; |
| 850 | } | 882 | } |
| 851 | preparedPairs.push_back(std::move(prepared)); | 883 | preparedPairs.push_back(std::move(prepared)); |
| @@ -853,8 +885,6 @@ HcclResult SimulateTask(const TaskNode *node, SemanticState &state) | |||
| 853 | for (const auto &prepared : preparedPairs) { | 885 | for (const auto &prepared : preparedPairs) { |
| 854 | HcclResult ret = ApplyPreparedSliceOpPair(prepared, state); | 886 | HcclResult ret = ApplyPreparedSliceOpPair(prepared, state); |
| 855 | if (ret != HCCL_SUCCESS) { | 887 | if (ret != HCCL_SUCCESS) { |
| 856 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Process slice op pair failed, node={}, pair={}", | ||
| 857 | - node->Describe(), prepared.pair.Describe()); | ||
| 858 | return ret; | 888 | return ret; |
| 859 | } | 889 | } |
| 860 | } | 890 | } |
| @@ -914,8 +944,12 @@ HcclResult CheckFinalOutput(const SemanticState &state) | |||
| 914 | case HCCL_CMD_BATCH_SEND_RECV: | 944 | case HCCL_CMD_BATCH_SEND_RECV: |
| 915 | return TaskCheckBatchSendRecvSemantics(allRankMemSemantics, state.dataSize); | 945 | return TaskCheckBatchSendRecvSemantics(allRankMemSemantics, state.dataSize); |
| 916 | default: | 946 | default: |
| 917 | - HCCL_VM_WARN("[TaskGraphSemanticCheckV3] Unsupported op type for final semantic check, op={}", | 947 | + HCCL_VM_WARN("{} Final output validation does not support this collective type yet, " |
| 918 | - static_cast<uint32_t>(state.param.cmdType)); | 948 | + "collectiveType={}, rankCount={}, dataType={}, elementCount={}, reduceType={}", |
| 949 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), | ||
| 950 | + HcclCmdTypeToString(state.param.cmdType), state.param.rankSize, | ||
| 951 | + HcclDataTypeToString(state.param.dataType), state.param.dataCount, | ||
| 952 | + DumpReduceOpToString(state.param.reduceType)); | ||
| 919 | return HCCL_E_NOT_SUPPORT; | 953 | return HCCL_E_NOT_SUPPORT; |
| 920 | } | 954 | } |
| 921 | } | 955 | } |
| @@ -944,7 +978,8 @@ void FillStats(const SemanticState &state, size_t handledNodeCount, SemanticChec | |||
| 944 | HcclResult CheckSemantics(const TaskNode *start, SemanticCheckStats *stats) | 978 | HcclResult CheckSemantics(const TaskNode *start, SemanticCheckStats *stats) |
| 945 | { | 979 | { |
| 946 | if (start == nullptr) { | 980 | if (start == nullptr) { |
| 947 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Null start node"); | 981 | + HCCL_VM_ERROR("{} Semantic check cannot start because the main start node is missing, " |
| 982 | + "mainStartNode=null", MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR)); | ||
| 948 | return HCCL_E_PTR; | 983 | return HCCL_E_PTR; |
| 949 | } | 984 | } |
| 950 | 985 | ||
| @@ -993,12 +1028,12 @@ HcclResult CheckSemantics(const TaskNode *start, SemanticCheckStats *stats) | |||
| 993 | ++handled; | 1028 | ++handled; |
| 994 | ret = SimulateTask(node, state); | 1029 | ret = SimulateTask(node, state); |
| 995 | if (ret != HCCL_SUCCESS) { | 1030 | if (ret != HCCL_SUCCESS) { |
| 996 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Semantic simulate failed, node={}", node->Describe()); | ||
| 997 | return ret; | 1031 | return ret; |
| 998 | } | 1032 | } |
| 999 | for (const TaskNode *child : node->GetChildren()) { | 1033 | for (const TaskNode *child : node->GetChildren()) { |
| 1000 | if (child == nullptr) { | 1034 | if (child == nullptr) { |
| 1001 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Invalid null child, parent={}", node->Describe()); | 1035 | + HCCL_VM_ERROR("{} Graph structure is broken because one child node is null, parent={}", |
| 1036 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), node->Describe()); | ||
| 1002 | return HCCL_E_PTR; | 1037 | return HCCL_E_PTR; |
| 1003 | } | 1038 | } |
| 1004 | auto iter = pendingParents.find(child->GetNodeId()); | 1039 | auto iter = pendingParents.find(child->GetNodeId()); |
| @@ -1013,14 +1048,26 @@ HcclResult CheckSemantics(const TaskNode *start, SemanticCheckStats *stats) | |||
| 1013 | } | 1048 | } |
| 1014 | 1049 | ||
| 1015 | if (handled != nodes.size()) { | 1050 | if (handled != nodes.size()) { |
| 1016 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Graph has unresolved dependencies, handled={}, total={}", | 1051 | + const TaskNode *firstRemainingNode = nullptr; |
| 1017 | - handled, nodes.size()); | 1052 | + for (const auto &entry : pendingParents) { |
| 1053 | + if (entry.second != 0) { | ||
| 1054 | + auto nodeIter = nodes.find(entry.first); | ||
| 1055 | + if (nodeIter != nodes.end()) { | ||
| 1056 | + firstRemainingNode = nodeIter->second; | ||
| 1057 | + break; | ||
| 1058 | + } | ||
| 1059 | + } | ||
| 1060 | + } | ||
| 1061 | + HCCL_VM_ERROR("{} Output simulation stopped because some tasks still have unresolved " | ||
| 1062 | + "dependencies, handledNodeCount={}, totalNodeCount={}, firstRemainingNode={}", | ||
| 1063 | + MakeErrorCodeText(ErrorCode::CHECKER_RUNTIME_ERROR), handled, nodes.size(), | ||
| 1064 | + firstRemainingNode == nullptr ? std::string("node=null") : firstRemainingNode->Describe()); | ||
| 1018 | return HCCL_E_INTERNAL; | 1065 | return HCCL_E_INTERNAL; |
| 1019 | } | 1066 | } |
| 1020 | 1067 | ||
| 1021 | ret = CheckFinalOutput(state); | 1068 | ret = CheckFinalOutput(state); |
| 1022 | if (ret != HCCL_SUCCESS && ret != HCCL_E_NOT_SUPPORT) { | 1069 | if (ret != HCCL_SUCCESS && ret != HCCL_E_NOT_SUPPORT) { |
| 1023 | - HCCL_VM_ERROR("[TaskGraphSemanticCheckV3] Final semantic check failed, ret={}", | 1070 | + HCCL_VM_ERROR("Final semantic check failed, ret={}", |
| 1024 | static_cast<uint32_t>(ret)); | 1071 | static_cast<uint32_t>(ret)); |
| 1025 | return ret; | 1072 | return ret; |
| 1026 | } | 1073 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_graph_single_task_check_v3.cc+116-52
| @@ -23,6 +23,8 @@ | |||
| 23 | 23 | ||
| 24 | 24 | ||
| 25 | 25 | ||
| 26 | + | ||
| 27 | + | ||
| 26 | 28 | ||
| 27 | namespace HcclSim { | 29 | namespace HcclSim { |
| 28 | namespace TaskGraphGeneratorV3 { | 30 | namespace TaskGraphGeneratorV3 { |
| @@ -80,15 +82,32 @@ const char *TaskTypeName(TaskType taskType) | |||
| 80 | std::string DescribePosition(const TaskPosition &loc) | 82 | std::string DescribePosition(const TaskPosition &loc) |
| 81 | { | 83 | { |
| 82 | std::ostringstream os; | 84 | std::ostringstream os; |
| 83 | - os << "rank=" << loc.rankId << ", stream=" << loc.streamId; | 85 | + os << "rankId="; |
| 86 | + if (loc.rankId == INVALID_RANK_ID) { | ||
| 87 | + os << "invalid"; | ||
| 88 | + } else { | ||
| 89 | + os << loc.rankId; | ||
| 90 | + } | ||
| 91 | + os << ", streamId="; | ||
| 92 | + if (loc.streamId == INVALID_STREAM_ID) { | ||
| 93 | + os << "invalid"; | ||
| 94 | + } else { | ||
| 95 | + os << loc.streamId; | ||
| 96 | + } | ||
| 84 | return os.str(); | 97 | return os.str(); |
| 85 | } | 98 | } |
| 86 | 99 | ||
| 87 | std::string DescribeMemSlice(const MemSlice &slice) | 100 | std::string DescribeMemSlice(const MemSlice &slice) |
| 88 | { | 101 | { |
| 89 | std::ostringstream os; | 102 | std::ostringstream os; |
| 90 | - os << "{rank=" << slice.rankId << ", mem=" << static_cast<uint32_t>(slice.memType) | 103 | + os << "{rankId="; |
| 91 | - << ", offset=0x" << std::hex << slice.offset << ", len=0x" << slice.len << std::dec << "}"; | 104 | + if (slice.rankId == INVALID_RANK_ID) { |
| 105 | + os << "invalid"; | ||
| 106 | + } else { | ||
| 107 | + os << slice.rankId; | ||
| 108 | + } | ||
| 109 | + os << ", memType=" << DescribeMemType(slice.memType) | ||
| 110 | + << ", offset=0x" << std::hex << slice.offset << ", length=0x" << slice.len << std::dec << "}"; | ||
| 92 | return os.str(); | 111 | return os.str(); |
| 93 | } | 112 | } |
| 94 | 113 | ||
| @@ -224,8 +243,10 @@ HcclResult AccumulateCoveredSizeByKey(const TaskNode *node, const std::vector<Me | |||
| 224 | continue; | 243 | continue; |
| 225 | } | 244 | } |
| 226 | if (slice.len > std::numeric_limits<uint64_t>::max() - slice.offset) { | 245 | if (slice.len > std::numeric_limits<uint64_t>::max() - slice.offset) { |
| 227 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Memory slice overflow while accumulating coverage, " | 246 | + HCCL_VM_ERROR("{} One memory slice is invalid because its end address overflows " |
| 228 | - "node={}, slice={}", node->Describe(), DescribeMemSlice(slice)); | 247 | + "while total coverage is being calculated, task={}, memorySlice={}, offset=0x{:x}, length=0x{:x}", |
| 248 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 249 | + DescribeMemSlice(slice), slice.offset, slice.len); | ||
| 229 | return HCCL_E_PARA; | 250 | return HCCL_E_PARA; |
| 230 | } | 251 | } |
| 231 | const MemoryKey key{slice.rankId, slice.memType}; | 252 | const MemoryKey key{slice.rankId, slice.memType}; |
| @@ -293,7 +314,7 @@ HcclResult BuildCoverageSummary(const TaskNode *node, const std::vector<std::vec | |||
| 293 | } | 314 | } |
| 294 | 315 | ||
| 295 | HcclResult CheckCoverageMultiple(const TaskNode *node, uint64_t srcSize, uint64_t dstSize, | 316 | HcclResult CheckCoverageMultiple(const TaskNode *node, uint64_t srcSize, uint64_t dstSize, |
| 296 | - const std::string &srcName, const std::string &dstName) | 317 | + const std::string &group) |
| 297 | { | 318 | { |
| 298 | if (node == nullptr) { | 319 | if (node == nullptr) { |
| 299 | return HCCL_E_PTR; | 320 | return HCCL_E_PTR; |
| @@ -302,15 +323,17 @@ HcclResult CheckCoverageMultiple(const TaskNode *node, uint64_t srcSize, uint64_ | |||
| 302 | if (srcSize == 0) { | 323 | if (srcSize == 0) { |
| 303 | return HCCL_SUCCESS; | 324 | return HCCL_SUCCESS; |
| 304 | } | 325 | } |
| 305 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Zero destination coverage with non-zero source coverage, " | 326 | + HCCL_VM_ERROR("{} Source data exists but the target range is empty, task={}, " |
| 306 | - "node={}, {}={}, {}={}", node->Describe(), srcName, srcSize, dstName, dstSize); | 327 | + "srcDataSize={}, dstDataSize={}, group={}", |
| 328 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), srcSize, dstSize, group); | ||
| 307 | return HCCL_E_INTERNAL; | 329 | return HCCL_E_INTERNAL; |
| 308 | } | 330 | } |
| 309 | if (srcSize % dstSize == 0) { | 331 | if (srcSize % dstSize == 0) { |
| 310 | return HCCL_SUCCESS; | 332 | return HCCL_SUCCESS; |
| 311 | } | 333 | } |
| 312 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Source coverage is not a multiple of destination coverage, " | 334 | + HCCL_VM_ERROR("{} Source data size is not an integer multiple of target data size, " |
| 313 | - "node={}, {}={}, {}={}", node->Describe(), srcName, srcSize, dstName, dstSize); | 335 | + "task={}, srcDataSize={}, dstDataSize={}, group={}", |
| 336 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), srcSize, dstSize, group); | ||
| 314 | return HCCL_E_INTERNAL; | 337 | return HCCL_E_INTERNAL; |
| 315 | } | 338 | } |
| 316 | 339 | ||
| @@ -323,8 +346,10 @@ HcclResult CheckSliceLenEqual(const TaskNode *node, const MemSlice &src, const M | |||
| 323 | if (src.len == dst.len) { | 346 | if (src.len == dst.len) { |
| 324 | return HCCL_SUCCESS; | 347 | return HCCL_SUCCESS; |
| 325 | } | 348 | } |
| 326 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Batch trans slice length mismatch, node={}, label={}, index={}, " | 349 | + HCCL_VM_ERROR("{} In one batch transfer pair, source length and target length are " |
| 327 | - "src={}, dst={}", node->Describe(), label, index, DescribeMemSlice(src), DescribeMemSlice(dst)); | 350 | + "different, task={}, group={}, pairIndex={}, sourceMemorySlice={}, targetMemorySlice={}", |
| 351 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), label, index, | ||
| 352 | + DescribeMemSlice(src), DescribeMemSlice(dst)); | ||
| 328 | return HCCL_E_INTERNAL; | 353 | return HCCL_E_INTERNAL; |
| 329 | } | 354 | } |
| 330 | 355 | ||
| @@ -365,8 +390,10 @@ HcclResult CheckBatchTransSlices(const TaskNode *node, const std::vector<MemSlic | |||
| 365 | // 原始 src/dst 负责逐对检查 memcpy 语义是否满足“长度一一对应”; | 390 | // 原始 src/dst 负责逐对检查 memcpy 语义是否满足“长度一一对应”; |
| 366 | // merged src/dst 只用于更快地检查 batch 节点内部是否存在自冲突。 | 391 | // merged src/dst 只用于更快地检查 batch 节点内部是否存在自冲突。 |
| 367 | if (srcs.size() != dsts.size()) { | 392 | if (srcs.size() != dsts.size()) { |
| 368 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Batch trans pair size mismatch, node={}, label={}, " | 393 | + HCCL_VM_ERROR("{} Batch transfer has different counts of source memory slices and " |
| 369 | - "srcCount={}, dstCount={}", node->Describe(), label, srcs.size(), dsts.size()); | 394 | + "target memory slices, task={}, group={}, sourceMemorySliceCount={}, targetMemorySliceCount={}", |
| 395 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), label, srcs.size(), | ||
| 396 | + dsts.size()); | ||
| 370 | return HCCL_E_PARA; | 397 | return HCCL_E_PARA; |
| 371 | } | 398 | } |
| 372 | for (size_t index = 0; index < srcs.size(); ++index) { | 399 | for (size_t index = 0; index < srcs.size(); ++index) { |
| @@ -409,9 +436,10 @@ HcclResult CheckBatchTransSlices(const TaskNode *node, const std::vector<MemSlic | |||
| 409 | } | 436 | } |
| 410 | } | 437 | } |
| 411 | } | 438 | } |
| 412 | - HCCL_VM_DEBUG("[TaskGraphSingleTaskCheckV3] Batch trans merged slices checked, " | 439 | + HCCL_VM_DEBUG("Batch transfer merged-slice check finished, " |
| 413 | - "nodeId={}, label={}, originalSrcCount={}, originalDstCount={}, mergedSrcCount={}, mergedDstCount={}, " | 440 | + "taskId={}, group={}, originalSourceMemorySliceCount={}, originalTargetMemorySliceCount={}, " |
| 414 | - "dedupMergedSrcCount={}, dedupMergedDstCount={}", node->GetNodeId(), label, srcs.size(), dsts.size(), | 441 | + "mergedSourceMemorySliceCount={}, mergedTargetMemorySliceCount={}, uniqueMergedSourceMemorySliceCount={}, " |
| 442 | + "uniqueMergedTargetMemorySliceCount={}", node->GetNodeId(), label, srcs.size(), dsts.size(), | ||
| 415 | mergedSrcs.size(), mergedDsts.size(), dedupMergedSrcs.size(), dedupMergedDsts.size()); | 443 | mergedSrcs.size(), mergedDsts.size(), dedupMergedSrcs.size(), dedupMergedDsts.size()); |
| 416 | return HCCL_SUCCESS; | 444 | return HCCL_SUCCESS; |
| 417 | } | 445 | } |
| @@ -426,8 +454,10 @@ HcclResult CheckBatchReduceSlices(const TaskNode *node, const std::vector<std::v | |||
| 426 | // batch reduce 仍然按原始 group 校验:每组 src 必须能落到同一个 dst, | 454 | // batch reduce 仍然按原始 group 校验:每组 src 必须能落到同一个 dst, |
| 427 | // 且所有 src 总覆盖量应当是 dst 总覆盖量的整倍数。 | 455 | // 且所有 src 总覆盖量应当是 dst 总覆盖量的整倍数。 |
| 428 | if (srcGroups.empty() || srcGroups.size() != dsts.size()) { | 456 | if (srcGroups.empty() || srcGroups.size() != dsts.size()) { |
| 429 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Batch reduce group size mismatch, node={}, label={}, " | 457 | + HCCL_VM_ERROR("{} Batch reduce has different counts of source groups and target " |
| 430 | - "srcGroupCount={}, dstCount={}", node->Describe(), label, srcGroups.size(), dsts.size()); | 458 | + "memory slices, task={}, group={}, sourceGroupCount={}, targetMemorySliceCount={}", |
| 459 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), label, srcGroups.size(), | ||
| 460 | + dsts.size()); | ||
| 431 | return HCCL_E_PARA; | 461 | return HCCL_E_PARA; |
| 432 | } | 462 | } |
| 433 | for (size_t groupIndex = 0; groupIndex < srcGroups.size(); ++groupIndex) { | 463 | for (size_t groupIndex = 0; groupIndex < srcGroups.size(); ++groupIndex) { |
| @@ -436,8 +466,9 @@ HcclResult CheckBatchReduceSlices(const TaskNode *node, const std::vector<std::v | |||
| 436 | return ret; | 466 | return ret; |
| 437 | } | 467 | } |
| 438 | if (srcGroups[groupIndex].empty()) { | 468 | if (srcGroups[groupIndex].empty()) { |
| 439 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Batch reduce source list is empty, node={}, label={}, " | 469 | + HCCL_VM_ERROR("{} One reduce group has no source data at all, task={}, group={}, " |
| 440 | - "groupIndex={}", node->Describe(), label, groupIndex); | 470 | + "groupIndex={}", MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), |
| 471 | + label, groupIndex); | ||
| 441 | return HCCL_E_PARA; | 472 | return HCCL_E_PARA; |
| 442 | } | 473 | } |
| 443 | for (const auto &src : srcGroups[groupIndex]) { | 474 | for (const auto &src : srcGroups[groupIndex]) { |
| @@ -460,8 +491,7 @@ HcclResult CheckBatchReduceSlices(const TaskNode *node, const std::vector<std::v | |||
| 460 | if (ret != HCCL_SUCCESS) { | 491 | if (ret != HCCL_SUCCESS) { |
| 461 | return ret; | 492 | return ret; |
| 462 | } | 493 | } |
| 463 | - return CheckCoverageMultiple(node, srcSummary.totalCoveredSize, dstSummary.totalCoveredSize, | 494 | + return CheckCoverageMultiple(node, srcSummary.totalCoveredSize, dstSummary.totalCoveredSize, label); |
| 464 | - label + " srcCoveredSize", label + " dstCoveredSize"); | ||
| 465 | } | 495 | } |
| 466 | 496 | ||
| 467 | HcclResult CheckSingleSlice(const TaskNode *node, const MemSlice &slice) | 497 | HcclResult CheckSingleSlice(const TaskNode *node, const MemSlice &slice) |
| @@ -473,21 +503,28 @@ HcclResult CheckSingleSlice(const TaskNode *node, const MemSlice &slice) | |||
| 473 | return HCCL_SUCCESS; | 503 | return HCCL_SUCCESS; |
| 474 | } | 504 | } |
| 475 | if (slice.rankId == INVALID_RANK_ID || slice.memType == MemType::INVALID) { | 505 | if (slice.rankId == INVALID_RANK_ID || slice.memType == MemType::INVALID) { |
| 476 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Invalid memory slice, node={}, slice={}", | 506 | + HCCL_VM_ERROR("{} One memory slice is missing a valid rank or memory type, task={}, " |
| 477 | - node->Describe(), DescribeMemSlice(slice)); | 507 | + "memorySlice={}, rankId={}, memType={}", |
| 508 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 509 | + DescribeMemSlice(slice), slice.rankId == INVALID_RANK_ID ? "invalid" : std::to_string(slice.rankId), | ||
| 510 | + DescribeMemType(slice.memType)); | ||
| 478 | return HCCL_E_PARA; | 511 | return HCCL_E_PARA; |
| 479 | } | 512 | } |
| 480 | if (slice.len > std::numeric_limits<uint64_t>::max() - slice.offset) { | 513 | if (slice.len > std::numeric_limits<uint64_t>::max() - slice.offset) { |
| 481 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Memory slice overflow, node={}, slice={}", | 514 | + HCCL_VM_ERROR("{} One memory slice is invalid because offset + length exceeds the " |
| 482 | - node->Describe(), DescribeMemSlice(slice)); | 515 | + "numeric limit, task={}, memorySlice={}, offset=0x{:x}, length=0x{:x}", |
| 516 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 517 | + DescribeMemSlice(slice), slice.offset, slice.len); | ||
| 483 | return HCCL_E_PARA; | 518 | return HCCL_E_PARA; |
| 484 | } | 519 | } |
| 485 | 520 | ||
| 486 | if (slice.memType == MemType::UB_AIV || slice.memType == MemType::FLAG_AIV) { | 521 | if (slice.memType == MemType::UB_AIV || slice.memType == MemType::FLAG_AIV) { |
| 487 | const uint64_t boundSize = GetAivSliceBoundSize(slice.memType); | 522 | const uint64_t boundSize = GetAivSliceBoundSize(slice.memType); |
| 488 | if (boundSize != 0 && slice.offset + slice.len > boundSize) { | 523 | if (boundSize != 0 && slice.offset + slice.len > boundSize) { |
| 489 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] AIV slice out-of-bounds, node={}, slice={}, boundSize={}", | 524 | + HCCL_VM_ERROR("{} One AIV memory slice goes past the valid AIV buffer boundary, " |
| 490 | - node->Describe(), DescribeMemSlice(slice), boundSize); | 525 | + "task={}, memorySlice={}, bufferLimit={}, memType={}", |
| 526 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 527 | + DescribeMemSlice(slice), boundSize, DescribeMemType(slice.memType)); | ||
| 491 | return HCCL_E_INTERNAL; | 528 | return HCCL_E_INTERNAL; |
| 492 | } | 529 | } |
| 493 | return HCCL_SUCCESS; | 530 | return HCCL_SUCCESS; |
| @@ -495,15 +532,20 @@ HcclResult CheckSingleSlice(const TaskNode *node, const MemSlice &slice) | |||
| 495 | 532 | ||
| 496 | const BufferType storageType = ConvertMemTypeToBufferType(slice.memType); | 533 | const BufferType storageType = ConvertMemTypeToBufferType(slice.memType); |
| 497 | if (storageType == BufferType::RESERVED) { | 534 | if (storageType == BufferType::RESERVED) { |
| 498 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Unsupported memory type, node={}, slice={}", | 535 | + HCCL_VM_ERROR("{} This task uses a memory type that the checker does not support, " |
| 499 | - node->Describe(), DescribeMemSlice(slice)); | 536 | + "task={}, memorySlice={}, memType={}", |
| 537 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 538 | + DescribeMemSlice(slice), DescribeMemType(slice.memType)); | ||
| 500 | return HCCL_E_NOT_SUPPORT; | 539 | return HCCL_E_NOT_SUPPORT; |
| 501 | } | 540 | } |
| 502 | 541 | ||
| 503 | const uint64_t blockSize = StorageManager::GetInstance().GetBlockSize(slice.rankId, storageType); | 542 | const uint64_t blockSize = StorageManager::GetInstance().GetBlockSize(slice.rankId, storageType); |
| 504 | if (slice.offset + slice.len > blockSize) { | 543 | if (slice.offset + slice.len > blockSize) { |
| 505 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Failed to check slice, {}, node={}, slice={}, blockSize={}", | 544 | + HCCL_VM_ERROR("{} One memory slice goes past the end of its buffer, task={}, " |
| 506 | - DescribePosition(node->GetPosition()), node->Describe(), DescribeMemSlice(slice), blockSize); | 545 | + "memorySlice={}, bufferType={}, bufferSize={}, position={}", |
| 546 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 547 | + DescribeMemSlice(slice), BufferTypeToString(storageType), blockSize, | ||
| 548 | + DescribePosition(node->GetPosition())); | ||
| 507 | return HCCL_E_INTERNAL; | 549 | return HCCL_E_INTERNAL; |
| 508 | } | 550 | } |
| 509 | return HCCL_SUCCESS; | 551 | return HCCL_SUCCESS; |
| @@ -519,15 +561,18 @@ HcclResult CheckTwoSliceOverlap(const TaskNode *node, const MemSlice &lhs, const | |||
| 519 | } | 561 | } |
| 520 | if (lhs.len > std::numeric_limits<uint64_t>::max() - lhs.offset || | 562 | if (lhs.len > std::numeric_limits<uint64_t>::max() - lhs.offset || |
| 521 | rhs.len > std::numeric_limits<uint64_t>::max() - rhs.offset) { | 563 | rhs.len > std::numeric_limits<uint64_t>::max() - rhs.offset) { |
| 522 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Memory slice overflow, node={}, sliceA={}, sliceB={}", | 564 | + HCCL_VM_ERROR("{} Two memory slices cannot be compared because one slice already " |
| 523 | - node->Describe(), DescribeMemSlice(lhs), DescribeMemSlice(rhs)); | 565 | + "overflowed, task={}, memorySlice1={}, memorySlice2={}", |
| 566 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 567 | + DescribeMemSlice(lhs), DescribeMemSlice(rhs)); | ||
| 524 | return HCCL_E_PARA; | 568 | return HCCL_E_PARA; |
| 525 | } | 569 | } |
| 526 | 570 | ||
| 527 | if (node->HasCcuTrace()) { | 571 | if (node->HasCcuTrace()) { |
| 528 | // CCU mode下,允许src == dst | 572 | // CCU mode下,允许src == dst |
| 529 | if (lhs.rankId == rhs.rankId && IsSameMemoryType(lhs, rhs) && lhs.offset == rhs.offset && lhs.len == rhs.len) { | 573 | if (lhs.rankId == rhs.rankId && IsSameMemoryType(lhs, rhs) && lhs.offset == rhs.offset && lhs.len == rhs.len) { |
| 530 | - HCCL_VM_WARN("[TaskGraphSingleTaskCheckV3] MemSlice are same (src == dst), which may affect performance, {}, node={}, sliceA={}, sliceB={}", | 574 | + HCCL_VM_WARN("Source and destination use the same memory slice, which may hurt " |
| 575 | + "performance, position={}, task={}, memorySlice1={}, memorySlice2={}", | ||
| 531 | DescribePosition(node->GetPosition()), node->Describe(), DescribeMemSlice(lhs), DescribeMemSlice(rhs)); | 576 | DescribePosition(node->GetPosition()), node->Describe(), DescribeMemSlice(lhs), DescribeMemSlice(rhs)); |
| 532 | return HCCL_SUCCESS; | 577 | return HCCL_SUCCESS; |
| 533 | } | 578 | } |
| @@ -536,8 +581,10 @@ HcclResult CheckTwoSliceOverlap(const TaskNode *node, const MemSlice &lhs, const | |||
| 536 | const bool conflictCase1 = lhs.offset >= rhs.offset && lhs.offset < (rhs.offset + rhs.len); | 581 | const bool conflictCase1 = lhs.offset >= rhs.offset && lhs.offset < (rhs.offset + rhs.len); |
| 537 | const bool conflictCase2 = rhs.offset >= lhs.offset && rhs.offset < (lhs.offset + lhs.len); | 582 | const bool conflictCase2 = rhs.offset >= lhs.offset && rhs.offset < (lhs.offset + lhs.len); |
| 538 | if (conflictCase1 || conflictCase2) { | 583 | if (conflictCase1 || conflictCase2) { |
| 539 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Slice conflict, {}, node={}, sliceA={}, sliceB={}", | 584 | + HCCL_VM_ERROR("{} Two memory slices overlap inside the same task, task={}, " |
| 540 | - DescribePosition(node->GetPosition()), node->Describe(), DescribeMemSlice(lhs), DescribeMemSlice(rhs)); | 585 | + "memorySlice1={}, memorySlice2={}, position={}", |
| 586 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_CONFLICT), node->Describe(), | ||
| 587 | + DescribeMemSlice(lhs), DescribeMemSlice(rhs), DescribePosition(node->GetPosition())); | ||
| 541 | return HCCL_E_INTERNAL; | 588 | return HCCL_E_INTERNAL; |
| 542 | } | 589 | } |
| 543 | return HCCL_SUCCESS; | 590 | return HCCL_SUCCESS; |
| @@ -566,14 +613,18 @@ HcclResult CheckAivFlagNodeMem(const TaskNode *node) | |||
| 566 | return HCCL_SUCCESS; | 613 | return HCCL_SUCCESS; |
| 567 | } | 614 | } |
| 568 | if (flag->flagOffset > (std::numeric_limits<uint64_t>::max() - sizeof(int32_t)) / AIV_FLAG_SLOT_BYTES) { | 615 | if (flag->flagOffset > (std::numeric_limits<uint64_t>::max() - sizeof(int32_t)) / AIV_FLAG_SLOT_BYTES) { |
| 569 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] AIV flag offset overflow, node={}", node->Describe()); | 616 | + HCCL_VM_ERROR("{} AIV flag address overflowed while calculating its byte offset, " |
| 617 | + "task={}, flagOffset={}", MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), | ||
| 618 | + flag->flagOffset); | ||
| 570 | return HCCL_E_PARA; | 619 | return HCCL_E_PARA; |
| 571 | } | 620 | } |
| 572 | const uint64_t flagByteOffset = flag->flagOffset * AIV_FLAG_SLOT_BYTES; | 621 | const uint64_t flagByteOffset = flag->flagOffset * AIV_FLAG_SLOT_BYTES; |
| 573 | const uint64_t flagBufferSize = GetAivSliceBoundSize(MemType::FLAG_AIV); | 622 | const uint64_t flagBufferSize = GetAivSliceBoundSize(MemType::FLAG_AIV); |
| 574 | if (flagBufferSize != 0 && flagByteOffset + sizeof(int32_t) > flagBufferSize) { | 623 | if (flagBufferSize != 0 && flagByteOffset + sizeof(int32_t) > flagBufferSize) { |
| 575 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] AIV flag offset out-of-bounds, node={}, " | 624 | + HCCL_VM_ERROR("{} AIV flag address goes past the valid flag buffer boundary, task={}, " |
| 576 | - "flagByteOffset={}, flagBufferSize={}", node->Describe(), flagByteOffset, flagBufferSize); | 625 | + "flagByteOffset={}, flagBufferSize={}", |
| 626 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe(), flagByteOffset, | ||
| 627 | + flagBufferSize); | ||
| 577 | return HCCL_E_INTERNAL; | 628 | return HCCL_E_INTERNAL; |
| 578 | } | 629 | } |
| 579 | return HCCL_SUCCESS; | 630 | return HCCL_SUCCESS; |
| @@ -611,7 +662,8 @@ HcclResult CheckDataMoveTaskMem(const TaskNode *node) | |||
| 611 | } | 662 | } |
| 612 | const std::vector<MemSlice> &srcs = reduce->GetSrcs(); | 663 | const std::vector<MemSlice> &srcs = reduce->GetSrcs(); |
| 613 | if (srcs.empty()) { | 664 | if (srcs.empty()) { |
| 614 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Reduce source list is empty, node={}", node->Describe()); | 665 | + HCCL_VM_ERROR("{} This reduce task has no source data at all, task={}", |
| 666 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLICE_INVALID), node->Describe()); | ||
| 615 | return HCCL_E_PARA; | 667 | return HCCL_E_PARA; |
| 616 | } | 668 | } |
| 617 | for (const auto &src : srcs) { | 669 | for (const auto &src : srcs) { |
| @@ -674,8 +726,8 @@ HcclResult CheckSlaveTaskQueue(const std::vector<std::unique_ptr<TaskNode>> &nod | |||
| 674 | const auto &stream = taskQueue[streamId]; | 726 | const auto &stream = taskQueue[streamId]; |
| 675 | const size_t taskSize = stream.size(); | 727 | const size_t taskSize = stream.size(); |
| 676 | if (StreamHasAivGraph(nodes, stream)) { | 728 | if (StreamHasAivGraph(nodes, stream)) { |
| 677 | - HCCL_VM_DEBUG("[TaskGraphSingleTaskCheckV3] Skip legacy slave stream shape check for AIV stream, " | 729 | + HCCL_VM_DEBUG("Skip this slave-stream structure check because the stream contains " |
| 678 | - "rank={}, stream={}", rankId, streamId); | 730 | + "an AIV graph task, rankId={}, streamId={}", rankId, streamId); |
| 679 | continue; | 731 | continue; |
| 680 | } | 732 | } |
| 681 | if (taskSize < 2) { | 733 | if (taskSize < 2) { |
| @@ -685,31 +737,43 @@ HcclResult CheckSlaveTaskQueue(const std::vector<std::unique_ptr<TaskNode>> &nod | |||
| 685 | const TaskNode *firstTask = GetQueueNode(nodes, stream.front()); | 737 | const TaskNode *firstTask = GetQueueNode(nodes, stream.front()); |
| 686 | const TaskNode *lastTask = GetQueueNode(nodes, stream.back()); | 738 | const TaskNode *lastTask = GetQueueNode(nodes, stream.back()); |
| 687 | if (firstTask == nullptr || lastTask == nullptr) { | 739 | if (firstTask == nullptr || lastTask == nullptr) { |
| 688 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Invalid slave stream node, rank={}, stream={}", | 740 | + HCCL_VM_ERROR("{} This slave stream is missing its start node " |
| 689 | - rankId, streamId); | 741 | + "or end node, rankId={}, streamId={}, taskCount={}, startNode={}, endNode={}", |
| 742 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLAVE_STREAM_INVALID), rankId, streamId, taskSize, | ||
| 743 | + firstTask == nullptr ? "null" : firstTask->Describe(), | ||
| 744 | + lastTask == nullptr ? "null" : lastTask->Describe()); | ||
| 690 | return HCCL_E_PTR; | 745 | return HCCL_E_PTR; |
| 691 | } | 746 | } |
| 692 | 747 | ||
| 693 | size_t backStep = 1; | 748 | size_t backStep = 1; |
| 749 | + size_t skippedEmptyLocalCopyCount = 0; | ||
| 694 | while (IsEmptyLocalCopy(lastTask) && backStep < taskSize) { | 750 | while (IsEmptyLocalCopy(lastTask) && backStep < taskSize) { |
| 695 | lastTask = GetQueueNode(nodes, stream[taskSize - 1 - backStep]); | 751 | lastTask = GetQueueNode(nodes, stream[taskSize - 1 - backStep]); |
| 696 | if (lastTask == nullptr) { | 752 | if (lastTask == nullptr) { |
| 697 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] Invalid slave stream tail node, rank={}, stream={}", | 753 | + HCCL_VM_ERROR("{} This slave stream still has no valid end " |
| 698 | - rankId, streamId); | 754 | + "node after empty local-copy tasks are skipped, rankId={}, streamId={}, " |
| 755 | + "skippedEmptyLocalCopyCount={}, currentTailNode={}", | ||
| 756 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLAVE_STREAM_INVALID), rankId, streamId, | ||
| 757 | + skippedEmptyLocalCopyCount, "null"); | ||
| 699 | return HCCL_E_PTR; | 758 | return HCCL_E_PTR; |
| 700 | } | 759 | } |
| 760 | + ++skippedEmptyLocalCopyCount; | ||
| 701 | ++backStep; | 761 | ++backStep; |
| 702 | } | 762 | } |
| 703 | 763 | ||
| 704 | if (!IsLocalAicpuWait(firstTask)) { | 764 | if (!IsLocalAicpuWait(firstTask)) { |
| 705 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] RankId:{}, streamId:{} first task type should be " | 765 | + HCCL_VM_ERROR("{} The first task in this slave stream is not a " |
| 706 | - "local WAIT, while is {}", rankId, streamId, TaskTypeName(firstTask->GetType())); | 766 | + "local WAIT task, rankId={}, streamId={}, actualFirstTaskType={}, firstTask={}", |
| 767 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLAVE_STREAM_INVALID), rankId, streamId, | ||
| 768 | + TaskTypeName(firstTask->GetType()), firstTask->Describe()); | ||
| 707 | return HCCL_E_INTERNAL; | 769 | return HCCL_E_INTERNAL; |
| 708 | } | 770 | } |
| 709 | 771 | ||
| 710 | if (!IsLocalAicpuRecord(lastTask)) { | 772 | if (!IsLocalAicpuRecord(lastTask)) { |
| 711 | - HCCL_VM_ERROR("[TaskGraphSingleTaskCheckV3] RankId:{}, streamId:{} last task type should be " | 773 | + HCCL_VM_ERROR("{} The last task in this slave stream is not a " |
| 712 | - "local RECORD, while is {}", rankId, streamId, TaskTypeName(lastTask->GetType())); | 774 | + "local RECORD task, rankId={}, streamId={}, actualLastTaskType={}, lastTask={}", |
| 775 | + MakeErrorCodeText(ErrorCode::SINGLETASK_SLAVE_STREAM_INVALID), rankId, streamId, | ||
| 776 | + TaskTypeName(lastTask->GetType()), lastTask->Describe()); | ||
| 713 | return HCCL_E_INTERNAL; | 777 | return HCCL_E_INTERNAL; |
| 714 | } | 778 | } |
| 715 | } | 779 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_meta_translator_v3.cc+66-9
| @@ -13,10 +13,12 @@ | |||
| 13 | 13 | ||
| 14 | 14 | ||
| 15 | 15 | ||
| 16 | + | ||
| 16 | 17 | ||
| 17 | 18 | ||
| 18 | 19 | ||
| 19 | 20 | ||
| 21 | + | ||
| 20 | 22 | ||
| 21 | namespace HcclSim { | 23 | namespace HcclSim { |
| 22 | namespace TaskGraphGeneratorV3 { | 24 | namespace TaskGraphGeneratorV3 { |
| @@ -112,6 +114,56 @@ CcuSqeParam MakeCcuSqeParam(const CcuTask &ccuTask) | |||
| 112 | return param; | 114 | return param; |
| 113 | } | 115 | } |
| 114 | 116 | ||
| 117 | +std::string DescribeTaskMetaForLog(const HcclTaskMetaData &taskMeta) | ||
| 118 | +{ | ||
| 119 | + std::ostringstream os; | ||
| 120 | + os << "{taskType=" << static_cast<int32_t>(taskMeta.taskType) | ||
| 121 | + << ", rankId=" << taskMeta.rankId | ||
| 122 | + << ", streamId=" << taskMeta.streamId; | ||
| 123 | + switch (taskMeta.taskType) { | ||
| 124 | + case HccLTaskMetaType::MEM_CPY: | ||
| 125 | + os << ", srcRankId=" << taskMeta.taskData.transMem.srcRankId | ||
| 126 | + << ", dstRankId=" << taskMeta.taskData.transMem.dstRankId | ||
| 127 | + << ", srcOffset=" << taskMeta.taskData.transMem.srcOffset | ||
| 128 | + << ", dstOffset=" << taskMeta.taskData.transMem.dstOffset | ||
| 129 | + << ", len=" << taskMeta.taskData.transMem.len | ||
| 130 | + << ", protocol=" << static_cast<uint32_t>(taskMeta.taskData.transMem.protocol); | ||
| 131 | + break; | ||
| 132 | + case HccLTaskMetaType::REDUCE: | ||
| 133 | + os << ", srcRankId=" << taskMeta.taskData.reduce.srcRankId | ||
| 134 | + << ", dstRankId=" << taskMeta.taskData.reduce.dstRankId | ||
| 135 | + << ", srcOffset=" << taskMeta.taskData.reduce.srcOffset | ||
| 136 | + << ", dstOffset=" << taskMeta.taskData.reduce.dstOffset | ||
| 137 | + << ", dataCount=" << taskMeta.taskData.reduce.dataCount | ||
| 138 | + << ", dataType=" << static_cast<uint32_t>(taskMeta.taskData.reduce.dataType) | ||
| 139 | + << ", reduceOp=" << static_cast<uint32_t>(taskMeta.taskData.reduce.reduceOp); | ||
| 140 | + break; | ||
| 141 | + case HccLTaskMetaType::NOTIFY_RECORD: | ||
| 142 | + case HccLTaskMetaType::NOTIFY_WAIT: | ||
| 143 | + os << ", srcRankId=" << taskMeta.taskData.notify.srcRankId | ||
| 144 | + << ", dstRankId=" << taskMeta.taskData.notify.dstRankId | ||
| 145 | + << ", notifyId=" << taskMeta.taskData.notify.notifyId | ||
| 146 | + << ", notifyCount=" << taskMeta.taskData.notify.notifyCount | ||
| 147 | + << ", protocol=" << static_cast<uint32_t>(taskMeta.taskData.notify.protocol); | ||
| 148 | + break; | ||
| 149 | + case HccLTaskMetaType::CCU_GRAPH: | ||
| 150 | + os << ", dieId=" << taskMeta.taskData.ccu.dieId | ||
| 151 | + << ", missionId=" << taskMeta.taskData.ccu.missionId | ||
| 152 | + << ", instStartId=" << taskMeta.taskData.ccu.instStartId | ||
| 153 | + << ", instCnt=" << taskMeta.taskData.ccu.instCnt | ||
| 154 | + << ", argSize=" << taskMeta.taskData.ccu.argSize | ||
| 155 | + << ", key=" << taskMeta.taskData.ccu.key; | ||
| 156 | + break; | ||
| 157 | + case HccLTaskMetaType::AIV_GRAPH: | ||
| 158 | + os << ", launchId=" << taskMeta.taskData.aiv.launchIdx; | ||
| 159 | + break; | ||
| 160 | + default: | ||
| 161 | + break; | ||
| 162 | + } | ||
| 163 | + os << "}"; | ||
| 164 | + return os.str(); | ||
| 165 | +} | ||
| 166 | + | ||
| 115 | bool IsContiguousCcuSqe(const TaskCcuGraph &missionNode, const CcuSqeParam &sqe) | 167 | bool IsContiguousCcuSqe(const TaskCcuGraph &missionNode, const CcuSqeParam &sqe) |
| 116 | { | 168 | { |
| 117 | constexpr size_t DEFAULT_CCU_QUEUE_ID = 0U; | 169 | constexpr size_t DEFAULT_CCU_QUEUE_ID = 0U; |
| @@ -160,14 +212,17 @@ HcclResult TaskMetaTranslatorV3::Translate(StorageManager &storage) | |||
| 160 | 212 | ||
| 161 | const HcclVmTaskMetaData &taskMetaData = storage.GetHvmTaskMetaData(); | 213 | const HcclVmTaskMetaData &taskMetaData = storage.GetHvmTaskMetaData(); |
| 162 | const auto &taskMetaVec = taskMetaData.task_meta; | 214 | const auto &taskMetaVec = taskMetaData.task_meta; |
| 163 | - HCCL_VM_INFO("[TaskMetaTranslatorV3][Translate] Start translate task meta, taskMetaCount={}", | 215 | + HCCL_VM_INFO("Start converting task metadata into graph nodes, taskMetaCount={}", |
| 164 | taskMetaVec.size()); | 216 | taskMetaVec.size()); |
| 165 | for (uint32_t i = 0; i < taskMetaVec.size(); ++i) { | 217 | for (uint32_t i = 0; i < taskMetaVec.size(); ++i) { |
| 166 | NodeId nodeId = INVALID_NODE_ID; | 218 | NodeId nodeId = INVALID_NODE_ID; |
| 167 | - const HcclResult ret = TranslateOneTaskMeta(taskMetaVec[i], storage, nodeId); | 219 | + const HcclResult ret = TranslateOneTaskMeta(taskMetaVec[i], storage, i, nodeId); |
| 168 | if (ret != HCCL_SUCCESS) { | 220 | if (ret != HCCL_SUCCESS) { |
| 169 | - HCCL_VM_ERROR("[TaskMetaTranslatorV3] Translate task meta failed, index={}, taskType={}, ret={}", | 221 | + HCCL_VM_ERROR("{} Failed to convert one task into a graph node, taskIndex={}, " |
| 170 | - i, static_cast<int32_t>(taskMetaVec[i].taskType), static_cast<uint32_t>(ret)); | 222 | + "taskType={}, rankId={}, streamId={}, ret={}, taskMeta={}", |
| 223 | + MakeErrorCodeText(ErrorCode::GRAPH_TRANSLATE_FAILED), i, | ||
| 224 | + static_cast<int32_t>(taskMetaVec[i].taskType), taskMetaVec[i].rankId, taskMetaVec[i].streamId, | ||
| 225 | + static_cast<uint32_t>(ret), DescribeTaskMetaForLog(taskMetaVec[i])); | ||
| 171 | return ret; | 226 | return ret; |
| 172 | } | 227 | } |
| 173 | } | 228 | } |
| @@ -180,8 +235,8 @@ HcclResult TaskMetaTranslatorV3::Translate(StorageManager &storage) | |||
| 180 | } | 235 | } |
| 181 | } | 236 | } |
| 182 | } | 237 | } |
| 183 | - HCCL_VM_INFO("[TaskMetaTranslatorV3][Translate] Translate task meta success, taskMetaCount={}, nodeCount={}, " | 238 | + HCCL_VM_INFO("Finished converting task metadata into graph nodes, taskMetaCount={}, nodeCount={}, " |
| 184 | - "rankCount={}, streamCount={}", taskMetaVec.size(), nodes_.size(), taskQueues_.size(), streamCount); | 239 | + "rankCount={}, nonEmptyStreamCount={}", taskMetaVec.size(), nodes_.size(), taskQueues_.size(), streamCount); |
| 185 | return HCCL_SUCCESS; | 240 | return HCCL_SUCCESS; |
| 186 | } | 241 | } |
| 187 | 242 | ||
| @@ -209,7 +264,7 @@ HcclResult TaskMetaTranslatorV3::AddTaskNode(const TaskPosition &position, std:: | |||
| 209 | } | 264 | } |
| 210 | 265 | ||
| 211 | HcclResult TaskMetaTranslatorV3::TranslateOneTaskMeta(const HcclTaskMetaData &taskMeta, StorageManager &storage, | 266 | HcclResult TaskMetaTranslatorV3::TranslateOneTaskMeta(const HcclTaskMetaData &taskMeta, StorageManager &storage, |
| 212 | - NodeId &nodeId) | 267 | + uint32_t taskIndex, NodeId &nodeId) |
| 213 | { | 268 | { |
| 214 | TaskPosition position; | 269 | TaskPosition position; |
| 215 | HcclResult ret = MakeTaskPosition(taskMeta, position); | 270 | HcclResult ret = MakeTaskPosition(taskMeta, position); |
| @@ -322,8 +377,10 @@ HcclResult TaskMetaTranslatorV3::TranslateOneTaskMeta(const HcclTaskMetaData &ta | |||
| 322 | case HccLTaskMetaType::EVENT_WAIT: | 377 | case HccLTaskMetaType::EVENT_WAIT: |
| 323 | case HccLTaskMetaType::EVENT_RECORD: | 378 | case HccLTaskMetaType::EVENT_RECORD: |
| 324 | default: | 379 | default: |
| 325 | - HCCL_VM_WARN("[TaskMetaTranslatorV3] Unsupported task meta type: {}", | 380 | + HCCL_VM_WARN("{} This task type is not supported for CheckerV3 graph generation, " |
| 326 | - static_cast<int32_t>(taskMeta.taskType)); | 381 | + "taskIndex={}, taskType={}, rankId={}, streamId={}, taskMeta={}", |
| 382 | + MakeErrorCodeText(ErrorCode::GRAPH_UNSUPPORTED), taskIndex, static_cast<int32_t>(taskMeta.taskType), | ||
| 383 | + taskMeta.rankId, taskMeta.streamId, DescribeTaskMetaForLog(taskMeta)); | ||
| 327 | return HCCL_E_NOT_SUPPORT; | 384 | return HCCL_E_NOT_SUPPORT; |
| 328 | } | 385 | } |
| 329 | } | 386 | } |
Mtest/hccl_vm/src/plugin/checker/src/framework/task_graph_generator_v3/task_meta_translator_v3.h+2-1
| @@ -59,7 +59,8 @@ private: | |||
| 59 | } | 59 | } |
| 60 | }; | 60 | }; |
| 61 | 61 | ||
| 62 | - HcclResult TranslateOneTaskMeta(const HcclTaskMetaData &taskMeta, StorageManager &storage, NodeId &nodeId); | 62 | + HcclResult TranslateOneTaskMeta(const HcclTaskMetaData &taskMeta, StorageManager &storage, uint32_t taskIndex, |
| 63 | + NodeId &nodeId); | ||
| 63 | HcclResult AddTaskNode(const TaskPosition &position, std::unique_ptr<TaskNode> node, NodeId &nodeId); | 64 | HcclResult AddTaskNode(const TaskPosition &position, std::unique_ptr<TaskNode> node, NodeId &nodeId); |
| 64 | 65 | ||
| 65 | std::vector<std::unique_ptr<TaskNode>> nodes_; | 66 | std::vector<std::unique_ptr<TaskNode>> nodes_; |
| @@ -30,19 +30,19 @@ void DlogPrintStub(int level, const char *msgBuffer) | |||
| 30 | { | 30 | { |
| 31 | switch (level) { | 31 | switch (level) { |
| 32 | case 0: | 32 | case 0: |
| 33 | - HCCL_VM_DEBUG(msgBuffer); | 33 | + HCCL_VM_DEBUG("{}", msgBuffer); |
| 34 | break; | 34 | break; |
| 35 | case 1: | 35 | case 1: |
| 36 | - HCCL_VM_INFO(msgBuffer); | 36 | + HCCL_VM_INFO("{}", msgBuffer); |
| 37 | break; | 37 | break; |
| 38 | case 2: | 38 | case 2: |
| 39 | - HCCL_VM_WARN(msgBuffer); | 39 | + HCCL_VM_WARN("{}", msgBuffer); |
| 40 | break; | 40 | break; |
| 41 | case 3: | 41 | case 3: |
| 42 | - HCCL_VM_ERROR(msgBuffer); | 42 | + HCCL_VM_ERROR("{}", msgBuffer); |
| 43 | break; | 43 | break; |
| 44 | default: | 44 | default: |
| 45 | - HCCL_VM_TRACE(msgBuffer); | 45 | + HCCL_VM_TRACE("{}", msgBuffer); |
| 46 | } | 46 | } |
| 47 | } | 47 | } |
| 48 | 48 | ||
| @@ -45,7 +45,7 @@ HcclResult FileHeaderRead(FILE *fp, FileHeader &header, uint32_t magic) | |||
| 45 | return HcclResult::HCCL_E_INTERNAL; | 45 | return HcclResult::HCCL_E_INTERNAL; |
| 46 | } | 46 | } |
| 47 | if (header.magic != magic) { | 47 | if (header.magic != magic) { |
| 48 | - HCCL_VM_ERROR("[FileHeaderRead] Unmatched magic number:{:#x}≠{:#x}", header.magic, magic); | 48 | + HCCL_VM_ERROR("Unmatched magic number:{:#x}≠{:#x}", header.magic, magic); |
| 49 | return HcclResult::HCCL_E_NOT_SUPPORT; | 49 | return HcclResult::HCCL_E_NOT_SUPPORT; |
| 50 | } | 50 | } |
| 51 | 51 | ||
| @@ -56,13 +56,13 @@ HcclResult HcclVmSynDataRead(FILE *fp, HcclVmSynData &synData, uint32_t magic) | |||
| 56 | { | 56 | { |
| 57 | auto ret = FileHeaderRead(fp, synData.header, magic); | 57 | auto ret = FileHeaderRead(fp, synData.header, magic); |
| 58 | if (ret != HcclResult::HCCL_SUCCESS) { | 58 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 59 | - HCCL_VM_ERROR("[HcclVmSynDataRead] Read file header failed. "); | 59 | + HCCL_VM_ERROR("Read file header failed. "); |
| 60 | return HcclResult::HCCL_E_INTERNAL; | 60 | return HcclResult::HCCL_E_INTERNAL; |
| 61 | } | 61 | } |
| 62 | 62 | ||
| 63 | ret = ModelInfoRead(fp, synData.model_info); | 63 | ret = ModelInfoRead(fp, synData.model_info); |
| 64 | if (ret != HcclResult::HCCL_SUCCESS) { | 64 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 65 | - HCCL_VM_ERROR("[HcclVmSynDataRead] Read file model info failed. "); | 65 | + HCCL_VM_ERROR("Read file model info failed. "); |
| 66 | return HcclResult::HCCL_E_INTERNAL; | 66 | return HcclResult::HCCL_E_INTERNAL; |
| 67 | } | 67 | } |
| 68 | 68 | ||
| @@ -71,7 +71,7 @@ HcclResult HcclVmSynDataRead(FILE *fp, HcclVmSynData &synData, uint32_t magic) | |||
| 71 | 71 | ||
| 72 | ret = MemLayoutRead(fp, synData.memory_info); | 72 | ret = MemLayoutRead(fp, synData.memory_info); |
| 73 | if (ret != HcclResult::HCCL_SUCCESS) { | 73 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 74 | - HCCL_VM_ERROR("[HcclVmSynDataRead] Read file memory layout info failed. "); | 74 | + HCCL_VM_ERROR("Read file memory layout info failed. "); |
| 75 | return HcclResult::HCCL_E_INTERNAL; | 75 | return HcclResult::HCCL_E_INTERNAL; |
| 76 | } | 76 | } |
| 77 | 77 | ||
| @@ -82,13 +82,13 @@ HcclResult HcclVmSynDataWrite(FILE *fp, const HcclVmSynData &synData) | |||
| 82 | { | 82 | { |
| 83 | auto ret = FileHeaderWrite(fp, synData.header); | 83 | auto ret = FileHeaderWrite(fp, synData.header); |
| 84 | if (ret != HcclResult::HCCL_SUCCESS) { | 84 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 85 | - HCCL_VM_ERROR("[HcclVmSynDataWrite] Write file header failed. "); | 85 | + HCCL_VM_ERROR("Write file header failed. "); |
| 86 | return HcclResult::HCCL_E_INTERNAL; | 86 | return HcclResult::HCCL_E_INTERNAL; |
| 87 | } | 87 | } |
| 88 | 88 | ||
| 89 | ret = ModelInfoWrite(fp, synData.model_info); | 89 | ret = ModelInfoWrite(fp, synData.model_info); |
| 90 | if (ret != HcclResult::HCCL_SUCCESS) { | 90 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 91 | - HCCL_VM_ERROR("[HcclVmSynDataWrite] Write file model info failed. "); | 91 | + HCCL_VM_ERROR("Write file model info failed. "); |
| 92 | return HcclResult::HCCL_E_INTERNAL; | 92 | return HcclResult::HCCL_E_INTERNAL; |
| 93 | } | 93 | } |
| 94 | 94 | ||
| @@ -97,7 +97,7 @@ HcclResult HcclVmSynDataWrite(FILE *fp, const HcclVmSynData &synData) | |||
| 97 | 97 | ||
| 98 | ret = MemLayoutWrite(fp, synData.memory_info); | 98 | ret = MemLayoutWrite(fp, synData.memory_info); |
| 99 | if (ret != HcclResult::HCCL_SUCCESS) { | 99 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 100 | - HCCL_VM_ERROR("[HcclVmSynDataWrite] Write file memory layout info failed. "); | 100 | + HCCL_VM_ERROR("Write file memory layout info failed. "); |
| 101 | return HcclResult::HCCL_E_INTERNAL; | 101 | return HcclResult::HCCL_E_INTERNAL; |
| 102 | } | 102 | } |
| 103 | 103 | ||
| @@ -108,7 +108,7 @@ HcclResult HcclVmInstrDataRead(FILE *fp, HcclVmInstrData &instrData, uint32_t ma | |||
| 108 | { | 108 | { |
| 109 | auto ret = FileHeaderRead(fp, instrData.header, magic); | 109 | auto ret = FileHeaderRead(fp, instrData.header, magic); |
| 110 | if (ret != HcclResult::HCCL_SUCCESS) { | 110 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 111 | - HCCL_VM_ERROR("[HcclVmInstrDataRead] Read file header failed. "); | 111 | + HCCL_VM_ERROR("Read file header failed. "); |
| 112 | return HcclResult::HCCL_E_INTERNAL; | 112 | return HcclResult::HCCL_E_INTERNAL; |
| 113 | } | 113 | } |
| 114 | 114 | ||
| @@ -116,7 +116,7 @@ HcclResult HcclVmInstrDataRead(FILE *fp, HcclVmInstrData &instrData, uint32_t ma | |||
| 116 | for (uint32_t i = 0; i < instrData.header.count; i++) { | 116 | for (uint32_t i = 0; i < instrData.header.count; i++) { |
| 117 | ret = MicrocodeInstrRead(fp, instrData.instr_data[i]); | 117 | ret = MicrocodeInstrRead(fp, instrData.instr_data[i]); |
| 118 | if (ret != HcclResult::HCCL_SUCCESS) { | 118 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 119 | - HCCL_VM_ERROR("[HcclVmInstrDataRead] Read file microcode instruction info failed. times= {}", i); | 119 | + HCCL_VM_ERROR("Read file microcode instruction info failed. times= {}", i); |
| 120 | return HcclResult::HCCL_E_INTERNAL; | 120 | return HcclResult::HCCL_E_INTERNAL; |
| 121 | } | 121 | } |
| 122 | } | 122 | } |
| @@ -128,14 +128,14 @@ HcclResult HcclVmInstrDataWrite(FILE *fp, const HcclVmInstrData &instrData) | |||
| 128 | { | 128 | { |
| 129 | auto ret = FileHeaderWrite(fp, instrData.header); | 129 | auto ret = FileHeaderWrite(fp, instrData.header); |
| 130 | if (ret != HcclResult::HCCL_SUCCESS) { | 130 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 131 | - HCCL_VM_ERROR("[HcclVmInstrDataWrite] Write file header failed. "); | 131 | + HCCL_VM_ERROR("Write file header failed. "); |
| 132 | return HcclResult::HCCL_E_INTERNAL; | 132 | return HcclResult::HCCL_E_INTERNAL; |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | for (uint32_t i = 0; i < instrData.header.count; i++) { | 135 | for (uint32_t i = 0; i < instrData.header.count; i++) { |
| 136 | ret = MicrocodeInstrWrite(fp, instrData.instr_data[i]); | 136 | ret = MicrocodeInstrWrite(fp, instrData.instr_data[i]); |
| 137 | if (ret != HcclResult::HCCL_SUCCESS) { | 137 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 138 | - HCCL_VM_ERROR("[HcclVmInstrDataWrite] Write file microcode instruction info failed. times= {}", i); | 138 | + HCCL_VM_ERROR("Write file microcode instruction info failed. times= {}", i); |
| 139 | return HcclResult::HCCL_E_INTERNAL; | 139 | return HcclResult::HCCL_E_INTERNAL; |
| 140 | } | 140 | } |
| 141 | } | 141 | } |
| @@ -268,19 +268,19 @@ HcclResult ModelInfoWrite(FILE *fp, const ModelInfoInner &modelInfo) | |||
| 268 | { | 268 | { |
| 269 | auto ret = ModelInfoCommWrite(fp, modelInfo.comm); | 269 | auto ret = ModelInfoCommWrite(fp, modelInfo.comm); |
| 270 | if (ret != HcclResult::HCCL_SUCCESS) { | 270 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 271 | - HCCL_VM_ERROR("[ModelInfoWrite] Write model info comm data failed. "); | 271 | + HCCL_VM_ERROR("Write model info comm data failed. "); |
| 272 | return HcclResult::HCCL_E_INTERNAL; | 272 | return HcclResult::HCCL_E_INTERNAL; |
| 273 | } | 273 | } |
| 274 | 274 | ||
| 275 | ret = VDataDesTagWrite(fp, modelInfo.vDataDes); | 275 | ret = VDataDesTagWrite(fp, modelInfo.vDataDes); |
| 276 | if (ret != HcclResult::HCCL_SUCCESS) { | 276 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 277 | - HCCL_VM_ERROR("[ModelInfoWrite] Write model info vdatades data failed. "); | 277 | + HCCL_VM_ERROR("Write model info vdatades data failed. "); |
| 278 | return HcclResult::HCCL_E_INTERNAL; | 278 | return HcclResult::HCCL_E_INTERNAL; |
| 279 | } | 279 | } |
| 280 | 280 | ||
| 281 | ret = All2AllDataDesTagWrite(fp, modelInfo.all2AllDataDes); | 281 | ret = All2AllDataDesTagWrite(fp, modelInfo.all2AllDataDes); |
| 282 | if (ret != HcclResult::HCCL_SUCCESS) { | 282 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 283 | - HCCL_VM_ERROR("[ModelInfoWrite] Write model info all2allDataDes data failed. "); | 283 | + HCCL_VM_ERROR("Write model info all2allDataDes data failed. "); |
| 284 | return HcclResult::HCCL_E_INTERNAL; | 284 | return HcclResult::HCCL_E_INTERNAL; |
| 285 | } | 285 | } |
| 286 | 286 | ||
| @@ -291,19 +291,19 @@ HcclResult ModelInfoRead(FILE *fp, ModelInfoInner &modelInfo) | |||
| 291 | { | 291 | { |
| 292 | auto ret = ModelInfoCommRead(fp, modelInfo.comm); | 292 | auto ret = ModelInfoCommRead(fp, modelInfo.comm); |
| 293 | if (ret != HcclResult::HCCL_SUCCESS) { | 293 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 294 | - HCCL_VM_ERROR("[ModelInfoRead] Read model info comm data failed "); | 294 | + HCCL_VM_ERROR("Read model info comm data failed "); |
| 295 | return HcclResult::HCCL_E_INTERNAL; | 295 | return HcclResult::HCCL_E_INTERNAL; |
| 296 | } | 296 | } |
| 297 | 297 | ||
| 298 | ret = VDataDesTagRead(fp, modelInfo.vDataDes); | 298 | ret = VDataDesTagRead(fp, modelInfo.vDataDes); |
| 299 | if (ret != HcclResult::HCCL_SUCCESS) { | 299 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 300 | - HCCL_VM_ERROR("[ModelInfoRead] Read model info vdatades data failed. "); | 300 | + HCCL_VM_ERROR("Read model info vdatades data failed. "); |
| 301 | return HcclResult::HCCL_E_INTERNAL; | 301 | return HcclResult::HCCL_E_INTERNAL; |
| 302 | } | 302 | } |
| 303 | 303 | ||
| 304 | ret = All2AllDataDesTagRead(fp, modelInfo.all2AllDataDes); | 304 | ret = All2AllDataDesTagRead(fp, modelInfo.all2AllDataDes); |
| 305 | if (ret != HcclResult::HCCL_SUCCESS) { | 305 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 306 | - HCCL_VM_ERROR("[ModelInfoRead] Read model info all2allDataDes data failed."); | 306 | + HCCL_VM_ERROR("Read model info all2allDataDes data failed."); |
| 307 | return HcclResult::HCCL_E_INTERNAL; | 307 | return HcclResult::HCCL_E_INTERNAL; |
| 308 | } | 308 | } |
| 309 | 309 | ||
| @@ -441,7 +441,7 @@ HcclResult HcclVmTaskMetaDataWrite(FILE *fp, const HcclVmTaskMetaData &taskMeta) | |||
| 441 | { | 441 | { |
| 442 | auto ret = FileHeaderWrite(fp, taskMeta.header); | 442 | auto ret = FileHeaderWrite(fp, taskMeta.header); |
| 443 | if (ret != HcclResult::HCCL_SUCCESS) { | 443 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 444 | - HCCL_VM_ERROR("[TaskMetaDataWrite] Read file header failed. "); | 444 | + HCCL_VM_ERROR("Read file header failed. "); |
| 445 | return HcclResult::HCCL_E_INTERNAL; | 445 | return HcclResult::HCCL_E_INTERNAL; |
| 446 | } | 446 | } |
| 447 | 447 | ||
| @@ -456,7 +456,7 @@ HcclResult HcclVmTaskMetaDataRead(FILE *fp, HcclVmTaskMetaData &taskMeta, uint32 | |||
| 456 | { | 456 | { |
| 457 | auto ret = FileHeaderRead(fp, taskMeta.header, magic); | 457 | auto ret = FileHeaderRead(fp, taskMeta.header, magic); |
| 458 | if (ret != HcclResult::HCCL_SUCCESS) { | 458 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 459 | - HCCL_VM_ERROR("[HcclVmTaskMetaDataRead] Read file header failed. "); | 459 | + HCCL_VM_ERROR("Read file header failed. "); |
| 460 | return HcclResult::HCCL_E_INTERNAL; | 460 | return HcclResult::HCCL_E_INTERNAL; |
| 461 | } | 461 | } |
| 462 | 462 | ||
| @@ -472,7 +472,7 @@ HcclResult HcclVmFlagDataWrite(FILE *fp, const HcclVmFlagData &flagData) | |||
| 472 | { | 472 | { |
| 473 | auto ret = FileHeaderWrite(fp, flagData.header); | 473 | auto ret = FileHeaderWrite(fp, flagData.header); |
| 474 | if (ret != HcclResult::HCCL_SUCCESS) { | 474 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 475 | - HCCL_VM_ERROR("[HcclVmFlagDataWrite] Read file header failed. "); | 475 | + HCCL_VM_ERROR("Read file header failed. "); |
| 476 | return HcclResult::HCCL_E_INTERNAL; | 476 | return HcclResult::HCCL_E_INTERNAL; |
| 477 | } | 477 | } |
| 478 | 478 | ||
| @@ -487,7 +487,7 @@ HcclResult HcclVmFlagDataRead(FILE *fp, HcclVmFlagData &flagData, uint32_t magic | |||
| 487 | { | 487 | { |
| 488 | auto ret = FileHeaderRead(fp, flagData.header, magic); | 488 | auto ret = FileHeaderRead(fp, flagData.header, magic); |
| 489 | if (ret != HcclResult::HCCL_SUCCESS) { | 489 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 490 | - HCCL_VM_ERROR("[HcclVmFlagDataRead] Read file header failed. "); | 490 | + HCCL_VM_ERROR("Read file header failed. "); |
| 491 | return HcclResult::HCCL_E_INTERNAL; | 491 | return HcclResult::HCCL_E_INTERNAL; |
| 492 | } | 492 | } |
| 493 | 493 | ||
| @@ -197,7 +197,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 197 | case TaskTypeStub::LOCAL_COPY: { | 197 | case TaskTypeStub::LOCAL_COPY: { |
| 198 | TaskStubLocalCopy *localCopyTask = dynamic_cast<TaskStubLocalCopy *>(task); | 198 | TaskStubLocalCopy *localCopyTask = dynamic_cast<TaskStubLocalCopy *>(task); |
| 199 | if (localCopyTask == nullptr) { | 199 | if (localCopyTask == nullptr) { |
| 200 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalCopy.", | 200 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalCopy.", |
| 201 | taskType); | 201 | taskType); |
| 202 | break; | 202 | break; |
| 203 | } | 203 | } |
| @@ -209,7 +209,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 209 | case TaskTypeStub::LOCAL_REDUCE: { | 209 | case TaskTypeStub::LOCAL_REDUCE: { |
| 210 | TaskStubLocalReduce *localReduceTask = dynamic_cast<TaskStubLocalReduce *>(task); | 210 | TaskStubLocalReduce *localReduceTask = dynamic_cast<TaskStubLocalReduce *>(task); |
| 211 | if (localReduceTask == nullptr) { | 211 | if (localReduceTask == nullptr) { |
| 212 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalReduce.", | 212 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalReduce.", |
| 213 | taskType); | 213 | taskType); |
| 214 | break; | 214 | break; |
| 215 | } | 215 | } |
| @@ -223,7 +223,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 223 | case TaskTypeStub::LOCAL_BATCH_REDUCE: { | 223 | case TaskTypeStub::LOCAL_BATCH_REDUCE: { |
| 224 | TaskStubLocalBatchReduce *localBatchReduceTask = dynamic_cast<TaskStubLocalBatchReduce *>(task); | 224 | TaskStubLocalBatchReduce *localBatchReduceTask = dynamic_cast<TaskStubLocalBatchReduce *>(task); |
| 225 | if (localBatchReduceTask == nullptr) { | 225 | if (localBatchReduceTask == nullptr) { |
| 226 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalBatchReduce.", | 226 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalBatchReduce.", |
| 227 | taskType); | 227 | taskType); |
| 228 | break; | 228 | break; |
| 229 | } | 229 | } |
| @@ -239,7 +239,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 239 | case TaskTypeStub::LOCAL_POST_TO: { | 239 | case TaskTypeStub::LOCAL_POST_TO: { |
| 240 | TaskStubLocalPostTo *localPostToTask = dynamic_cast<TaskStubLocalPostTo *>(task); | 240 | TaskStubLocalPostTo *localPostToTask = dynamic_cast<TaskStubLocalPostTo *>(task); |
| 241 | if (localPostToTask == nullptr) { | 241 | if (localPostToTask == nullptr) { |
| 242 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalPostTo.", | 242 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalPostTo.", |
| 243 | taskType); | 243 | taskType); |
| 244 | break; | 244 | break; |
| 245 | } | 245 | } |
| @@ -255,7 +255,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 255 | case TaskTypeStub::LOCAL_WAIT_FROM: { | 255 | case TaskTypeStub::LOCAL_WAIT_FROM: { |
| 256 | TaskStubLocalWaitFrom *localWaitFromTask = dynamic_cast<TaskStubLocalWaitFrom *>(task); | 256 | TaskStubLocalWaitFrom *localWaitFromTask = dynamic_cast<TaskStubLocalWaitFrom *>(task); |
| 257 | if (localWaitFromTask == nullptr) { | 257 | if (localWaitFromTask == nullptr) { |
| 258 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalWaitFrom.", | 258 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalWaitFrom.", |
| 259 | taskType); | 259 | taskType); |
| 260 | break; | 260 | break; |
| 261 | } | 261 | } |
| @@ -265,7 +265,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 265 | case TaskTypeStub::POST: { | 265 | case TaskTypeStub::POST: { |
| 266 | TaskStubPost *postTask = dynamic_cast<TaskStubPost *>(task); | 266 | TaskStubPost *postTask = dynamic_cast<TaskStubPost *>(task); |
| 267 | if (postTask == nullptr) { | 267 | if (postTask == nullptr) { |
| 268 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubPost.", | 268 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubPost.", |
| 269 | taskType); | 269 | taskType); |
| 270 | break; | 270 | break; |
| 271 | } | 271 | } |
| @@ -279,7 +279,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 279 | case TaskTypeStub::WAIT: { | 279 | case TaskTypeStub::WAIT: { |
| 280 | TaskStubWait *waitTask = dynamic_cast<TaskStubWait *>(task); | 280 | TaskStubWait *waitTask = dynamic_cast<TaskStubWait *>(task); |
| 281 | if (waitTask == nullptr) { | 281 | if (waitTask == nullptr) { |
| 282 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubWait.", | 282 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubWait.", |
| 283 | taskType); | 283 | taskType); |
| 284 | break; | 284 | break; |
| 285 | } | 285 | } |
| @@ -292,7 +292,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 292 | case TaskTypeStub::READ: { | 292 | case TaskTypeStub::READ: { |
| 293 | TaskStubRead *readTask = dynamic_cast<TaskStubRead *>(task); | 293 | TaskStubRead *readTask = dynamic_cast<TaskStubRead *>(task); |
| 294 | if (readTask == nullptr) { | 294 | if (readTask == nullptr) { |
| 295 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubRead.", | 295 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubRead.", |
| 296 | taskType); | 296 | taskType); |
| 297 | break; | 297 | break; |
| 298 | } | 298 | } |
| @@ -306,7 +306,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 306 | case TaskTypeStub::READ_REDUCE: { | 306 | case TaskTypeStub::READ_REDUCE: { |
| 307 | TaskStubReadReduce *readReduceTask = dynamic_cast<TaskStubReadReduce *>(task); | 307 | TaskStubReadReduce *readReduceTask = dynamic_cast<TaskStubReadReduce *>(task); |
| 308 | if (readReduceTask == nullptr) { | 308 | if (readReduceTask == nullptr) { |
| 309 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubReadReduce.", | 309 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubReadReduce.", |
| 310 | taskType); | 310 | taskType); |
| 311 | break; | 311 | break; |
| 312 | } | 312 | } |
| @@ -322,7 +322,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 322 | case TaskTypeStub::WRITE: { | 322 | case TaskTypeStub::WRITE: { |
| 323 | TaskStubWrite *writeTask = dynamic_cast<TaskStubWrite *>(task); | 323 | TaskStubWrite *writeTask = dynamic_cast<TaskStubWrite *>(task); |
| 324 | if (writeTask == nullptr) { | 324 | if (writeTask == nullptr) { |
| 325 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubWrite.", | 325 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubWrite.", |
| 326 | taskType); | 326 | taskType); |
| 327 | break; | 327 | break; |
| 328 | } | 328 | } |
| @@ -336,7 +336,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 336 | case TaskTypeStub::WRITE_REDUCE: { | 336 | case TaskTypeStub::WRITE_REDUCE: { |
| 337 | TaskStubWriteReduce *writeReduceTask = dynamic_cast<TaskStubWriteReduce *>(task); | 337 | TaskStubWriteReduce *writeReduceTask = dynamic_cast<TaskStubWriteReduce *>(task); |
| 338 | if (writeReduceTask == nullptr) { | 338 | if (writeReduceTask == nullptr) { |
| 339 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubWriteReduce.", | 339 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubWriteReduce.", |
| 340 | taskType); | 340 | taskType); |
| 341 | break; | 341 | break; |
| 342 | } | 342 | } |
| @@ -352,7 +352,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 352 | case TaskTypeStub::LOCAL_POST_TO_SHADOW: { | 352 | case TaskTypeStub::LOCAL_POST_TO_SHADOW: { |
| 353 | TaskStubLocalPostToShadow *localPostToShadowTask = dynamic_cast<TaskStubLocalPostToShadow *>(task); | 353 | TaskStubLocalPostToShadow *localPostToShadowTask = dynamic_cast<TaskStubLocalPostToShadow *>(task); |
| 354 | if (localPostToShadowTask == nullptr) { | 354 | if (localPostToShadowTask == nullptr) { |
| 355 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalPostToShadow.", | 355 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalPostToShadow.", |
| 356 | taskType); | 356 | taskType); |
| 357 | break; | 357 | break; |
| 358 | } | 358 | } |
| @@ -362,7 +362,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 362 | case TaskTypeStub::LOCAL_WAIT_FROM_SHADOW: { | 362 | case TaskTypeStub::LOCAL_WAIT_FROM_SHADOW: { |
| 363 | TaskStubLocalWaitFromShadow *localWaitFromShadowTask = dynamic_cast<TaskStubLocalWaitFromShadow *>(task); | 363 | TaskStubLocalWaitFromShadow *localWaitFromShadowTask = dynamic_cast<TaskStubLocalWaitFromShadow *>(task); |
| 364 | if (localWaitFromShadowTask == nullptr) { | 364 | if (localWaitFromShadowTask == nullptr) { |
| 365 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLocalWaitFromShadow.", | 365 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLocalWaitFromShadow.", |
| 366 | taskType); | 366 | taskType); |
| 367 | break; | 367 | break; |
| 368 | } | 368 | } |
| @@ -372,7 +372,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 372 | case TaskTypeStub::BEING_READ: { | 372 | case TaskTypeStub::BEING_READ: { |
| 373 | TaskStubBeingRead *beingReadTask = dynamic_cast<TaskStubBeingRead *>(task); | 373 | TaskStubBeingRead *beingReadTask = dynamic_cast<TaskStubBeingRead *>(task); |
| 374 | if (beingReadTask == nullptr) { | 374 | if (beingReadTask == nullptr) { |
| 375 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubBeingRead.", | 375 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubBeingRead.", |
| 376 | taskType); | 376 | taskType); |
| 377 | break; | 377 | break; |
| 378 | } | 378 | } |
| @@ -386,7 +386,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 386 | case TaskTypeStub::BEING_READ_REDUCE: { | 386 | case TaskTypeStub::BEING_READ_REDUCE: { |
| 387 | TaskStubBeingReadReduce *beingReadReduceTask = dynamic_cast<TaskStubBeingReadReduce *>(task); | 387 | TaskStubBeingReadReduce *beingReadReduceTask = dynamic_cast<TaskStubBeingReadReduce *>(task); |
| 388 | if (beingReadReduceTask == nullptr) { | 388 | if (beingReadReduceTask == nullptr) { |
| 389 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubBeingReadReduce.", | 389 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubBeingReadReduce.", |
| 390 | taskType); | 390 | taskType); |
| 391 | break; | 391 | break; |
| 392 | } | 392 | } |
| @@ -402,7 +402,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 402 | case TaskTypeStub::BEING_WRITTEN: { | 402 | case TaskTypeStub::BEING_WRITTEN: { |
| 403 | TaskStubBeingWritten *beingWrittenTask = dynamic_cast<TaskStubBeingWritten *>(task); | 403 | TaskStubBeingWritten *beingWrittenTask = dynamic_cast<TaskStubBeingWritten *>(task); |
| 404 | if (beingWrittenTask == nullptr) { | 404 | if (beingWrittenTask == nullptr) { |
| 405 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubBeingWritten.", | 405 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubBeingWritten.", |
| 406 | taskType); | 406 | taskType); |
| 407 | break; | 407 | break; |
| 408 | } | 408 | } |
| @@ -416,7 +416,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 416 | case TaskTypeStub::BEING_WRITTEN_REDUCE: { | 416 | case TaskTypeStub::BEING_WRITTEN_REDUCE: { |
| 417 | TaskStubBeingWrittenReduce *beingWrittenReduceTask = dynamic_cast<TaskStubBeingWrittenReduce *>(task); | 417 | TaskStubBeingWrittenReduce *beingWrittenReduceTask = dynamic_cast<TaskStubBeingWrittenReduce *>(task); |
| 418 | if (beingWrittenReduceTask == nullptr) { | 418 | if (beingWrittenReduceTask == nullptr) { |
| 419 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubBeingWrittenReduce.", | 419 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubBeingWrittenReduce.", |
| 420 | taskType); | 420 | taskType); |
| 421 | break; | 421 | break; |
| 422 | } | 422 | } |
| @@ -432,7 +432,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 432 | case TaskTypeStub::CCU_GRAPH: { | 432 | case TaskTypeStub::CCU_GRAPH: { |
| 433 | TaskStubCcuGraph *ccuGraphTask = dynamic_cast<TaskStubCcuGraph *>(task); | 433 | TaskStubCcuGraph *ccuGraphTask = dynamic_cast<TaskStubCcuGraph *>(task); |
| 434 | if (ccuGraphTask == nullptr) { | 434 | if (ccuGraphTask == nullptr) { |
| 435 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubCcuGraph.", | 435 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubCcuGraph.", |
| 436 | taskType); | 436 | taskType); |
| 437 | break; | 437 | break; |
| 438 | } | 438 | } |
| @@ -444,7 +444,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 444 | case TaskTypeStub::LOOP_START: { | 444 | case TaskTypeStub::LOOP_START: { |
| 445 | TaskStubLoopStart *loopStartTask = dynamic_cast<TaskStubLoopStart *>(task); | 445 | TaskStubLoopStart *loopStartTask = dynamic_cast<TaskStubLoopStart *>(task); |
| 446 | if (loopStartTask == nullptr) { | 446 | if (loopStartTask == nullptr) { |
| 447 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLoopStart.", | 447 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLoopStart.", |
| 448 | taskType); | 448 | taskType); |
| 449 | break; | 449 | break; |
| 450 | } | 450 | } |
| @@ -455,7 +455,7 @@ static Json MakeTaskStubFields(TaskStub *task, const std::map<TaskNode *, std::s | |||
| 455 | case TaskTypeStub::LOOP_END: { | 455 | case TaskTypeStub::LOOP_END: { |
| 456 | TaskStubLoopEnd *loopEndTask = dynamic_cast<TaskStubLoopEnd *>(task); | 456 | TaskStubLoopEnd *loopEndTask = dynamic_cast<TaskStubLoopEnd *>(task); |
| 457 | if (loopEndTask == nullptr) { | 457 | if (loopEndTask == nullptr) { |
| 458 | - HCCL_VM_WARN("[MakeTaskStubFields] failed to cast task [{}] to TaskStubLoopEnd.", | 458 | + HCCL_VM_WARN("failed to cast task [{}] to TaskStubLoopEnd.", |
| 459 | taskType); | 459 | taskType); |
| 460 | break; | 460 | break; |
| 461 | } | 461 | } |
| @@ -568,7 +568,7 @@ void CollectTaskNodesWithGlobalIds(TaskNode *dummyStart, std::vector<TaskNode *> | |||
| 568 | } | 568 | } |
| 569 | std::string nodeId = node->task->GetTaskId(); | 569 | std::string nodeId = node->task->GetTaskId(); |
| 570 | if (nodeId.empty()) { | 570 | if (nodeId.empty()) { |
| 571 | - HCCL_VM_ERROR("[CollectTaskNodesWithGlobalIds] task id not found, fallback to local node id."); | 571 | + HCCL_VM_ERROR("task id not found, fallback to local node id."); |
| 572 | const uint64_t nodeIndex = rankNodeCounters[node->rankIdx]++; | 572 | const uint64_t nodeIndex = rankNodeCounters[node->rankIdx]++; |
| 573 | nodeId = BuildNodeId(node->rankIdx, nodeIndex); | 573 | nodeId = BuildNodeId(node->rankIdx, nodeIndex); |
| 574 | } | 574 | } |
| @@ -726,7 +726,7 @@ static nlohmann::json DumpTaskNodeToJson(TaskNode *node, const std::string &node | |||
| 726 | } | 726 | } |
| 727 | auto iter = nodeIdMap.find(parent); | 727 | auto iter = nodeIdMap.find(parent); |
| 728 | if (iter == nodeIdMap.end()) { | 728 | if (iter == nodeIdMap.end()) { |
| 729 | - HCCL_VM_ERROR("[TaskNodeToJson] parent node id not found."); | 729 | + HCCL_VM_ERROR("parent node id not found."); |
| 730 | continue; | 730 | continue; |
| 731 | } | 731 | } |
| 732 | nodeJson["parents"].push_back(iter->second); | 732 | nodeJson["parents"].push_back(iter->second); |
| @@ -740,7 +740,7 @@ static nlohmann::json DumpTaskNodeToJson(TaskNode *node, const std::string &node | |||
| 740 | } | 740 | } |
| 741 | auto iter = nodeIdMap.find(child); | 741 | auto iter = nodeIdMap.find(child); |
| 742 | if (iter == nodeIdMap.end()) { | 742 | if (iter == nodeIdMap.end()) { |
| 743 | - HCCL_VM_ERROR("[TaskNodeToJson] child node id not found."); | 743 | + HCCL_VM_ERROR("child node id not found."); |
| 744 | continue; | 744 | continue; |
| 745 | } | 745 | } |
| 746 | nodeJson["children"].push_back(iter->second); | 746 | nodeJson["children"].push_back(iter->second); |
| @@ -793,7 +793,7 @@ static Json DumpCcuSubGraph(TaskStubCcuGraph *ccuGraphTask, const std::map<TaskN | |||
| 793 | continue; | 793 | continue; |
| 794 | } | 794 | } |
| 795 | if (node->queIdx >= queueBuckets.size()) { | 795 | if (node->queIdx >= queueBuckets.size()) { |
| 796 | - HCCL_VM_WARN("[DumpCcuSubGraph] queue id[%u] is out of range[%zu], skip node.", | 796 | + HCCL_VM_WARN("queue id[{}] is out of range[{}], skip node.", |
| 797 | node->queIdx, queueBuckets.size()); | 797 | node->queIdx, queueBuckets.size()); |
| 798 | continue; | 798 | continue; |
| 799 | } | 799 | } |
| @@ -805,7 +805,7 @@ static Json DumpCcuSubGraph(TaskStubCcuGraph *ccuGraphTask, const std::map<TaskN | |||
| 805 | for (auto *node : queueBuckets[queueIdx]) { | 805 | for (auto *node : queueBuckets[queueIdx]) { |
| 806 | auto nodeIdIter = resolvedNodeIdMap->find(node); | 806 | auto nodeIdIter = resolvedNodeIdMap->find(node); |
| 807 | if (nodeIdIter == resolvedNodeIdMap->end()) { | 807 | if (nodeIdIter == resolvedNodeIdMap->end()) { |
| 808 | - HCCL_VM_ERROR("[DumpCcuSubGraph] node id not found."); | 808 | + HCCL_VM_ERROR("node id not found."); |
| 809 | continue; | 809 | continue; |
| 810 | } | 810 | } |
| 811 | queueJson.push_back(DumpTaskNodeToJson(node, nodeIdIter->second, *resolvedNodeIdMap, | 811 | queueJson.push_back(DumpTaskNodeToJson(node, nodeIdIter->second, *resolvedNodeIdMap, |
| @@ -869,7 +869,7 @@ static std::map<RankId, Json> BuildTaskGraphJsonMap(TaskNode *dummyStart) | |||
| 869 | for (auto *currentNode : graphNodes) { | 869 | for (auto *currentNode : graphNodes) { |
| 870 | auto currentNodeIdIter = nodeIdMap.find(currentNode); | 870 | auto currentNodeIdIter = nodeIdMap.find(currentNode); |
| 871 | if (currentNodeIdIter == nodeIdMap.end()) { | 871 | if (currentNodeIdIter == nodeIdMap.end()) { |
| 872 | - HCCL_VM_ERROR("[DumpTaskGraph] current node id not found."); | 872 | + HCCL_VM_ERROR("current node id not found."); |
| 873 | continue; | 873 | continue; |
| 874 | } | 874 | } |
| 875 | 875 | ||
| @@ -889,7 +889,7 @@ static std::map<RankId, Json> BuildTaskGraphJsonMap(TaskNode *dummyStart) | |||
| 889 | } | 889 | } |
| 890 | auto parentIdIter = nodeIdMap.find(parent); | 890 | auto parentIdIter = nodeIdMap.find(parent); |
| 891 | if (parentIdIter == nodeIdMap.end()) { | 891 | if (parentIdIter == nodeIdMap.end()) { |
| 892 | - HCCL_VM_ERROR("[DumpTaskGraph] parent node id not found."); | 892 | + HCCL_VM_ERROR("parent node id not found."); |
| 893 | continue; | 893 | continue; |
| 894 | } | 894 | } |
| 895 | 895 | ||
| @@ -908,7 +908,7 @@ static std::map<RankId, Json> BuildTaskGraphJsonMap(TaskNode *dummyStart) | |||
| 908 | } | 908 | } |
| 909 | auto childIdIter = nodeIdMap.find(child); | 909 | auto childIdIter = nodeIdMap.find(child); |
| 910 | if (childIdIter == nodeIdMap.end()) { | 910 | if (childIdIter == nodeIdMap.end()) { |
| 911 | - HCCL_VM_ERROR("[DumpTaskGraph] child node id not found."); | 911 | + HCCL_VM_ERROR("child node id not found."); |
| 912 | continue; | 912 | continue; |
| 913 | } | 913 | } |
| 914 | 914 | ||
| @@ -926,7 +926,7 @@ static HcclResult DumpRankJsonMapByStage(const std::map<RankId, Json> &rankJsonM | |||
| 926 | const std::string &stage) | 926 | const std::string &stage) |
| 927 | { | 927 | { |
| 928 | if (stage.empty()) { | 928 | if (stage.empty()) { |
| 929 | - HCCL_VM_WARN("[DumpRankJsonMapByStage] stage is empty."); | 929 | + HCCL_VM_WARN("stage is empty."); |
| 930 | return HcclResult::HCCL_E_PARA; | 930 | return HcclResult::HCCL_E_PARA; |
| 931 | } | 931 | } |
| 932 | 932 | ||
| @@ -962,14 +962,14 @@ static HcclResult DumpRankJsonMapByStage(const std::map<RankId, Json> &rankJsonM | |||
| 962 | } | 962 | } |
| 963 | 963 | ||
| 964 | if (unlink(childPath.c_str()) != 0) { | 964 | if (unlink(childPath.c_str()) != 0) { |
| 965 | - HCCL_VM_WARN("[DumpRankJsonMapByStage] failed to remove file[%s], errno[%d].", | 965 | + HCCL_VM_WARN("failed to remove file[{}], errno[{}].", |
| 966 | childPath.c_str(), errno); | 966 | childPath.c_str(), errno); |
| 967 | } | 967 | } |
| 968 | } | 968 | } |
| 969 | 969 | ||
| 970 | closedir(dir); | 970 | closedir(dir); |
| 971 | if (rmdir(path.c_str()) != 0 && errno != ENOENT) { | 971 | if (rmdir(path.c_str()) != 0 && errno != ENOENT) { |
| 972 | - HCCL_VM_WARN("[DumpRankJsonMapByStage] failed to remove dir[%s], errno[%d].", | 972 | + HCCL_VM_WARN("failed to remove dir[{}], errno[{}].", |
| 973 | path.c_str(), errno); | 973 | path.c_str(), errno); |
| 974 | } | 974 | } |
| 975 | }; | 975 | }; |
| @@ -993,7 +993,7 @@ static HcclResult DumpRankJsonMapByStage(const std::map<RankId, Json> &rankJsonM | |||
| 993 | const std::string rankDumpPath = BuildStageRankDumpPath(stage, rankJsonPair.first); | 993 | const std::string rankDumpPath = BuildStageRankDumpPath(stage, rankJsonPair.first); |
| 994 | const HcclResult writeRet = dumpManager.Write(rankDumpPath, rankDumpJson); | 994 | const HcclResult writeRet = dumpManager.Write(rankDumpPath, rankDumpJson); |
| 995 | if (writeRet != HcclResult::HCCL_SUCCESS) { | 995 | if (writeRet != HcclResult::HCCL_SUCCESS) { |
| 996 | - HCCL_VM_WARN("[DumpRankJsonMapByStage] failed to dump rank[%u], stage[%s], ret[%u].", | 996 | + HCCL_VM_WARN("failed to dump rank[{}], stage[{}], ret[{}].", |
| 997 | rankJsonPair.first, stage.c_str(), static_cast<u32>(writeRet)); | 997 | rankJsonPair.first, stage.c_str(), static_cast<u32>(writeRet)); |
| 998 | return writeRet; | 998 | return writeRet; |
| 999 | } | 999 | } |
| @@ -25,21 +25,21 @@ HcclResult DumpManager::Initialize(const std::string& dataId) | |||
| 25 | { | 25 | { |
| 26 | std::lock_guard<std::mutex> lock(m_mutex); | 26 | std::lock_guard<std::mutex> lock(m_mutex); |
| 27 | if (dataId.empty()) { | 27 | if (dataId.empty()) { |
| 28 | - HCCL_VM_ERROR("[DumpManager::Initialize] dataId is empty."); | 28 | + HCCL_VM_ERROR("dataId is empty."); |
| 29 | return HcclResult::HCCL_E_PARA; | 29 | return HcclResult::HCCL_E_PARA; |
| 30 | } | 30 | } |
| 31 | 31 | ||
| 32 | m_dataId = dataId; | 32 | m_dataId = dataId; |
| 33 | m_pluginRootDir = HcclSim::GetCurrentPath(); | 33 | m_pluginRootDir = HcclSim::GetCurrentPath(); |
| 34 | if (m_pluginRootDir.empty()) { | 34 | if (m_pluginRootDir.empty()) { |
| 35 | - HCCL_VM_ERROR("[DumpManager::Initialize] failed to get current path."); | 35 | + HCCL_VM_ERROR("failed to get current path."); |
| 36 | return HcclResult::HCCL_E_INTERNAL; | 36 | return HcclResult::HCCL_E_INTERNAL; |
| 37 | } | 37 | } |
| 38 | m_dataDir = HcclSim::JoinPath(m_pluginRootDir, DATA_DIR_NAME); | 38 | m_dataDir = HcclSim::JoinPath(m_pluginRootDir, DATA_DIR_NAME); |
| 39 | m_dumpRootDir = HcclSim::JoinPath(HcclSim::JoinPath(m_dataDir, INSIGHT_DIR_NAME), m_dataId); | 39 | m_dumpRootDir = HcclSim::JoinPath(HcclSim::JoinPath(m_dataDir, INSIGHT_DIR_NAME), m_dataId); |
| 40 | 40 | ||
| 41 | if (!SettingManager::GetInstance().IsInsightDumpEnabled()) { | 41 | if (!SettingManager::GetInstance().IsInsightDumpEnabled()) { |
| 42 | - HCCL_VM_INFO("[DumpManager] insight dump disabled by manifest setting."); | 42 | + HCCL_VM_INFO("insight dump disabled by manifest setting."); |
| 43 | return HcclResult::HCCL_SUCCESS; | 43 | return HcclResult::HCCL_SUCCESS; |
| 44 | } | 44 | } |
| 45 | return PrepareDumpDirs(); | 45 | return PrepareDumpDirs(); |
| @@ -141,7 +141,7 @@ HcclResult DumpManager::WriteMsgpackFile(const std::string& dumpRootDir, const s | |||
| 141 | 141 | ||
| 142 | std::ofstream out(fullPath.c_str(), std::ios::out | std::ios::trunc | std::ios::binary); | 142 | std::ofstream out(fullPath.c_str(), std::ios::out | std::ios::trunc | std::ios::binary); |
| 143 | if (!out.is_open()) { | 143 | if (!out.is_open()) { |
| 144 | - HCCL_VM_ERROR("[DumpManager::WriteMsgpackFile] failed to open file: {}", fullPath); | 144 | + HCCL_VM_ERROR("failed to open file: {}", fullPath); |
| 145 | return HcclResult::HCCL_E_INTERNAL; | 145 | return HcclResult::HCCL_E_INTERNAL; |
| 146 | } | 146 | } |
| 147 | 147 | ||
| @@ -149,7 +149,7 @@ HcclResult DumpManager::WriteMsgpackFile(const std::string& dumpRootDir, const s | |||
| 149 | try { | 149 | try { |
| 150 | nlohmann::json::to_msgpack(document, nlohmann::detail::output_adapter<char>(out)); | 150 | nlohmann::json::to_msgpack(document, nlohmann::detail::output_adapter<char>(out)); |
| 151 | } catch (const std::exception &ex) { | 151 | } catch (const std::exception &ex) { |
| 152 | - HCCL_VM_ERROR("[DumpManager::WriteMsgpackFile] stream msgpack serialize failed: {}, file: {}", ex.what(), | 152 | + HCCL_VM_ERROR("stream msgpack serialize failed: {}, file: {}", ex.what(), |
| 153 | fullPath); | 153 | fullPath); |
| 154 | return HcclResult::HCCL_E_INTERNAL; | 154 | return HcclResult::HCCL_E_INTERNAL; |
| 155 | } | 155 | } |
| @@ -158,7 +158,7 @@ HcclResult DumpManager::WriteMsgpackFile(const std::string& dumpRootDir, const s | |||
| 158 | const uint64_t writeBeginMs = NowMs(); | 158 | const uint64_t writeBeginMs = NowMs(); |
| 159 | out.flush(); | 159 | out.flush(); |
| 160 | if (!out.good()) { | 160 | if (!out.good()) { |
| 161 | - HCCL_VM_ERROR("[DumpManager::WriteMsgpackFile] failed to flush file: {}", fullPath); | 161 | + HCCL_VM_ERROR("failed to flush file: {}", fullPath); |
| 162 | return HcclResult::HCCL_E_INTERNAL; | 162 | return HcclResult::HCCL_E_INTERNAL; |
| 163 | } | 163 | } |
| 164 | const uint64_t writeCostMs = NowMs() - writeBeginMs; | 164 | const uint64_t writeCostMs = NowMs() - writeBeginMs; |
| @@ -167,7 +167,7 @@ HcclResult DumpManager::WriteMsgpackFile(const std::string& dumpRootDir, const s | |||
| 167 | static_cast<uint64_t>(finalPos) : 0; | 167 | static_cast<uint64_t>(finalPos) : 0; |
| 168 | 168 | ||
| 169 | if (relativePath.compare(0, std::string("memory/").size(), "memory/") == 0) { | 169 | if (relativePath.compare(0, std::string("memory/").size(), "memory/") == 0) { |
| 170 | - HCCL_VM_INFO("[DumpManager::WriteMsgpackFile][perf] path={}, ensure_dir_ms={}, serialize_ms={}, " | 170 | + HCCL_VM_INFO("path={}, ensure_dir_ms={}, serialize_ms={}, " |
| 171 | "write_file_ms={}, msgpack_bytes={}", relativePath, ensureDirCostMs, serializeCostMs, writeCostMs, | 171 | "write_file_ms={}, msgpack_bytes={}", relativePath, ensureDirCostMs, serializeCostMs, writeCostMs, |
| 172 | msgpackBytes); | 172 | msgpackBytes); |
| 173 | } | 173 | } |
| @@ -185,19 +185,19 @@ HcclResult DumpManager::WriteJsonFile(const std::string& dumpRootDir, const std: | |||
| 185 | 185 | ||
| 186 | std::ofstream out(fullPath.c_str(), std::ios::out | std::ios::trunc); | 186 | std::ofstream out(fullPath.c_str(), std::ios::out | std::ios::trunc); |
| 187 | if (!out.is_open()) { | 187 | if (!out.is_open()) { |
| 188 | - HCCL_VM_ERROR("[DumpManager::WriteJsonFile] failed to open file: {}", fullPath); | 188 | + HCCL_VM_ERROR("failed to open file: {}", fullPath); |
| 189 | return HcclResult::HCCL_E_INTERNAL; | 189 | return HcclResult::HCCL_E_INTERNAL; |
| 190 | } | 190 | } |
| 191 | 191 | ||
| 192 | try { | 192 | try { |
| 193 | out << document.dump(2) << std::endl; | 193 | out << document.dump(2) << std::endl; |
| 194 | } catch (const std::exception &ex) { | 194 | } catch (const std::exception &ex) { |
| 195 | - HCCL_VM_ERROR("[DumpManager::WriteJsonFile] json serialize failed: {}, file: {}", ex.what(), fullPath); | 195 | + HCCL_VM_ERROR("json serialize failed: {}, file: {}", ex.what(), fullPath); |
| 196 | return HcclResult::HCCL_E_INTERNAL; | 196 | return HcclResult::HCCL_E_INTERNAL; |
| 197 | } | 197 | } |
| 198 | 198 | ||
| 199 | if (!out.good()) { | 199 | if (!out.good()) { |
| 200 | - HCCL_VM_ERROR("[DumpManager::WriteJsonFile] failed to write file: {}", fullPath); | 200 | + HCCL_VM_ERROR("failed to write file: {}", fullPath); |
| 201 | return HcclResult::HCCL_E_INTERNAL; | 201 | return HcclResult::HCCL_E_INTERNAL; |
| 202 | } | 202 | } |
| 203 | return HcclResult::HCCL_SUCCESS; | 203 | return HcclResult::HCCL_SUCCESS; |
| @@ -221,7 +221,7 @@ HcclResult DumpManager::WriteMsgpackStreamFile(const std::string& dumpRootDir, c | |||
| 221 | 221 | ||
| 222 | std::ofstream out(fullPath.c_str(), std::ios::out | std::ios::trunc | std::ios::binary); | 222 | std::ofstream out(fullPath.c_str(), std::ios::out | std::ios::trunc | std::ios::binary); |
| 223 | if (!out.is_open()) { | 223 | if (!out.is_open()) { |
| 224 | - HCCL_VM_ERROR("[DumpManager::WriteMsgpackStreamFile] failed to open file: {}", fullPath); | 224 | + HCCL_VM_ERROR("failed to open file: {}", fullPath); |
| 225 | return HcclResult::HCCL_E_INTERNAL; | 225 | return HcclResult::HCCL_E_INTERNAL; |
| 226 | } | 226 | } |
| 227 | 227 | ||
| @@ -229,14 +229,14 @@ HcclResult DumpManager::WriteMsgpackStreamFile(const std::string& dumpRootDir, c | |||
| 229 | ret = writer(out); | 229 | ret = writer(out); |
| 230 | const uint64_t serializeCostMs = NowMs() - serializeBeginMs; | 230 | const uint64_t serializeCostMs = NowMs() - serializeBeginMs; |
| 231 | if (ret != HcclResult::HCCL_SUCCESS) { | 231 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 232 | - HCCL_VM_ERROR("[DumpManager::WriteMsgpackStreamFile] custom writer failed. path: {}", fullPath); | 232 | + HCCL_VM_ERROR("custom writer failed. path: {}", fullPath); |
| 233 | return ret; | 233 | return ret; |
| 234 | } | 234 | } |
| 235 | 235 | ||
| 236 | const uint64_t writeBeginMs = NowMs(); | 236 | const uint64_t writeBeginMs = NowMs(); |
| 237 | out.flush(); | 237 | out.flush(); |
| 238 | if (!out.good()) { | 238 | if (!out.good()) { |
| 239 | - HCCL_VM_ERROR("[DumpManager::WriteMsgpackStreamFile] failed to flush file: {}", fullPath); | 239 | + HCCL_VM_ERROR("failed to flush file: {}", fullPath); |
| 240 | return HcclResult::HCCL_E_INTERNAL; | 240 | return HcclResult::HCCL_E_INTERNAL; |
| 241 | } | 241 | } |
| 242 | const uint64_t writeCostMs = NowMs() - writeBeginMs; | 242 | const uint64_t writeCostMs = NowMs() - writeBeginMs; |
| @@ -245,7 +245,7 @@ HcclResult DumpManager::WriteMsgpackStreamFile(const std::string& dumpRootDir, c | |||
| 245 | static_cast<uint64_t>(finalPos) : 0; | 245 | static_cast<uint64_t>(finalPos) : 0; |
| 246 | 246 | ||
| 247 | if (relativePath.compare(0, std::string("memory/").size(), "memory/") == 0) { | 247 | if (relativePath.compare(0, std::string("memory/").size(), "memory/") == 0) { |
| 248 | - HCCL_VM_INFO("[DumpManager::WriteMsgpackStreamFile][perf] path={}, ensure_dir_ms={}, serialize_ms={}, " | 248 | + HCCL_VM_INFO("path={}, ensure_dir_ms={}, serialize_ms={}, " |
| 249 | "write_file_ms={}, msgpack_bytes={}", relativePath, ensureDirCostMs, serializeCostMs, writeCostMs, | 249 | "write_file_ms={}, msgpack_bytes={}", relativePath, ensureDirCostMs, serializeCostMs, writeCostMs, |
| 250 | msgpackBytes); | 250 | msgpackBytes); |
| 251 | } | 251 | } |
| @@ -285,7 +285,7 @@ HcclResult MemoryTimelineDumpStream::Initialize(const std::vector<RankId> &rankI | |||
| 285 | 285 | ||
| 286 | const EventIdToEventMap eventIdToEvent = BuildEventIdToEventMap(timelineEvents); | 286 | const EventIdToEventMap eventIdToEvent = BuildEventIdToEventMap(timelineEvents); |
| 287 | m_globalStepToEvent = BuildGlobalStepToEventMap(globalStepToEventId, eventIdToEvent); | 287 | m_globalStepToEvent = BuildGlobalStepToEventMap(globalStepToEventId, eventIdToEvent); |
| 288 | - HCCL_VM_INFO("[DumpMemoryTimelines] start dump memory timeline, rank_count={}, event_count={}, step_event_count={}", | 288 | + HCCL_VM_INFO("start dump memory timeline, rank_count={}, event_count={}, step_event_count={}", |
| 289 | rankIds.size(), eventIdToEvent.size(), m_globalStepToEvent.size()); | 289 | rankIds.size(), eventIdToEvent.size(), m_globalStepToEvent.size()); |
| 290 | 290 | ||
| 291 | for (RankId rankId : rankIds) { | 291 | for (RankId rankId : rankIds) { |
| @@ -312,7 +312,7 @@ HcclResult MemoryTimelineDumpStream::FlushOpenChunk(RankId rankId, RankDumpState | |||
| 312 | rankDumpState.openChunkSnapshotEntries); | 312 | rankDumpState.openChunkSnapshotEntries); |
| 313 | }); | 313 | }); |
| 314 | if (ret != HcclResult::HCCL_SUCCESS) { | 314 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 315 | - HCCL_VM_WARN("[DumpMemoryTimelines] failed to dump rank [%u] snapshot chunk [%zu].", rankId, chunkId); | 315 | + HCCL_VM_WARN("failed to dump rank [{}] snapshot chunk [{}].", rankId, chunkId); |
| 316 | return ret; | 316 | return ret; |
| 317 | } | 317 | } |
| 318 | 318 | ||
| @@ -397,7 +397,7 @@ HcclResult MemoryTimelineDumpStream::Finalize() | |||
| 397 | const std::string snapshotManifestPath = BuildRankSnapshotManifestPath(rankId); | 397 | const std::string snapshotManifestPath = BuildRankSnapshotManifestPath(rankId); |
| 398 | ret = DumpManager::GetInstance().Write(snapshotManifestPath, snapshotManifestJson); | 398 | ret = DumpManager::GetInstance().Write(snapshotManifestPath, snapshotManifestJson); |
| 399 | if (ret != HcclResult::HCCL_SUCCESS) { | 399 | if (ret != HcclResult::HCCL_SUCCESS) { |
| 400 | - HCCL_VM_WARN("[DumpMemoryTimelines] failed to dump rank [%u] snapshot manifest.", rankId); | 400 | + HCCL_VM_WARN("failed to dump rank [{}] snapshot manifest.", rankId); |
| 401 | return ret; | 401 | return ret; |
| 402 | } | 402 | } |
| 403 | 403 | ||
| @@ -413,7 +413,7 @@ HcclResult MemoryTimelineDumpStream::Finalize() | |||
| 413 | memoryStatsJson["snapshot_count"] = totalSnapshotCount; | 413 | memoryStatsJson["snapshot_count"] = totalSnapshotCount; |
| 414 | memoryStatsJson["chunk_count"] = totalChunkCount; | 414 | memoryStatsJson["chunk_count"] = totalChunkCount; |
| 415 | DumpRunManifest::GetInstance().SetMemorySnapshotStats(memoryStatsJson); | 415 | DumpRunManifest::GetInstance().SetMemorySnapshotStats(memoryStatsJson); |
| 416 | - HCCL_VM_INFO("[DumpMemoryTimelines] finish dump memory timeline, rank_count={}", m_rankDumpStates.size()); | 416 | + HCCL_VM_INFO("finish dump memory timeline, rank_count={}", m_rankDumpStates.size()); |
| 417 | return HcclResult::HCCL_SUCCESS; | 417 | return HcclResult::HCCL_SUCCESS; |
| 418 | } | 418 | } |
| 419 | } // namespace HcclSim | 419 | } // namespace HcclSim |
| @@ -25,6 +25,7 @@ | |||
| 25 | 25 | ||
| 26 | 26 | ||
| 27 | 27 | ||
| 28 | + | ||
| 28 | 29 | ||
| 29 | namespace HcclSim { | 30 | namespace HcclSim { |
| 30 | namespace { | 31 | namespace { |
| @@ -521,7 +522,9 @@ HcclResult CollectTopoNodes(const TaskGraphGeneratorV3::TaskGraphGeneratorV3 &gr | |||
| 521 | topoNodes.clear(); | 522 | topoNodes.clear(); |
| 522 | const TaskNode *mainStart = graph.GetMainStartNode(); | 523 | const TaskNode *mainStart = graph.GetMainStartNode(); |
| 523 | if (!IsMainGraphStartNode(mainStart)) { | 524 | if (!IsMainGraphStartNode(mainStart)) { |
| 524 | - HCCL_VM_ERROR("[DumpV3Dag] invalid V3 main start node."); | 525 | + HCCL_VM_ERROR("{} Main start node of the V3 graph is invalid, mainStartNodeId={}", |
| 526 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), | ||
| 527 | + mainStart == nullptr ? MAIN_START_NODE_ID : mainStart->GetNodeId()); | ||
| 525 | return HCCL_E_PARA; | 528 | return HCCL_E_PARA; |
| 526 | } | 529 | } |
| 527 | 530 | ||
| @@ -529,7 +532,9 @@ HcclResult CollectTopoNodes(const TaskGraphGeneratorV3::TaskGraphGeneratorV3 &gr | |||
| 529 | std::vector<NodeId> topoOrder; | 532 | std::vector<NodeId> topoOrder; |
| 530 | HcclResult ret = GenReachabilityClosure(mainStart, closure, &topoOrder); | 533 | HcclResult ret = GenReachabilityClosure(mainStart, closure, &topoOrder); |
| 531 | if (ret != HCCL_SUCCESS) { | 534 | if (ret != HCCL_SUCCESS) { |
| 532 | - HCCL_VM_ERROR("[DumpV3Dag] failed to build topo order, ret={}.", static_cast<uint32_t>(ret)); | 535 | + HCCL_VM_ERROR("{} Failed to build the V3 graph topological order for dump output, " |
| 536 | + "mainStartNodeId={}, ret={}", MakeErrorCodeText(ErrorCode::DUMP_FAILED), | ||
| 537 | + mainStart->GetNodeId(), static_cast<uint32_t>(ret)); | ||
| 533 | return ret; | 538 | return ret; |
| 534 | } | 539 | } |
| 535 | 540 | ||
| @@ -540,7 +545,8 @@ HcclResult CollectTopoNodes(const TaskGraphGeneratorV3::TaskGraphGeneratorV3 &gr | |||
| 540 | } | 545 | } |
| 541 | const TaskNode *node = graph.GetNode(nodeId); | 546 | const TaskNode *node = graph.GetNode(nodeId); |
| 542 | if (node == nullptr) { | 547 | if (node == nullptr) { |
| 543 | - HCCL_VM_ERROR("[DumpV3Dag] topo node is null, nodeId={}.", nodeId); | 548 | + HCCL_VM_ERROR("{} One node in topological order is missing, nodeId={}, " |
| 549 | + "processedTopoNodeCount={}", MakeErrorCodeText(ErrorCode::DUMP_FAILED), nodeId, topoNodes.size()); | ||
| 544 | return HCCL_E_PTR; | 550 | return HCCL_E_PTR; |
| 545 | } | 551 | } |
| 546 | topoNodes.push_back(node); | 552 | topoNodes.push_back(node); |
| @@ -616,7 +622,9 @@ HcclResult DumpV3Dag(const TaskGraphGeneratorV3::TaskGraphGeneratorV3 &graph, co | |||
| 616 | return HCCL_SUCCESS; | 622 | return HCCL_SUCCESS; |
| 617 | } | 623 | } |
| 618 | if (stage.empty()) { | 624 | if (stage.empty()) { |
| 619 | - HCCL_VM_ERROR("[DumpV3Dag] stage is empty."); | 625 | + HCCL_VM_ERROR("{} Dump stage name is empty, dataId={}, graphOutputPath={}, layoutOutputPath={}", |
| 626 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), StorageManager::GetInstance().GetDataId(), kGraphPath, | ||
| 627 | + kLayoutPath); | ||
| 620 | return HCCL_E_PARA; | 628 | return HCCL_E_PARA; |
| 621 | } | 629 | } |
| 622 | 630 | ||
| @@ -847,8 +855,8 @@ HcclResult DumpV3Dag(const TaskGraphGeneratorV3::TaskGraphGeneratorV3 &graph, co | |||
| 847 | MakeStageStats(graphDoc["lanes"].get<std::vector<Json>>(), | 855 | MakeStageStats(graphDoc["lanes"].get<std::vector<Json>>(), |
| 848 | graphDoc["nodes"].get<std::vector<Json>>(), graphDoc["edges"].get<std::vector<Json>>())); | 856 | graphDoc["nodes"].get<std::vector<Json>>(), graphDoc["edges"].get<std::vector<Json>>())); |
| 849 | 857 | ||
| 850 | - HCCL_VM_INFO("[DumpV3Dag] dumped V3 DAG, stage={}, laneCount={}, nodeCount={}, edgeCount={}", | 858 | + HCCL_VM_INFO("V3 DAG dump finished, laneCount={}, nodeCount={}, edgeCount={}", |
| 851 | - stage, laneIdByKey.size(), topoNodes.size(), edgeInfos.size()); | 859 | + laneIdByKey.size(), topoNodes.size(), edgeInfos.size()); |
| 852 | return HCCL_SUCCESS; | 860 | return HCCL_SUCCESS; |
| 853 | } | 861 | } |
| 854 | } // namespace HcclSim | 862 | } // namespace HcclSim |
| @@ -15,6 +15,7 @@ | |||
| 15 | 15 | ||
| 16 | 16 | ||
| 17 | 17 | ||
| 18 | + | ||
| 18 | 19 | ||
| 19 | namespace { | 20 | namespace { |
| 20 | const std::string kDataDirName = "data"; | 21 | const std::string kDataDirName = "data"; |
| @@ -26,14 +27,16 @@ HcclResult DumpV3Manager::Initialize(const std::string &dataId) | |||
| 26 | { | 27 | { |
| 27 | std::lock_guard<std::mutex> lock(m_mutex); | 28 | std::lock_guard<std::mutex> lock(m_mutex); |
| 28 | if (dataId.empty()) { | 29 | if (dataId.empty()) { |
| 29 | - HCCL_VM_ERROR("[DumpV3Manager::Initialize] dataId is empty."); | 30 | + HCCL_VM_ERROR("{} dataId is empty, the dump output path cannot be built, outputRoot=null", |
| 31 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED)); | ||
| 30 | return HcclResult::HCCL_E_PARA; | 32 | return HcclResult::HCCL_E_PARA; |
| 31 | } | 33 | } |
| 32 | 34 | ||
| 33 | m_dataId = dataId; | 35 | m_dataId = dataId; |
| 34 | m_pluginRootDir = HcclSim::GetCurrentPath(); | 36 | m_pluginRootDir = HcclSim::GetCurrentPath(); |
| 35 | if (m_pluginRootDir.empty()) { | 37 | if (m_pluginRootDir.empty()) { |
| 36 | - HCCL_VM_ERROR("[DumpV3Manager::Initialize] failed to get current path."); | 38 | + HCCL_VM_ERROR("{} Failed to get the current working directory for dump output, dataId={}", |
| 39 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), m_dataId); | ||
| 37 | return HcclResult::HCCL_E_INTERNAL; | 40 | return HcclResult::HCCL_E_INTERNAL; |
| 38 | } | 41 | } |
| 39 | 42 | ||
| @@ -41,7 +44,7 @@ HcclResult DumpV3Manager::Initialize(const std::string &dataId) | |||
| 41 | m_rootDir = HcclSim::JoinPath(HcclSim::JoinPath(m_dataDir, kInsightDirName), m_dataId); | 44 | m_rootDir = HcclSim::JoinPath(HcclSim::JoinPath(m_dataDir, kInsightDirName), m_dataId); |
| 42 | 45 | ||
| 43 | if (!SettingManager::GetInstance().IsInsightDumpEnabled()) { | 46 | if (!SettingManager::GetInstance().IsInsightDumpEnabled()) { |
| 44 | - HCCL_VM_INFO("[DumpV3Manager] insight dump disabled by manifest setting."); | 47 | + HCCL_VM_INFO("Insight dump is disabled by manifest setting"); |
| 45 | return HcclResult::HCCL_SUCCESS; | 48 | return HcclResult::HCCL_SUCCESS; |
| 46 | } | 49 | } |
| 47 | return PrepareDirs(); | 50 | return PrepareDirs(); |
| @@ -86,21 +89,23 @@ HcclResult DumpV3Manager::WriteMsgpack(const std::string &relativePath, const nl | |||
| 86 | 89 | ||
| 87 | std::ofstream out(fullPath, std::ios::out | std::ios::trunc | std::ios::binary); | 90 | std::ofstream out(fullPath, std::ios::out | std::ios::trunc | std::ios::binary); |
| 88 | if (!out.is_open()) { | 91 | if (!out.is_open()) { |
| 89 | - HCCL_VM_ERROR("[DumpV3Manager::WriteMsgpack] failed to open file: {}", fullPath); | 92 | + HCCL_VM_ERROR("{} Failed to open dump file, file={}, format=msgpack, dataId={}", |
| 93 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), fullPath, m_dataId); | ||
| 90 | return HcclResult::HCCL_E_INTERNAL; | 94 | return HcclResult::HCCL_E_INTERNAL; |
| 91 | } | 95 | } |
| 92 | 96 | ||
| 93 | try { | 97 | try { |
| 94 | nlohmann::json::to_msgpack(document, nlohmann::detail::output_adapter<char>(out)); | 98 | nlohmann::json::to_msgpack(document, nlohmann::detail::output_adapter<char>(out)); |
| 95 | } catch (const std::exception &ex) { | 99 | } catch (const std::exception &ex) { |
| 96 | - HCCL_VM_ERROR("[DumpV3Manager::WriteMsgpack] msgpack serialize failed: {}, file: {}", ex.what(), | 100 | + HCCL_VM_ERROR("{} Failed to serialize dump content into msgpack, file={}, dataId={}, reason={}", |
| 97 | - fullPath); | 101 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), fullPath, m_dataId, ex.what()); |
| 98 | return HcclResult::HCCL_E_INTERNAL; | 102 | return HcclResult::HCCL_E_INTERNAL; |
| 99 | } | 103 | } |
| 100 | 104 | ||
| 101 | out.flush(); | 105 | out.flush(); |
| 102 | if (!out.good()) { | 106 | if (!out.good()) { |
| 103 | - HCCL_VM_ERROR("[DumpV3Manager::WriteMsgpack] failed to flush file: {}", fullPath); | 107 | + HCCL_VM_ERROR("{} Failed to flush dump file to disk, file={}, format=msgpack, dataId={}", |
| 108 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), fullPath, m_dataId); | ||
| 104 | return HcclResult::HCCL_E_INTERNAL; | 109 | return HcclResult::HCCL_E_INTERNAL; |
| 105 | } | 110 | } |
| 106 | return HcclResult::HCCL_SUCCESS; | 111 | return HcclResult::HCCL_SUCCESS; |
| @@ -119,19 +124,22 @@ HcclResult DumpV3Manager::WriteJson(const std::string &relativePath, const nlohm | |||
| 119 | 124 | ||
| 120 | std::ofstream out(fullPath, std::ios::out | std::ios::trunc); | 125 | std::ofstream out(fullPath, std::ios::out | std::ios::trunc); |
| 121 | if (!out.is_open()) { | 126 | if (!out.is_open()) { |
| 122 | - HCCL_VM_ERROR("[DumpV3Manager::WriteJson] failed to open file: {}", fullPath); | 127 | + HCCL_VM_ERROR("{} Failed to open dump file, file={}, format=json, dataId={}", |
| 128 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), fullPath, m_dataId); | ||
| 123 | return HcclResult::HCCL_E_INTERNAL; | 129 | return HcclResult::HCCL_E_INTERNAL; |
| 124 | } | 130 | } |
| 125 | 131 | ||
| 126 | try { | 132 | try { |
| 127 | out << document.dump(2) << std::endl; | 133 | out << document.dump(2) << std::endl; |
| 128 | } catch (const std::exception &ex) { | 134 | } catch (const std::exception &ex) { |
| 129 | - HCCL_VM_ERROR("[DumpV3Manager::WriteJson] json serialize failed: {}, file: {}", ex.what(), fullPath); | 135 | + HCCL_VM_ERROR("{} Failed to serialize dump content into JSON, file={}, dataId={}, reason={}", |
| 136 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), fullPath, m_dataId, ex.what()); | ||
| 130 | return HcclResult::HCCL_E_INTERNAL; | 137 | return HcclResult::HCCL_E_INTERNAL; |
| 131 | } | 138 | } |
| 132 | 139 | ||
| 133 | if (!out.good()) { | 140 | if (!out.good()) { |
| 134 | - HCCL_VM_ERROR("[DumpV3Manager::WriteJson] failed to write file: {}", fullPath); | 141 | + HCCL_VM_ERROR("{} Failed to write dump file content, file={}, format=json, dataId={}", |
| 142 | + MakeErrorCodeText(ErrorCode::DUMP_FAILED), fullPath, m_dataId); | ||
| 135 | return HcclResult::HCCL_E_INTERNAL; | 143 | return HcclResult::HCCL_E_INTERNAL; |
| 136 | } | 144 | } |
| 137 | return HcclResult::HCCL_SUCCESS; | 145 | return HcclResult::HCCL_SUCCESS; |
| @@ -0,0 +1,69 @@ | |||
| 1 | +/** | ||
| 2 | + * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | + * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | + * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | + * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | + * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | + * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | + */ | ||
| 10 | + | ||
| 11 | + | ||
| 12 | + | ||
| 13 | + | ||
| 14 | + | ||
| 15 | + | ||
| 16 | + | ||
| 17 | +namespace HcclSim { | ||
| 18 | + | ||
| 19 | +enum class ErrorCode { | ||
| 20 | + GRAPH_TRANSLATE_FAILED = 101, | ||
| 21 | + GRAPH_DEADLOCK = 102, | ||
| 22 | + GRAPH_UNMATCHED = 103, | ||
| 23 | + GRAPH_MEMBER_MISSING = 104, | ||
| 24 | + GRAPH_STRUCTURE_INVALID = 105, | ||
| 25 | + GRAPH_SNAPSHOT_MISMATCH = 106, | ||
| 26 | + GRAPH_RESOURCE_NOT_FOUND = 107, | ||
| 27 | + GRAPH_REGISTER_UNINITIALIZED = 108, | ||
| 28 | + GRAPH_OUT_OF_RANGE = 109, | ||
| 29 | + GRAPH_ADDRESS_INVALID = 110, | ||
| 30 | + GRAPH_UNSUPPORTED = 111, | ||
| 31 | + GRAPH_REMOTE_RANK_MISMATCH = 112, | ||
| 32 | + GRAPH_LOOP_MERGE_ERROR = 113, | ||
| 33 | + | ||
| 34 | + SINGLETASK_SLICE_INVALID = 201, | ||
| 35 | + SINGLETASK_SLICE_CONFLICT = 202, | ||
| 36 | + SINGLETASK_SLAVE_STREAM_INVALID = 203, | ||
| 37 | + | ||
| 38 | + MEMCONFLICT_DAG_INVALID = 301, | ||
| 39 | + MEMCONFLICT_DETECTED = 302, | ||
| 40 | + | ||
| 41 | + SEMANTIC_BUFFER_EMPTY = 401, | ||
| 42 | + SEMANTIC_GAP = 402, | ||
| 43 | + SEMANTIC_REDUCE_ERROR = 403, | ||
| 44 | + SEMANTIC_SIMULATE_FAILED = 404, | ||
| 45 | + SEMANTIC_FINAL_CHECK_FAILED = 405, | ||
| 46 | + SEMANTIC_FINAL_MISSING = 406, | ||
| 47 | + SEMANTIC_FINAL_SRC_ERROR = 407, | ||
| 48 | + SEMANTIC_FINAL_SIZE_ERROR = 408, | ||
| 49 | + SEMANTIC_FINAL_REDUCE_ERROR = 409, | ||
| 50 | + | ||
| 51 | + DUMP_FAILED = 501, | ||
| 52 | + | ||
| 53 | + CHECKER_RUNTIME_ERROR = 901, | ||
| 54 | + SETTING_WARNING = 902 | ||
| 55 | +}; | ||
| 56 | + | ||
| 57 | +constexpr uint32_t ToErrorCodeValue(ErrorCode code) | ||
| 58 | +{ | ||
| 59 | + return static_cast<uint32_t>(code); | ||
| 60 | +} | ||
| 61 | + | ||
| 62 | +inline std::string MakeErrorCodeText(ErrorCode code) | ||
| 63 | +{ | ||
| 64 | + return "[ErrorCode: " + std::to_string(ToErrorCodeValue(code)) + "]"; | ||
| 65 | +} | ||
| 66 | + | ||
| 67 | +} // namespace HcclSim | ||
| 68 | + | ||
| 69 | + | ||
| @@ -23,7 +23,7 @@ std::string GetCurrentPath() | |||
| 23 | { | 23 | { |
| 24 | char absPath[PATH_MAX]; | 24 | char absPath[PATH_MAX]; |
| 25 | if (realpath(".", absPath) == nullptr) { | 25 | if (realpath(".", absPath) == nullptr) { |
| 26 | - HCCL_VM_ERROR("[GetCurrentPath] realpath failed for current path."); | 26 | + HCCL_VM_ERROR("realpath failed for current path."); |
| 27 | return ""; | 27 | return ""; |
| 28 | } | 28 | } |
| 29 | return std::string(absPath); | 29 | return std::string(absPath); |
| @@ -45,7 +45,7 @@ std::string GetRootPath(const std::string& anchorPath, int maxDepth) | |||
| 45 | currentSearchPath = GetParentPath(currentSearchPath); | 45 | currentSearchPath = GetParentPath(currentSearchPath); |
| 46 | } | 46 | } |
| 47 | 47 | ||
| 48 | - HCCL_VM_INFO("[GetRootPath] RootPath NOT found."); | 48 | + HCCL_VM_INFO("RootPath NOT found."); |
| 49 | return ""; | 49 | return ""; |
| 50 | } | 50 | } |
| 51 | 51 | ||
| @@ -91,7 +91,7 @@ HcclResult EnsureDirectory(const std::string& path) | |||
| 91 | current = JoinPath(current, part); | 91 | current = JoinPath(current, part); |
| 92 | if (!DirExists(current)) { | 92 | if (!DirExists(current)) { |
| 93 | if (mkdir(current.c_str(), 0755) != 0 && errno != EEXIST) { | 93 | if (mkdir(current.c_str(), 0755) != 0 && errno != EEXIST) { |
| 94 | - HCCL_VM_ERROR("[EnsureDirectory] mkdir failed for {}: {}", current, strerror(errno)); | 94 | + HCCL_VM_ERROR("mkdir failed for {}: {}", current, strerror(errno)); |
| 95 | return HcclResult::HCCL_E_INTERNAL; | 95 | return HcclResult::HCCL_E_INTERNAL; |
| 96 | } | 96 | } |
| 97 | } | 97 | } |
| @@ -33,7 +33,7 @@ HcclResult SettingManager::Refresh() | |||
| 33 | { | 33 | { |
| 34 | const std::string pluginRootDir = GetCurrentPath(); | 34 | const std::string pluginRootDir = GetCurrentPath(); |
| 35 | if (pluginRootDir.empty()) { | 35 | if (pluginRootDir.empty()) { |
| 36 | - HCCL_VM_ERROR("[SettingManager::Refresh] failed to get current path."); | 36 | + HCCL_VM_ERROR("failed to get current path."); |
| 37 | return HcclResult::HCCL_E_INTERNAL; | 37 | return HcclResult::HCCL_E_INTERNAL; |
| 38 | } | 38 | } |
| 39 | 39 | ||
| @@ -41,7 +41,7 @@ HcclResult SettingManager::Refresh() | |||
| 41 | CheckerSettings newSettings; | 41 | CheckerSettings newSettings; |
| 42 | 42 | ||
| 43 | if (!FileExists(manifestPath)) { | 43 | if (!FileExists(manifestPath)) { |
| 44 | - HCCL_VM_WARN("[SettingManager::Refresh] manifest.json not found, use default settings."); | 44 | + HCCL_VM_WARN("manifest.json not found, use default settings."); |
| 45 | std::lock_guard<std::mutex> lock(m_mutex); | 45 | std::lock_guard<std::mutex> lock(m_mutex); |
| 46 | m_settings = newSettings; | 46 | m_settings = newSettings; |
| 47 | m_pluginRootDir = pluginRootDir; | 47 | m_pluginRootDir = pluginRootDir; |
| @@ -51,7 +51,7 @@ HcclResult SettingManager::Refresh() | |||
| 51 | 51 | ||
| 52 | std::ifstream manifestStream(manifestPath.c_str()); | 52 | std::ifstream manifestStream(manifestPath.c_str()); |
| 53 | if (!manifestStream.is_open()) { | 53 | if (!manifestStream.is_open()) { |
| 54 | - HCCL_VM_ERROR("[SettingManager::Refresh] failed to open manifest file: {}", manifestPath); | 54 | + HCCL_VM_ERROR("failed to open manifest file: {}", manifestPath); |
| 55 | return HcclResult::HCCL_E_INTERNAL; | 55 | return HcclResult::HCCL_E_INTERNAL; |
| 56 | } | 56 | } |
| 57 | 57 | ||
| @@ -64,7 +64,7 @@ HcclResult SettingManager::Refresh() | |||
| 64 | newSettings.enableNewChecker = settings.value(SETTING_KEY_ENABLE_NEW_CHECKER, true); | 64 | newSettings.enableNewChecker = settings.value(SETTING_KEY_ENABLE_NEW_CHECKER, true); |
| 65 | newSettings.enableOldChecker = settings.value(SETTING_KEY_ENABLE_OLD_CHECKER, true); | 65 | newSettings.enableOldChecker = settings.value(SETTING_KEY_ENABLE_OLD_CHECKER, true); |
| 66 | } catch (const std::exception &ex) { | 66 | } catch (const std::exception &ex) { |
| 67 | - HCCL_VM_ERROR("[SettingManager::Refresh] parse manifest failed: {}", ex.what()); | 67 | + HCCL_VM_ERROR("parse manifest failed: {}", ex.what()); |
| 68 | return HcclResult::HCCL_E_INTERNAL; | 68 | return HcclResult::HCCL_E_INTERNAL; |
| 69 | } | 69 | } |
| 70 | 70 | ||
| @@ -75,7 +75,7 @@ HcclResult SettingManager::Refresh() | |||
| 75 | m_manifestPath = manifestPath; | 75 | m_manifestPath = manifestPath; |
| 76 | } | 76 | } |
| 77 | 77 | ||
| 78 | - HCCL_VM_INFO("[SettingManager::Refresh] settings refreshed: insight_dump={}, memory_snapshot_dump={}, " | 78 | + HCCL_VM_INFO("settings refreshed: insight_dump={}, memory_snapshot_dump={}, " |
| 79 | "new_checker={}, old_checker={}", newSettings.enableInsightDump, newSettings.enableMemorySnapshotDump, | 79 | "new_checker={}, old_checker={}", newSettings.enableInsightDump, newSettings.enableMemorySnapshotDump, |
| 80 | newSettings.enableNewChecker, newSettings.enableOldChecker); | 80 | newSettings.enableNewChecker, newSettings.enableOldChecker); |
| 81 | return HcclResult::HCCL_SUCCESS; | 81 | return HcclResult::HCCL_SUCCESS; |
| @@ -97,7 +97,7 @@ HcclResult StorageManager::Trans2CheckerParam(sim::OpDetailTab& detailTab, ::OpD | |||
| 97 | m_checker_param.all2AllDataDes.sendCountMatrix = currentMatrix; | 97 | m_checker_param.all2AllDataDes.sendCountMatrix = currentMatrix; |
| 98 | } | 98 | } |
| 99 | 99 | ||
| 100 | - HCCL_VM_INFO("[Trans2CheckerParam] opIter={}, rankId={}", detailTab.opIter, detailTab.rankId); | 100 | + HCCL_VM_INFO("opIter={}, rankId={}", detailTab.opIter, detailTab.rankId); |
| 101 | return HcclResult::HCCL_SUCCESS; | 101 | return HcclResult::HCCL_SUCCESS; |
| 102 | } | 102 | } |
| 103 | 103 | ||
| @@ -125,7 +125,7 @@ void StorageManager::MergeAll2AllVSendCountMatrix() | |||
| 125 | m_checker_param.all2AllDataDes.count = matrixSize; | 125 | m_checker_param.all2AllDataDes.count = matrixSize; |
| 126 | m_all2AllvSendMatrices.clear(); | 126 | m_all2AllvSendMatrices.clear(); |
| 127 | 127 | ||
| 128 | - HCCL_VM_INFO("[MergeAll2AllVSendCountMatrix] Merged {} ranks into {}x{} matrix", numRanks, rankSize, rankSize); | 128 | + HCCL_VM_INFO("Merged {} ranks into {}x{} matrix", numRanks, rankSize, rankSize); |
| 129 | } | 129 | } |
| 130 | 130 | ||
| 131 | HcclResult StorageManager::LoadHcclVmSynthesisData(uint32_t rankId, sim::OpMemInfoTab memInfo, std::vector<sim::CcuChannelTab>& channels) | 131 | HcclResult StorageManager::LoadHcclVmSynthesisData(uint32_t rankId, sim::OpMemInfoTab memInfo, std::vector<sim::CcuChannelTab>& channels) |
| @@ -188,7 +188,7 @@ HcclResult StorageManager::LoadHcclVmInstrData(std::vector<sim::CcuInstrResTab>& | |||
| 188 | { | 188 | { |
| 189 | for (auto& instr : instrRes) { | 189 | for (auto& instr : instrRes) { |
| 190 | if (instr.instrCount == 0 || instr.instrCount > 32 * 1024) { | 190 | if (instr.instrCount == 0 || instr.instrCount > 32 * 1024) { |
| 191 | - HCCL_VM_WARN("[LoadHcclVmInstrData] invalid instrCount={}, rankId={}, dieId={}", | 191 | + HCCL_VM_WARN("invalid instrCount={}, rankId={}, dieId={}", |
| 192 | instr.instrCount, instr.rankId, static_cast<uint32_t>(instr.dieId)); | 192 | instr.instrCount, instr.rankId, static_cast<uint32_t>(instr.dieId)); |
| 193 | continue; | 193 | continue; |
| 194 | } | 194 | } |
| @@ -207,7 +207,7 @@ HcclResult StorageManager::LoadHcclVmInstrData(std::vector<sim::CcuInstrResTab>& | |||
| 207 | 207 | ||
| 208 | m_instrData.instr_data.push_back(std::move(instrInner)); | 208 | m_instrData.instr_data.push_back(std::move(instrInner)); |
| 209 | 209 | ||
| 210 | - HCCL_VM_INFO("[LoadHcclVmInstrData] rankId={}, dieId={}, count={}", | 210 | + HCCL_VM_INFO("rankId={}, dieId={}, count={}", |
| 211 | instr.rankId, static_cast<uint32_t>(instr.dieId), instr.instrCount); | 211 | instr.rankId, static_cast<uint32_t>(instr.dieId), instr.instrCount); |
| 212 | } | 212 | } |
| 213 | 213 | ||
| @@ -220,7 +220,7 @@ HcclResult StorageManager::LoadHcclVmTaskMetaData(std::vector<std::vector<sim::O | |||
| 220 | for (const auto& rankTasks : allTasks) { | 220 | for (const auto& rankTasks : allTasks) { |
| 221 | totalTasks += rankTasks.size(); | 221 | totalTasks += rankTasks.size(); |
| 222 | } | 222 | } |
| 223 | - HCCL_VM_INFO("[LoadHcclVmTaskMetaData] total tasks: {}, ranks: {}", totalTasks, allTasks.size()); | 223 | + HCCL_VM_INFO("total tasks: {}, ranks: {}", totalTasks, allTasks.size()); |
| 224 | HcclVmTaskMetaData taskMeataData; | 224 | HcclVmTaskMetaData taskMeataData; |
| 225 | for (const auto& rankTasks : allTasks) { | 225 | for (const auto& rankTasks : allTasks) { |
| 226 | for (const auto &task : rankTasks) { | 226 | for (const auto &task : rankTasks) { |
| @@ -229,7 +229,7 @@ HcclResult StorageManager::LoadHcclVmTaskMetaData(std::vector<std::vector<sim::O | |||
| 229 | std::memcpy(&metaData, task.optaskMeta.data(), sizeof(HcclTaskMetaData)); | 229 | std::memcpy(&metaData, task.optaskMeta.data(), sizeof(HcclTaskMetaData)); |
| 230 | taskMeataData.task_meta.push_back(metaData); | 230 | taskMeataData.task_meta.push_back(metaData); |
| 231 | } else { | 231 | } else { |
| 232 | - HCCL_VM_WARN("[LoadHcclVmTaskMetaData] optaskMeta too small, taskSeq={} src:{:d}, dst:{:d}", task.taskSeq, task.optaskMeta.size(), sizeof(HcclTaskMetaData)); | 232 | + HCCL_VM_WARN("optaskMeta too small, taskSeq={} src:{:d}, dst:{:d}", task.taskSeq, task.optaskMeta.size(), sizeof(HcclTaskMetaData)); |
| 233 | } | 233 | } |
| 234 | } | 234 | } |
| 235 | } | 235 | } |
| @@ -269,21 +269,21 @@ uint64_t StorageManager::GetBlockSize(uint32_t rankId, BufferType bufferType) { | |||
| 269 | // 1. 定位 Rank | 269 | // 1. 定位 Rank |
| 270 | auto rankIt = m_mem_layout.find(rankId); | 270 | auto rankIt = m_mem_layout.find(rankId); |
| 271 | if (rankIt == m_mem_layout.end()) { | 271 | if (rankIt == m_mem_layout.end()) { |
| 272 | - HCCL_VM_INFO("[GetBlockSize] Cannot find rank id from memory layout"); | 272 | + HCCL_VM_INFO("Cannot find rank id from memory layout"); |
| 273 | return 0; | 273 | return 0; |
| 274 | } | 274 | } |
| 275 | 275 | ||
| 276 | // 2. 定位 BufferType | 276 | // 2. 定位 BufferType |
| 277 | auto typeIt = rankIt->second.find(bufferType); | 277 | auto typeIt = rankIt->second.find(bufferType); |
| 278 | if (typeIt == rankIt->second.end()) { | 278 | if (typeIt == rankIt->second.end()) { |
| 279 | - HCCL_VM_INFO("[GetBlockSize] Cannot find buffer type from memory layout"); | 279 | + HCCL_VM_INFO("Cannot find buffer type from memory layout"); |
| 280 | return 0; | 280 | return 0; |
| 281 | } | 281 | } |
| 282 | 282 | ||
| 283 | // 3. 获取该类型下的最后一个块 (map 的 rbegin) | 283 | // 3. 获取该类型下的最后一个块 (map 的 rbegin) |
| 284 | const auto& addrMap = typeIt->second; | 284 | const auto& addrMap = typeIt->second; |
| 285 | if (addrMap.empty()) { | 285 | if (addrMap.empty()) { |
| 286 | - HCCL_VM_INFO("[GetBlockSize] Cannot find addr from memory layout"); | 286 | + HCCL_VM_INFO("Cannot find addr from memory layout"); |
| 287 | return 0; | 287 | return 0; |
| 288 | } | 288 | } |
| 289 | 289 | ||
| @@ -302,7 +302,7 @@ DataSlice StorageManager::GetDataSlice(uint32_t rankId, uint64_t addr, uint64_t | |||
| 302 | // 1. 定位该 Rank 的内存布局 | 302 | // 1. 定位该 Rank 的内存布局 |
| 303 | auto rankIter = m_mem_layout.find(rankId); | 303 | auto rankIter = m_mem_layout.find(rankId); |
| 304 | if (rankIter == m_mem_layout.end()) { | 304 | if (rankIter == m_mem_layout.end()) { |
| 305 | - HCCL_VM_ERROR("[GetDataSlice] Cannot find rank {} from memory layout.", rankId); | 305 | + HCCL_VM_ERROR("Cannot find rank {} from memory layout.", rankId); |
| 306 | return slice; | 306 | return slice; |
| 307 | } | 307 | } |
| 308 | 308 | ||
| @@ -332,7 +332,7 @@ DataSlice StorageManager::GetDataSlice(uint32_t rankId, uint64_t addr, uint64_t | |||
| 332 | } | 332 | } |
| 333 | } | 333 | } |
| 334 | } | 334 | } |
| 335 | - HCCL_VM_ERROR("[GetDataSlice] Cannot find rank {}, addr {}, size {} from memory layout.", rankId, addr, size); | 335 | + HCCL_VM_ERROR("Cannot find rank {}, addr {}, size {} from memory layout.", rankId, addr, size); |
| 336 | return slice; // 未找到匹配的物理区间 | 336 | return slice; // 未找到匹配的物理区间 |
| 337 | } | 337 | } |
| 338 | 338 | ||
| @@ -371,7 +371,7 @@ HcclResult StorageManager::GetSlice(uint64_t addr, uint64_t len, DataSlice& data | |||
| 371 | } | 371 | } |
| 372 | } | 372 | } |
| 373 | 373 | ||
| 374 | - HCCL_VM_ERROR("[GetSlice] Cannot find addr...:addr= {}, len={}", addr, len); | 374 | + HCCL_VM_ERROR("Cannot find addr...:addr= {}, len={}", addr, len); |
| 375 | return HcclResult::HCCL_E_MEMORY; | 375 | return HcclResult::HCCL_E_MEMORY; |
| 376 | } | 376 | } |
| 377 | 377 | ||
| @@ -391,14 +391,14 @@ std::string StorageManager::FindRootPath() | |||
| 391 | } | 391 | } |
| 392 | } else { | 392 | } else { |
| 393 | // 如果 realpath 失败(例如路径被删除或权限不足) | 393 | // 如果 realpath 失败(例如路径被删除或权限不足) |
| 394 | - HCCL_VM_ERROR("[FindRootPath] Iteration {}: realpath failed for {}", i, current_search_path); | 394 | + HCCL_VM_ERROR("Iteration {}: realpath failed for {}", i, current_search_path); |
| 395 | } | 395 | } |
| 396 | 396 | ||
| 397 | // 没找到,将探测路径向上推一级 | 397 | // 没找到,将探测路径向上推一级 |
| 398 | current_search_path += "/.."; | 398 | current_search_path += "/.."; |
| 399 | } | 399 | } |
| 400 | 400 | ||
| 401 | - HCCL_VM_INFO("[FindRootPath] RootPath NOT found."); | 401 | + HCCL_VM_INFO("RootPath NOT found."); |
| 402 | return ""; | 402 | return ""; |
| 403 | } | 403 | } |
| 404 | 404 | ||
| @@ -352,7 +352,7 @@ std::shared_ptr<TaskStub> ConvertTask(const HcclSim::StorageManager& storage, Hc | |||
| 352 | memcpy(ccuParam.args, hcclTask.taskData.ccu.args, sizeof(uint64_t) * ccuParam.argSize); | 352 | memcpy(ccuParam.args, hcclTask.taskData.ccu.args, sizeof(uint64_t) * ccuParam.argSize); |
| 353 | missionParam.push_back(ccuParam); | 353 | missionParam.push_back(ccuParam); |
| 354 | HCCL_VM_INFO("rank {}, dieId= {}", hcclTask.rankId, static_cast<uint32_t>(dieId)); | 354 | HCCL_VM_INFO("rank {}, dieId= {}", hcclTask.rankId, static_cast<uint32_t>(dieId)); |
| 355 | - HCCL_VM_INFO("[ConvertTask] Get sqe info: missionId= {:d}, startId= {:d}, cnt= {:d}, argSize= {:d}", | 355 | + HCCL_VM_INFO("Get sqe info: missionId= {:d}, startId= {:d}, cnt= {:d}, argSize= {:d}", |
| 356 | missionId, ccuParam.instStartId, ccuParam.instCnt, ccuParam.argSize); | 356 | missionId, ccuParam.instStartId, ccuParam.instCnt, ccuParam.argSize); |
| 357 | 357 | ||
| 358 | if (g_missionTask.find(hcclTask.rankId) != g_missionTask.end()) { | 358 | if (g_missionTask.find(hcclTask.rankId) != g_missionTask.end()) { |
| @@ -378,7 +378,7 @@ std::shared_ptr<TaskStub> ConvertTask(const HcclSim::StorageManager& storage, Hc | |||
| 378 | } | 378 | } |
| 379 | hcomm::CcuRep::CcuInstrInfo ccuInstrInfo; | 379 | hcomm::CcuRep::CcuInstrInfo ccuInstrInfo; |
| 380 | ccuInstrInfo.instrVec = ccuInstr.data; | 380 | ccuInstrInfo.instrVec = ccuInstr.data; |
| 381 | - HCCL_VM_INFO("[ConvertTask] Create new ccu graph base node, rank_id= {}, die_id= {} - {}", | 381 | + HCCL_VM_INFO("Create new ccu graph base node, rank_id= {}, die_id= {} - {}", |
| 382 | ccuInstr.desc.rank_id, static_cast<uint32_t>(ccuInstr.desc.die_id), static_cast<uint32_t>(dieId)); | 382 | ccuInstr.desc.rank_id, static_cast<uint32_t>(ccuInstr.desc.die_id), static_cast<uint32_t>(dieId)); |
| 383 | auto taskPtr = std::make_shared<HcclSim::TaskStubCcuGraph>(ccuInstrInfo, missionParam, ccuInstr.desc.rank_id); | 383 | auto taskPtr = std::make_shared<HcclSim::TaskStubCcuGraph>(ccuInstrInfo, missionParam, ccuInstr.desc.rank_id); |
| 384 | g_missionTask[hcclTask.rankId][dieId][missionId] = taskPtr; | 384 | g_missionTask[hcclTask.rankId][dieId][missionId] = taskPtr; |
| @@ -410,7 +410,7 @@ void ConvertTaskQueue(AllRankTaskQueues& allRankTaskQueues) | |||
| 410 | continue; | 410 | continue; |
| 411 | } | 411 | } |
| 412 | if (size > 10000 && i % outputInterval == 0) { | 412 | if (size > 10000 && i % outputInterval == 0) { |
| 413 | - HCCL_VM_INFO("[ConvertTaskQueue] Processing at index: {:d} / {:d} ({:f}% complete)", | 413 | + HCCL_VM_INFO("Processing at index: {:d} / {:d} ({:f}% complete)", |
| 414 | i, size, i * 100.0 / size); | 414 | i, size, i * 100.0 / size); |
| 415 | } | 415 | } |
| 416 | uint32_t rankId = taskMetaVec[i].rankId; | 416 | uint32_t rankId = taskMetaVec[i].rankId; |
| @@ -29,6 +29,7 @@ | |||
| 29 | 29 | ||
| 30 | 30 | ||
| 31 | 31 | ||
| 32 | + | ||
| 32 | 33 | ||
| 33 | 34 | ||
| 34 | 35 | ||
| @@ -69,7 +70,7 @@ bool HcclPluginManager::IsMatchingPlugin(const std::string& manifestPath, const | |||
| 69 | if (currentPluginName == targetTag) { | 70 | if (currentPluginName == targetTag) { |
| 70 | // 如果必需字段缺失,认为是一个损坏的插件,返回 false 并记录日志 | 71 | // 如果必需字段缺失,认为是一个损坏的插件,返回 false 并记录日志 |
| 71 | if (!data.contains(HcclPlugin::Manifest::pluginEntry)) { | 72 | if (!data.contains(HcclPlugin::Manifest::pluginEntry)) { |
| 72 | - HCCL_VM_ERROR("[Plugin] [{}] '{}' missing entry field.", __func__, targetTag); | 73 | + HCCL_VM_ERROR("[{}] '{}' missing entry field.", __func__, targetTag); |
| 73 | return false; | 74 | return false; |
| 74 | } | 75 | } |
| 75 | return true; | 76 | return true; |
| @@ -77,11 +78,11 @@ bool HcclPluginManager::IsMatchingPlugin(const std::string& manifestPath, const | |||
| 77 | } | 78 | } |
| 78 | } catch (const nlohmann::json::parse_error& e) { | 79 | } catch (const nlohmann::json::parse_error& e) { |
| 79 | // 生产环境建议记录具体的解析错误位置 | 80 | // 生产环境建议记录具体的解析错误位置 |
| 80 | - HCCL_VM_ERROR("[Plugin] [{}] JSON Parse Error at byte {}", __func__, e.byte); | 81 | + HCCL_VM_ERROR("[{}] JSON Parse Error at byte {}", __func__, e.byte); |
| 81 | return false; | 82 | return false; |
| 82 | } catch (const std::exception& e) { | 83 | } catch (const std::exception& e) { |
| 83 | // 处理其他可能的异常(如文件读取中途断开等) | 84 | // 处理其他可能的异常(如文件读取中途断开等) |
| 84 | - HCCL_VM_ERROR("[Plugin] [{}] Error: ", __func__, e.what()); | 85 | + HCCL_VM_ERROR("[{}] Error: {}", __func__, e.what()); |
| 85 | return false; | 86 | return false; |
| 86 | } | 87 | } |
| 87 | 88 | ||
| @@ -90,7 +91,7 @@ bool HcclPluginManager::IsMatchingPlugin(const std::string& manifestPath, const | |||
| 90 | 91 | ||
| 91 | bool HcclPluginManager::GetPluginFolderPath(const std::string& pluginTag, std::string& pluginFolderPath) | 92 | bool HcclPluginManager::GetPluginFolderPath(const std::string& pluginTag, std::string& pluginFolderPath) |
| 92 | { | 93 | { |
| 93 | - std::string rootDir = GetBinLocation() + HcclPlugin::PLUGIN_PATH; | 94 | + std::string rootDir = InstallPath::ResolveToInstallRoot(HcclPlugin::PLUGIN_PATH.substr(1)); |
| 94 | std::string foundPath = ""; | 95 | std::string foundPath = ""; |
| 95 | bool hasConflict = false; | 96 | bool hasConflict = false; |
| 96 | 97 | ||
| @@ -115,7 +116,7 @@ bool HcclPluginManager::GetPluginFolderPath(const std::string& pluginTag, std::s | |||
| 115 | 116 | ||
| 116 | // 冲突检查:如果之前已经找到了一个路径,且不是当前路径 | 117 | // 冲突检查:如果之前已经找到了一个路径,且不是当前路径 |
| 117 | if (!foundPath.empty() && foundPath != currentAbsPath) { | 118 | if (!foundPath.empty() && foundPath != currentAbsPath) { |
| 118 | - HCCL_VM_ERROR("[Plugin] [{}] Conflict detected: Tag '{}' exists in both: {} and {}", | 119 | + HCCL_VM_ERROR("[{}] Conflict detected: Tag '{}' exists in both: {} and {}", |
| 119 | __func__, pluginTag, foundPath, currentAbsPath); | 120 | __func__, pluginTag, foundPath, currentAbsPath); |
| 120 | hasConflict = true; | 121 | hasConflict = true; |
| 121 | break; | 122 | break; |
| @@ -203,7 +204,7 @@ HcclVmResult HcclPluginManager::RegisterPlugin(const std::string& pluginTag) | |||
| 203 | 204 | ||
| 204 | // 防止重复注册 | 205 | // 防止重复注册 |
| 205 | if (m_plugins.find(pluginTag) != m_plugins.end()) { | 206 | if (m_plugins.find(pluginTag) != m_plugins.end()) { |
| 206 | - HCCL_VM_WARN("[PluginManager] [{}] Plugin {} already registered", __func__, pluginTag); | 207 | + HCCL_VM_WARN("[{}] Plugin {} already registered", __func__, pluginTag); |
| 207 | return HcclSim::HcclVmResult::HCCL_SIM_SUCCESS; | 208 | return HcclSim::HcclVmResult::HCCL_SIM_SUCCESS; |
| 208 | } | 209 | } |
| 209 | 210 | ||
| @@ -217,7 +218,7 @@ HcclVmResult HcclPluginManager::RegisterPlugin(const std::string& pluginTag) | |||
| 217 | auto plugin = std::make_shared<HcclPlugin>(folderPath); | 218 | auto plugin = std::make_shared<HcclPlugin>(folderPath); |
| 218 | m_plugins[pluginTag] = plugin; | 219 | m_plugins[pluginTag] = plugin; |
| 219 | } catch (const std::exception& e) { | 220 | } catch (const std::exception& e) { |
| 220 | - HCCL_VM_ERROR("[PluginManager] [{}] Failed to register plugin {}: {}", __func__, pluginTag, e.what()); | 221 | + HCCL_VM_ERROR("[{}] Failed to register plugin {}: {}", __func__, pluginTag, e.what()); |
| 221 | return HcclSim::HcclVmResult::HCCL_SIM_E_INTERNAL; | 222 | return HcclSim::HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 222 | } | 223 | } |
| 223 | 224 | ||
| @@ -229,12 +230,12 @@ HcclVmResult HcclPluginManager::SendMessageToPlugin(const std::string& pluginTag | |||
| 229 | std::lock_guard<std::mutex> lock(m_mutex); | 230 | std::lock_guard<std::mutex> lock(m_mutex); |
| 230 | auto it = m_plugins.find(pluginTag); | 231 | auto it = m_plugins.find(pluginTag); |
| 231 | if (it == m_plugins.end()) { | 232 | if (it == m_plugins.end()) { |
| 232 | - HCCL_VM_ERROR("[PluginManager] [{}] Plugin {} not found", __func__, pluginTag); | 233 | + HCCL_VM_ERROR("[{}] Plugin {} not found", __func__, pluginTag); |
| 233 | return HcclVmResult::HCCL_SIM_E_NOT_FOUND; | 234 | return HcclVmResult::HCCL_SIM_E_NOT_FOUND; |
| 234 | } | 235 | } |
| 235 | HcclVmResult ret = it->second->SendMessage(PLUGIN_MESSAGE_TYPE::BROADCAST, action, payload); | 236 | HcclVmResult ret = it->second->SendMessage(PLUGIN_MESSAGE_TYPE::BROADCAST, action, payload); |
| 236 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 237 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 237 | - HCCL_VM_ERROR("[PluginManager] [{}] Failed to send message to plugin {}: {}", __func__, pluginTag, static_cast<int>(ret)); | 238 | + HCCL_VM_ERROR("[{}] Failed to send message to plugin {}: {}", __func__, pluginTag, static_cast<int>(ret)); |
| 238 | } | 239 | } |
| 239 | return ret; | 240 | return ret; |
| 240 | } | 241 | } |
| @@ -341,7 +342,7 @@ HcclSim::HcclVmResult HcclPluginManager::StopAllPlugins() { | |||
| 341 | 342 | ||
| 342 | HcclPluginManager::HcclPluginManager() | 343 | HcclPluginManager::HcclPluginManager() |
| 343 | { | 344 | { |
| 344 | - HCCL_VM_INFO("[PluginManager]HcclPluginManager"); | 345 | + HCCL_VM_INFO("HcclPluginManager"); |
| 345 | m_monitorThreadStop = false; | 346 | m_monitorThreadStop = false; |
| 346 | m_monitorThread = std::thread([this]() { | 347 | m_monitorThread = std::thread([this]() { |
| 347 | this->MonitorThread(); | 348 | this->MonitorThread(); |
| @@ -369,10 +370,10 @@ void HcclPluginManager::MonitorThread() | |||
| 369 | if (it.second->GetPid() == terminatedPid) { | 370 | if (it.second->GetPid() == terminatedPid) { |
| 370 | // 解析退出原因 | 371 | // 解析退出原因 |
| 371 | if (WIFEXITED(status)) { | 372 | if (WIFEXITED(status)) { |
| 372 | - HCCL_VM_INFO("[PluginManager] [{}] Plugin [{}] (PID: {}) exit normally. Code {}", __func__, | 373 | + HCCL_VM_INFO("[{}] Plugin [{}] (PID: {}) exit normally. Code {}", __func__, |
| 373 | it.second->GetTag(), terminatedPid, WEXITSTATUS(status)); | 374 | it.second->GetTag(), terminatedPid, WEXITSTATUS(status)); |
| 374 | } else if (WIFSIGNALED(status)) { | 375 | } else if (WIFSIGNALED(status)) { |
| 375 | - HCCL_VM_ERROR("[PluginManager] [{}] Plugin [{}] (PID: {}) exit with failure. Code {} ({})", __func__, | 376 | + HCCL_VM_ERROR("[{}] Plugin [{}] (PID: {}) exit with failure. Code {} ({})", __func__, |
| 376 | it.second->GetTag(), terminatedPid, WTERMSIG(status), strsignal(WTERMSIG(status))); | 377 | it.second->GetTag(), terminatedPid, WTERMSIG(status), strsignal(WTERMSIG(status))); |
| 377 | } | 378 | } |
| 378 | m_plugins.erase(it.first); | 379 | m_plugins.erase(it.first); |
| @@ -85,10 +85,10 @@ void RecordRunVirtualRuntimeTime(RunVirtualRuntimeTimeStats& stats, uint64_t cos | |||
| 85 | void DumpRunVirtualRuntimeTimeStats(const RunVirtualRuntimeTimeStats& stats) | 85 | void DumpRunVirtualRuntimeTimeStats(const RunVirtualRuntimeTimeStats& stats) |
| 86 | { | 86 | { |
| 87 | if (stats.runCount == 0) { | 87 | if (stats.runCount == 0) { |
| 88 | - HCCL_VM_INFO("[Runner][StageTime] RunVirtualRuntime no execution, runCount=0"); | 88 | + HCCL_VM_INFO("RunVirtualRuntime no execution, runCount=0"); |
| 89 | return; | 89 | return; |
| 90 | } | 90 | } |
| 91 | - HCCL_VM_INFO("[Runner][StageTime] RunVirtualRuntime summary, runCount={}, totalCostMs={}, lastCostMs={}, " | 91 | + HCCL_VM_INFO("RunVirtualRuntime summary, runCount={}, totalCostMs={}, lastCostMs={}, " |
| 92 | "maxCostMs={}", stats.runCount, stats.totalCostMs, stats.lastCostMs, stats.maxCostMs); | 92 | "maxCostMs={}", stats.runCount, stats.totalCostMs, stats.lastCostMs, stats.maxCostMs); |
| 93 | } | 93 | } |
| 94 | } | 94 | } |
| @@ -152,12 +152,12 @@ void ProcessCommand(const std::string& line) { | |||
| 152 | std::string action = j.value("action", ""); | 152 | std::string action = j.value("action", ""); |
| 153 | 153 | ||
| 154 | if (action == "stop") { | 154 | if (action == "stop") { |
| 155 | - HCCL_VM_INFO("[Runner] Stop signal received. Initiating shutdown..."); | 155 | + HCCL_VM_INFO("Stop signal received. Initiating shutdown..."); |
| 156 | g_keep_running.store(false); | 156 | g_keep_running.store(false); |
| 157 | } | 157 | } |
| 158 | // 未来可扩展其他 action... | 158 | // 未来可扩展其他 action... |
| 159 | } catch (const std::exception& e) { | 159 | } catch (const std::exception& e) { |
| 160 | - HCCL_VM_ERROR("[Runner] Command processing failed: {}", e.what()); | 160 | + HCCL_VM_ERROR("Command processing failed: {}", e.what()); |
| 161 | } | 161 | } |
| 162 | } | 162 | } |
| 163 | 163 | ||
| @@ -210,10 +210,10 @@ void RunVirtualRuntime(HcclSim::StorageManager& storage) { | |||
| 210 | 210 | ||
| 211 | HcclSim::CheckerParam param = storage.GetCheckerParam(); | 211 | HcclSim::CheckerParam param = storage.GetCheckerParam(); |
| 212 | HcclCMDType cmdType = param.cmdType; | 212 | HcclCMDType cmdType = param.cmdType; |
| 213 | - HCCL_VM_INFO("[RunVirtualRuntime] start check semantic ....: cmdType = {}, reduceOp = {}", | 213 | + HCCL_VM_INFO("start check semantic ....: cmdType = {}, reduceOp = {}", |
| 214 | static_cast<uint32_t>(cmdType), static_cast<uint32_t>(param.reduceType)); | 214 | static_cast<uint32_t>(cmdType), static_cast<uint32_t>(param.reduceType)); |
| 215 | 215 | ||
| 216 | - HCCL_VM_INFO("[Runner][OpInfo] syncIter = {}, op = {}, rankSize = {}, dataType = {}, dataCount = {}, " | 216 | + HCCL_VM_INFO("syncIter = {}, op = {}, rankSize = {}, dataType = {}, dataCount = {}, " |
| 217 | "reduceType = {}, srcRank = {}, dstRank = {}, root = {}", | 217 | "reduceType = {}, srcRank = {}, dstRank = {}, root = {}", |
| 218 | records[0].syncIter, HcclCmdTypeToString(cmdType), param.rankSize, | 218 | records[0].syncIter, HcclCmdTypeToString(cmdType), param.rankSize, |
| 219 | HcclDataTypeToString(param.dataType), param.dataCount, | 219 | HcclDataTypeToString(param.dataType), param.dataCount, |
| @@ -275,7 +275,7 @@ int main(int argc, char* argv[]) | |||
| 275 | } | 275 | } |
| 276 | // EOF 或 stop 命令后,确保标志被设置 | 276 | // EOF 或 stop 命令后,确保标志被设置 |
| 277 | g_keep_running.store(false); | 277 | g_keep_running.store(false); |
| 278 | - HCCL_VM_INFO("[Runner] stdin reader thread exiting."); | 278 | + HCCL_VM_INFO("stdin reader thread exiting."); |
| 279 | }).detach(); | 279 | }).detach(); |
| 280 | 280 | ||
| 281 | while (g_keep_running.load()) { | 281 | while (g_keep_running.load()) { |
| @@ -286,16 +286,16 @@ int main(int argc, char* argv[]) | |||
| 286 | continue; | 286 | continue; |
| 287 | } | 287 | } |
| 288 | if (targetRound == sim::kRunnerExitSignal) { | 288 | if (targetRound == sim::kRunnerExitSignal) { |
| 289 | - HCCL_VM_INFO("[Runner] Exit signal received from hccl-vm."); | 289 | + HCCL_VM_INFO("Exit signal received from hccl-vm."); |
| 290 | break; | 290 | break; |
| 291 | } | 291 | } |
| 292 | - HCCL_VM_INFO("[Runner] Task received, targetRound={}", targetRound); | 292 | + HCCL_VM_INFO("Task received, targetRound={}", targetRound); |
| 293 | RunVirtualRuntime(storage); | 293 | RunVirtualRuntime(storage); |
| 294 | syncer.notifyProxyToContinue(targetRound); | 294 | syncer.notifyProxyToContinue(targetRound); |
| 295 | - HCCL_VM_INFO("[Runner] Round {} completed.", targetRound); | 295 | + HCCL_VM_INFO("Round {} completed.", targetRound); |
| 296 | FlushLog(); // 将本轮完整日志落盘 | 296 | FlushLog(); // 将本轮完整日志落盘 |
| 297 | } | 297 | } |
| 298 | 298 | ||
| 299 | - HCCL_VM_INFO("[Runner] Exiting..."); | 299 | + HCCL_VM_INFO("Exiting..."); |
| 300 | return 0; | 300 | return 0; |
| 301 | } | 301 | } |
| @@ -89,3 +89,6 @@ install(FILES "${CMAKE_CURRENT_SOURCE_DIR}/../manifest.json" | |||
| 89 | GROUP_READ GROUP_EXECUTE | 89 | GROUP_READ GROUP_EXECUTE |
| 90 | WORLD_READ WORLD_EXECUTE | 90 | WORLD_READ WORLD_EXECUTE |
| 91 | ) | 91 | ) |
| 92 | + | ||
| 93 | +# 日志染色: 模块 tag | ||
| 94 | +target_compile_definitions(runner PRIVATE HCCL_VM_MODULE="RUNNER") | ||
| @@ -56,3 +56,6 @@ target_link_libraries(aiv_executor | |||
| 56 | log | 56 | log |
| 57 | rt | 57 | rt |
| 58 | ) | 58 | ) |
| 59 | + | ||
| 60 | +# 日志染色: 模块 tag | ||
| 61 | +target_compile_definitions(aiv_executor PRIVATE HCCL_VM_MODULE="RUNNER") | ||
| @@ -78,7 +78,7 @@ AivBlock::~AivBlock() { | |||
| 78 | } | 78 | } |
| 79 | 79 | ||
| 80 | bool AivGraphExecutor::Init(uint32_t rankId, uint32_t launchIdx) { | 80 | bool AivGraphExecutor::Init(uint32_t rankId, uint32_t launchIdx) { |
| 81 | - HCCL_VM_DEBUG("[AivGraphExecutor][Init] begin, launchIdx={}, rankId={}", launchIdx_, rankId_); | 81 | + HCCL_VM_DEBUG("begin, launchIdx={}, rankId={}", launchIdx_, rankId_); |
| 82 | rankId_ = rankId; | 82 | rankId_ = rankId; |
| 83 | launchIdx_ = launchIdx; | 83 | launchIdx_ = launchIdx; |
| 84 | 84 | ||
| @@ -86,7 +86,7 @@ bool AivGraphExecutor::Init(uint32_t rankId, uint32_t launchIdx) { | |||
| 86 | std::string errorMessage; | 86 | std::string errorMessage; |
| 87 | if (!AivTaskSnapshotLoader::LoadRuntimeTaskSnapshotByLaunchDirect( | 87 | if (!AivTaskSnapshotLoader::LoadRuntimeTaskSnapshotByLaunchDirect( |
| 88 | rankId_, static_cast<uint32_t>(launchIdx_), taskSnapshot, &errorMessage)) { | 88 | rankId_, static_cast<uint32_t>(launchIdx_), taskSnapshot, &errorMessage)) { |
| 89 | - HCCL_VM_ERROR("[AivGraphExecutor][Init] failed to load runtime snapshot, launchIdx={}, rankId={}, reason={}", | 89 | + HCCL_VM_ERROR("failed to load runtime snapshot, launchIdx={}, rankId={}, reason={}", |
| 90 | launchIdx_, rankId_, errorMessage); | 90 | launchIdx_, rankId_, errorMessage); |
| 91 | return false; | 91 | return false; |
| 92 | } | 92 | } |
| @@ -99,7 +99,7 @@ bool AivGraphExecutor::Init(uint32_t rankId, uint32_t launchIdx) { | |||
| 99 | 99 | ||
| 100 | rankId_ = taskSnapshot.rankId; | 100 | rankId_ = taskSnapshot.rankId; |
| 101 | rankSize_ = taskSnapshot.rankSize; | 101 | rankSize_ = taskSnapshot.rankSize; |
| 102 | - HCCL_VM_DEBUG("[AivGraphExecutor][Init] snapshot loaded, launchIdx={}, rankId={}, rankSize={}, file={}, aivBlockNum={}", | 102 | + HCCL_VM_DEBUG("snapshot loaded, launchIdx={}, rankId={}, rankSize={}, file={}, aivBlockNum={}", |
| 103 | launchIdx_, rankId_, rankSize_, taskSnapshot.filePath, taskSnapshot.blocks.size()); | 103 | launchIdx_, rankId_, rankSize_, taskSnapshot.filePath, taskSnapshot.blocks.size()); |
| 104 | 104 | ||
| 105 | uint32_t maxTaskId = 0; | 105 | uint32_t maxTaskId = 0; |
| @@ -116,7 +116,7 @@ bool AivGraphExecutor::Init(uint32_t rankId, uint32_t launchIdx) { | |||
| 116 | aivTaskQueues_.emplace_back(); | 116 | aivTaskQueues_.emplace_back(); |
| 117 | AppendPipeTasksToQueue(block.mte3Tasks, aivTaskQueues_.back(), maxTaskId, maxEventId, maxSyncRound); | 117 | AppendPipeTasksToQueue(block.mte3Tasks, aivTaskQueues_.back(), maxTaskId, maxEventId, maxSyncRound); |
| 118 | 118 | ||
| 119 | - HCCL_VM_TRACE("[AivGraphExecutor][Init] blockIdx={}, scalarTaskCount={}, mte2TaskCount={}, mte3TaskCount={}, maxEventId={}", | 119 | + HCCL_VM_TRACE("blockIdx={}, scalarTaskCount={}, mte2TaskCount={}, mte3TaskCount={}, maxEventId={}", |
| 120 | block.blockIdx, block.scalarTasks.size(), block.mte2Tasks.size(), block.mte3Tasks.size(), maxEventId); | 120 | block.blockIdx, block.scalarTasks.size(), block.mte2Tasks.size(), block.mte3Tasks.size(), maxEventId); |
| 121 | aivBlocks_.emplace_back(std::make_unique<AivBlock>(block.blockIdx, maxEventId, AivSim::AIV_UB_SIZE)); | 121 | aivBlocks_.emplace_back(std::make_unique<AivBlock>(block.blockIdx, maxEventId, AivSim::AIV_UB_SIZE)); |
| 122 | } | 122 | } |
| @@ -129,11 +129,11 @@ bool AivGraphExecutor::Init(uint32_t rankId, uint32_t launchIdx) { | |||
| 129 | syncAllRegisters_.emplace_back(pipeSize, false); | 129 | syncAllRegisters_.emplace_back(pipeSize, false); |
| 130 | } | 130 | } |
| 131 | 131 | ||
| 132 | - HCCL_VM_DEBUG("[AivGraphExecutor][Init] queueCount={}, maxTaskId={}, pipeBarrierRegisterCount={}, syncAllRegisterCount={}, aivBlockNum={}", | 132 | + HCCL_VM_DEBUG("queueCount={}, maxTaskId={}, pipeBarrierRegisterCount={}, syncAllRegisterCount={}, aivBlockNum={}", |
| 133 | aivTaskQueues_.size(), maxTaskId, pipeBarrierRegisters_.size(), syncAllRegisters_.size(), aivBlocks_.size()); | 133 | aivTaskQueues_.size(), maxTaskId, pipeBarrierRegisters_.size(), syncAllRegisters_.size(), aivBlocks_.size()); |
| 134 | 134 | ||
| 135 | isInitialized_ = true; | 135 | isInitialized_ = true; |
| 136 | - HCCL_VM_INFO("[AivGraphExecutor][Init] success, launchIdx={}, rankId={}", launchIdx_, rankId_); | 136 | + HCCL_VM_INFO("success, launchIdx={}, rankId={}", launchIdx_, rankId_); |
| 137 | return true; | 137 | return true; |
| 138 | } | 138 | } |
| 139 | 139 | ||
| @@ -204,7 +204,7 @@ HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTask> task | |||
| 204 | } | 204 | } |
| 205 | 205 | ||
| 206 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskMemCopy> task) { | 206 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskMemCopy> task) { |
| 207 | - HCCL_VM_DEBUG(task->Describe()); | 207 | + HCCL_VM_DEBUG("{}", task->Describe()); |
| 208 | 208 | ||
| 209 | if (task->GetSrc().GetSize() != task->GetDst().GetSize()) { | 209 | if (task->GetSrc().GetSize() != task->GetDst().GetSize()) { |
| 210 | HCCL_VM_ERROR("task invalid, len not match, taskId={:d}", task->GetTaskId()); | 210 | HCCL_VM_ERROR("task invalid, len not match, taskId={:d}", task->GetTaskId()); |
| @@ -270,7 +270,7 @@ void* AivGraphExecutor::GetMemPtr(std::shared_ptr<T> task, bool isSrc) { | |||
| 270 | } | 270 | } |
| 271 | 271 | ||
| 272 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskReduce> task) { | 272 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskReduce> task) { |
| 273 | - HCCL_VM_DEBUG(task->Describe()); | 273 | + HCCL_VM_DEBUG("{}", task->Describe()); |
| 274 | 274 | ||
| 275 | if (task->GetSrc().GetSize() != task->GetDst().GetSize()) { | 275 | if (task->GetSrc().GetSize() != task->GetDst().GetSize()) { |
| 276 | HCCL_VM_ERROR("task invalid, len not match, taskId={:d}", task->GetTaskId()); | 276 | HCCL_VM_ERROR("task invalid, len not match, taskId={:d}", task->GetTaskId()); |
| @@ -399,7 +399,7 @@ HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskSyncAl | |||
| 399 | } | 399 | } |
| 400 | 400 | ||
| 401 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskSendFlag> task) { | 401 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskSendFlag> task) { |
| 402 | - HCCL_VM_DEBUG(task->Describe()); | 402 | + HCCL_VM_DEBUG("{}", task->Describe()); |
| 403 | 403 | ||
| 404 | auto* rankResource = AivResourceManager::GetInstance().GetRankResource(task->GetRank()); | 404 | auto* rankResource = AivResourceManager::GetInstance().GetRankResource(task->GetRank()); |
| 405 | if (rankResource == nullptr) { | 405 | if (rankResource == nullptr) { |
| @@ -417,7 +417,7 @@ HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskSendFl | |||
| 417 | } | 417 | } |
| 418 | 418 | ||
| 419 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskRecvFlag> task) { | 419 | HcclVmResult AivGraphExecutor::ExecuteTask(std::shared_ptr<AivSim::AivTaskRecvFlag> task) { |
| 420 | - HCCL_VM_DEBUG(task->Describe()); | 420 | + HCCL_VM_DEBUG("{}", task->Describe()); |
| 421 | 421 | ||
| 422 | auto* rankResource = AivResourceManager::GetInstance().GetRankResource(task->GetRank()); | 422 | auto* rankResource = AivResourceManager::GetInstance().GetRankResource(task->GetRank()); |
| 423 | if (rankResource == nullptr) { | 423 | if (rankResource == nullptr) { |
| @@ -456,5 +456,5 @@ void AivGraphExecutor::ShowCurrentTaskStatus() { | |||
| 456 | ss << "]"; | 456 | ss << "]"; |
| 457 | } | 457 | } |
| 458 | ss << "}"; | 458 | ss << "}"; |
| 459 | - HCCL_VM_DEBUG(ss.str()); | 459 | + HCCL_VM_DEBUG("{}", ss.str()); |
| 460 | } | 460 | } |
| @@ -74,7 +74,7 @@ HcclVmResult AivResourceManager::Init(uint32_t rankId, const sim::OpMemInfoTab & | |||
| 74 | return ret; | 74 | return ret; |
| 75 | } | 75 | } |
| 76 | if (rankId >= rankSize) { | 76 | if (rankId >= rankSize) { |
| 77 | - HCCL_VM_ERROR("[AivResourceManager][Init] rankId={} out of range, rankSize={}", rankId, rankSize); | 77 | + HCCL_VM_ERROR("rankId={} out of range, rankSize={}", rankId, rankSize); |
| 78 | return HcclVmResult::HCCL_SIM_E_PARA; | 78 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 79 | } | 79 | } |
| 80 | 80 | ||
| @@ -91,7 +91,7 @@ HcclVmResult AivResourceManager::Init(uint32_t rankId, const sim::OpMemInfoTab & | |||
| 91 | return ret; | 91 | return ret; |
| 92 | } | 92 | } |
| 93 | 93 | ||
| 94 | - HCCL_VM_DEBUG("[AivResourceManager][Init] init rank op mem success, rankId={}, rankSize={}, " | 94 | + HCCL_VM_DEBUG("init rank op mem success, rankId={}, rankSize={}, " |
| 95 | "input={:x}/{} output={:x}/{} ccl={:x}/{}", | 95 | "input={:x}/{} output={:x}/{} ccl={:x}/{}", |
| 96 | rankId, | 96 | rankId, |
| 97 | rankSize, | 97 | rankSize, |
| @@ -130,7 +130,7 @@ const AivRankResource *AivResourceManager::GetRankResource(uint32_t rankId) cons | |||
| 130 | HcclVmResult AivResourceManager::EnsureInitialized(uint32_t rankSize) | 130 | HcclVmResult AivResourceManager::EnsureInitialized(uint32_t rankSize) |
| 131 | { | 131 | { |
| 132 | if (rankSize == 0) { | 132 | if (rankSize == 0) { |
| 133 | - HCCL_VM_ERROR("[AivResourceManager][EnsureInitialized] rankSize is 0"); | 133 | + HCCL_VM_ERROR("rankSize is 0"); |
| 134 | return HcclVmResult::HCCL_SIM_E_PARA; | 134 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 135 | } | 135 | } |
| 136 | 136 | ||
| @@ -141,12 +141,12 @@ HcclVmResult AivResourceManager::EnsureInitialized(uint32_t rankSize) | |||
| 141 | Reset(); | 141 | Reset(); |
| 142 | return ret; | 142 | return ret; |
| 143 | } | 143 | } |
| 144 | - HCCL_VM_INFO("[AivResourceManager][EnsureInitialized] init base resource success, rankSize={}", rankSize); | 144 | + HCCL_VM_INFO("init base resource success, rankSize={}", rankSize); |
| 145 | return HcclVmResult::HCCL_SIM_SUCCESS; | 145 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 146 | } | 146 | } |
| 147 | 147 | ||
| 148 | if (rankResources_.size() != rankSize) { | 148 | if (rankResources_.size() != rankSize) { |
| 149 | - HCCL_VM_ERROR("[AivResourceManager][EnsureInitialized] rankSize mismatch, current={}, incoming={}", | 149 | + HCCL_VM_ERROR("rankSize mismatch, current={}, incoming={}", |
| 150 | rankResources_.size(), rankSize); | 150 | rankResources_.size(), rankSize); |
| 151 | return HcclVmResult::HCCL_SIM_E_PARA; | 151 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 152 | } | 152 | } |
| @@ -157,7 +157,7 @@ HcclVmResult AivResourceManager::MapBuffer( | |||
| 157 | uint32_t rankSize, uint32_t rankId, uint8_t bufferType, uint64_t startAddr, uint64_t size, bool allowDuplicateSame) | 157 | uint32_t rankSize, uint32_t rankId, uint8_t bufferType, uint64_t startAddr, uint64_t size, bool allowDuplicateSame) |
| 158 | { | 158 | { |
| 159 | if (rankId >= rankSize) { | 159 | if (rankId >= rankSize) { |
| 160 | - HCCL_VM_ERROR("[AivResourceManager][MapBuffer] rankId={} out of range, rankSize={}", | 160 | + HCCL_VM_ERROR("rankId={} out of range, rankSize={}", |
| 161 | rankId, rankSize); | 161 | rankId, rankSize); |
| 162 | return HcclVmResult::HCCL_SIM_E_PARA; | 162 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 163 | } | 163 | } |
| @@ -165,18 +165,18 @@ HcclVmResult AivResourceManager::MapBuffer( | |||
| 165 | auto &rankResource = rankResources_[rankId]; | 165 | auto &rankResource = rankResources_[rankId]; |
| 166 | AivBufferResource *bufferSlot = GetBufferSlot(rankResource, bufferType); | 166 | AivBufferResource *bufferSlot = GetBufferSlot(rankResource, bufferType); |
| 167 | if (bufferSlot == nullptr) { | 167 | if (bufferSlot == nullptr) { |
| 168 | - HCCL_VM_ERROR("[AivResourceManager][MapBuffer] unsupported buffer type={}, rankId={}", | 168 | + HCCL_VM_ERROR("unsupported buffer type={}, rankId={}", |
| 169 | bufferType, rankId); | 169 | bufferType, rankId); |
| 170 | return HcclVmResult::HCCL_SIM_E_PARA; | 170 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 171 | } | 171 | } |
| 172 | if (bufferSlot->realAddr != nullptr) { | 172 | if (bufferSlot->realAddr != nullptr) { |
| 173 | if (allowDuplicateSame && bufferSlot->virtualAddr == startAddr && bufferSlot->size == size) { | 173 | if (allowDuplicateSame && bufferSlot->virtualAddr == startAddr && bufferSlot->size == size) { |
| 174 | - HCCL_VM_DEBUG("[AivResourceManager][MapBuffer] duplicate same {} buffer skipped, " | 174 | + HCCL_VM_DEBUG("duplicate same {} buffer skipped, " |
| 175 | "rankId={}, virtualAddr={:x}, size={}", | 175 | "rankId={}, virtualAddr={:x}, size={}", |
| 176 | GetBufferTypeName(bufferType), rankId, startAddr, size); | 176 | GetBufferTypeName(bufferType), rankId, startAddr, size); |
| 177 | return HcclVmResult::HCCL_SIM_SUCCESS; | 177 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 178 | } | 178 | } |
| 179 | - HCCL_VM_ERROR("[AivResourceManager][MapBuffer] duplicate {} buffer found, rankId={}, virtualAddr={:x}, size={}", | 179 | + HCCL_VM_ERROR("duplicate {} buffer found, rankId={}, virtualAddr={:x}, size={}", |
| 180 | GetBufferTypeName(bufferType), rankId, startAddr, size); | 180 | GetBufferTypeName(bufferType), rankId, startAddr, size); |
| 181 | return HcclVmResult::HCCL_SIM_E_PARA; | 181 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 182 | } | 182 | } |
| @@ -184,7 +184,7 @@ HcclVmResult AivResourceManager::MapBuffer( | |||
| 184 | sim::PhyMemBlock phyMem {}; | 184 | sim::PhyMemBlock phyMem {}; |
| 185 | void *realAddr = sim::AcquireDevPtrInNoHostProcess(reinterpret_cast<void *>(startAddr), phyMem); | 185 | void *realAddr = sim::AcquireDevPtrInNoHostProcess(reinterpret_cast<void *>(startAddr), phyMem); |
| 186 | if (realAddr == nullptr) { | 186 | if (realAddr == nullptr) { |
| 187 | - HCCL_VM_ERROR("[AivResourceManager][MapBuffer] translate addr failed, rankId={}, type={}, addr={:x}", | 187 | + HCCL_VM_ERROR("translate addr failed, rankId={}, type={}, addr={:x}", |
| 188 | rankId, bufferType, startAddr); | 188 | rankId, bufferType, startAddr); |
| 189 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 189 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 190 | } | 190 | } |
| @@ -197,7 +197,7 @@ HcclVmResult AivResourceManager::MapBuffer( | |||
| 197 | *bufferSlot = bufferResource; | 197 | *bufferSlot = bufferResource; |
| 198 | 198 | ||
| 199 | acquiredPhyMemBlocks_.push_back(phyMem); | 199 | acquiredPhyMemBlocks_.push_back(phyMem); |
| 200 | - HCCL_VM_DEBUG("[AivResourceManager][MapBuffer] rankId={}, type={}, virtualAddr={:x}, realAddr={}, size={}", | 200 | + HCCL_VM_DEBUG("rankId={}, type={}, virtualAddr={:x}, realAddr={}, size={}", |
| 201 | rankId, bufferType, startAddr, realAddr, size); | 201 | rankId, bufferType, startAddr, realAddr, size); |
| 202 | return HcclVmResult::HCCL_SIM_SUCCESS; | 202 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 203 | } | 203 | } |
| @@ -207,7 +207,7 @@ HcclVmResult AivResourceManager::MapOpMemBuffer( | |||
| 207 | { | 207 | { |
| 208 | if (startAddr == 0 || size == 0) { | 208 | if (startAddr == 0 || size == 0) { |
| 209 | if (startAddr != 0 || size != 0) { | 209 | if (startAddr != 0 || size != 0) { |
| 210 | - HCCL_VM_WARN("[AivResourceManager][MapOpMemBuffer] incomplete {} buffer skipped, " | 210 | + HCCL_VM_WARN("incomplete {} buffer skipped, " |
| 211 | "rankId={}, virtualAddr={:x}, size={}", | 211 | "rankId={}, virtualAddr={:x}, size={}", |
| 212 | GetBufferTypeName(bufferType), rankId, startAddr, size); | 212 | GetBufferTypeName(bufferType), rankId, startAddr, size); |
| 213 | } | 213 | } |
| @@ -223,7 +223,7 @@ HcclVmResult AivResourceManager::InitFlagBuffers(uint32_t rankSize) | |||
| 223 | for (uint32_t rankId = 0; rankId < rankSize; ++rankId) { | 223 | for (uint32_t rankId = 0; rankId < rankSize; ++rankId) { |
| 224 | auto flagBuffer = std::make_unique<uint8_t[]>(AIV_FLAG_BUFFER_SIZE); | 224 | auto flagBuffer = std::make_unique<uint8_t[]>(AIV_FLAG_BUFFER_SIZE); |
| 225 | if (flagBuffer == nullptr) { | 225 | if (flagBuffer == nullptr) { |
| 226 | - HCCL_VM_ERROR("[AivResourceManager][InitFlagBuffers] alloc flag buffer failed, rankId={}, size={}", | 226 | + HCCL_VM_ERROR("alloc flag buffer failed, rankId={}, size={}", |
| 227 | rankId, AIV_FLAG_BUFFER_SIZE); | 227 | rankId, AIV_FLAG_BUFFER_SIZE); |
| 228 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 228 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 229 | } | 229 | } |
| @@ -232,7 +232,7 @@ HcclVmResult AivResourceManager::InitFlagBuffers(uint32_t rankSize) | |||
| 232 | rankResources_[rankId].flagBuffer.size = AIV_FLAG_BUFFER_SIZE; | 232 | rankResources_[rankId].flagBuffer.size = AIV_FLAG_BUFFER_SIZE; |
| 233 | flagBufferOwners_[rankId] = std::move(flagBuffer); | 233 | flagBufferOwners_[rankId] = std::move(flagBuffer); |
| 234 | 234 | ||
| 235 | - HCCL_VM_DEBUG("[AivResourceManager][InitFlagBuffers] rankId={}, realAddr={}, size={}", | 235 | + HCCL_VM_DEBUG("rankId={}, realAddr={}, size={}", |
| 236 | rankId, | 236 | rankId, |
| 237 | rankResources_[rankId].flagBuffer.realAddr, | 237 | rankResources_[rankId].flagBuffer.realAddr, |
| 238 | rankResources_[rankId].flagBuffer.size); | 238 | rankResources_[rankId].flagBuffer.size); |
| @@ -9,6 +9,7 @@ | |||
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | 11 | ||
| 12 | + | ||
| 12 | 13 | ||
| 13 | 14 | ||
| 14 | 15 | ||
| @@ -33,13 +34,7 @@ static void SetCommonError(std::string *errorMessage, const std::string &message | |||
| 33 | 34 | ||
| 34 | static bool ResolveAivSnapshotDirectory(fs::path &snapshotDir, std::string *errorMessage) | 35 | static bool ResolveAivSnapshotDirectory(fs::path &snapshotDir, std::string *errorMessage) |
| 35 | { | 36 | { |
| 36 | - const char *installDirEnv = std::getenv("HCCL_VM_INSTALL_DIR"); | 37 | + snapshotDir = fs::path(InstallPath::ResolveToInstallRoot("data")); |
| 37 | - if (installDirEnv == nullptr || installDirEnv[0] == '\0') { | ||
| 38 | - SetCommonError(errorMessage, "HCCL_VM_INSTALL_DIR is not set"); | ||
| 39 | - return false; | ||
| 40 | - } | ||
| 41 | - | ||
| 42 | - snapshotDir = fs::path(installDirEnv) / "data"; | ||
| 43 | std::error_code ec; | 38 | std::error_code ec; |
| 44 | if (!fs::exists(snapshotDir, ec) || ec || !fs::is_directory(snapshotDir, ec)) { | 39 | if (!fs::exists(snapshotDir, ec) || ec || !fs::is_directory(snapshotDir, ec)) { |
| 45 | SetCommonError(errorMessage, "AIV data directory does not exist: " + snapshotDir.string()); | 40 | SetCommonError(errorMessage, "AIV data directory does not exist: " + snapshotDir.string()); |
| @@ -42,3 +42,6 @@ target_link_libraries(ccu_executor PRIVATE | |||
| 42 | log | 42 | log |
| 43 | rt | 43 | rt |
| 44 | ) | 44 | ) |
| 45 | + | ||
| 46 | +# 日志染色: 模块 tag | ||
| 47 | +target_compile_definitions(ccu_executor PRIVATE HCCL_VM_MODULE="RUNNER") | ||
| @@ -26,7 +26,7 @@ using namespace hcomm::CcuRep; | |||
| 26 | 26 | ||
| 27 | void CcuResourceManager::Reset() | 27 | void CcuResourceManager::Reset() |
| 28 | { | 28 | { |
| 29 | - HCCL_VM_DEBUG("[CcuResourceManager][Reset] Clearing all CCU resource state (KE/XN/MS/simulators)"); | 29 | + HCCL_VM_DEBUG("Clearing all CCU resource state (KE/XN/MS/simulators)"); |
| 30 | for (size_t rankId = 0; rankId < ccuResData_.v1Res.size(); rankId++) { | 30 | for (size_t rankId = 0; rankId < ccuResData_.v1Res.size(); rankId++) { |
| 31 | if (ccuResData_.v1Res[rankId] != nullptr) { | 31 | if (ccuResData_.v1Res[rankId] != nullptr) { |
| 32 | ccuResData_.v1Res[rankId]->Reset(); | 32 | ccuResData_.v1Res[rankId]->Reset(); |
| @@ -43,7 +43,7 @@ void CcuResourceManager::Init(int rankId, int rankSize, RunnerCcuVersion version | |||
| 43 | } | 43 | } |
| 44 | ccuResData_.v1Res[rankId] = std::make_unique<CcuResourceV1>(rankId, rankSize); | 44 | ccuResData_.v1Res[rankId] = std::make_unique<CcuResourceV1>(rankId, rankSize); |
| 45 | } else { | 45 | } else { |
| 46 | - HCCL_VM_ERROR("[CcuResourceManager][Init] ccu version {} not supported", static_cast<int>(ccuResData_.version)); | 46 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(ccuResData_.version)); |
| 47 | } | 47 | } |
| 48 | ccuResourceBaseAddr_ = ccuResourceBaseAddr; | 48 | ccuResourceBaseAddr_ = ccuResourceBaseAddr; |
| 49 | } | 49 | } |
| @@ -54,7 +54,7 @@ void CcuResourceManager::InitInstrInfo(int rankId, int dieId, const CcuInstrData | |||
| 54 | if (version == RunnerCcuVersion::CCU_V1) { | 54 | if (version == RunnerCcuVersion::CCU_V1) { |
| 55 | ccuResData_.v1Res[rankId]->instrSpace_[dieId] = ccuInstrInfo; | 55 | ccuResData_.v1Res[rankId]->instrSpace_[dieId] = ccuInstrInfo; |
| 56 | } else { | 56 | } else { |
| 57 | - HCCL_VM_ERROR("[CcuResourceManager][InitInstrInfo] ccu version {} not supported", static_cast<int>(version)); | 57 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 58 | } | 58 | } |
| 59 | } | 59 | } |
| 60 | 60 | ||
| @@ -64,7 +64,7 @@ void CcuResourceManager::InitChannelInfo(int rankId, const RankChannelInfo &chan | |||
| 64 | if (version == RunnerCcuVersion::CCU_V1) { | 64 | if (version == RunnerCcuVersion::CCU_V1) { |
| 65 | ccuResData_.v1Res[rankId]->channelId2RmtRankMap_ = channelInfo; | 65 | ccuResData_.v1Res[rankId]->channelId2RmtRankMap_ = channelInfo; |
| 66 | } else { | 66 | } else { |
| 67 | - HCCL_VM_ERROR("[CcuResourceManager][InitInstrInfo] ccu version {} not supported", static_cast<int>(version)); | 67 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 68 | } | 68 | } |
| 69 | } | 69 | } |
| 70 | 70 | ||
| @@ -72,14 +72,14 @@ void CcuResourceManager::AddTaskInfo(int rankId, const HcclTaskMetaData &task) | |||
| 72 | { | 72 | { |
| 73 | auto dieId = task.taskData.ccu.dieId; | 73 | auto dieId = task.taskData.ccu.dieId; |
| 74 | if (dieId >= HcclSim::DIE_NUM) { | 74 | if (dieId >= HcclSim::DIE_NUM) { |
| 75 | - HCCL_VM_ERROR("[CcuResourceManager][AddTaskInfo] invalid dieId[{}].", dieId); | 75 | + HCCL_VM_ERROR("invalid dieId[{}].", dieId); |
| 76 | return; | 76 | return; |
| 77 | } | 77 | } |
| 78 | auto version = ccuResData_.version; | 78 | auto version = ccuResData_.version; |
| 79 | if (version == RunnerCcuVersion::CCU_V1) { | 79 | if (version == RunnerCcuVersion::CCU_V1) { |
| 80 | ccuResData_.v1Res[rankId]->ccuTaskInfos_[dieId] = task.taskData.ccu; | 80 | ccuResData_.v1Res[rankId]->ccuTaskInfos_[dieId] = task.taskData.ccu; |
| 81 | } else { | 81 | } else { |
| 82 | - HCCL_VM_ERROR("[CcuResourceManager][AddTaskInfo] ccu version {} not supported", static_cast<int>(version)); | 82 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 83 | } | 83 | } |
| 84 | } | 84 | } |
| 85 | 85 | ||
| @@ -89,7 +89,7 @@ uint64_t CcuResourceManager::GetSqeArgValue(int rankId, int dieId, uint16_t argI | |||
| 89 | if (version == RunnerCcuVersion::CCU_V1) { | 89 | if (version == RunnerCcuVersion::CCU_V1) { |
| 90 | return ccuResData_.v1Res[rankId]->ccuTaskInfos_[dieId].args[argId]; | 90 | return ccuResData_.v1Res[rankId]->ccuTaskInfos_[dieId].args[argId]; |
| 91 | } else { | 91 | } else { |
| 92 | - HCCL_VM_ERROR("[CcuResourceManager][GetSqeArgValue] ccu version {} not supported", static_cast<int>(version)); | 92 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 93 | return U64_INVALID; | 93 | return U64_INVALID; |
| 94 | } | 94 | } |
| 95 | } | 95 | } |
| @@ -100,7 +100,7 @@ uint64_t CcuResourceManager::GetXnValue(int rankId, int dieId, uint16_t xnId) co | |||
| 100 | if (version == RunnerCcuVersion::CCU_V1) { | 100 | if (version == RunnerCcuVersion::CCU_V1) { |
| 101 | return ccuResData_.v1Res[rankId]->xn_[dieId][xnId]; | 101 | return ccuResData_.v1Res[rankId]->xn_[dieId][xnId]; |
| 102 | } else { | 102 | } else { |
| 103 | - HCCL_VM_ERROR("[CcuResourceManager][GetXnValue] ccu version {} not supported", static_cast<int>(version)); | 103 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 104 | return U64_INVALID; | 104 | return U64_INVALID; |
| 105 | } | 105 | } |
| 106 | } | 106 | } |
| @@ -111,7 +111,7 @@ uint64_t CcuResourceManager::GetGsaValue(int rankId, int dieId, uint16_t gsaId) | |||
| 111 | if (version == RunnerCcuVersion::CCU_V1) { | 111 | if (version == RunnerCcuVersion::CCU_V1) { |
| 112 | return ccuResData_.v1Res[rankId]->gsa_[dieId][gsaId]; | 112 | return ccuResData_.v1Res[rankId]->gsa_[dieId][gsaId]; |
| 113 | } else { | 113 | } else { |
| 114 | - HCCL_VM_ERROR("[CcuResourceManager][GetGsaValue] ccu version {} not supported", static_cast<int>(version)); | 114 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 115 | return U64_INVALID; | 115 | return U64_INVALID; |
| 116 | } | 116 | } |
| 117 | } | 117 | } |
| @@ -122,7 +122,7 @@ uint16_t CcuResourceManager::GetCkeValue(int rankId, int dieId, uint16_t ckeId) | |||
| 122 | if (version == RunnerCcuVersion::CCU_V1) { | 122 | if (version == RunnerCcuVersion::CCU_V1) { |
| 123 | return ccuResData_.v1Res[rankId]->cke_[dieId][ckeId]; | 123 | return ccuResData_.v1Res[rankId]->cke_[dieId][ckeId]; |
| 124 | } else { | 124 | } else { |
| 125 | - HCCL_VM_ERROR("[CcuResourceManager][GetCkeValue] ccu version {} not supported", static_cast<int>(version)); | 125 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 126 | return U16_INVALID; | 126 | return U16_INVALID; |
| 127 | } | 127 | } |
| 128 | } | 128 | } |
| @@ -133,7 +133,7 @@ std::pair<int, int> CcuResourceManager::GetRmtCcu(int rankId, int dieId, uint16_ | |||
| 133 | int rmtDie = S32_INVALID; | 133 | int rmtDie = S32_INVALID; |
| 134 | 134 | ||
| 135 | if (channelId >= SimCcuV1::MAX_CCU_CHANNEL_NUM) { | 135 | if (channelId >= SimCcuV1::MAX_CCU_CHANNEL_NUM) { |
| 136 | - HCCL_VM_ERROR("[CcuResourceManager][GetRmtCcu] invalid channelId[{}], max={}", channelId, SimCcuV1::MAX_CCU_CHANNEL_NUM); | 136 | + HCCL_VM_ERROR("invalid channelId[{}], max={}", channelId, SimCcuV1::MAX_CCU_CHANNEL_NUM); |
| 137 | return std::make_pair(S32_INVALID, S32_INVALID); | 137 | return std::make_pair(S32_INVALID, S32_INVALID); |
| 138 | } | 138 | } |
| 139 | 139 | ||
| @@ -142,12 +142,12 @@ std::pair<int, int> CcuResourceManager::GetRmtCcu(int rankId, int dieId, uint16_ | |||
| 142 | rmtRank = ccuResData_.v1Res[rankId]->channelId2RmtRankMap_[dieId][channelId].rankId; | 142 | rmtRank = ccuResData_.v1Res[rankId]->channelId2RmtRankMap_[dieId][channelId].rankId; |
| 143 | rmtDie = ccuResData_.v1Res[rankId]->channelId2RmtRankMap_[dieId][channelId].dieId; | 143 | rmtDie = ccuResData_.v1Res[rankId]->channelId2RmtRankMap_[dieId][channelId].dieId; |
| 144 | } else { | 144 | } else { |
| 145 | - HCCL_VM_ERROR("[CcuResourceManager][GetRmtCcu] ccu version {} not supported", static_cast<int>(version)); | 145 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 146 | return std::make_pair(S32_INVALID, S32_INVALID); | 146 | return std::make_pair(S32_INVALID, S32_INVALID); |
| 147 | } | 147 | } |
| 148 | 148 | ||
| 149 | if (dieId >= HcclSim::DIE_NUM || rmtRank == S32_INVALID || rmtDie == S32_INVALID) { | 149 | if (dieId >= HcclSim::DIE_NUM || rmtRank == S32_INVALID || rmtDie == S32_INVALID) { |
| 150 | - HCCL_VM_ERROR("[CcuResourceManager][GetRmtCcu] get invalid value, dieId{}, rmtRank{}, rmtDie{}", dieId, rmtRank, rmtDie); | 150 | + HCCL_VM_ERROR("get invalid value, dieId{}, rmtRank{}, rmtDie{}", dieId, rmtRank, rmtDie); |
| 151 | return std::make_pair(S32_INVALID, S32_INVALID); | 151 | return std::make_pair(S32_INVALID, S32_INVALID); |
| 152 | } | 152 | } |
| 153 | 153 | ||
| @@ -160,7 +160,7 @@ void CcuResourceManager::UpdateXnValue(int rankId, int dieId, uint16_t xnId, uin | |||
| 160 | if (version == RunnerCcuVersion::CCU_V1) { | 160 | if (version == RunnerCcuVersion::CCU_V1) { |
| 161 | ccuResData_.v1Res[rankId]->xn_[dieId][xnId] = value; | 161 | ccuResData_.v1Res[rankId]->xn_[dieId][xnId] = value; |
| 162 | } else { | 162 | } else { |
| 163 | - HCCL_VM_ERROR("[CcuResourceManager][UpdateXnValue] ccu version {} not supported", static_cast<int>(version)); | 163 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 164 | } | 164 | } |
| 165 | } | 165 | } |
| 166 | 166 | ||
| @@ -170,7 +170,7 @@ void CcuResourceManager::UpdateGsaValue(int rankId, int dieId, uint16_t gsaId, u | |||
| 170 | if (version == RunnerCcuVersion::CCU_V1) { | 170 | if (version == RunnerCcuVersion::CCU_V1) { |
| 171 | ccuResData_.v1Res[rankId]->gsa_[dieId][gsaId] = value; | 171 | ccuResData_.v1Res[rankId]->gsa_[dieId][gsaId] = value; |
| 172 | } else { | 172 | } else { |
| 173 | - HCCL_VM_ERROR("[CcuResourceManager][UpdateGsaValue] ccu version {} not supported", static_cast<int>(version)); | 173 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 174 | } | 174 | } |
| 175 | } | 175 | } |
| 176 | 176 | ||
| @@ -180,34 +180,34 @@ void CcuResourceManager::UpdateCkeValue(int rankId, int dieId, uint16_t ckeId, u | |||
| 180 | if (version == RunnerCcuVersion::CCU_V1) { | 180 | if (version == RunnerCcuVersion::CCU_V1) { |
| 181 | ccuResData_.v1Res[rankId]->cke_[dieId][ckeId] = value; | 181 | ccuResData_.v1Res[rankId]->cke_[dieId][ckeId] = value; |
| 182 | } else { | 182 | } else { |
| 183 | - HCCL_VM_ERROR("[CcuResourceManager][UpdateCkeValue] ccu version {} not supported", static_cast<int>(version)); | 183 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 184 | } | 184 | } |
| 185 | } | 185 | } |
| 186 | 186 | ||
| 187 | void CcuResourceManager::TransMemToMem(void *srcBuf, void *dstBuf, uint64_t length, bool reduceEn, uint16_t reduceOp, uint16_t dataType) | 187 | void CcuResourceManager::TransMemToMem(void *srcBuf, void *dstBuf, uint64_t length, bool reduceEn, uint16_t reduceOp, uint16_t dataType) |
| 188 | { | 188 | { |
| 189 | - HCCL_VM_DEBUG("[CcuResourceManager][TransMemToMem] srcBuf={}, dstBuf={}, length={}", srcBuf, dstBuf, length); | 189 | + HCCL_VM_DEBUG("srcBuf={}, dstBuf={}, length={}", srcBuf, dstBuf, length); |
| 190 | if (srcBuf == nullptr) { | 190 | if (srcBuf == nullptr) { |
| 191 | - HCCL_VM_ERROR("[CcuResourceManager][TransMemToMem] srcBuf invalid..."); | 191 | + HCCL_VM_ERROR("srcBuf invalid..."); |
| 192 | return; | 192 | return; |
| 193 | } | 193 | } |
| 194 | 194 | ||
| 195 | if (dstBuf == nullptr) { | 195 | if (dstBuf == nullptr) { |
| 196 | - HCCL_VM_ERROR("[CcuResourceManager][TransMemToMem] dstBuf invalid..."); | 196 | + HCCL_VM_ERROR("dstBuf invalid..."); |
| 197 | return; | 197 | return; |
| 198 | } | 198 | } |
| 199 | 199 | ||
| 200 | sim::PhyMemBlock srcPhyMem{}; | 200 | sim::PhyMemBlock srcPhyMem{}; |
| 201 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)srcBuf, srcPhyMem); | 201 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)srcBuf, srcPhyMem); |
| 202 | if (srcAddr == nullptr) { | 202 | if (srcAddr == nullptr) { |
| 203 | - HCCL_VM_ERROR("[TransMemToMem] 无法获取srcBuf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(srcBuf)); | 203 | + HCCL_VM_ERROR("无法获取srcBuf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(srcBuf)); |
| 204 | return; | 204 | return; |
| 205 | } | 205 | } |
| 206 | 206 | ||
| 207 | sim::PhyMemBlock dstPhyMem{}; | 207 | sim::PhyMemBlock dstPhyMem{}; |
| 208 | auto dstAddr = sim::AcquireDevPtrInNoHostProcess((void*)dstBuf, dstPhyMem); | 208 | auto dstAddr = sim::AcquireDevPtrInNoHostProcess((void*)dstBuf, dstPhyMem); |
| 209 | if (dstAddr == nullptr) { | 209 | if (dstAddr == nullptr) { |
| 210 | - HCCL_VM_ERROR("[TransMemToMem] 无法获取dstBuf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(dstBuf)); | 210 | + HCCL_VM_ERROR("无法获取dstBuf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(dstBuf)); |
| 211 | return; | 211 | return; |
| 212 | } | 212 | } |
| 213 | 213 | ||
| @@ -229,14 +229,14 @@ void CcuResourceManager::TransMSToMS(int srcRank, int srcDie, int dstRank, int d | |||
| 229 | bool CcuResourceManager::TransMSToMem(int rankId, int dieId, uint16_t msId, void *buf, uint16_t length) | 229 | bool CcuResourceManager::TransMSToMem(int rankId, int dieId, uint16_t msId, void *buf, uint16_t length) |
| 230 | { | 230 | { |
| 231 | if (buf == nullptr) { | 231 | if (buf == nullptr) { |
| 232 | - HCCL_VM_ERROR("[CcuResourceManager][TransMSToMem] param invalid..."); | 232 | + HCCL_VM_ERROR("param invalid..."); |
| 233 | return false; | 233 | return false; |
| 234 | } | 234 | } |
| 235 | 235 | ||
| 236 | sim::PhyMemBlock dstPhyMem{}; | 236 | sim::PhyMemBlock dstPhyMem{}; |
| 237 | auto dstAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, dstPhyMem); | 237 | auto dstAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, dstPhyMem); |
| 238 | if (dstAddr == nullptr) { | 238 | if (dstAddr == nullptr) { |
| 239 | - HCCL_VM_ERROR("[TransMSToMem] 无法获取buf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(buf)); | 239 | + HCCL_VM_ERROR("无法获取buf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(buf)); |
| 240 | return false; | 240 | return false; |
| 241 | } | 241 | } |
| 242 | 242 | ||
| @@ -251,7 +251,7 @@ bool CcuResourceManager::TransMemToXn(int rankId, int dieId, uint16_t xnId, uint | |||
| 251 | sim::PhyMemBlock srcPhyMem{}; | 251 | sim::PhyMemBlock srcPhyMem{}; |
| 252 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, srcPhyMem); | 252 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, srcPhyMem); |
| 253 | if (srcAddr == nullptr) { | 253 | if (srcAddr == nullptr) { |
| 254 | - HCCL_VM_ERROR("[TransMemToMS] 无法获取buf的设备地址(addr= {:x})!", buf); | 254 | + HCCL_VM_ERROR("无法获取buf的设备地址(addr= {:x})!", buf); |
| 255 | return false; | 255 | return false; |
| 256 | } | 256 | } |
| 257 | memcpy(GetXnAddr(rankId, dieId, xnId), srcAddr, length); | 257 | memcpy(GetXnAddr(rankId, dieId, xnId), srcAddr, length); |
| @@ -265,7 +265,7 @@ bool CcuResourceManager::TransXnToMem(int rankId, int dieId, uint16_t xnId, uint | |||
| 265 | sim::PhyMemBlock dstPhyMem{}; | 265 | sim::PhyMemBlock dstPhyMem{}; |
| 266 | auto dstAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, dstPhyMem); | 266 | auto dstAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, dstPhyMem); |
| 267 | if (dstAddr == nullptr) { | 267 | if (dstAddr == nullptr) { |
| 268 | - HCCL_VM_ERROR("[TransXnToMem] 无法获取buf的设备地址(addr= {:x})!", buf); | 268 | + HCCL_VM_ERROR("无法获取buf的设备地址(addr= {:x})!", buf); |
| 269 | return false; | 269 | return false; |
| 270 | } | 270 | } |
| 271 | memcpy(dstAddr, GetXnAddr(rankId, dieId, xnId), length); | 271 | memcpy(dstAddr, GetXnAddr(rankId, dieId, xnId), length); |
| @@ -277,14 +277,14 @@ bool CcuResourceManager::TransXnToMem(int rankId, int dieId, uint16_t xnId, uint | |||
| 277 | bool CcuResourceManager::TransMemToMS(int rankId, int dieId, uint16_t msId, void *buf, uint16_t length) | 277 | bool CcuResourceManager::TransMemToMS(int rankId, int dieId, uint16_t msId, void *buf, uint16_t length) |
| 278 | { | 278 | { |
| 279 | if (buf == nullptr) { | 279 | if (buf == nullptr) { |
| 280 | - HCCL_VM_ERROR("[CcuResourceManager][TransMemToMS] param invalid..."); | 280 | + HCCL_VM_ERROR("param invalid..."); |
| 281 | return false; | 281 | return false; |
| 282 | } | 282 | } |
| 283 | 283 | ||
| 284 | sim::PhyMemBlock srcPhyMem{}; | 284 | sim::PhyMemBlock srcPhyMem{}; |
| 285 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, srcPhyMem); | 285 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)buf, srcPhyMem); |
| 286 | if (srcAddr == nullptr) { | 286 | if (srcAddr == nullptr) { |
| 287 | - HCCL_VM_ERROR("[TransMemToMS] 无法获取buf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(buf)); | 287 | + HCCL_VM_ERROR("无法获取buf的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>(buf)); |
| 288 | return false; | 288 | return false; |
| 289 | } | 289 | } |
| 290 | memcpy(GetMsAddr(rankId, dieId, msId), srcAddr, length); | 290 | memcpy(GetMsAddr(rankId, dieId, msId), srcAddr, length); |
| @@ -300,7 +300,7 @@ char *CcuResourceManager::GetMsAddr(int rankId, int dieId, uint16_t msId) const | |||
| 300 | if (version == RunnerCcuVersion::CCU_V1) { | 300 | if (version == RunnerCcuVersion::CCU_V1) { |
| 301 | return (ccuResData_.v1Res[rankId]->ms_[dieId].data() + offset); | 301 | return (ccuResData_.v1Res[rankId]->ms_[dieId].data() + offset); |
| 302 | } else { | 302 | } else { |
| 303 | - HCCL_VM_ERROR("[CcuResourceManager][GetMsAddr] ccu version {} not supported", static_cast<int>(version)); | 303 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 304 | return nullptr; | 304 | return nullptr; |
| 305 | } | 305 | } |
| 306 | } | 306 | } |
| @@ -312,7 +312,7 @@ uint64_t *CcuResourceManager::GetXnAddr(int rankId, int dieId, uint16_t xnId) co | |||
| 312 | if (version == RunnerCcuVersion::CCU_V1) { | 312 | if (version == RunnerCcuVersion::CCU_V1) { |
| 313 | return (ccuResData_.v1Res[rankId]->xn_[dieId].data() + offset); | 313 | return (ccuResData_.v1Res[rankId]->xn_[dieId].data() + offset); |
| 314 | } else { | 314 | } else { |
| 315 | - HCCL_VM_ERROR("[CcuResourceManager][GetXnAddr] ccu version {} not supported", static_cast<int>(version)); | 315 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 316 | return nullptr; | 316 | return nullptr; |
| 317 | } | 317 | } |
| 318 | } | 318 | } |
| @@ -323,7 +323,7 @@ uint16_t CcuResourceManager::GetInstrCnt(int rankId, int dieId) const | |||
| 323 | if (version == RunnerCcuVersion::CCU_V1) { | 323 | if (version == RunnerCcuVersion::CCU_V1) { |
| 324 | return ccuResData_.v1Res[rankId]->instrSpace_[dieId].instrCnt; | 324 | return ccuResData_.v1Res[rankId]->instrSpace_[dieId].instrCnt; |
| 325 | } else { | 325 | } else { |
| 326 | - HCCL_VM_ERROR("[CcuResourceManager][GetInstrCnt] ccu version {} not supported", static_cast<int>(version)); | 326 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 327 | return U16_INVALID; | 327 | return U16_INVALID; |
| 328 | } | 328 | } |
| 329 | } | 329 | } |
| @@ -334,7 +334,7 @@ std::vector<hcomm::CcuRep::CcuInstr> CcuResourceManager::GetInstrData(int rankId | |||
| 334 | if (version == RunnerCcuVersion::CCU_V1) { | 334 | if (version == RunnerCcuVersion::CCU_V1) { |
| 335 | return ccuResData_.v1Res[rankId]->instrSpace_[dieId].instrData; | 335 | return ccuResData_.v1Res[rankId]->instrSpace_[dieId].instrData; |
| 336 | } else { | 336 | } else { |
| 337 | - HCCL_VM_ERROR("[CcuResourceManager][GetInstrData] ccu version {} not supported", static_cast<int>(version)); | 337 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 338 | return {}; | 338 | return {}; |
| 339 | } | 339 | } |
| 340 | } | 340 | } |
| @@ -346,7 +346,7 @@ std::string CcuResourceManager::GetInstrDescribe(int rankId, int dieId, int inst | |||
| 346 | if (version == RunnerCcuVersion::CCU_V1) { | 346 | if (version == RunnerCcuVersion::CCU_V1) { |
| 347 | // return hcomm::CcuRep::ParseInstr(&(ccuResData_.v1Res[rankId]->instrSpace_[dieId].instrData[instrId])); | 347 | // return hcomm::CcuRep::ParseInstr(&(ccuResData_.v1Res[rankId]->instrSpace_[dieId].instrData[instrId])); |
| 348 | } else { | 348 | } else { |
| 349 | - HCCL_VM_ERROR("[CcuResourceManager][GetInstrDescribe] ccu version {} not supported", static_cast<int>(version)); | 349 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 350 | return ""; | 350 | return ""; |
| 351 | } | 351 | } |
| 352 | 352 | ||
| @@ -365,7 +365,7 @@ std::shared_ptr<CcuSimulator> CcuResourceManager::InitSimulator(int rankId, int | |||
| 365 | return ccuResData_.v1Res[rankId]->simulators_[dieId]; | 365 | return ccuResData_.v1Res[rankId]->simulators_[dieId]; |
| 366 | } | 366 | } |
| 367 | } else { | 367 | } else { |
| 368 | - HCCL_VM_ERROR("[CcuResourceManager][GetInstrData] ccu version {} not supported", static_cast<int>(version)); | 368 | + HCCL_VM_ERROR("ccu version {} not supported", static_cast<int>(version)); |
| 369 | return nullptr; | 369 | return nullptr; |
| 370 | } | 370 | } |
| 371 | } | 371 | } |
| @@ -440,7 +440,7 @@ void CcuResourceManager::DumpCcuCkeResouceInfo(int rankId) const | |||
| 440 | void CcuResourceManager::DumpCcuChannelResouceInfo(int rankId) const | 440 | void CcuResourceManager::DumpCcuChannelResouceInfo(int rankId) const |
| 441 | { | 441 | { |
| 442 | for (int dieId = 0; dieId < HcclSim::DIE_NUM; dieId++) { | 442 | for (int dieId = 0; dieId < HcclSim::DIE_NUM; dieId++) { |
| 443 | - HCCL_VM_DEBUG("\n==========================All CCU CHANNEL Resouce Info Start: rank[{}], dieId[{}]===========================", rankId, dieId); | 443 | + HCCL_VM_DEBUG("==========================All CCU CHANNEL Resouce Info Start: rank[{}], dieId[{}]===========================", rankId, dieId); |
| 444 | HCCL_VM_DEBUG("-------------------------DieId[{}] CHANNEL Resouce Info Start-------------------------", dieId); | 444 | HCCL_VM_DEBUG("-------------------------DieId[{}] CHANNEL Resouce Info Start-------------------------", dieId); |
| 445 | for (uint32_t chId = 0; chId < SimCcuV1::MAX_CCU_CHANNEL_NUM; chId++) { | 445 | for (uint32_t chId = 0; chId < SimCcuV1::MAX_CCU_CHANNEL_NUM; chId++) { |
| 446 | auto rmtCcu = GetRmtCcu(rankId, dieId, chId); | 446 | auto rmtCcu = GetRmtCcu(rankId, dieId, chId); |
| @@ -449,7 +449,7 @@ void CcuResourceManager::DumpCcuChannelResouceInfo(int rankId) const | |||
| 449 | } | 449 | } |
| 450 | } | 450 | } |
| 451 | HCCL_VM_DEBUG("-------------------------DieId[{}] CHANNEL Resouce Info End-------------------------", dieId); | 451 | HCCL_VM_DEBUG("-------------------------DieId[{}] CHANNEL Resouce Info End-------------------------", dieId); |
| 452 | - HCCL_VM_DEBUG("==========================All CCU CHANNEL Resouce Info End: rank[{}], dieId[{}]===========================\n", rankId, dieId); | 452 | + HCCL_VM_DEBUG("==========================All CCU CHANNEL Resouce Info End: rank[{}], dieId[{}]===========================", rankId, dieId); |
| 453 | } | 453 | } |
| 454 | } | 454 | } |
| 455 | 455 | ||
| @@ -465,7 +465,7 @@ void CcuResourceManager::DumpChannelId2RmtRank(int rankId, int dieId) const | |||
| 465 | if (rmtCcu.first == INT32_MAX || rmtCcu.second == INT32_MAX) { | 465 | if (rmtCcu.first == INT32_MAX || rmtCcu.second == INT32_MAX) { |
| 466 | continue; | 466 | continue; |
| 467 | } | 467 | } |
| 468 | - HCCL_VM_DEBUG("[CcuResourceManager][DumpChannelId2RmtRank] channelId[{}], ccu[{}:{} --> {}:{}]", i, rankId, dieId, rmtCcu.first, rmtCcu.second); | 468 | + HCCL_VM_DEBUG("channelId[{}], ccu[{}:{} --> {}:{}]", i, rankId, dieId, rmtCcu.first, rmtCcu.second); |
| 469 | } | 469 | } |
| 470 | } | 470 | } |
| 471 | 471 | ||
| @@ -28,13 +28,13 @@ void CcuSimulator::InitLoopGroupInfo(uint16_t startLoopId, uint64_t offsetCfg, u | |||
| 28 | loopGroupInfo_.loopNum_ = (repeatCfg >> 41) & 0x7F; // 0x7F: 取loop指令个数,即[47:41]位 | 28 | loopGroupInfo_.loopNum_ = (repeatCfg >> 41) & 0x7F; // 0x7F: 取loop指令个数,即[47:41]位 |
| 29 | loopGroupInfo_.loopOffset_ = (repeatCfg >> 48) & 0x7F; // 0x7F: 取loop偏移,即[54:48]位 | 29 | loopGroupInfo_.loopOffset_ = (repeatCfg >> 48) & 0x7F; // 0x7F: 取loop偏移,即[54:48]位 |
| 30 | loopGroupInfo_.loopExtendNum_ = (repeatCfg >> 55) & 0x7F; // 0x7F: 取loop展开次数,即[61:55]位 | 30 | loopGroupInfo_.loopExtendNum_ = (repeatCfg >> 55) & 0x7F; // 0x7F: 取loop展开次数,即[61:55]位 |
| 31 | - HCCL_VM_DEBUG("[CcuSimulator][InitLoopGroupInfo] locCcu[{}:{}], curInstrId_=[{}], loopNum=[{}], loopOffset=[{}], loopExtendNum=[{}]", | 31 | + HCCL_VM_DEBUG("locCcu[{}:{}], curInstrId_=[{}], loopNum=[{}], loopOffset=[{}], loopExtendNum=[{}]", |
| 32 | rankId_, dieId_, curInstrId_, loopGroupInfo_.loopNum_, loopGroupInfo_.loopOffset_, loopGroupInfo_.loopExtendNum_); | 32 | rankId_, dieId_, curInstrId_, loopGroupInfo_.loopNum_, loopGroupInfo_.loopOffset_, loopGroupInfo_.loopExtendNum_); |
| 33 | 33 | ||
| 34 | loopGroupInfo_.ckeOffset_ = offsetCfg & 0x3FF; // 0x3FF: 取低[9:0]位 | 34 | loopGroupInfo_.ckeOffset_ = offsetCfg & 0x3FF; // 0x3FF: 取低[9:0]位 |
| 35 | loopGroupInfo_.msOffset_ = (offsetCfg >> 10) & 0x7FF; // 0x7FF: 取低[20:10]位 | 35 | loopGroupInfo_.msOffset_ = (offsetCfg >> 10) & 0x7FF; // 0x7FF: 取低[20:10]位 |
| 36 | loopGroupInfo_.gsaOffset_ = (offsetCfg >> 21) & 0xFFFFFFFF; // 0xFFFFFFFF: 取低[52:21]位 | 36 | loopGroupInfo_.gsaOffset_ = (offsetCfg >> 21) & 0xFFFFFFFF; // 0xFFFFFFFF: 取低[52:21]位 |
| 37 | - HCCL_VM_DEBUG("[CcuSimulator][InitLoopGroupInfo] locCcu[{}:{}], ckeOffset=[{}], msOffset=[{}], gsaOffset=[{}]", | 37 | + HCCL_VM_DEBUG("locCcu[{}:{}], ckeOffset=[{}], msOffset=[{}], gsaOffset=[{}]", |
| 38 | rankId_, dieId_, loopGroupInfo_.ckeOffset_, loopGroupInfo_.msOffset_, loopGroupInfo_.gsaOffset_); | 38 | rankId_, dieId_, loopGroupInfo_.ckeOffset_, loopGroupInfo_.msOffset_, loopGroupInfo_.gsaOffset_); |
| 39 | } | 39 | } |
| 40 | 40 | ||
| @@ -47,7 +47,7 @@ void CcuSimulator::InitLoopInfo(uint16_t startInstrId, uint16_t endInstrId, uint | |||
| 47 | loopGroupInfo_.loopStatus_.curLoopRound = 0; | 47 | loopGroupInfo_.loopStatus_.curLoopRound = 0; |
| 48 | loopGroupInfo_.loopStatus_.loopGsaIterStep = addrStep; | 48 | loopGroupInfo_.loopStatus_.loopGsaIterStep = addrStep; |
| 49 | 49 | ||
| 50 | - HCCL_VM_DEBUG("[CcuSimulator][InitLoopInfo] locCcu[{}:{}], loopStartInstrId=[{}], loopCurInstrId=[{}], " | 50 | + HCCL_VM_DEBUG("locCcu[{}:{}], loopStartInstrId=[{}], loopCurInstrId=[{}], " |
| 51 | "loopEndInstrId=[{}], loopExecCount=[{}], loopGsaIterStep=[{}]", | 51 | "loopEndInstrId=[{}], loopExecCount=[{}], loopGsaIterStep=[{}]", |
| 52 | rankId_, dieId_, startInstrId, startInstrId, endInstrId, execCount, addrStep); | 52 | rankId_, dieId_, startInstrId, startInstrId, endInstrId, execCount, addrStep); |
| 53 | } | 53 | } |
| @@ -192,16 +192,16 @@ bool CcuSimulator::ExecuteInstr(uint16_t curInstrId) | |||
| 192 | { | 192 | { |
| 193 | auto &ccuResMgr = CcuResourceManager::GetInstance(); | 193 | auto &ccuResMgr = CcuResourceManager::GetInstance(); |
| 194 | auto instrData = ccuResMgr.GetInstrData(rankId_, dieId_); | 194 | auto instrData = ccuResMgr.GetInstrData(rankId_, dieId_); |
| 195 | - HCCL_VM_DEBUG("[CcuSimulator][ExecuteInstr] locCcu[{}:{}], current instr[{}], state=[{}]", rankId_, dieId_, curInstrId, static_cast<int>(state_)); | 195 | + HCCL_VM_DEBUG("locCcu[{}:{}], current instr[{}], state=[{}]", rankId_, dieId_, curInstrId, static_cast<int>(state_)); |
| 196 | if (curInstrId >= endInstrId_) { | 196 | if (curInstrId >= endInstrId_) { |
| 197 | state_ = CcuExecState::EXEC_FAIL; | 197 | state_ = CcuExecState::EXEC_FAIL; |
| 198 | - HCCL_VM_ERROR("[CcuSimulator][ExecuteInstr] locCcu[{}:{}], Invalid curInstrId_[{}], endInstrId[{}]", rankId_, dieId_, curInstrId, endInstrId_); | 198 | + HCCL_VM_ERROR("locCcu[{}:{}], Invalid curInstrId_[{}], endInstrId[{}]", rankId_, dieId_, curInstrId, endInstrId_); |
| 199 | return false; | 199 | return false; |
| 200 | } | 200 | } |
| 201 | auto executor = CcuExecutorFactory::MakeCcuExecutorInstance(version_, instrData[curInstrId].header.header, 0, rankId_, dieId_, instrData[curInstrId], this); | 201 | auto executor = CcuExecutorFactory::MakeCcuExecutorInstance(version_, instrData[curInstrId].header.header, 0, rankId_, dieId_, instrData[curInstrId], this); |
| 202 | if (executor == nullptr) { | 202 | if (executor == nullptr) { |
| 203 | UpdateLoopStatus(); | 203 | UpdateLoopStatus(); |
| 204 | - HCCL_VM_DEBUG("[CcuSimulator][ExecuteInstr] locCcu[{}:{}], curInstrId_[{}] executor is null.", rankId_, dieId_, curInstrId); | 204 | + HCCL_VM_DEBUG("locCcu[{}:{}], curInstrId_[{}] executor is null.", rankId_, dieId_, curInstrId); |
| 205 | return true; // 让还未添加的Executor先跑下去,看下是否能从start->loop->end | 205 | return true; // 让还未添加的Executor先跑下去,看下是否能从start->loop->end |
| 206 | } | 206 | } |
| 207 | executor->Parser(); | 207 | executor->Parser(); |
| @@ -210,7 +210,7 @@ bool CcuSimulator::ExecuteInstr(uint16_t curInstrId) | |||
| 210 | return false; | 210 | return false; |
| 211 | } | 211 | } |
| 212 | 212 | ||
| 213 | - HCCL_VM_DEBUG("[CcuSimulator][ExecuteInstr] locCcu[{}:{}], end instr[{}], {}", rankId_, dieId_, curInstrId, executor->Describe()); | 213 | + HCCL_VM_DEBUG("locCcu[{}:{}], end instr[{}], {}", rankId_, dieId_, curInstrId, executor->Describe()); |
| 214 | return true; | 214 | return true; |
| 215 | } | 215 | } |
| 216 | 216 | ||
| @@ -219,7 +219,7 @@ bool CcuSimulator::ExecuteLoop() | |||
| 219 | { | 219 | { |
| 220 | auto instrCnt = loopGroupInfo_.loopStatus_.loopEndInstrId - loopGroupInfo_.loopStatus_.loopStartInstrId + 1; | 220 | auto instrCnt = loopGroupInfo_.loopStatus_.loopEndInstrId - loopGroupInfo_.loopStatus_.loopStartInstrId + 1; |
| 221 | for (uint32_t i = 0; i < loopGroupInfo_.loopStatus_.loopExecCount; i++) { | 221 | for (uint32_t i = 0; i < loopGroupInfo_.loopStatus_.loopExecCount; i++) { |
| 222 | - HCCL_VM_DEBUG("[CcuSimulator][ExecuteLoop] locCcu[{}:{}], loopStartInstrId=[{}], loopCurInstrId=[{}], instrCnt=[{}], loopRound=[{}].", | 222 | + HCCL_VM_DEBUG("locCcu[{}:{}], loopStartInstrId=[{}], loopCurInstrId=[{}], instrCnt=[{}], loopRound=[{}].", |
| 223 | rankId_, dieId_, loopGroupInfo_.loopStatus_.loopStartInstrId, curInstrId_, instrCnt, i); | 223 | rankId_, dieId_, loopGroupInfo_.loopStatus_.loopStartInstrId, curInstrId_, instrCnt, i); |
| 224 | auto simulator = std::make_unique<CcuSimulator>( | 224 | auto simulator = std::make_unique<CcuSimulator>( |
| 225 | rankId_, dieId_, loopGroupInfo_.loopStatus_.loopStartInstrId, loopGroupInfo_.loopStatus_.loopEndInstrId + 1, instrCnt, version_); | 225 | rankId_, dieId_, loopGroupInfo_.loopStatus_.loopStartInstrId, loopGroupInfo_.loopStatus_.loopEndInstrId + 1, instrCnt, version_); |
| @@ -234,7 +234,7 @@ bool CcuSimulator::ExecuteLoop() | |||
| 234 | // loopGroup内所有展开的loop执行,后续改为多线程并行执行 | 234 | // loopGroup内所有展开的loop执行,后续改为多线程并行执行 |
| 235 | bool CcuSimulator::ExecuteLoopGroup() | 235 | bool CcuSimulator::ExecuteLoopGroup() |
| 236 | { | 236 | { |
| 237 | - HCCL_VM_DEBUG("[CcuSimulator][ExecuteLoopGroup] locCcu[{}:{}], lopNum=[{}], loopExtendNum=[{}], loopOffset=[{}].", | 237 | + HCCL_VM_DEBUG("locCcu[{}:{}], lopNum=[{}], loopExtendNum=[{}], loopOffset=[{}].", |
| 238 | rankId_, dieId_, loopGroupInfo_.loopNum_, loopGroupInfo_.loopExtendNum_, loopGroupInfo_.loopOffset_); | 238 | rankId_, dieId_, loopGroupInfo_.loopNum_, loopGroupInfo_.loopExtendNum_, loopGroupInfo_.loopOffset_); |
| 239 | if (loopGroupInfo_.loopNum_ == 0) { | 239 | if (loopGroupInfo_.loopNum_ == 0) { |
| 240 | return true; | 240 | return true; |
| @@ -39,7 +39,7 @@ void LoadImdToXnExecutor::Run() | |||
| 39 | { | 39 | { |
| 40 | auto &ccuResMgr = CcuResourceManager::GetInstance(); | 40 | auto &ccuResMgr = CcuResourceManager::GetInstance(); |
| 41 | ccuResMgr.UpdateXnValue(rankId_, dieId_, xnId_, immediate_); | 41 | ccuResMgr.UpdateXnValue(rankId_, dieId_, xnId_, immediate_); |
| 42 | - HCCL_VM_DEBUG("[LoadImdToXnExecutor] Load immediate: locCcu[{}:{}], XnId=[{}], immediate=[{}]", | 42 | + HCCL_VM_DEBUG("Load immediate: locCcu[{}:{}], XnId=[{}], immediate=[{}]", |
| 43 | rankId_, dieId_, xnId_, immediate_); | 43 | rankId_, dieId_, xnId_, immediate_); |
| 44 | } | 44 | } |
| 45 | 45 | ||
| @@ -37,7 +37,7 @@ void LoadSqeArgsToGsaExecutor::Run() { | |||
| 37 | uint64_t sqeArgValue = ccuResMgr.GetSqeArgValue(rankId_, dieId_, sqeArgId_); | 37 | uint64_t sqeArgValue = ccuResMgr.GetSqeArgValue(rankId_, dieId_, sqeArgId_); |
| 38 | 38 | ||
| 39 | ccuResMgr.UpdateXnValue(rankId_, dieId_, gsaId_, sqeArgValue); | 39 | ccuResMgr.UpdateXnValue(rankId_, dieId_, gsaId_, sqeArgValue); |
| 40 | - HCCL_VM_DEBUG("[LoadSqeArgsToGsaExecutor] Load SqeArg: locCcu[{}:{}], GSAId=[{}], sqeArgValue=[{}]", | 40 | + HCCL_VM_DEBUG("Load SqeArg: locCcu[{}:{}], GSAId=[{}], sqeArgValue=[{}]", |
| 41 | rankId_, dieId_, gsaId_, sqeArgValue); | 41 | rankId_, dieId_, gsaId_, sqeArgValue); |
| 42 | } | 42 | } |
| 43 | 43 | ||
| @@ -30,7 +30,7 @@ void LoadSqeArgsToXnExecutor::Parser() { | |||
| 30 | xnId_ = instr_.v1.loadSqeArgsToXn.xnId; | 30 | xnId_ = instr_.v1.loadSqeArgsToXn.xnId; |
| 31 | sqeArgId_ = instr_.v1.loadSqeArgsToXn.sqeArgsId; | 31 | sqeArgId_ = instr_.v1.loadSqeArgsToXn.sqeArgsId; |
| 32 | } else { | 32 | } else { |
| 33 | - HCCL_VM_ERROR("[LoadSqeArgsToXnExecutor] Unsupported CCU version: {}", RunnerCcuVersionToString(version_)); | 33 | + HCCL_VM_ERROR("Unsupported CCU version: {}", RunnerCcuVersionToString(version_)); |
| 34 | } | 34 | } |
| 35 | } | 35 | } |
| 36 | 36 | ||
| @@ -39,7 +39,7 @@ void LoadSqeArgsToXnExecutor::Run() { | |||
| 39 | auto &ccuResMgr = CcuResourceManager::GetInstance(); | 39 | auto &ccuResMgr = CcuResourceManager::GetInstance(); |
| 40 | uint64_t sqeArgValue = ccuResMgr.GetSqeArgValue(rankId_, dieId_, sqeArgId_); | 40 | uint64_t sqeArgValue = ccuResMgr.GetSqeArgValue(rankId_, dieId_, sqeArgId_); |
| 41 | 41 | ||
| 42 | - HCCL_VM_DEBUG("[LoadSqeArgsToXnExecutor] Load arg: locCcu[{}:{}], XnId=[{}], argId=[{}], value=[{}]", | 42 | + HCCL_VM_DEBUG("Load arg: locCcu[{}:{}], XnId=[{}], argId=[{}], value=[{}]", |
| 43 | rankId_, dieId_, xnId_, sqeArgId_, sqeArgValue); | 43 | rankId_, dieId_, xnId_, sqeArgId_, sqeArgValue); |
| 44 | ccuResMgr.UpdateXnValue(rankId_, dieId_, xnId_, sqeArgValue); | 44 | ccuResMgr.UpdateXnValue(rankId_, dieId_, xnId_, sqeArgValue); |
| 45 | } | 45 | } |
| @@ -41,7 +41,7 @@ void LoadXnXnExecutor::Run() | |||
| 41 | uint64_t val = xn1 + xn2; | 41 | uint64_t val = xn1 + xn2; |
| 42 | ccuResMgr.UpdateXnValue(rankId_, dieId_, xdId_, val); | 42 | ccuResMgr.UpdateXnValue(rankId_, dieId_, xdId_, val); |
| 43 | 43 | ||
| 44 | - HCCL_VM_DEBUG("[LoadXnXnExecutor] Load Xn[%u] + Xn[%u] to Xn[%u]", | 44 | + HCCL_VM_DEBUG("Load Xn[{}] + Xn[{}] to Xn[{}]", |
| 45 | xmId_, xnId_, xdId_); | 45 | xmId_, xnId_, xdId_); |
| 46 | } | 46 | } |
| 47 | 47 | ||
| @@ -51,7 +51,7 @@ void ReduceAddExecutor::Parser() | |||
| 51 | // Reduce Add操作 | 51 | // Reduce Add操作 |
| 52 | void ReduceAddExecutor::Process(CcuResourceManager &ccuResMgr) | 52 | void ReduceAddExecutor::Process(CcuResourceManager &ccuResMgr) |
| 53 | { | 53 | { |
| 54 | - HCCL_VM_DEBUG("[ReduceAddExecutor][Process] Reduce Add info, locCcu[{}:{}], count=[{}], castEn=[{}], dataType=[{}]", | 54 | + HCCL_VM_DEBUG("Reduce Add info, locCcu[{}:{}], count=[{}], castEn=[{}], dataType=[{}]", |
| 55 | rankId_, dieId_, count_, castEn_, dataType_); | 55 | rankId_, dieId_, count_, castEn_, dataType_); |
| 56 | for (uint32_t i = 0; i < CCU_REDUCE_MAX_MS; i++) { | 56 | for (uint32_t i = 0; i < CCU_REDUCE_MAX_MS; i++) { |
| 57 | HCCL_VM_TRACE("msId_[{}]:dieId[{}], msId[{}]", i, msId_[i] >> 15, msId_[i] & 0x7FFF); | 57 | HCCL_VM_TRACE("msId_[{}]:dieId[{}], msId[{}]", i, msId_[i] >> 15, msId_[i] & 0x7FFF); |
| @@ -68,7 +68,7 @@ void ReduceAddExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 68 | for (uint32_t i = 0; i < hcomm::CcuRep::CCU_REDUCE_MAX_MS; i++) { | 68 | for (uint32_t i = 0; i < hcomm::CcuRep::CCU_REDUCE_MAX_MS; i++) { |
| 69 | msId_[i] += msOffset; | 69 | msId_[i] += msOffset; |
| 70 | } | 70 | } |
| 71 | - HCCL_VM_DEBUG("[ReduceAddExecutor][Process] locCcu[{}:{}], ms offset=[{}], ckeOffset[{}]", rankId_, dieId_, msOffset, ckeOffset); | 71 | + HCCL_VM_DEBUG("locCcu[{}:{}], ms offset=[{}], ckeOffset[{}]", rankId_, dieId_, msOffset, ckeOffset); |
| 72 | } | 72 | } |
| 73 | // 2. reduce操作 | 73 | // 2. reduce操作 |
| 74 | ReduceAddDataType type = static_cast<ReduceAddDataType>(dataType_); | 74 | ReduceAddDataType type = static_cast<ReduceAddDataType>(dataType_); |
| @@ -49,7 +49,7 @@ void ReduceMaxExecutor::Parser() | |||
| 49 | // Reduce Max操作 | 49 | // Reduce Max操作 |
| 50 | void ReduceMaxExecutor::Process(CcuResourceManager &ccuResMgr) | 50 | void ReduceMaxExecutor::Process(CcuResourceManager &ccuResMgr) |
| 51 | { | 51 | { |
| 52 | - HCCL_VM_DEBUG("[ReduceMaxExecutor][Process] Reduce Max info, locCcu[{}:{}], count:[{}], dataType:[{}]", rankId_, dieId_, count_, dataType_); | 52 | + HCCL_VM_DEBUG("Reduce Max info, locCcu[{}:{}], count:[{}], dataType:[{}]", rankId_, dieId_, count_, dataType_); |
| 53 | if (dataType_ >= ReduceMaxMinDataType::MAX_MIN_RESERVED4 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED1 || | 53 | if (dataType_ >= ReduceMaxMinDataType::MAX_MIN_RESERVED4 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED1 || |
| 54 | dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED2 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED3) { | 54 | dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED2 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED3) { |
| 55 | return; | 55 | return; |
| @@ -66,7 +66,7 @@ void ReduceMaxExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 66 | for (uint32_t i = 0; i < hcomm::CcuRep::CCU_REDUCE_MAX_MS; i++) { | 66 | for (uint32_t i = 0; i < hcomm::CcuRep::CCU_REDUCE_MAX_MS; i++) { |
| 67 | msId_[i] += msOffset; | 67 | msId_[i] += msOffset; |
| 68 | } | 68 | } |
| 69 | - HCCL_VM_DEBUG("[ReduceMaxExecutor][Process] locCcu[{}:{}], ms offset=[{}]", rankId_, dieId_, msOffset); | 69 | + HCCL_VM_DEBUG("locCcu[{}:{}], ms offset=[{}]", rankId_, dieId_, msOffset); |
| 70 | } | 70 | } |
| 71 | // 2. reduce操作 | 71 | // 2. reduce操作 |
| 72 | ReduceMaxMinDataType type = static_cast<ReduceMaxMinDataType>(dataType_); | 72 | ReduceMaxMinDataType type = static_cast<ReduceMaxMinDataType>(dataType_); |
| @@ -49,7 +49,7 @@ void ReduceMinExecutor::Parser() | |||
| 49 | // Reduce Min操作 | 49 | // Reduce Min操作 |
| 50 | void ReduceMinExecutor::Process(CcuResourceManager &ccuResMgr) | 50 | void ReduceMinExecutor::Process(CcuResourceManager &ccuResMgr) |
| 51 | { | 51 | { |
| 52 | - HCCL_VM_DEBUG("[ReduceMinExecutor][Process] Reduce Min info, locCcu[{}:{}], count:[{}], dataType:[{}]", rankId_, dieId_, count_, dataType_); | 52 | + HCCL_VM_DEBUG("Reduce Min info, locCcu[{}:{}], count:[{}], dataType:[{}]", rankId_, dieId_, count_, dataType_); |
| 53 | if (dataType_ >= ReduceMaxMinDataType::MAX_MIN_RESERVED4 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED1 || | 53 | if (dataType_ >= ReduceMaxMinDataType::MAX_MIN_RESERVED4 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED1 || |
| 54 | dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED2 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED3) { | 54 | dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED2 || dataType_ == ReduceMaxMinDataType::MAX_MIN_RESERVED3) { |
| 55 | return; | 55 | return; |
| @@ -66,7 +66,7 @@ void ReduceMinExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 66 | for (uint32_t i = 0; i < hcomm::CcuRep::CCU_REDUCE_MAX_MS; i++) { | 66 | for (uint32_t i = 0; i < hcomm::CcuRep::CCU_REDUCE_MAX_MS; i++) { |
| 67 | msId_[i] += msOffset; | 67 | msId_[i] += msOffset; |
| 68 | } | 68 | } |
| 69 | - HCCL_VM_DEBUG("[ReduceMinExecutor][Process] locCcu[{}:{}], ms offset=[{}]", rankId_, dieId_, msOffset); | 69 | + HCCL_VM_DEBUG("locCcu[{}:{}], ms offset=[{}]", rankId_, dieId_, msOffset); |
| 70 | } | 70 | } |
| 71 | // 2. reduce操作 | 71 | // 2. reduce操作 |
| 72 | ReduceMaxMinDataType type = static_cast<ReduceMaxMinDataType>(dataType_); | 72 | ReduceMaxMinDataType type = static_cast<ReduceMaxMinDataType>(dataType_); |
| @@ -47,7 +47,7 @@ void SyncCkeExecutor::Process(CcuResourceManager &ccuResMgr) { | |||
| 47 | uint16_t newRmtCke = rmtCke | (locCKE & locCKEMask_); | 47 | uint16_t newRmtCke = rmtCke | (locCKE & locCKEMask_); |
| 48 | ccuResMgr.UpdateCkeValue(rmtCcu.first, rmtCcu.second, rmtCKEId_, newRmtCke); | 48 | ccuResMgr.UpdateCkeValue(rmtCcu.first, rmtCcu.second, rmtCKEId_, newRmtCke); |
| 49 | 49 | ||
| 50 | - HCCL_VM_DEBUG("[SyncCkeExecutor][Process] locCcu[{}:{}], channelId=[{}], " | 50 | + HCCL_VM_DEBUG("locCcu[{}:{}], channelId=[{}], " |
| 51 | "rmtDevice[{}:{}], locCKE=[{}:{:04x}], " | 51 | "rmtDevice[{}:{}], locCKE=[{}:{:04x}], " |
| 52 | "value=[{}], rmtCKE=[{}:{:04x}], value=[{} --> {}].", | 52 | "value=[{}], rmtCKE=[{}:{:04x}], value=[{} --> {}].", |
| 53 | rankId_, dieId_, channelId_, rmtCcu.first, rmtCcu.second, locCKEId_, | 53 | rankId_, dieId_, channelId_, rmtCcu.first, rmtCcu.second, locCKEId_, |
| @@ -42,7 +42,7 @@ void SyncXnExecutor::Process(CcuResourceManager &ccuResMgr) { | |||
| 42 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); | 42 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); |
| 43 | // 将本端的xn内容写到目的端的xn中 | 43 | // 将本端的xn内容写到目的端的xn中 |
| 44 | auto locXnValue = ccuResMgr.GetXnValue(rankId_, dieId_, locXnId_); | 44 | auto locXnValue = ccuResMgr.GetXnValue(rankId_, dieId_, locXnId_); |
| 45 | - HCCL_VM_INFO("[SyncXnExecutor]sync xn, rankId{}, dieId{}, xnId{}, channelId{}, locXnValue: {}", rmtCcu.first, rmtCcu.second, rmtXnId_, channelId_, locXnValue); | 45 | + HCCL_VM_INFO("sync xn, rankId{}, dieId{}, xnId{}, channelId{}, locXnValue: {}", rmtCcu.first, rmtCcu.second, rmtXnId_, channelId_, locXnValue); |
| 46 | ccuResMgr.UpdateXnValue(rmtCcu.first, rmtCcu.second, rmtXnId_, locXnValue); | 46 | ccuResMgr.UpdateXnValue(rmtCcu.first, rmtCcu.second, rmtXnId_, locXnValue); |
| 47 | 47 | ||
| 48 | // 设置目的端的cke | 48 | // 设置目的端的cke |
Mtest/hccl_vm/src/plugin/runner/ccu_executor/trans_executor/trans_loc_mem_to_loc_mem_executor.cc+2-2
| @@ -56,9 +56,9 @@ void TransLocMemToLocMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 56 | srcLocAddr += gsaOffset; | 56 | srcLocAddr += gsaOffset; |
| 57 | dstLocAddr += gsaOffset; | 57 | dstLocAddr += gsaOffset; |
| 58 | setCKEId_ += ckeOffset; | 58 | setCKEId_ += ckeOffset; |
| 59 | - HCCL_VM_DEBUG("[TransLocMemToLocMemExecutor][Process] locCcu[{}:{}], Get gsa addr offset = [{:x}], cke id offset = [{}]", rankId_, dieId_, gsaOffset, ckeOffset); | 59 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa addr offset = [{:x}], cke id offset = [{}]", rankId_, dieId_, gsaOffset, ckeOffset); |
| 60 | } | 60 | } |
| 61 | - HCCL_VM_DEBUG("[TransLocMemToLocMemExecutor][Process] locCcu[{}:{}] Trans data from srcLocGSAId_[{}] srcLocAddr[{:x}] to dstLocGSAId_[{}] " | 61 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data from srcLocGSAId_[{}] srcLocAddr[{:x}] to dstLocGSAId_[{}] " |
| 62 | "dstLocAddr[{:x}], with lengthXnId[{}] transLength[{}].", | 62 | "dstLocAddr[{:x}], with lengthXnId[{}] transLength[{}].", |
| 63 | rankId_, dieId_, srcGSAId_, srcLocAddr, dstGSAId_, dstLocAddr, lengthXnId_, transLength_); | 63 | rankId_, dieId_, srcGSAId_, srcLocAddr, dstGSAId_, dstLocAddr, lengthXnId_, transLength_); |
| 64 | ccuResMgr.TransMemToMem(reinterpret_cast<void *>(srcLocAddr), reinterpret_cast<void *>(dstLocAddr), transLength_, false, 0, 0); | 64 | ccuResMgr.TransMemToMem(reinterpret_cast<void *>(srcLocAddr), reinterpret_cast<void *>(dstLocAddr), transLength_, false, 0, 0); |
| @@ -56,11 +56,11 @@ void TransLocMemToLocMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 56 | locAddr += addrOffset; | 56 | locAddr += addrOffset; |
| 57 | locMSId_ += msOffset; | 57 | locMSId_ += msOffset; |
| 58 | setCKEId_ += ckeOffset; | 58 | setCKEId_ += ckeOffset; |
| 59 | - HCCL_VM_DEBUG("[TransLocMemToLocMSExecutor][Process] locCcu[{}:{}], Get gsa " | 59 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa " |
| 60 | "addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", | 60 | "addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", |
| 61 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); | 61 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); |
| 62 | } | 62 | } |
| 63 | - HCCL_VM_DEBUG("[TransLocMemToLocMSExecutor][Process] locCcu[{}:{}] Trans data " | 63 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 64 | "from locGSAId_[{}] locAddr[{:x}] to locMsId[{:04x}], " | 64 | "from locGSAId_[{}] locAddr[{:x}] to locMsId[{:04x}], " |
| 65 | "with lengthXnId[{}] transLength[{}], lengthEn_[{}].", | 65 | "with lengthXnId[{}] transLength[{}], lengthEn_[{}].", |
| 66 | rankId_, dieId_, locGSAId_, locAddr, locMSId_, lengthXnId_, | 66 | rankId_, dieId_, locGSAId_, locAddr, locMSId_, lengthXnId_, |
Mtest/hccl_vm/src/plugin/runner/ccu_executor/trans_executor/trans_loc_mem_to_rmt_mem_executor.cc+2-2
| @@ -59,12 +59,12 @@ void TransLocMemToRmtMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 59 | rmtAddr += offset; | 59 | rmtAddr += offset; |
| 60 | locAddr += offset; | 60 | locAddr += offset; |
| 61 | setCKEId_ += ckeOffset; | 61 | setCKEId_ += ckeOffset; |
| 62 | - HCCL_VM_DEBUG("[TransLocMemToRmtMemExecutor][Process] locCcu[{}:{}], Get gsa " | 62 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa " |
| 63 | "addr offset = [{:04x}], cke offset = [{:04x}]", | 63 | "addr offset = [{:04x}], cke offset = [{:04x}]", |
| 64 | rankId_, dieId_, offset, ckeOffset); | 64 | rankId_, dieId_, offset, ckeOffset); |
| 65 | } | 65 | } |
| 66 | // 3.搬运动作 | 66 | // 3.搬运动作 |
| 67 | - HCCL_VM_DEBUG("[TransLocMemToRmtMemExecutor][Process] locCcu[{}:{}] Trans data " | 67 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 68 | "from locGSAId_[{}] locAddr[{:x}] to rmtGSAId_[{}] rmtAddr[{:x}], " | 68 | "from locGSAId_[{}] locAddr[{:x}] to rmtGSAId_[{}] rmtAddr[{:x}], " |
| 69 | "with lengthXnId[{}] transLength[{}], reduceEn[{}], reduceOp[{}].", | 69 | "with lengthXnId[{}] transLength[{}], reduceEn[{}], reduceOp[{}].", |
| 70 | rankId_, dieId_, locGSAId_, locAddr, rmtGSAId_, rmtAddr, lengthXnId_, transLength_, reduceEn_, reduceOpCode_); | 70 | rankId_, dieId_, locGSAId_, locAddr, rmtGSAId_, rmtAddr, lengthXnId_, transLength_, reduceEn_, reduceOpCode_); |
| @@ -56,10 +56,10 @@ void TransLocMSToLocMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 56 | locAddr += addrOffset; | 56 | locAddr += addrOffset; |
| 57 | locMSId_ += msOffset; | 57 | locMSId_ += msOffset; |
| 58 | setCKEId_ += ckeOffset; | 58 | setCKEId_ += ckeOffset; |
| 59 | - HCCL_VM_DEBUG("[TransLocMSToLocMemExecutor][Process] locCcu[{}:{}], Get gsa addr offset = [{:04x}], ms offset = [{}], " | 59 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa addr offset = [{:04x}], ms offset = [{}], " |
| 60 | "cke offset = [{:04x}]", rankId_, dieId_, addrOffset, msOffset, ckeOffset); | 60 | "cke offset = [{:04x}]", rankId_, dieId_, addrOffset, msOffset, ckeOffset); |
| 61 | } | 61 | } |
| 62 | - HCCL_VM_DEBUG("[TransLocMSToLocMemExecutor][Process] locCcu[{}:{}] Trans data " | 62 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 63 | "from locMsId[{}] to locGSAId[{}] locAddr[{:x}], " | 63 | "from locMsId[{}] to locGSAId[{}] locAddr[{:x}], " |
| 64 | "with lengthXnId[{}] transLength[{}], lengthEn_[{}].", | 64 | "with lengthXnId[{}] transLength[{}], lengthEn_[{}].", |
| 65 | rankId_, dieId_, locMSId_, locGSAId_, locAddr, lengthXnId_, | 65 | rankId_, dieId_, locMSId_, locGSAId_, locAddr, lengthXnId_, |
| @@ -54,10 +54,10 @@ void TransLocMSToLocMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 54 | srcMSId_ += msOffset; | 54 | srcMSId_ += msOffset; |
| 55 | dstMSId_ += msOffset; | 55 | dstMSId_ += msOffset; |
| 56 | setCKEId_ += ckeOffset; | 56 | setCKEId_ += ckeOffset; |
| 57 | - HCCL_VM_DEBUG("[TransLocMSToLocMSExecutor][Process] locCcu[{}:{}], Get ms offset = [{:04x}], cke offset = [{:04x}]", | 57 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get ms offset = [{:04x}], cke offset = [{:04x}]", |
| 58 | rankId_, dieId_, msOffset, ckeOffset); | 58 | rankId_, dieId_, msOffset, ckeOffset); |
| 59 | } | 59 | } |
| 60 | - HCCL_VM_DEBUG("[TransLocMSToLocMSExecutor][Process] locCcu[{}:{}] Trans data " | 60 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 61 | "from locSrcMsId[{}] to locDstMsId[{}], " | 61 | "from locSrcMsId[{}] to locDstMsId[{}], " |
| 62 | "with lengthXnId[{}] transLength[{}].", | 62 | "with lengthXnId[{}] transLength[{}].", |
| 63 | rankId_, dieId_, srcMSId_, dstMSId_, lengthXnId_, transLength_); | 63 | rankId_, dieId_, srcMSId_, dstMSId_, lengthXnId_, transLength_); |
| @@ -56,12 +56,12 @@ void TransLocMSToRmtMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 56 | rmtAddr += addrOffset; | 56 | rmtAddr += addrOffset; |
| 57 | locMSId_ += msOffset; | 57 | locMSId_ += msOffset; |
| 58 | setCKEId_ += ckeOffset; | 58 | setCKEId_ += ckeOffset; |
| 59 | - HCCL_VM_DEBUG("[TransLocMSToRmtMemExecutor][Process] locCcu[{}:{}], Get gsa " | 59 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa " |
| 60 | "addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", | 60 | "addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", |
| 61 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); | 61 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); |
| 62 | } | 62 | } |
| 63 | // 3.搬运动作 | 63 | // 3.搬运动作 |
| 64 | - HCCL_VM_DEBUG("[TransLocMSToRmtMemExecutor][Process] locCcu[{}:{}] Trans data " | 64 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 65 | "from locMsId[{}] to rmtGSAId_[{}] rmtAddr[{:x}], " | 65 | "from locMsId[{}] to rmtGSAId_[{}] rmtAddr[{:x}], " |
| 66 | "with lengthXnId[{}] transLength[{}].", | 66 | "with lengthXnId[{}] transLength[{}].", |
| 67 | rankId_, dieId_, locMSId_, rmtGSAId_, rmtAddr, lengthXnId_, transLength_); | 67 | rankId_, dieId_, locMSId_, rmtGSAId_, rmtAddr, lengthXnId_, transLength_); |
| @@ -48,7 +48,7 @@ void TransLocMSToRmtMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 48 | // 1.根据channel id获取remote rank id | 48 | // 1.根据channel id获取remote rank id |
| 49 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); | 49 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); |
| 50 | if (rmtCcu.second != rmtDieId_) { | 50 | if (rmtCcu.second != rmtDieId_) { |
| 51 | - HCCL_VM_WARN("[TransLocMSToRmtMSExecutor][Process] dieId[{}] from channel is not same as rmtDieId[{}]. curCcu[{}:{}], rmtCcu[{}:{}]", | 51 | + HCCL_VM_WARN("dieId[{}] from channel is not same as rmtDieId[{}]. curCcu[{}:{}], rmtCcu[{}:{}]", |
| 52 | rmtCcu.second, rmtDieId_, rankId_, dieId_, rmtCcu.first, rmtCcu.second); | 52 | rmtCcu.second, rmtDieId_, rankId_, dieId_, rmtCcu.first, rmtCcu.second); |
| 53 | return; | 53 | return; |
| 54 | } | 54 | } |
| @@ -59,13 +59,13 @@ void TransLocMSToRmtMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 59 | rmtMSId_ += msOffset; | 59 | rmtMSId_ += msOffset; |
| 60 | locMSId_ += msOffset; | 60 | locMSId_ += msOffset; |
| 61 | setCKEId_ += ckeOffset; | 61 | setCKEId_ += ckeOffset; |
| 62 | - HCCL_VM_DEBUG("[TransLocMSToRmtMSExecutor][Process] curCcu[{}:{}], Get ms offset = [{:04x}], cke offset = [{:04x}]", | 62 | + HCCL_VM_DEBUG("curCcu[{}:{}], Get ms offset = [{:04x}], cke offset = [{:04x}]", |
| 63 | rankId_, dieId_, msOffset, ckeOffset); | 63 | rankId_, dieId_, msOffset, ckeOffset); |
| 64 | } | 64 | } |
| 65 | // 3.要搬运的本端内存地址及数据长度 | 65 | // 3.要搬运的本端内存地址及数据长度 |
| 66 | transLength_ = (lengthEn_ == 0) ? HcclSim::BYTE_NUM_4K : ccuResMgr.GetXnValue(rankId_, dieId_, lengthXnId_); | 66 | transLength_ = (lengthEn_ == 0) ? HcclSim::BYTE_NUM_4K : ccuResMgr.GetXnValue(rankId_, dieId_, lengthXnId_); |
| 67 | // 4.搬运动作 | 67 | // 4.搬运动作 |
| 68 | - HCCL_VM_DEBUG("[TransLocMSToRmtMSExecutor][Process] ccuId[{}:{}-{}:{}] Trans data " | 68 | + HCCL_VM_DEBUG("ccuId[{}:{}-{}:{}] Trans data " |
| 69 | "from locSrcMsId[{}] to rmtDstMsId[{}], " | 69 | "from locSrcMsId[{}] to rmtDstMsId[{}], " |
| 70 | "with lengthXnId[{}] transLength[{}].", | 70 | "with lengthXnId[{}] transLength[{}].", |
| 71 | rankId_, dieId_, rmtCcu.first, rmtCcu.second, locMSId_, rmtMSId_, lengthXnId_, transLength_); | 71 | rankId_, dieId_, rmtCcu.first, rmtCcu.second, locMSId_, rmtMSId_, lengthXnId_, transLength_); |
Mtest/hccl_vm/src/plugin/runner/ccu_executor/trans_executor/trans_rmt_mem_to_loc_mem_executor.cc+2-2
| @@ -59,11 +59,11 @@ void TransRmtMemToLocMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 59 | rmtAddr += offset; | 59 | rmtAddr += offset; |
| 60 | locAddr += offset; | 60 | locAddr += offset; |
| 61 | setCKEId_ += ckeOffset; | 61 | setCKEId_ += ckeOffset; |
| 62 | - HCCL_VM_DEBUG("[TransRmtMemToLocMemExecutor][Process] locCcu[{}:{}], Get gsa addr offset = [{:04x}], cke offset = [{:04x}]", | 62 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa addr offset = [{:04x}], cke offset = [{:04x}]", |
| 63 | rankId_, dieId_, offset, ckeOffset); | 63 | rankId_, dieId_, offset, ckeOffset); |
| 64 | } | 64 | } |
| 65 | // 3.搬运动作 | 65 | // 3.搬运动作 |
| 66 | - HCCL_VM_DEBUG("[TransRmtMemToLocMemExecutor][Process] locCcu[{}:{}] Trans data " | 66 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 67 | "from rmtGSAId[{}] rmtAddr[{:x}] to locGSAId[{}] locAddr[{:x}], " | 67 | "from rmtGSAId[{}] rmtAddr[{:x}] to locGSAId[{}] locAddr[{:x}], " |
| 68 | "with lengthXnId[{}] transLength[{}], reduceEn[{}], reduceOp[{}].", | 68 | "with lengthXnId[{}] transLength[{}], reduceEn[{}], reduceOp[{}].", |
| 69 | rankId_, dieId_, rmtGSAId_, rmtAddr, locGSAId_, locAddr, lengthXnId_, transLength_, reduceEn_, reduceOpCode_); | 69 | rankId_, dieId_, rmtGSAId_, rmtAddr, locGSAId_, locAddr, lengthXnId_, transLength_, reduceEn_, reduceOpCode_); |
| @@ -56,11 +56,11 @@ void TransRmtMemToLocMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 56 | rmtAddr += addrOffset; | 56 | rmtAddr += addrOffset; |
| 57 | locMSId_ += msOffset; | 57 | locMSId_ += msOffset; |
| 58 | setCKEId_ += ckeOffset; | 58 | setCKEId_ += ckeOffset; |
| 59 | - HCCL_VM_DEBUG("[TransRmtMemToLocMSExecutor][Process] locCcu[{}:{}], Get gsa addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", | 59 | + HCCL_VM_DEBUG("locCcu[{}:{}], Get gsa addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", |
| 60 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); | 60 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); |
| 61 | } | 61 | } |
| 62 | // 3.搬运动作 | 62 | // 3.搬运动作 |
| 63 | - HCCL_VM_DEBUG("[TransRmtMemToLocMSExecutor][Process] locCcu[{}:{}] Trans data " | 63 | + HCCL_VM_DEBUG("locCcu[{}:{}] Trans data " |
| 64 | "from rmtGSAId[{}] rmtAddr[{:x}] to locMsId[{}], " | 64 | "from rmtGSAId[{}] rmtAddr[{:x}] to locMsId[{}], " |
| 65 | "with lengthXnId[{}] transLength[{}].", | 65 | "with lengthXnId[{}] transLength[{}].", |
| 66 | rankId_, dieId_, rmtGSAId_, rmtAddr, locMSId_, lengthXnId_, transLength_); | 66 | rankId_, dieId_, rmtGSAId_, rmtAddr, locMSId_, lengthXnId_, transLength_); |
| @@ -49,7 +49,7 @@ void TransRmtMSToLocMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 49 | // 1.根据channel id获取remote rank id | 49 | // 1.根据channel id获取remote rank id |
| 50 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); | 50 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); |
| 51 | if (rmtCcu.second != rmtDieId_) { | 51 | if (rmtCcu.second != rmtDieId_) { |
| 52 | - HCCL_VM_WARN("[TransRmtMSToLocMemExecutor][Process] dieId[{}] from channel is not same as rmtDieId[{}]. curCcu[{}:{}], rmtCcu[{}:{}]", | 52 | + HCCL_VM_WARN("dieId[{}] from channel is not same as rmtDieId[{}]. curCcu[{}:{}], rmtCcu[{}:{}]", |
| 53 | rmtCcu.second, rmtDieId_, rankId_, dieId_, rmtCcu.first, rmtCcu.second); | 53 | rmtCcu.second, rmtDieId_, rankId_, dieId_, rmtCcu.first, rmtCcu.second); |
| 54 | return; | 54 | return; |
| 55 | } | 55 | } |
| @@ -63,13 +63,13 @@ void TransRmtMSToLocMemExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 63 | locAddr += addrOffset; | 63 | locAddr += addrOffset; |
| 64 | rmtMSId_ += msOffset; | 64 | rmtMSId_ += msOffset; |
| 65 | setCKEId_ += ckeOffset; | 65 | setCKEId_ += ckeOffset; |
| 66 | - HCCL_VM_DEBUG("[TransRmtMSToLocMemExecutor][Process] ccuId=[{}:{}], Get gsa addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", | 66 | + HCCL_VM_DEBUG("ccuId=[{}:{}], Get gsa addr offset = [{:04x}], ms offset = [{:04x}], cke offset = [{:04x}]", |
| 67 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); | 67 | rankId_, dieId_, addrOffset, msOffset, ckeOffset); |
| 68 | } | 68 | } |
| 69 | // 4.要搬运的本端内存地址及数据长度 | 69 | // 4.要搬运的本端内存地址及数据长度 |
| 70 | transLength_ = (lengthEn_ == 0) ? HcclSim::BYTE_NUM_4K : ccuResMgr.GetXnValue(rankId_, dieId_, lengthXnId_); | 70 | transLength_ = (lengthEn_ == 0) ? HcclSim::BYTE_NUM_4K : ccuResMgr.GetXnValue(rankId_, dieId_, lengthXnId_); |
| 71 | // 5.搬运动作 | 71 | // 5.搬运动作 |
| 72 | - HCCL_VM_DEBUG("[TransRmtMSToLocMemExecutor][Process] ccuId=[{}:{}-{}:{}] Trans data " | 72 | + HCCL_VM_DEBUG("ccuId=[{}:{}-{}:{}] Trans data " |
| 73 | "from rmtMsId[{}] to locGSAId[{}] locAddr[{:x}], " | 73 | "from rmtMsId[{}] to locGSAId[{}] locAddr[{:x}], " |
| 74 | "with lengthXnId[{}] transLength[{}].", | 74 | "with lengthXnId[{}] transLength[{}].", |
| 75 | rankId_, dieId_, rmtCcu.first, rmtCcu.second, rmtMSId_, locGSAId_, locAddr, lengthXnId_, transLength_); | 75 | rankId_, dieId_, rmtCcu.first, rmtCcu.second, rmtMSId_, locGSAId_, locAddr, lengthXnId_, transLength_); |
| @@ -46,7 +46,7 @@ void TransRmtMSToLocMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 46 | // 1.根据channel id获取remote rank id | 46 | // 1.根据channel id获取remote rank id |
| 47 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); | 47 | auto rmtCcu = ccuResMgr.GetRmtCcu(rankId_, dieId_, channelId_); |
| 48 | if (rmtCcu.second != rmtDieId_) { | 48 | if (rmtCcu.second != rmtDieId_) { |
| 49 | - HCCL_VM_WARN("[TransRmtMSToLocMSExecutor][Process] dieId[{}] from channel is not same as rmtDieId[{}]. curCcu[{}:{}], rmtCcu[{}:{}]", | 49 | + HCCL_VM_WARN("dieId[{}] from channel is not same as rmtDieId[{}]. curCcu[{}:{}], rmtCcu[{}:{}]", |
| 50 | rmtCcu.second, rmtDieId_, rankId_, dieId_, rmtCcu.first, rmtCcu.second); | 50 | rmtCcu.second, rmtDieId_, rankId_, dieId_, rmtCcu.first, rmtCcu.second); |
| 51 | return; | 51 | return; |
| 52 | } | 52 | } |
| @@ -57,13 +57,13 @@ void TransRmtMSToLocMSExecutor::Process(CcuResourceManager &ccuResMgr) | |||
| 57 | locMSId_ += msOffset; | 57 | locMSId_ += msOffset; |
| 58 | rmtMSId_ += msOffset; | 58 | rmtMSId_ += msOffset; |
| 59 | setCKEId_ += ckeOffset; | 59 | setCKEId_ += ckeOffset; |
| 60 | - HCCL_VM_DEBUG("[TransRmtMSToLocMSExecutor][Process] ccuId=[{}:{}], Get ms offset = [{:04x}], cke offset = [{:04x}]", | 60 | + HCCL_VM_DEBUG("ccuId=[{}:{}], Get ms offset = [{:04x}], cke offset = [{:04x}]", |
| 61 | rankId_, dieId_, msOffset, ckeOffset); | 61 | rankId_, dieId_, msOffset, ckeOffset); |
| 62 | } | 62 | } |
| 63 | // 3.要搬运的本端内存地址及数据长度 | 63 | // 3.要搬运的本端内存地址及数据长度 |
| 64 | transLength_ = (lengthEn_ == 0) ? HcclSim::BYTE_NUM_4K : ccuResMgr.GetXnValue(rankId_, dieId_, lengthXnId_); | 64 | transLength_ = (lengthEn_ == 0) ? HcclSim::BYTE_NUM_4K : ccuResMgr.GetXnValue(rankId_, dieId_, lengthXnId_); |
| 65 | // 4.搬运动作 | 65 | // 4.搬运动作 |
| 66 | - HCCL_VM_DEBUG("[TransRmtMSToLocMSExecutor][Process] ccuId=[{}:{}-{}:{}] Trans data " | 66 | + HCCL_VM_DEBUG("ccuId=[{}:{}-{}:{}] Trans data " |
| 67 | "from rmtMsId[{}] to locMsId[{}], " | 67 | "from rmtMsId[{}] to locMsId[{}], " |
| 68 | "with lengthXnId[{}] transLength[{}].", | 68 | "with lengthXnId[{}] transLength[{}].", |
| 69 | rankId_, dieId_, rmtCcu.first, rmtCcu.second, rmtMSId_, locMSId_, lengthXnId_, transLength_); | 69 | rankId_, dieId_, rmtCcu.first, rmtCcu.second, rmtMSId_, locMSId_, lengthXnId_, transLength_); |
| @@ -70,16 +70,16 @@ HcclVmResult StorageManager::PrintAllRankInputBuffer() | |||
| 70 | sim::PhyMemBlock srcPhyMem{}; | 70 | sim::PhyMemBlock srcPhyMem{}; |
| 71 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)startAddr, srcPhyMem); | 71 | auto srcAddr = sim::AcquireDevPtrInNoHostProcess((void*)startAddr, srcPhyMem); |
| 72 | if (srcAddr == nullptr) { | 72 | if (srcAddr == nullptr) { |
| 73 | - HCCL_VM_ERROR("[PrintAllRankInputBuffer] 无法获取startAddr的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>((void*)startAddr)); | 73 | + HCCL_VM_ERROR("无法获取startAddr的设备地址(addr= {:x})!", reinterpret_cast<uintptr_t>((void*)startAddr)); |
| 74 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 74 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 75 | } | 75 | } |
| 76 | m_allPhyMem.push_back(srcPhyMem); | 76 | m_allPhyMem.push_back(srcPhyMem); |
| 77 | 77 | ||
| 78 | auto size = m_synData.memory_info.data[i].size; | 78 | auto size = m_synData.memory_info.data[i].size; |
| 79 | - HCCL_VM_DEBUG("[PrintAllRankInputBuffer] rankId={}, input buffer= {:x}, size={}", | 79 | + HCCL_VM_DEBUG("rankId={}, input buffer= {:x}, size={}", |
| 80 | m_synData.memory_info.data[i].rank_id, startAddr, size); | 80 | m_synData.memory_info.data[i].rank_id, startAddr, size); |
| 81 | for (uint32_t idx = 0; idx < size; idx++) { | 81 | for (uint32_t idx = 0; idx < size; idx++) { |
| 82 | - HCCL_VM_TRACE("[PrintAllRankInputBuffer] idx={}, value={:x}", idx, *((char*)srcAddr + idx)); | 82 | + HCCL_VM_TRACE("idx={}, value={:x}", idx, *((char*)srcAddr + idx)); |
| 83 | } | 83 | } |
| 84 | } | 84 | } |
| 85 | } | 85 | } |
| @@ -116,7 +116,7 @@ HcclVmResult StorageManager::Trans2CheckerParam(sim::OpDetailTab& detailTab, ::O | |||
| 116 | } | 116 | } |
| 117 | } | 117 | } |
| 118 | 118 | ||
| 119 | - HCCL_VM_DEBUG("[Trans2CheckerParam] Success"); | 119 | + HCCL_VM_DEBUG("Success"); |
| 120 | 120 | ||
| 121 | return HcclVmResult::HCCL_SIM_SUCCESS; | 121 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 122 | } | 122 | } |
| @@ -124,7 +124,7 @@ HcclVmResult StorageManager::Trans2CheckerParam(sim::OpDetailTab& detailTab, ::O | |||
| 124 | HcclVmResult StorageManager::LoadHcclVmSynthesisData(sim::OpDetailTab& detailTab, std::vector<sim::CcuChannelTab>& channels) | 124 | HcclVmResult StorageManager::LoadHcclVmSynthesisData(sim::OpDetailTab& detailTab, std::vector<sim::CcuChannelTab>& channels) |
| 125 | { | 125 | { |
| 126 | if (detailTab.opDetail.size() < sizeof(::OpDetails)) { | 126 | if (detailTab.opDetail.size() < sizeof(::OpDetails)) { |
| 127 | - HCCL_VM_ERROR("[LoadHcclVmSynthesisData] opDetail BLOB too small"); | 127 | + HCCL_VM_ERROR("opDetail BLOB too small"); |
| 128 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 128 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 129 | } | 129 | } |
| 130 | ::OpDetails opDetails{}; | 130 | ::OpDetails opDetails{}; |
| @@ -151,7 +151,7 @@ HcclVmResult StorageManager::LoadHcclVmSynthesisData(sim::OpDetailTab& detailTab | |||
| 151 | HcclVmResult StorageManager::LoadHcclVmInstrData(std::vector<sim::CcuInstrResTab>& instrRes) | 151 | HcclVmResult StorageManager::LoadHcclVmInstrData(std::vector<sim::CcuInstrResTab>& instrRes) |
| 152 | { | 152 | { |
| 153 | for (auto &ccuInstr : instrRes) { | 153 | for (auto &ccuInstr : instrRes) { |
| 154 | - HCCL_VM_DEBUG("[LoadHcclVmInstrData] rankId={}, dieId={}, count={}", | 154 | + HCCL_VM_DEBUG("rankId={}, dieId={}, count={}", |
| 155 | ccuInstr.rankId, ccuInstr.dieId, ccuInstr.instrCount); | 155 | ccuInstr.rankId, ccuInstr.dieId, ccuInstr.instrCount); |
| 156 | } | 156 | } |
| 157 | 157 | ||
| @@ -167,11 +167,11 @@ HcclVmResult StorageManager::LoadHcclVmTaskMetaData(std::vector<sim::OpTaskTab>& | |||
| 167 | std::memcpy(&metaData, task.optaskMeta.data(), sizeof(HcclTaskMetaData)); | 167 | std::memcpy(&metaData, task.optaskMeta.data(), sizeof(HcclTaskMetaData)); |
| 168 | taskMeataData.task_meta.push_back(metaData); | 168 | taskMeataData.task_meta.push_back(metaData); |
| 169 | } else { | 169 | } else { |
| 170 | - HCCL_VM_WARN("[LoadHcclVmTaskMetaData] optaskMeta too small, taskSeq={} src:{:d}, dst:{:d}", task.taskSeq, task.optaskMeta.size(), sizeof(HcclTaskMetaData)); | 170 | + HCCL_VM_WARN("optaskMeta too small, taskSeq={} src:{:d}, dst:{:d}", task.taskSeq, task.optaskMeta.size(), sizeof(HcclTaskMetaData)); |
| 171 | } | 171 | } |
| 172 | } | 172 | } |
| 173 | for (auto &taskMeta : taskMeataData.task_meta) { | 173 | for (auto &taskMeta : taskMeataData.task_meta) { |
| 174 | - HCCL_VM_DEBUG("[LoadHcclVmTaskMetaData] rankId={}, dieId={}, instrCnt={}, argSize={}, streamId={}", | 174 | + HCCL_VM_DEBUG("rankId={}, dieId={}, instrCnt={}, argSize={}, streamId={}", |
| 175 | taskMeta.rankId, static_cast<uint32_t>(taskMeta.taskData.ccu.dieId), | 175 | taskMeta.rankId, static_cast<uint32_t>(taskMeta.taskData.ccu.dieId), |
| 176 | taskMeta.taskData.ccu.instCnt, taskMeta.taskData.ccu.argSize, taskMeta.streamId); | 176 | taskMeta.taskData.ccu.instCnt, taskMeta.taskData.ccu.argSize, taskMeta.streamId); |
| 177 | } | 177 | } |
| @@ -184,20 +184,20 @@ HcclVmResult StorageManager::GetHcclVmFlagData(HcclSim::HcclVmFlagData &waitFlag | |||
| 184 | // 1. 构造路径 | 184 | // 1. 构造路径 |
| 185 | std::string rootPath = FindRootPath(); | 185 | std::string rootPath = FindRootPath(); |
| 186 | if (rootPath.empty()) { | 186 | if (rootPath.empty()) { |
| 187 | - HCCL_VM_ERROR("[StorageManager][GetHcclVmFlagData] Failed to find root path"); | 187 | + HCCL_VM_ERROR("Failed to find root path"); |
| 188 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 188 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 189 | } | 189 | } |
| 190 | std::string fullPath = rootPath + DATA_FILE_PATH + HCCLVM_FLAG_DATA_FILE; | 190 | std::string fullPath = rootPath + DATA_FILE_PATH + HCCLVM_FLAG_DATA_FILE; |
| 191 | // 2. 写文件,通知runner启动 | 191 | // 2. 写文件,通知runner启动 |
| 192 | FILE *fp = fopen(fullPath.c_str(), "rb"); | 192 | FILE *fp = fopen(fullPath.c_str(), "rb"); |
| 193 | if (!fp) { | 193 | if (!fp) { |
| 194 | - HCCL_VM_ERROR("[StorageManager][GetHcclVmFlagData] Open file failed: {}", fullPath); | 194 | + HCCL_VM_ERROR("Open file failed: {}", fullPath); |
| 195 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 195 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 196 | } | 196 | } |
| 197 | 197 | ||
| 198 | auto ret = HcclVmFlagDataRead(fp, waitFlag, HCCLVM_FLAG_FILE_MAGIC); | 198 | auto ret = HcclVmFlagDataRead(fp, waitFlag, HCCLVM_FLAG_FILE_MAGIC); |
| 199 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 199 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 200 | - HCCL_VM_ERROR("[StorageManager][GetHcclVmFlagData] HcclVmFlagDataRead failed"); | 200 | + HCCL_VM_ERROR("HcclVmFlagDataRead failed"); |
| 201 | fclose(fp); | 201 | fclose(fp); |
| 202 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 202 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 203 | } | 203 | } |
| @@ -207,18 +207,18 @@ HcclVmResult StorageManager::GetHcclVmFlagData(HcclSim::HcclVmFlagData &waitFlag | |||
| 207 | 207 | ||
| 208 | HcclVmResult StorageManager::DumpHcclVmFlagData(uint16_t status) | 208 | HcclVmResult StorageManager::DumpHcclVmFlagData(uint16_t status) |
| 209 | { | 209 | { |
| 210 | - HCCL_VM_DEBUG("[StorageManager][DumpHcclVmFlagData] Start dumping hccl vm flag data..."); | 210 | + HCCL_VM_DEBUG("Start dumping hccl vm flag data..."); |
| 211 | // 1. 构造完整路径 | 211 | // 1. 构造完整路径 |
| 212 | std::string rootPath = FindRootPath(); | 212 | std::string rootPath = FindRootPath(); |
| 213 | if (rootPath.empty()) { | 213 | if (rootPath.empty()) { |
| 214 | - HCCL_VM_ERROR("[StorageManager][GetHcclVmFlagData] Failed to find root path"); | 214 | + HCCL_VM_ERROR("Failed to find root path"); |
| 215 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 215 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 216 | } | 216 | } |
| 217 | std::string fullPath = rootPath + DATA_FILE_PATH + HCCLVM_FLAG_DATA_FILE; | 217 | std::string fullPath = rootPath + DATA_FILE_PATH + HCCLVM_FLAG_DATA_FILE; |
| 218 | // 2. 写文件,通知runner启动 | 218 | // 2. 写文件,通知runner启动 |
| 219 | FILE *fp = fopen(fullPath.c_str(), "wb"); | 219 | FILE *fp = fopen(fullPath.c_str(), "wb"); |
| 220 | if (!fp) { | 220 | if (!fp) { |
| 221 | - HCCL_VM_ERROR("[StorageManager][DumpHcclVmFlagData] Open file failed: {}", fullPath); | 221 | + HCCL_VM_ERROR("Open file failed: {}", fullPath); |
| 222 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 222 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 223 | } | 223 | } |
| 224 | 224 | ||
| @@ -227,7 +227,7 @@ HcclVmResult StorageManager::DumpHcclVmFlagData(uint16_t status) | |||
| 227 | flagData.runner_status = status; | 227 | flagData.runner_status = status; |
| 228 | auto ret = HcclVmFlagDataWrite(fp, flagData); | 228 | auto ret = HcclVmFlagDataWrite(fp, flagData); |
| 229 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 229 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 230 | - HCCL_VM_ERROR("[StorageManager][DumpHcclVmFlagData] HcclVmFlagDataWrite failed"); | 230 | + HCCL_VM_ERROR("HcclVmFlagDataWrite failed"); |
| 231 | fclose(fp); | 231 | fclose(fp); |
| 232 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 232 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 233 | } | 233 | } |
| @@ -252,14 +252,14 @@ std::string StorageManager::FindRootPath() | |||
| 252 | } | 252 | } |
| 253 | } else { | 253 | } else { |
| 254 | // 如果 realpath 失败(例如路径被删除或权限不足) | 254 | // 如果 realpath 失败(例如路径被删除或权限不足) |
| 255 | - HCCL_VM_WARN("[FindRootPath] Iteration {}: realpath failed for {}", i, current_search_path); | 255 | + HCCL_VM_WARN("Iteration {}: realpath failed for {}", i, current_search_path); |
| 256 | } | 256 | } |
| 257 | 257 | ||
| 258 | // 没找到,将探测路径向上推一级 | 258 | // 没找到,将探测路径向上推一级 |
| 259 | current_search_path += "/.."; | 259 | current_search_path += "/.."; |
| 260 | } | 260 | } |
| 261 | 261 | ||
| 262 | - HCCL_VM_WARN("[FindRootPath] RootPath NOT found"); | 262 | + HCCL_VM_WARN("RootPath NOT found"); |
| 263 | return ""; | 263 | return ""; |
| 264 | } | 264 | } |
| 265 | 265 | ||
| @@ -267,14 +267,14 @@ HcclVmResult StorageManager::ConvertTaskQueue(const HcclVmTaskMetaData &taskMeat | |||
| 267 | { | 267 | { |
| 268 | auto rankSize = GetRankSize(); | 268 | auto rankSize = GetRankSize(); |
| 269 | m_allRankTaskQueues.resize(rankSize); | 269 | m_allRankTaskQueues.resize(rankSize); |
| 270 | - HCCL_VM_INFO("[ConvertTaskQueue] rankSize={}", rankSize); | 270 | + HCCL_VM_INFO("rankSize={}", rankSize); |
| 271 | 271 | ||
| 272 | auto taskMetaVec = taskMeataData.task_meta; | 272 | auto taskMetaVec = taskMeataData.task_meta; |
| 273 | uint32_t size = taskMetaVec.size(); | 273 | uint32_t size = taskMetaVec.size(); |
| 274 | for (uint32_t i = 0; i < size; i++) { | 274 | for (uint32_t i = 0; i < size; i++) { |
| 275 | uint32_t rankId = taskMetaVec[i].rankId; | 275 | uint32_t rankId = taskMetaVec[i].rankId; |
| 276 | if (rankId >= rankSize) { | 276 | if (rankId >= rankSize) { |
| 277 | - HCCL_VM_ERROR("[ConvertTaskQueue] rankId={} is out of range. rankSize={}", rankId, rankSize); | 277 | + HCCL_VM_ERROR("rankId={} is out of range. rankSize={}", rankId, rankSize); |
| 278 | return HcclVmResult::HCCL_SIM_E_PARA; | 278 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 279 | } | 279 | } |
| 280 | uint32_t streamId = (taskMetaVec[i].streamId); | 280 | uint32_t streamId = (taskMetaVec[i].streamId); |
| @@ -315,7 +315,7 @@ HcclVmResult StorageManager::InitCcuResource(std::vector<sim::CcuInstrResTab>& i | |||
| 315 | if (devType_ == DevType::DEV_TYPE_950) { | 315 | if (devType_ == DevType::DEV_TYPE_950) { |
| 316 | ccuVersion = RunnerCcuVersion::CCU_V1; | 316 | ccuVersion = RunnerCcuVersion::CCU_V1; |
| 317 | } else { | 317 | } else { |
| 318 | - HCCL_VM_ERROR("[InitCcuResource] Wrong devive type: {:d}", static_cast<int>(devType_)); | 318 | + HCCL_VM_ERROR("Wrong devive type: {:d}", static_cast<int>(devType_)); |
| 319 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 319 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 320 | } | 320 | } |
| 321 | std::vector<uint64_t> ccuResourceBaseAddr{}; | 321 | std::vector<uint64_t> ccuResourceBaseAddr{}; |
| @@ -346,7 +346,7 @@ HcclVmResult StorageManager::InitAivResourceFromCompositeOpDetail(const sim::Com | |||
| 346 | { | 346 | { |
| 347 | auto ret = AivResourceManager::GetInstance().Init(opDetail.rankId, opDetail.memInfo, GetRankSize()); | 347 | auto ret = AivResourceManager::GetInstance().Init(opDetail.rankId, opDetail.memInfo, GetRankSize()); |
| 348 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 348 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 349 | - HCCL_VM_ERROR("[InitAivResourceFromCompositeOpDetail] init aiv resource failed, " | 349 | + HCCL_VM_ERROR("init aiv resource failed, " |
| 350 | "rankId={}, opDetailId={}, memInfoId={}, ret={}", | 350 | "rankId={}, opDetailId={}, memInfoId={}, ret={}", |
| 351 | opDetail.rankId, opDetail.detail.id, opDetail.memInfo.id, static_cast<int>(ret)); | 351 | opDetail.rankId, opDetail.detail.id, opDetail.memInfo.id, static_cast<int>(ret)); |
| 352 | return ret; | 352 | return ret; |
| @@ -364,27 +364,27 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 364 | { | 364 | { |
| 365 | const char* enableDumpData = std::getenv("HCCLVM_ENABLE_DUMP_DATA"); | 365 | const char* enableDumpData = std::getenv("HCCLVM_ENABLE_DUMP_DATA"); |
| 366 | if (enableDumpData == nullptr || std::string(enableDumpData).empty() || std::string(enableDumpData) == "0") { | 366 | if (enableDumpData == nullptr || std::string(enableDumpData).empty() || std::string(enableDumpData) == "0") { |
| 367 | - HCCL_VM_DEBUG("[DumpAllRankInputOutput] HCCLVM_ENABLE_DUMP_DATA is not set"); | 367 | + HCCL_VM_DEBUG("HCCLVM_ENABLE_DUMP_DATA is not set"); |
| 368 | return; | 368 | return; |
| 369 | } | 369 | } |
| 370 | auto rankSize = GetRankSize(); | 370 | auto rankSize = GetRankSize(); |
| 371 | if (compositeDataMap.empty() || compositeDataMap[0].empty()) { | 371 | if (compositeDataMap.empty() || compositeDataMap[0].empty()) { |
| 372 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] compositeDataMap is empty"); | 372 | + HCCL_VM_ERROR("compositeDataMap is empty"); |
| 373 | return; | 373 | return; |
| 374 | } | 374 | } |
| 375 | std::string rootPath = FindRootPath(); | 375 | std::string rootPath = FindRootPath(); |
| 376 | if (rootPath.empty()) { | 376 | if (rootPath.empty()) { |
| 377 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] Failed to find root path"); | 377 | + HCCL_VM_ERROR("Failed to find root path"); |
| 378 | return; | 378 | return; |
| 379 | } | 379 | } |
| 380 | std::string fullPath = rootPath + HCCLVM_ALL_RANK_INPUT_OUTPUT_FILE; | 380 | std::string fullPath = rootPath + HCCLVM_ALL_RANK_INPUT_OUTPUT_FILE; |
| 381 | std::ofstream ofs(fullPath); | 381 | std::ofstream ofs(fullPath); |
| 382 | if (!ofs.is_open()) { | 382 | if (!ofs.is_open()) { |
| 383 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] Open file failed: {}", fullPath); | 383 | + HCCL_VM_ERROR("Open file failed: {}", fullPath); |
| 384 | return; | 384 | return; |
| 385 | } | 385 | } |
| 386 | 386 | ||
| 387 | - HCCL_VM_INFO("[DumpAllRankInputOutput] Dump All Rank Input Output Data. rankSize={}", rankSize); | 387 | + HCCL_VM_INFO("Dump All Rank Input Output Data. rankSize={}", rankSize); |
| 388 | for (size_t opIdx = 0; opIdx < compositeDataMap.size(); opIdx++) { | 388 | for (size_t opIdx = 0; opIdx < compositeDataMap.size(); opIdx++) { |
| 389 | auto& rankTask = compositeDataMap[opIdx]; | 389 | auto& rankTask = compositeDataMap[opIdx]; |
| 390 | if (rankTask.empty()) { | 390 | if (rankTask.empty()) { |
| @@ -399,7 +399,7 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 399 | sim::PhyMemBlock srcPhyMem{}; | 399 | sim::PhyMemBlock srcPhyMem{}; |
| 400 | auto startAddr = sim::AcquireDevPtrInNoHostProcess((void*)(memInfo.inputAddr), srcPhyMem); | 400 | auto startAddr = sim::AcquireDevPtrInNoHostProcess((void*)(memInfo.inputAddr), srcPhyMem); |
| 401 | if (startAddr == nullptr) { | 401 | if (startAddr == nullptr) { |
| 402 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] fail addr(type={}, addr={:x})!", | 402 | + HCCL_VM_ERROR("fail addr(type={}, addr={:x})!", |
| 403 | static_cast<int>(BufferType::INPUT), | 403 | static_cast<int>(BufferType::INPUT), |
| 404 | memInfo.inputAddr); | 404 | memInfo.inputAddr); |
| 405 | return; | 405 | return; |
| @@ -408,7 +408,7 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 408 | allRankInput[it.first].virtualAddr = memInfo.inputAddr; | 408 | allRankInput[it.first].virtualAddr = memInfo.inputAddr; |
| 409 | allRankInput[it.first].phyAddr = (char*)startAddr; | 409 | allRankInput[it.first].phyAddr = (char*)startAddr; |
| 410 | allRankInput[it.first].size = memInfo.inputSize; | 410 | allRankInput[it.first].size = memInfo.inputSize; |
| 411 | - HCCL_VM_INFO("[DumpAllRankInputOutput] opIdx={}, rankId={}, inputAddr={:x}, size={}", | 411 | + HCCL_VM_INFO("opIdx={}, rankId={}, inputAddr={:x}, size={}", |
| 412 | opIdx, it.first, memInfo.inputAddr, memInfo.inputSize); | 412 | opIdx, it.first, memInfo.inputAddr, memInfo.inputSize); |
| 413 | } | 413 | } |
| 414 | 414 | ||
| @@ -416,7 +416,7 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 416 | sim::PhyMemBlock srcPhyMem{}; | 416 | sim::PhyMemBlock srcPhyMem{}; |
| 417 | auto startAddr = sim::AcquireDevPtrInNoHostProcess((void*)(memInfo.outputAddr), srcPhyMem); | 417 | auto startAddr = sim::AcquireDevPtrInNoHostProcess((void*)(memInfo.outputAddr), srcPhyMem); |
| 418 | if (startAddr == nullptr) { | 418 | if (startAddr == nullptr) { |
| 419 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] fail addr(type={}, addr={:x})!", | 419 | + HCCL_VM_ERROR("fail addr(type={}, addr={:x})!", |
| 420 | static_cast<int>(BufferType::OUTPUT), | 420 | static_cast<int>(BufferType::OUTPUT), |
| 421 | memInfo.outputAddr); | 421 | memInfo.outputAddr); |
| 422 | return; | 422 | return; |
| @@ -425,7 +425,7 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 425 | allRankOutput[it.first].virtualAddr = memInfo.outputAddr; | 425 | allRankOutput[it.first].virtualAddr = memInfo.outputAddr; |
| 426 | allRankOutput[it.first].phyAddr = (char*)startAddr; | 426 | allRankOutput[it.first].phyAddr = (char*)startAddr; |
| 427 | allRankOutput[it.first].size = memInfo.outputSize; | 427 | allRankOutput[it.first].size = memInfo.outputSize; |
| 428 | - HCCL_VM_INFO("[DumpAllRankInputOutput] opIdx={}, rankId={}, outputAddr={:x}, size={}", | 428 | + HCCL_VM_INFO("opIdx={}, rankId={}, outputAddr={:x}, size={}", |
| 429 | opIdx, it.first, memInfo.outputAddr, memInfo.outputSize); | 429 | opIdx, it.first, memInfo.outputAddr, memInfo.outputSize); |
| 430 | } | 430 | } |
| 431 | 431 | ||
| @@ -433,7 +433,7 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 433 | sim::PhyMemBlock srcPhyMem{}; | 433 | sim::PhyMemBlock srcPhyMem{}; |
| 434 | auto startAddr = sim::AcquireDevPtrInNoHostProcess((void*)(memInfo.cclAddr), srcPhyMem); | 434 | auto startAddr = sim::AcquireDevPtrInNoHostProcess((void*)(memInfo.cclAddr), srcPhyMem); |
| 435 | if (startAddr == nullptr) { | 435 | if (startAddr == nullptr) { |
| 436 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] fail addr(type={}, addr={:x})!", | 436 | + HCCL_VM_ERROR("fail addr(type={}, addr={:x})!", |
| 437 | static_cast<int>(BufferType::CCL), | 437 | static_cast<int>(BufferType::CCL), |
| 438 | memInfo.cclAddr); | 438 | memInfo.cclAddr); |
| 439 | return; | 439 | return; |
| @@ -442,14 +442,14 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 442 | allRankCCL[it.first].virtualAddr = memInfo.cclAddr; | 442 | allRankCCL[it.first].virtualAddr = memInfo.cclAddr; |
| 443 | allRankCCL[it.first].phyAddr = (char*)startAddr; | 443 | allRankCCL[it.first].phyAddr = (char*)startAddr; |
| 444 | allRankCCL[it.first].size = memInfo.cclSize; | 444 | allRankCCL[it.first].size = memInfo.cclSize; |
| 445 | - HCCL_VM_INFO("[DumpAllRankInputOutput] opIdx={}, rankId={}, cclAddr={:x}, size={}", | 445 | + HCCL_VM_INFO("opIdx={}, rankId={}, cclAddr={:x}, size={}", |
| 446 | opIdx, it.first, memInfo.cclAddr, memInfo.cclSize); | 446 | opIdx, it.first, memInfo.cclAddr, memInfo.cclSize); |
| 447 | } | 447 | } |
| 448 | } | 448 | } |
| 449 | 449 | ||
| 450 | const auto& firstComp = rankTask.begin()->second; | 450 | const auto& firstComp = rankTask.begin()->second; |
| 451 | if (firstComp.detail.opDetail.size() < sizeof(::OpDetails)) { | 451 | if (firstComp.detail.opDetail.size() < sizeof(::OpDetails)) { |
| 452 | - HCCL_VM_ERROR("[DumpAllRankInputOutput] opDetail BLOB too small, opIdx={}, size={}", | 452 | + HCCL_VM_ERROR("opDetail BLOB too small, opIdx={}, size={}", |
| 453 | opIdx, firstComp.detail.opDetail.size()); | 453 | opIdx, firstComp.detail.opDetail.size()); |
| 454 | continue; | 454 | continue; |
| 455 | } | 455 | } |
| @@ -475,7 +475,7 @@ void StorageManager::DumpAllRankInputOutput(std::vector<std::map<uint32_t, sim:: | |||
| 475 | } | 475 | } |
| 476 | } | 476 | } |
| 477 | ofs.close(); | 477 | ofs.close(); |
| 478 | - HCCL_VM_INFO("[DumpAllRankInputOutput] Dump All Rank Input Output Data success."); | 478 | + HCCL_VM_INFO("Dump All Rank Input Output Data success."); |
| 479 | } | 479 | } |
| 480 | 480 | ||
| 481 | void StorageManager::FlexiblePrintData(std::ofstream &ofs, const BufferInfo &buffer) | 481 | void StorageManager::FlexiblePrintData(std::ofstream &ofs, const BufferInfo &buffer) |
| @@ -57,8 +57,8 @@ HcclVmResult SqeuentialExecutor::Execute() | |||
| 57 | } | 57 | } |
| 58 | } | 58 | } |
| 59 | } | 59 | } |
| 60 | - | 60 | + |
| 61 | - return HcclVmResult::HCCL_SIM_SUCCESS; | 61 | + return HcclVmResult::HCCL_SIM_SUCCESS; |
| 62 | } | 62 | } |
| 63 | 63 | ||
| 64 | HcclVmResult SqeuentialExecutor::ExecuteOneTask(HcclTaskMetaData& task) | 64 | HcclVmResult SqeuentialExecutor::ExecuteOneTask(HcclTaskMetaData& task) |
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -21,11 +24,6 @@ | |||
| 21 | 24 | ||
| 22 | 25 | ||
| 23 | 26 | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | 27 | ||
| 30 | 28 | ||
| 31 | extern "C" { | 29 | extern "C" { |
| @@ -40,7 +38,7 @@ aclError aclrtCreateContext(aclrtContext *context, int32_t deviceId) | |||
| 40 | return d.logic_id == (uint32_t)deviceId; | 38 | return d.logic_id == (uint32_t)deviceId; |
| 41 | }); | 39 | }); |
| 42 | if (!ret.second) { | 40 | if (!ret.second) { |
| 43 | - CTX_STUB_ERROR("device not found:{:d}", deviceId); | 41 | + HCCL_VM_ERROR("device not found:{:d}", deviceId); |
| 44 | return HcclResult::HCCL_E_NOT_FOUND; | 42 | return HcclResult::HCCL_E_NOT_FOUND; |
| 45 | } | 43 | } |
| 46 | // 找不到context,则默认新增一个context | 44 | // 找不到context,则默认新增一个context |
| @@ -67,7 +65,7 @@ aclError aclrtDestroyContext(aclrtContext context) | |||
| 67 | return stm.ctx_id == ctxId && stm.is_primary_default == 1; | 65 | return stm.ctx_id == ctxId && stm.is_primary_default == 1; |
| 68 | }); | 66 | }); |
| 69 | if (!ret.second) { | 67 | if (!ret.second) { |
| 70 | - CTX_STUB_ERROR("stream not found ctxId:{:d}", ctxId); | 68 | + HCCL_VM_ERROR("stream not found ctxId:{:d}", ctxId); |
| 71 | return HcclResult::HCCL_E_NOT_FOUND; | 69 | return HcclResult::HCCL_E_NOT_FOUND; |
| 72 | } | 70 | } |
| 73 | 71 | ||
| @@ -104,7 +102,7 @@ aclError aclrtCtxGetCurrentDefaultStream(aclrtStream *stream) | |||
| 104 | return stm.ctx_id == curCtxId && stm.is_primary_default == 1; | 102 | return stm.ctx_id == curCtxId && stm.is_primary_default == 1; |
| 105 | }); | 103 | }); |
| 106 | if (!stm.second) { | 104 | if (!stm.second) { |
| 107 | - CTX_STUB_ERROR("stream not found ctxId:{:d}", curCtxId); | 105 | + HCCL_VM_ERROR("stream not found ctxId:{:d}", curCtxId); |
| 108 | return HcclResult::HCCL_E_NOT_FOUND; | 106 | return HcclResult::HCCL_E_NOT_FOUND; |
| 109 | } | 107 | } |
| 110 | 108 | ||
| @@ -123,7 +121,7 @@ aclError aclrtGetPrimaryCtxState(int32_t deviceId, uint32_t *flags, int32_t *act | |||
| 123 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 121 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 124 | if (!currCtx.has_value()) { | 122 | if (!currCtx.has_value()) { |
| 125 | // not find | 123 | // not find |
| 126 | - CTX_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 124 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 127 | return ACL_ERROR_INVALID_PARAM; | 125 | return ACL_ERROR_INVALID_PARAM; |
| 128 | } | 126 | } |
| 129 | 127 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -30,11 +33,6 @@ uint64_t g_host_id; | |||
| 30 | extern uint64_t g_cur_server_key; | 33 | extern uint64_t g_cur_server_key; |
| 31 | 34 | ||
| 32 | extern pid_t g_devicePid; | 35 | extern pid_t g_devicePid; |
| 33 | - | ||
| 34 | - | ||
| 35 | - | ||
| 36 | - | ||
| 37 | - | ||
| 38 | 36 | ||
| 39 | 37 | ||
| 40 | extern "C" { | 38 | extern "C" { |
| @@ -56,13 +54,13 @@ HcclResult hrtGetDeviceIndexByPhyId(uint32_t devicePhyId, uint32_t &deviceLogicI | |||
| 56 | return d.server_id == g_cur_server_key && d.physical_id == (uint32_t)devicePhyId; | 54 | return d.server_id == g_cur_server_key && d.physical_id == (uint32_t)devicePhyId; |
| 57 | }); | 55 | }); |
| 58 | if (!ret.second) { | 56 | if (!ret.second) { |
| 59 | - DEVICE_STUB_ERROR("device not found by phyId:{:d}", devicePhyId); | 57 | + HCCL_VM_ERROR("device not found by phyId:{:d}", devicePhyId); |
| 60 | return HcclResult::HCCL_E_NOT_FOUND; | 58 | return HcclResult::HCCL_E_NOT_FOUND; |
| 61 | } | 59 | } |
| 62 | deviceLogicId = ret.first.logic_id; | 60 | deviceLogicId = ret.first.logic_id; |
| 63 | return HCCL_SUCCESS; | 61 | return HCCL_SUCCESS; |
| 64 | } catch (const std::exception &e) { | 62 | } catch (const std::exception &e) { |
| 65 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 63 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 66 | return HcclResult::HCCL_E_INTERNAL; | 64 | return HcclResult::HCCL_E_INTERNAL; |
| 67 | } | 65 | } |
| 68 | } | 66 | } |
| @@ -70,11 +68,11 @@ HcclResult hrtGetDeviceIndexByPhyId(uint32_t devicePhyId, uint32_t &deviceLogicI | |||
| 70 | aclError aclrtSetDevice(int32_t deviceId) | 68 | aclError aclrtSetDevice(int32_t deviceId) |
| 71 | { | 69 | { |
| 72 | try { | 70 | try { |
| 73 | - DEVICE_STUB_DEBUG("set id:{:d}", deviceId); | 71 | + HCCL_VM_DEBUG("set id:{:d}", deviceId); |
| 74 | uint32_t rankId; | 72 | uint32_t rankId; |
| 75 | uint64_t serverId = 0; | 73 | uint64_t serverId = 0; |
| 76 | if (!sim::GetRankIdByMPI(rankId, serverId)) { | 74 | if (!sim::GetRankIdByMPI(rankId, serverId)) { |
| 77 | - DEVICE_STUB_ERROR("get rankId by MPI fail serverId:{:d}", serverId); | 75 | + HCCL_VM_ERROR("get rankId by MPI fail serverId:{:d}", serverId); |
| 78 | return ACL_ERROR_INVALID_PARAM; | 76 | return ACL_ERROR_INVALID_PARAM; |
| 79 | } | 77 | } |
| 80 | 78 | ||
| @@ -82,21 +80,21 @@ aclError aclrtSetDevice(int32_t deviceId) | |||
| 82 | if (serverId == 0) { | 80 | if (serverId == 0) { |
| 83 | auto devRet = sim::GetDeviceByRankId(rankId, device); | 81 | auto devRet = sim::GetDeviceByRankId(rankId, device); |
| 84 | if (devRet != ACL_SUCCESS) { | 82 | if (devRet != ACL_SUCCESS) { |
| 85 | - DEVICE_STUB_ERROR("device not found by rankId:{:d}", rankId); | 83 | + HCCL_VM_ERROR("device not found by rankId:{:d}", rankId); |
| 86 | return devRet; | 84 | return devRet; |
| 87 | } | 85 | } |
| 88 | serverId = device.server_id; | 86 | serverId = device.server_id; |
| 89 | - DEVICE_STUB_DEBUG("logicId:{:d} rankId:{:d} key:{:d}", device.logic_id, rankId, device.id); | 87 | + HCCL_VM_DEBUG("logicId:{:d} rankId:{:d} key:{:d}", device.logic_id, rankId, device.id); |
| 90 | } else { | 88 | } else { |
| 91 | auto ret = RunnerDB::GetOneByPred<sim::Device>([serverId, deviceId](const sim::Device &d) { | 89 | auto ret = RunnerDB::GetOneByPred<sim::Device>([serverId, deviceId](const sim::Device &d) { |
| 92 | return d.server_id == serverId && d.logic_id == (uint32_t)deviceId; | 90 | return d.server_id == serverId && d.logic_id == (uint32_t)deviceId; |
| 93 | }); | 91 | }); |
| 94 | if (!ret.second) { | 92 | if (!ret.second) { |
| 95 | - DEVICE_STUB_ERROR("device not found logicId:{:d} serverId:{:d}", deviceId, serverId); | 93 | + HCCL_VM_ERROR("device not found logicId:{:d} serverId:{:d}", deviceId, serverId); |
| 96 | return ACL_ERROR_INVALID_PARAM; | 94 | return ACL_ERROR_INVALID_PARAM; |
| 97 | } | 95 | } |
| 98 | device = ret.first; | 96 | device = ret.first; |
| 99 | - DEVICE_STUB_DEBUG("logicId:{:d} serverId:{:d} key:{:d}", device.logic_id, serverId, device.id); | 97 | + HCCL_VM_DEBUG("logicId:{:d} serverId:{:d} key:{:d}", device.logic_id, serverId, device.id); |
| 100 | } | 98 | } |
| 101 | 99 | ||
| 102 | sim::Runner runner{}; | 100 | sim::Runner runner{}; |
| @@ -132,7 +130,7 @@ aclError aclrtSetDevice(int32_t deviceId) | |||
| 132 | sim::SetCurrCtxTls(currCtxId); | 130 | sim::SetCurrCtxTls(currCtxId); |
| 133 | return ACL_SUCCESS; | 131 | return ACL_SUCCESS; |
| 134 | } catch (const std::exception &e) { | 132 | } catch (const std::exception &e) { |
| 135 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 133 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 136 | return ACL_ERROR_INTERNAL_ERROR; | 134 | return ACL_ERROR_INTERNAL_ERROR; |
| 137 | } | 135 | } |
| 138 | } | 136 | } |
| @@ -140,7 +138,7 @@ aclError aclrtSetDevice(int32_t deviceId) | |||
| 140 | aclError aclrtResetDevice(int32_t deviceId) | 138 | aclError aclrtResetDevice(int32_t deviceId) |
| 141 | { | 139 | { |
| 142 | try { | 140 | try { |
| 143 | - DEVICE_STUB_INFO("deviceId:{:d}", deviceId); | 141 | + HCCL_VM_INFO("deviceId:{:d}", deviceId); |
| 144 | sim::Runner runner{}; | 142 | sim::Runner runner{}; |
| 145 | if (!sim::GetCurrRunnerTls(0, runner)) { | 143 | if (!sim::GetCurrRunnerTls(0, runner)) { |
| 146 | return ACL_ERROR_INVALID_PARAM; | 144 | return ACL_ERROR_INVALID_PARAM; |
| @@ -151,7 +149,7 @@ aclError aclrtResetDevice(int32_t deviceId) | |||
| 151 | } | 149 | } |
| 152 | auto currCtx = RunnerDB::GetById<sim::Context>(curCtxId); | 150 | auto currCtx = RunnerDB::GetById<sim::Context>(curCtxId); |
| 153 | if (!currCtx.has_value()) { | 151 | if (!currCtx.has_value()) { |
| 154 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 152 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 155 | return ACL_ERROR_INVALID_PARAM; | 153 | return ACL_ERROR_INVALID_PARAM; |
| 156 | } | 154 | } |
| 157 | 155 | ||
| @@ -159,7 +157,7 @@ aclError aclrtResetDevice(int32_t deviceId) | |||
| 159 | return stream.ctx_id == curCtxId && stream.is_primary_default == 1; | 157 | return stream.ctx_id == curCtxId && stream.is_primary_default == 1; |
| 160 | }); | 158 | }); |
| 161 | if (!streamRet.second) { | 159 | if (!streamRet.second) { |
| 162 | - DEVICE_STUB_ERROR("stream not found ctxId:{:d}", curCtxId); | 160 | + HCCL_VM_ERROR("stream not found ctxId:{:d}", curCtxId); |
| 163 | return ACL_ERROR_INVALID_PARAM; | 161 | return ACL_ERROR_INVALID_PARAM; |
| 164 | } | 162 | } |
| 165 | 163 | ||
| @@ -174,7 +172,7 @@ aclError aclrtResetDevice(int32_t deviceId) | |||
| 174 | sim::SetCurrCtxTls(curCtxId); | 172 | sim::SetCurrCtxTls(curCtxId); |
| 175 | return ACL_SUCCESS; | 173 | return ACL_SUCCESS; |
| 176 | } catch (const std::exception &e) { | 174 | } catch (const std::exception &e) { |
| 177 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 175 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 178 | return ACL_ERROR_INTERNAL_ERROR; | 176 | return ACL_ERROR_INTERNAL_ERROR; |
| 179 | } | 177 | } |
| 180 | } | 178 | } |
| @@ -182,7 +180,7 @@ aclError aclrtResetDevice(int32_t deviceId) | |||
| 182 | aclError aclrtResetDeviceForce(int32_t deviceId) | 180 | aclError aclrtResetDeviceForce(int32_t deviceId) |
| 183 | { | 181 | { |
| 184 | try { | 182 | try { |
| 185 | - DEVICE_STUB_DEBUG("stream not found deviceId:{:d}", deviceId); | 183 | + HCCL_VM_DEBUG("stream not found deviceId:{:d}", deviceId); |
| 186 | sim::Runner runner{}; | 184 | sim::Runner runner{}; |
| 187 | if (!sim::GetCurrRunnerTls(0, runner)) { | 185 | if (!sim::GetCurrRunnerTls(0, runner)) { |
| 188 | return ACL_ERROR_INVALID_PARAM; | 186 | return ACL_ERROR_INVALID_PARAM; |
| @@ -193,7 +191,7 @@ aclError aclrtResetDeviceForce(int32_t deviceId) | |||
| 193 | } | 191 | } |
| 194 | auto currCtx = RunnerDB::GetById<sim::Context>(curCtxId); | 192 | auto currCtx = RunnerDB::GetById<sim::Context>(curCtxId); |
| 195 | if (!currCtx.has_value()) { | 193 | if (!currCtx.has_value()) { |
| 196 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 194 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 197 | return ACL_ERROR_INVALID_PARAM; | 195 | return ACL_ERROR_INVALID_PARAM; |
| 198 | } | 196 | } |
| 199 | 197 | ||
| @@ -201,7 +199,7 @@ aclError aclrtResetDeviceForce(int32_t deviceId) | |||
| 201 | return stream.ctx_id == curCtxId && stream.is_primary_default == 1; | 199 | return stream.ctx_id == curCtxId && stream.is_primary_default == 1; |
| 202 | }); | 200 | }); |
| 203 | if (!streamRet.second) { | 201 | if (!streamRet.second) { |
| 204 | - DEVICE_STUB_ERROR("stream not found ctxId:{:d}", curCtxId); | 202 | + HCCL_VM_ERROR("stream not found ctxId:{:d}", curCtxId); |
| 205 | return ACL_ERROR_INVALID_PARAM; | 203 | return ACL_ERROR_INVALID_PARAM; |
| 206 | } | 204 | } |
| 207 | 205 | ||
| @@ -211,7 +209,7 @@ aclError aclrtResetDeviceForce(int32_t deviceId) | |||
| 211 | sim::SetCurrCtxTls(curCtxId); | 209 | sim::SetCurrCtxTls(curCtxId); |
| 212 | return ACL_SUCCESS; | 210 | return ACL_SUCCESS; |
| 213 | } catch (const std::exception &e) { | 211 | } catch (const std::exception &e) { |
| 214 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 212 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 215 | return ACL_ERROR_INTERNAL_ERROR; | 213 | return ACL_ERROR_INTERNAL_ERROR; |
| 216 | } | 214 | } |
| 217 | } | 215 | } |
| @@ -225,20 +223,20 @@ aclError aclrtGetDevice(int32_t* device) | |||
| 225 | } | 223 | } |
| 226 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 224 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 227 | if (!currCtx.has_value()) { | 225 | if (!currCtx.has_value()) { |
| 228 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 226 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 229 | return ACL_ERROR_INVALID_PARAM; | 227 | return ACL_ERROR_INVALID_PARAM; |
| 230 | } | 228 | } |
| 231 | 229 | ||
| 232 | auto devRes = RunnerDB::GetById<sim::Device>(currCtx->device_id); | 230 | auto devRes = RunnerDB::GetById<sim::Device>(currCtx->device_id); |
| 233 | if (!devRes.has_value()) { | 231 | if (!devRes.has_value()) { |
| 234 | - DEVICE_STUB_ERROR("device not found:{:d}", currCtx->device_id); | 232 | + HCCL_VM_ERROR("device not found:{:d}", currCtx->device_id); |
| 235 | return ACL_ERROR_INVALID_PARAM; | 233 | return ACL_ERROR_INVALID_PARAM; |
| 236 | } | 234 | } |
| 237 | *device = devRes->logic_id; | 235 | *device = devRes->logic_id; |
| 238 | - DEVICE_STUB_DEBUG("id:{:d}", devRes->logic_id); | 236 | + HCCL_VM_DEBUG("id:{:d}", devRes->logic_id); |
| 239 | return ACL_SUCCESS; | 237 | return ACL_SUCCESS; |
| 240 | } catch (const std::exception &e) { | 238 | } catch (const std::exception &e) { |
| 241 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 239 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 242 | return ACL_ERROR_INTERNAL_ERROR; | 240 | return ACL_ERROR_INTERNAL_ERROR; |
| 243 | } | 241 | } |
| 244 | } | 242 | } |
| @@ -262,13 +260,13 @@ aclError aclrtSetTsDevice(aclrtTsId tsId) | |||
| 262 | } | 260 | } |
| 263 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 261 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 264 | if (!currCtx.has_value()) { | 262 | if (!currCtx.has_value()) { |
| 265 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 263 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 266 | return ACL_ERROR_INVALID_PARAM; | 264 | return ACL_ERROR_INVALID_PARAM; |
| 267 | } | 265 | } |
| 268 | 266 | ||
| 269 | auto devRes = RunnerDB::GetById<sim::Device>(currCtx->device_id); | 267 | auto devRes = RunnerDB::GetById<sim::Device>(currCtx->device_id); |
| 270 | if (!devRes.has_value()) { | 268 | if (!devRes.has_value()) { |
| 271 | - DEVICE_STUB_ERROR("device not found:{:d}", currCtx->device_id); | 269 | + HCCL_VM_ERROR("device not found:{:d}", currCtx->device_id); |
| 272 | return ACL_ERROR_INVALID_PARAM; | 270 | return ACL_ERROR_INVALID_PARAM; |
| 273 | } | 271 | } |
| 274 | 272 | ||
| @@ -279,7 +277,7 @@ aclError aclrtSetTsDevice(aclrtTsId tsId) | |||
| 279 | 277 | ||
| 280 | return ACL_SUCCESS; | 278 | return ACL_SUCCESS; |
| 281 | } catch (const std::exception &e) { | 279 | } catch (const std::exception &e) { |
| 282 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 280 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 283 | return ACL_ERROR_INTERNAL_ERROR; | 281 | return ACL_ERROR_INTERNAL_ERROR; |
| 284 | } | 282 | } |
| 285 | } | 283 | } |
| @@ -292,13 +290,13 @@ aclError aclrtGetDeviceCount(uint32_t *count) | |||
| 292 | }); | 290 | }); |
| 293 | 291 | ||
| 294 | if (devs.empty()) { | 292 | if (devs.empty()) { |
| 295 | - DEVICE_STUB_ERROR("devices not found"); | 293 | + HCCL_VM_ERROR("devices not found"); |
| 296 | return ACL_ERROR_INVALID_PARAM; | 294 | return ACL_ERROR_INVALID_PARAM; |
| 297 | } | 295 | } |
| 298 | *count = devs.size(); | 296 | *count = devs.size(); |
| 299 | return ACL_SUCCESS; | 297 | return ACL_SUCCESS; |
| 300 | } catch (const std::exception &e) { | 298 | } catch (const std::exception &e) { |
| 301 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 299 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 302 | return ACL_ERROR_INTERNAL_ERROR; | 300 | return ACL_ERROR_INTERNAL_ERROR; |
| 303 | } | 301 | } |
| 304 | } | 302 | } |
| @@ -310,7 +308,7 @@ aclError aclrtGetDeviceUtilizationRate(int32_t deviceId, aclrtUtilizationInfo *u | |||
| 310 | return d.logic_id == deviceId; | 308 | return d.logic_id == deviceId; |
| 311 | }); | 309 | }); |
| 312 | if (!ret.second) { | 310 | if (!ret.second) { |
| 313 | - DEVICE_STUB_ERROR("device not found by phyId:{:d}", deviceId); | 311 | + HCCL_VM_ERROR("device not found by phyId:{:d}", deviceId); |
| 314 | return 0; | 312 | return 0; |
| 315 | } | 313 | } |
| 316 | 314 | ||
| @@ -320,7 +318,7 @@ aclError aclrtGetDeviceUtilizationRate(int32_t deviceId, aclrtUtilizationInfo *u | |||
| 320 | utilizationInfo->memoryUtilization = 20; | 318 | utilizationInfo->memoryUtilization = 20; |
| 321 | return ACL_SUCCESS; | 319 | return ACL_SUCCESS; |
| 322 | } catch (const std::exception &e) { | 320 | } catch (const std::exception &e) { |
| 323 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 321 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 324 | return ACL_ERROR_INTERNAL_ERROR; | 322 | return ACL_ERROR_INTERNAL_ERROR; |
| 325 | } | 323 | } |
| 326 | } | 324 | } |
| @@ -332,13 +330,13 @@ aclError aclrtQueryDeviceStatus(int32_t deviceId, aclrtDeviceStatus *deviceStatu | |||
| 332 | return d.logic_id == (uint32_t)deviceId; | 330 | return d.logic_id == (uint32_t)deviceId; |
| 333 | }); | 331 | }); |
| 334 | if (!ret.second) { | 332 | if (!ret.second) { |
| 335 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", deviceId); | 333 | + HCCL_VM_ERROR("device not found logicId:{:d}", deviceId); |
| 336 | return HcclResult::HCCL_E_NOT_FOUND; | 334 | return HcclResult::HCCL_E_NOT_FOUND; |
| 337 | } | 335 | } |
| 338 | *deviceStatus = (aclrtDeviceStatus)ret.first.status; | 336 | *deviceStatus = (aclrtDeviceStatus)ret.first.status; |
| 339 | return HcclVmResult::HCCL_SIM_SUCCESS; | 337 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 340 | } catch (const std::exception &e) { | 338 | } catch (const std::exception &e) { |
| 341 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 339 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 342 | return ACL_ERROR_INTERNAL_ERROR; | 340 | return ACL_ERROR_INTERNAL_ERROR; |
| 343 | } | 341 | } |
| 344 | } | 342 | } |
| @@ -351,17 +349,17 @@ const char *aclrtGetSocName() | |||
| 351 | return d.server_id == 1; | 349 | return d.server_id == 1; |
| 352 | }); | 350 | }); |
| 353 | if (!devRes.second) { | 351 | if (!devRes.second) { |
| 354 | - DEVICE_STUB_ERROR("device not found serverId:1"); | 352 | + HCCL_VM_ERROR("device not found serverId:1"); |
| 355 | return ""; | 353 | return ""; |
| 356 | } | 354 | } |
| 357 | 355 | ||
| 358 | thread_local static char SocName[128] = {0}; | 356 | thread_local static char SocName[128] = {0}; |
| 359 | memcpy(SocName, devRes.first.soc_version, strlen(devRes.first.soc_version)); | 357 | memcpy(SocName, devRes.first.soc_version, strlen(devRes.first.soc_version)); |
| 360 | SocName[strlen(devRes.first.soc_version)] = '\0'; | 358 | SocName[strlen(devRes.first.soc_version)] = '\0'; |
| 361 | - DEVICE_STUB_DEBUG("soc:{}", devRes.first.soc_version); | 359 | + HCCL_VM_DEBUG("soc:{}", devRes.first.soc_version); |
| 362 | return SocName; | 360 | return SocName; |
| 363 | } catch (const std::exception &e) { | 361 | } catch (const std::exception &e) { |
| 364 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 362 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 365 | static thread_local char s_errMsg[] = ""; | 363 | static thread_local char s_errMsg[] = ""; |
| 366 | return s_errMsg; | 364 | return s_errMsg; |
| 367 | } | 365 | } |
| @@ -376,7 +374,7 @@ aclError aclrtSetDeviceSatMode(aclrtFloatOverflowMode mode) | |||
| 376 | } | 374 | } |
| 377 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 375 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 378 | if (!currCtx.has_value()) { | 376 | if (!currCtx.has_value()) { |
| 379 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 377 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 380 | return ACL_ERROR_INVALID_PARAM; | 378 | return ACL_ERROR_INVALID_PARAM; |
| 381 | } | 379 | } |
| 382 | 380 | ||
| @@ -384,7 +382,7 @@ aclError aclrtSetDeviceSatMode(aclrtFloatOverflowMode mode) | |||
| 384 | RunnerDB::Update<sim::Device>(curDevId, [curDevId, mode](sim::Device &dev) { dev.overflow_mode = mode; }); | 382 | RunnerDB::Update<sim::Device>(curDevId, [curDevId, mode](sim::Device &dev) { dev.overflow_mode = mode; }); |
| 385 | return ACL_SUCCESS; | 383 | return ACL_SUCCESS; |
| 386 | } catch (const std::exception &e) { | 384 | } catch (const std::exception &e) { |
| 387 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 385 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 388 | return ACL_ERROR_INTERNAL_ERROR; | 386 | return ACL_ERROR_INTERNAL_ERROR; |
| 389 | } | 387 | } |
| 390 | } | 388 | } |
| @@ -398,19 +396,19 @@ aclError aclrtGetDeviceSatMode(aclrtFloatOverflowMode *mode) | |||
| 398 | } | 396 | } |
| 399 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 397 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 400 | if (!currCtx.has_value()) { | 398 | if (!currCtx.has_value()) { |
| 401 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 399 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 402 | return ACL_ERROR_INVALID_PARAM; | 400 | return ACL_ERROR_INVALID_PARAM; |
| 403 | } | 401 | } |
| 404 | 402 | ||
| 405 | auto dev = RunnerDB::GetById<sim::Device>(currCtx->device_id); | 403 | auto dev = RunnerDB::GetById<sim::Device>(currCtx->device_id); |
| 406 | if (!dev.has_value()) { | 404 | if (!dev.has_value()) { |
| 407 | - DEVICE_STUB_ERROR("device not found:{:d}", currCtx->device_id); | 405 | + HCCL_VM_ERROR("device not found:{:d}", currCtx->device_id); |
| 408 | return ACL_ERROR_INVALID_PARAM; | 406 | return ACL_ERROR_INVALID_PARAM; |
| 409 | } | 407 | } |
| 410 | *mode = (aclrtFloatOverflowMode)dev->overflow_mode; | 408 | *mode = (aclrtFloatOverflowMode)dev->overflow_mode; |
| 411 | return ACL_SUCCESS; | 409 | return ACL_SUCCESS; |
| 412 | } catch (const std::exception &e) { | 410 | } catch (const std::exception &e) { |
| 413 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 411 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 414 | return ACL_ERROR_INTERNAL_ERROR; | 412 | return ACL_ERROR_INTERNAL_ERROR; |
| 415 | } | 413 | } |
| 416 | } | 414 | } |
| @@ -422,7 +420,7 @@ aclError aclrtDeviceCanAccessPeer(int32_t *canAccessPeer, int32_t deviceId, int3 | |||
| 422 | return d.logic_id == deviceId; | 420 | return d.logic_id == deviceId; |
| 423 | }); | 421 | }); |
| 424 | if (!dev1.second) { | 422 | if (!dev1.second) { |
| 425 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", deviceId); | 423 | + HCCL_VM_ERROR("device not found logicId:{:d}", deviceId); |
| 426 | return ACL_ERROR_INVALID_PARAM; | 424 | return ACL_ERROR_INVALID_PARAM; |
| 427 | } | 425 | } |
| 428 | 426 | ||
| @@ -430,7 +428,7 @@ aclError aclrtDeviceCanAccessPeer(int32_t *canAccessPeer, int32_t deviceId, int3 | |||
| 430 | return d.logic_id == peerDeviceId; | 428 | return d.logic_id == peerDeviceId; |
| 431 | }); | 429 | }); |
| 432 | if (!dev2.second) { | 430 | if (!dev2.second) { |
| 433 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", peerDeviceId); | 431 | + HCCL_VM_ERROR("device not found logicId:{:d}", peerDeviceId); |
| 434 | return ACL_ERROR_INVALID_PARAM; | 432 | return ACL_ERROR_INVALID_PARAM; |
| 435 | } | 433 | } |
| 436 | 434 | ||
| @@ -441,14 +439,14 @@ aclError aclrtDeviceCanAccessPeer(int32_t *canAccessPeer, int32_t deviceId, int3 | |||
| 441 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; | 439 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; |
| 442 | }); | 440 | }); |
| 443 | if (!ret.second) { | 441 | if (!ret.second) { |
| 444 | - DEVICE_STUB_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); | 442 | + HCCL_VM_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); |
| 445 | return HcclResult::HCCL_E_NOT_FOUND; | 443 | return HcclResult::HCCL_E_NOT_FOUND; |
| 446 | } | 444 | } |
| 447 | 445 | ||
| 448 | *canAccessPeer = (int32_t)ret.first.access_by_remote; | 446 | *canAccessPeer = (int32_t)ret.first.access_by_remote; |
| 449 | return HcclVmResult::HCCL_SIM_SUCCESS; | 447 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 450 | } catch (const std::exception &e) { | 448 | } catch (const std::exception &e) { |
| 451 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 449 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 452 | return ACL_ERROR_INTERNAL_ERROR; | 450 | return ACL_ERROR_INTERNAL_ERROR; |
| 453 | } | 451 | } |
| 454 | } | 452 | } |
| @@ -463,18 +461,18 @@ aclError aclrtDeviceEnablePeerAccess(int32_t peerDeviceId, uint32_t flags) | |||
| 463 | } | 461 | } |
| 464 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 462 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 465 | if (!currCtx.has_value()) { | 463 | if (!currCtx.has_value()) { |
| 466 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 464 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 467 | return ACL_ERROR_INVALID_PARAM; | 465 | return ACL_ERROR_INVALID_PARAM; |
| 468 | } | 466 | } |
| 469 | 467 | ||
| 470 | auto dev1 = RunnerDB::GetById<sim::Device>(currCtx->device_id); | 468 | auto dev1 = RunnerDB::GetById<sim::Device>(currCtx->device_id); |
| 471 | if (!dev1.has_value()) { | 469 | if (!dev1.has_value()) { |
| 472 | - DEVICE_STUB_ERROR("device not found:{:d}", currCtx->device_id); | 470 | + HCCL_VM_ERROR("device not found:{:d}", currCtx->device_id); |
| 473 | return ACL_ERROR_INVALID_PARAM; | 471 | return ACL_ERROR_INVALID_PARAM; |
| 474 | } | 472 | } |
| 475 | 473 | ||
| 476 | if (dev1->logic_id == peerDeviceId) { | 474 | if (dev1->logic_id == peerDeviceId) { |
| 477 | - DEVICE_STUB_ERROR("invalid peerId:{:d}", peerDeviceId); | 475 | + HCCL_VM_ERROR("invalid peerId:{:d}", peerDeviceId); |
| 478 | return ACL_ERROR_INVALID_PARAM; | 476 | return ACL_ERROR_INVALID_PARAM; |
| 479 | } | 477 | } |
| 480 | 478 | ||
| @@ -482,7 +480,7 @@ aclError aclrtDeviceEnablePeerAccess(int32_t peerDeviceId, uint32_t flags) | |||
| 482 | return d.logic_id == peerDeviceId; | 480 | return d.logic_id == peerDeviceId; |
| 483 | }); | 481 | }); |
| 484 | if (!dev2.second) { | 482 | if (!dev2.second) { |
| 485 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", peerDeviceId); | 483 | + HCCL_VM_ERROR("device not found logicId:{:d}", peerDeviceId); |
| 486 | return ACL_ERROR_INVALID_PARAM; | 484 | return ACL_ERROR_INVALID_PARAM; |
| 487 | } | 485 | } |
| 488 | 486 | ||
| @@ -492,14 +490,14 @@ aclError aclrtDeviceEnablePeerAccess(int32_t peerDeviceId, uint32_t flags) | |||
| 492 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; | 490 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; |
| 493 | }); | 491 | }); |
| 494 | if (!ret.second) { | 492 | if (!ret.second) { |
| 495 | - DEVICE_STUB_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); | 493 | + HCCL_VM_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); |
| 496 | return HcclResult::HCCL_E_NOT_FOUND; | 494 | return HcclResult::HCCL_E_NOT_FOUND; |
| 497 | } | 495 | } |
| 498 | 496 | ||
| 499 | RunnerDB::Update<sim::DeviceConnection>(ret.first.id, [](sim::DeviceConnection &devConn) { devConn.access_by_remote = 1;}); | 497 | RunnerDB::Update<sim::DeviceConnection>(ret.first.id, [](sim::DeviceConnection &devConn) { devConn.access_by_remote = 1;}); |
| 500 | return ACL_SUCCESS; | 498 | return ACL_SUCCESS; |
| 501 | } catch (const std::exception &e) { | 499 | } catch (const std::exception &e) { |
| 502 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 500 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 503 | return ACL_ERROR_INTERNAL_ERROR; | 501 | return ACL_ERROR_INTERNAL_ERROR; |
| 504 | } | 502 | } |
| 505 | } | 503 | } |
| @@ -513,18 +511,18 @@ aclError aclrtDeviceDisablePeerAccess(int32_t peerDeviceId) | |||
| 513 | } | 511 | } |
| 514 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 512 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 515 | if (!currCtx.has_value()) { | 513 | if (!currCtx.has_value()) { |
| 516 | - DEVICE_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 514 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 517 | return ACL_ERROR_INVALID_PARAM; | 515 | return ACL_ERROR_INVALID_PARAM; |
| 518 | } | 516 | } |
| 519 | 517 | ||
| 520 | auto dev1 = RunnerDB::GetById<sim::Device>(currCtx->device_id); | 518 | auto dev1 = RunnerDB::GetById<sim::Device>(currCtx->device_id); |
| 521 | if (!dev1.has_value()) { | 519 | if (!dev1.has_value()) { |
| 522 | - DEVICE_STUB_ERROR("device not found:{:d}", currCtx->device_id); | 520 | + HCCL_VM_ERROR("device not found:{:d}", currCtx->device_id); |
| 523 | return ACL_ERROR_INVALID_PARAM; | 521 | return ACL_ERROR_INVALID_PARAM; |
| 524 | } | 522 | } |
| 525 | 523 | ||
| 526 | if (dev1->logic_id == peerDeviceId) { | 524 | if (dev1->logic_id == peerDeviceId) { |
| 527 | - DEVICE_STUB_ERROR("invalid peerId:{:d}", peerDeviceId); | 525 | + HCCL_VM_ERROR("invalid peerId:{:d}", peerDeviceId); |
| 528 | return ACL_ERROR_INVALID_PARAM; | 526 | return ACL_ERROR_INVALID_PARAM; |
| 529 | } | 527 | } |
| 530 | 528 | ||
| @@ -532,7 +530,7 @@ aclError aclrtDeviceDisablePeerAccess(int32_t peerDeviceId) | |||
| 532 | return d.logic_id == peerDeviceId; | 530 | return d.logic_id == peerDeviceId; |
| 533 | }); | 531 | }); |
| 534 | if (!dev2.second) { | 532 | if (!dev2.second) { |
| 535 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", peerDeviceId); | 533 | + HCCL_VM_ERROR("device not found logicId:{:d}", peerDeviceId); |
| 536 | return ACL_ERROR_INVALID_PARAM; | 534 | return ACL_ERROR_INVALID_PARAM; |
| 537 | } | 535 | } |
| 538 | 536 | ||
| @@ -542,14 +540,14 @@ aclError aclrtDeviceDisablePeerAccess(int32_t peerDeviceId) | |||
| 542 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; | 540 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; |
| 543 | }); | 541 | }); |
| 544 | if (!ret.second) { | 542 | if (!ret.second) { |
| 545 | - DEVICE_STUB_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); | 543 | + HCCL_VM_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); |
| 546 | return ACL_ERROR_INVALID_PARAM; | 544 | return ACL_ERROR_INVALID_PARAM; |
| 547 | } | 545 | } |
| 548 | 546 | ||
| 549 | RunnerDB::Update<sim::DeviceConnection>(ret.first.id, [](sim::DeviceConnection &devConn) { devConn.access_by_remote = 0;}); | 547 | RunnerDB::Update<sim::DeviceConnection>(ret.first.id, [](sim::DeviceConnection &devConn) { devConn.access_by_remote = 0;}); |
| 550 | return ACL_SUCCESS; | 548 | return ACL_SUCCESS; |
| 551 | } catch (const std::exception &e) { | 549 | } catch (const std::exception &e) { |
| 552 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 550 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 553 | return ACL_ERROR_INTERNAL_ERROR; | 551 | return ACL_ERROR_INTERNAL_ERROR; |
| 554 | } | 552 | } |
| 555 | } | 553 | } |
| @@ -561,13 +559,13 @@ aclError aclrtGetOverflowStatus(void *outputAddr, size_t outputSize, aclrtStream | |||
| 561 | uint64_t streamIdx = (uint64_t)(uintptr_t)stream; | 559 | uint64_t streamIdx = (uint64_t)(uintptr_t)stream; |
| 562 | auto stmRes = RunnerDB::GetById<sim::Stream>(streamIdx); | 560 | auto stmRes = RunnerDB::GetById<sim::Stream>(streamIdx); |
| 563 | if (!stmRes.has_value()) { | 561 | if (!stmRes.has_value()) { |
| 564 | - DEVICE_STUB_ERROR("stream not found:{:d}", streamIdx); | 562 | + HCCL_VM_ERROR("stream not found:{:d}", streamIdx); |
| 565 | return ACL_ERROR_INVALID_PARAM; | 563 | return ACL_ERROR_INVALID_PARAM; |
| 566 | } | 564 | } |
| 567 | 565 | ||
| 568 | auto ctxRes = RunnerDB::GetById<sim::Context>(stmRes->ctx_id); | 566 | auto ctxRes = RunnerDB::GetById<sim::Context>(stmRes->ctx_id); |
| 569 | if (!ctxRes.has_value()) { | 567 | if (!ctxRes.has_value()) { |
| 570 | - DEVICE_STUB_ERROR("ctx not found:{:d}", stmRes->ctx_id); | 568 | + HCCL_VM_ERROR("ctx not found:{:d}", stmRes->ctx_id); |
| 571 | return ACL_ERROR_INVALID_PARAM; | 569 | return ACL_ERROR_INVALID_PARAM; |
| 572 | } | 570 | } |
| 573 | 571 | ||
| @@ -576,7 +574,7 @@ aclError aclrtGetOverflowStatus(void *outputAddr, size_t outputSize, aclrtStream | |||
| 576 | return dev.device_id == deviceIdx; | 574 | return dev.device_id == deviceIdx; |
| 577 | }); | 575 | }); |
| 578 | if (!devStatusRes.second) { | 576 | if (!devStatusRes.second) { |
| 579 | - DEVICE_STUB_ERROR("device not found:{:d}", deviceIdx); | 577 | + HCCL_VM_ERROR("device not found:{:d}", deviceIdx); |
| 580 | return ACL_ERROR_INVALID_PARAM; | 578 | return ACL_ERROR_INVALID_PARAM; |
| 581 | } | 579 | } |
| 582 | 580 | ||
| @@ -585,7 +583,7 @@ aclError aclrtGetOverflowStatus(void *outputAddr, size_t outputSize, aclrtStream | |||
| 585 | 583 | ||
| 586 | return ACL_SUCCESS; | 584 | return ACL_SUCCESS; |
| 587 | } catch (const std::exception &e) { | 585 | } catch (const std::exception &e) { |
| 588 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 586 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 589 | return ACL_ERROR_INTERNAL_ERROR; | 587 | return ACL_ERROR_INTERNAL_ERROR; |
| 590 | } | 588 | } |
| 591 | } | 589 | } |
| @@ -596,13 +594,13 @@ aclError aclrtResetOverflowStatus(aclrtStream stream) | |||
| 596 | uint64_t streamIdx = (uint64_t)(uintptr_t)stream; | 594 | uint64_t streamIdx = (uint64_t)(uintptr_t)stream; |
| 597 | auto stmRes = RunnerDB::GetById<sim::Stream>(streamIdx); | 595 | auto stmRes = RunnerDB::GetById<sim::Stream>(streamIdx); |
| 598 | if (!stmRes.has_value()) { | 596 | if (!stmRes.has_value()) { |
| 599 | - DEVICE_STUB_ERROR("stream not found:{:d}", streamIdx); | 597 | + HCCL_VM_ERROR("stream not found:{:d}", streamIdx); |
| 600 | return ACL_ERROR_INVALID_PARAM; | 598 | return ACL_ERROR_INVALID_PARAM; |
| 601 | } | 599 | } |
| 602 | 600 | ||
| 603 | auto ctxRes = RunnerDB::GetById<sim::Context>(stmRes->ctx_id); | 601 | auto ctxRes = RunnerDB::GetById<sim::Context>(stmRes->ctx_id); |
| 604 | if (!ctxRes.has_value()) { | 602 | if (!ctxRes.has_value()) { |
| 605 | - DEVICE_STUB_ERROR("ctx not found:{:d}", stmRes->ctx_id); | 603 | + HCCL_VM_ERROR("ctx not found:{:d}", stmRes->ctx_id); |
| 606 | return ACL_ERROR_INVALID_PARAM; | 604 | return ACL_ERROR_INVALID_PARAM; |
| 607 | } | 605 | } |
| 608 | 606 | ||
| @@ -610,13 +608,13 @@ aclError aclrtResetOverflowStatus(aclrtStream stream) | |||
| 610 | auto devStatusRes = RunnerDB::GetOneByPred<sim::DeviceStatus>( | 608 | auto devStatusRes = RunnerDB::GetOneByPred<sim::DeviceStatus>( |
| 611 | [deviceIdx](const sim::DeviceStatus &dev) { return dev.device_id == deviceIdx; }); | 609 | [deviceIdx](const sim::DeviceStatus &dev) { return dev.device_id == deviceIdx; }); |
| 612 | if (!devStatusRes.second) { | 610 | if (!devStatusRes.second) { |
| 613 | - DEVICE_STUB_ERROR("device not found:{:d}", deviceIdx); | 611 | + HCCL_VM_ERROR("device not found:{:d}", deviceIdx); |
| 614 | return ACL_ERROR_INVALID_PARAM; | 612 | return ACL_ERROR_INVALID_PARAM; |
| 615 | } | 613 | } |
| 616 | RunnerDB::Update<sim::DeviceStatus>(devStatusRes.first.id, [](sim::DeviceStatus &devStatus) { devStatus.overflow_status = 0;}); | 614 | RunnerDB::Update<sim::DeviceStatus>(devStatusRes.first.id, [](sim::DeviceStatus &devStatus) { devStatus.overflow_status = 0;}); |
| 617 | return ACL_SUCCESS; | 615 | return ACL_SUCCESS; |
| 618 | } catch (const std::exception &e) { | 616 | } catch (const std::exception &e) { |
| 619 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 617 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 620 | return ACL_ERROR_INTERNAL_ERROR; | 618 | return ACL_ERROR_INTERNAL_ERROR; |
| 621 | } | 619 | } |
| 622 | } | 620 | } |
| @@ -668,7 +666,7 @@ aclError aclrtGetDevicesTopo(uint32_t deviceId, uint32_t otherDeviceId, uint64_t | |||
| 668 | return d.logic_id == deviceId; | 666 | return d.logic_id == deviceId; |
| 669 | }); | 667 | }); |
| 670 | if (!dev1.second) { | 668 | if (!dev1.second) { |
| 671 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", deviceId); | 669 | + HCCL_VM_ERROR("device not found logicId:{:d}", deviceId); |
| 672 | return ACL_ERROR_INVALID_PARAM; | 670 | return ACL_ERROR_INVALID_PARAM; |
| 673 | } | 671 | } |
| 674 | 672 | ||
| @@ -676,7 +674,7 @@ aclError aclrtGetDevicesTopo(uint32_t deviceId, uint32_t otherDeviceId, uint64_t | |||
| 676 | return d.logic_id == otherDeviceId; | 674 | return d.logic_id == otherDeviceId; |
| 677 | }); | 675 | }); |
| 678 | if (!dev2.second) { | 676 | if (!dev2.second) { |
| 679 | - DEVICE_STUB_ERROR("device not found logicId:{:d}", otherDeviceId); | 677 | + HCCL_VM_ERROR("device not found logicId:{:d}", otherDeviceId); |
| 680 | return ACL_ERROR_INVALID_PARAM; | 678 | return ACL_ERROR_INVALID_PARAM; |
| 681 | } | 679 | } |
| 682 | 680 | ||
| @@ -687,14 +685,14 @@ aclError aclrtGetDevicesTopo(uint32_t deviceId, uint32_t otherDeviceId, uint64_t | |||
| 687 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; | 685 | return devConn.src_dev_id == dev1Id && devConn.dst_dev_id == dev2Id; |
| 688 | }); | 686 | }); |
| 689 | if (!ret.second) { | 687 | if (!ret.second) { |
| 690 | - DEVICE_STUB_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); | 688 | + HCCL_VM_ERROR("connection not found src:{:d} dst:{:d}", dev1Id, dev2Id); |
| 691 | return HcclResult::HCCL_E_NOT_FOUND; | 689 | return HcclResult::HCCL_E_NOT_FOUND; |
| 692 | } | 690 | } |
| 693 | 691 | ||
| 694 | *value = (uint64_t)ret.first.link_type; | 692 | *value = (uint64_t)ret.first.link_type; |
| 695 | return HcclVmResult::HCCL_SIM_SUCCESS; | 693 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 696 | } catch (const std::exception &e) { | 694 | } catch (const std::exception &e) { |
| 697 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 695 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 698 | return ACL_ERROR_INTERNAL_ERROR; | 696 | return ACL_ERROR_INTERNAL_ERROR; |
| 699 | } | 697 | } |
| 700 | } | 698 | } |
| @@ -727,7 +725,7 @@ aclError aclrtGetPhyDevIdByLogicDevId(int32_t logicDevId, int32_t *const phyDevI | |||
| 727 | } | 725 | } |
| 728 | 726 | ||
| 729 | *phyDevId = (int32_t)device.physical_id; | 727 | *phyDevId = (int32_t)device.physical_id; |
| 730 | - DEVICE_STUB_DEBUG("server:{:d} logicId:{:d} phyId:{:d}",g_cur_server_key, logicDevId, *phyDevId); | 728 | + HCCL_VM_DEBUG("server:{:d} logicId:{:d} phyId:{:d}",g_cur_server_key, logicDevId, *phyDevId); |
| 731 | return ACL_SUCCESS; | 729 | return ACL_SUCCESS; |
| 732 | } | 730 | } |
| 733 | 731 | ||
| @@ -778,13 +776,13 @@ rtError_t rtGetDeviceIndexByPhyId(uint32_t phyId, uint32_t *devIndex) | |||
| 778 | return d.physical_id == phyId; | 776 | return d.physical_id == phyId; |
| 779 | }); | 777 | }); |
| 780 | if (!ret.second) { | 778 | if (!ret.second) { |
| 781 | - DEVICE_STUB_ERROR("device not found by phyId:{:d}", phyId); | 779 | + HCCL_VM_ERROR("device not found by phyId:{:d}", phyId); |
| 782 | return HcclResult::HCCL_E_NOT_FOUND; | 780 | return HcclResult::HCCL_E_NOT_FOUND; |
| 783 | } | 781 | } |
| 784 | *devIndex = ret.first.logic_id; | 782 | *devIndex = ret.first.logic_id; |
| 785 | return ACL_SUCCESS; | 783 | return ACL_SUCCESS; |
| 786 | } catch (const std::exception &e) { | 784 | } catch (const std::exception &e) { |
| 787 | - DEVICE_STUB_ERROR("exception:{}", e.what()); | 785 | + HCCL_VM_ERROR("exception:{}", e.what()); |
| 788 | return ACL_ERROR_INTERNAL_ERROR; | 786 | return ACL_ERROR_INTERNAL_ERROR; |
| 789 | } | 787 | } |
| 790 | } | 788 | } |
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -21,11 +24,6 @@ | |||
| 21 | 24 | ||
| 22 | 25 | ||
| 23 | 26 | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | 27 | ||
| 30 | 28 | ||
| 31 | extern "C" { | 29 | extern "C" { |
| @@ -40,7 +38,7 @@ aclError aclrtCreateEventWithFlag(aclrtEvent *event, uint32_t flag) | |||
| 40 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 38 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 41 | if (!currCtx.has_value()) { | 39 | if (!currCtx.has_value()) { |
| 42 | // not find | 40 | // not find |
| 43 | - EVENT_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 41 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 44 | return ACL_ERROR_INVALID_PARAM; | 42 | return ACL_ERROR_INVALID_PARAM; |
| 45 | } | 43 | } |
| 46 | 44 | ||
| @@ -57,7 +55,7 @@ aclError aclrtCreateEventWithFlag(aclrtEvent *event, uint32_t flag) | |||
| 57 | auto res = RunnerDB::Add<sim::Event>(tmp); | 55 | auto res = RunnerDB::Add<sim::Event>(tmp); |
| 58 | 56 | ||
| 59 | *event = (aclrtEvent)res; | 57 | *event = (aclrtEvent)res; |
| 60 | - EVENT_STUB_INFO("id:{:d}", res); | 58 | + HCCL_VM_INFO("id:{:d}", res); |
| 61 | return ACL_SUCCESS; | 59 | return ACL_SUCCESS; |
| 62 | } | 60 | } |
| 63 | 61 | ||
| @@ -74,7 +72,7 @@ aclError aclrtCreateEvent(aclrtEvent *event) | |||
| 74 | aclError aclrtDestroyEvent(aclrtEvent event) | 72 | aclError aclrtDestroyEvent(aclrtEvent event) |
| 75 | { | 73 | { |
| 76 | uint64_t eventId = (uint32_t)(uintptr_t)event; | 74 | uint64_t eventId = (uint32_t)(uintptr_t)event; |
| 77 | - EVENT_STUB_INFO("id:{:d}", eventId); | 75 | + HCCL_VM_INFO("id:{:d}", eventId); |
| 78 | RunnerDB::Delete<sim::Event>(eventId); | 76 | RunnerDB::Delete<sim::Event>(eventId); |
| 79 | return ACL_SUCCESS; | 77 | return ACL_SUCCESS; |
| 80 | } | 78 | } |
| @@ -87,14 +85,14 @@ aclError aclrtRecordEvent(aclrtEvent event, aclrtStream stream) | |||
| 87 | auto currEvent = RunnerDB::GetById<sim::Event>(eventIdx); | 85 | auto currEvent = RunnerDB::GetById<sim::Event>(eventIdx); |
| 88 | if (!currEvent.has_value()) { | 86 | if (!currEvent.has_value()) { |
| 89 | // not find | 87 | // not find |
| 90 | - EVENT_STUB_ERROR("event not found:{:d}", eventIdx); | 88 | + HCCL_VM_ERROR("event not found:{:d}", eventIdx); |
| 91 | return ACL_ERROR_INVALID_PARAM; | 89 | return ACL_ERROR_INVALID_PARAM; |
| 92 | } | 90 | } |
| 93 | 91 | ||
| 94 | auto currStm = RunnerDB::GetById<sim::Stream>(streamIdx); | 92 | auto currStm = RunnerDB::GetById<sim::Stream>(streamIdx); |
| 95 | if (!currStm.has_value()) { | 93 | if (!currStm.has_value()) { |
| 96 | // not find | 94 | // not find |
| 97 | - EVENT_STUB_ERROR("stream not found:{:d}", streamIdx); | 95 | + HCCL_VM_ERROR("stream not found:{:d}", streamIdx); |
| 98 | return ACL_ERROR_INVALID_PARAM; | 96 | return ACL_ERROR_INVALID_PARAM; |
| 99 | } | 97 | } |
| 100 | 98 | ||
| @@ -110,7 +108,7 @@ aclError aclrtResetEvent(aclrtEvent event, aclrtStream stream) | |||
| 110 | uint64_t eventIdx = (uint32_t)(uintptr_t)event; | 108 | uint64_t eventIdx = (uint32_t)(uintptr_t)event; |
| 111 | auto currEvent = RunnerDB::GetById<sim::Event>(eventIdx); | 109 | auto currEvent = RunnerDB::GetById<sim::Event>(eventIdx); |
| 112 | if (!currEvent.has_value()) { | 110 | if (!currEvent.has_value()) { |
| 113 | - EVENT_STUB_ERROR("event not found:{:d}", eventIdx); | 111 | + HCCL_VM_ERROR("event not found:{:d}", eventIdx); |
| 114 | return ACL_ERROR_INVALID_PARAM; | 112 | return ACL_ERROR_INVALID_PARAM; |
| 115 | } | 113 | } |
| 116 | RunnerDB::Update<sim::Event>(eventIdx, [](sim::Event &evt) { evt.status = ACL_EVENT_RECORDED_STATUS_COMPLETE;}); | 114 | RunnerDB::Update<sim::Event>(eventIdx, [](sim::Event &evt) { evt.status = ACL_EVENT_RECORDED_STATUS_COMPLETE;}); |
| @@ -122,7 +120,7 @@ aclError aclrtQueryEventStatus(aclrtEvent event, aclrtEventRecordedStatus *statu | |||
| 122 | uint64_t eventId = (uint32_t)(uintptr_t)event; | 120 | uint64_t eventId = (uint32_t)(uintptr_t)event; |
| 123 | auto res = RunnerDB::GetById<sim::Event>(eventId); | 121 | auto res = RunnerDB::GetById<sim::Event>(eventId); |
| 124 | if (!res.has_value()) { | 122 | if (!res.has_value()) { |
| 125 | - EVENT_STUB_ERROR("event not found:{:d}", eventId); | 123 | + HCCL_VM_ERROR("event not found:{:d}", eventId); |
| 126 | return ACL_ERROR_INVALID_PARAM; | 124 | return ACL_ERROR_INVALID_PARAM; |
| 127 | } | 125 | } |
| 128 | 126 | ||
| @@ -197,7 +195,7 @@ aclError aclrtGetEventAvailNum(uint32_t *eventCount) | |||
| 197 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 195 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 198 | if (!currCtx.has_value()) { | 196 | if (!currCtx.has_value()) { |
| 199 | // not find | 197 | // not find |
| 200 | - EVENT_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 198 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 201 | return ACL_ERROR_INVALID_PARAM; | 199 | return ACL_ERROR_INVALID_PARAM; |
| 202 | } | 200 | } |
| 203 | 201 | ||
| @@ -205,7 +203,7 @@ aclError aclrtGetEventAvailNum(uint32_t *eventCount) | |||
| 205 | 203 | ||
| 206 | auto device = RunnerDB::GetById<sim::Device>(devId); | 204 | auto device = RunnerDB::GetById<sim::Device>(devId); |
| 207 | if (!device.has_value()) { | 205 | if (!device.has_value()) { |
| 208 | - EVENT_STUB_ERROR("device not found:{:d}", devId); | 206 | + HCCL_VM_ERROR("device not found:{:d}", devId); |
| 209 | return ACL_ERROR_INVALID_PARAM; | 207 | return ACL_ERROR_INVALID_PARAM; |
| 210 | } | 208 | } |
| 211 | 209 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -21,11 +24,6 @@ | |||
| 21 | 24 | ||
| 22 | 25 | ||
| 23 | 26 | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | 27 | ||
| 30 | 28 | ||
| 31 | extern "C" { | 29 | extern "C" { |
| @@ -34,56 +32,56 @@ extern "C" { | |||
| 34 | aclError aclrtSetExceptionInfoCallback(aclrtExceptionInfoCallback callback) | 32 | aclError aclrtSetExceptionInfoCallback(aclrtExceptionInfoCallback callback) |
| 35 | { | 33 | { |
| 36 | (void) callback; | 34 | (void) callback; |
| 37 | - EXC_STUB_TRACE("not supported"); | 35 | + HCCL_VM_TRACE("not supported"); |
| 38 | return ACL_SUCCESS; | 36 | return ACL_SUCCESS; |
| 39 | } | 37 | } |
| 40 | 38 | ||
| 41 | uint32_t aclrtGetTaskIdFromExceptionInfo(const aclrtExceptionInfo *info) | 39 | uint32_t aclrtGetTaskIdFromExceptionInfo(const aclrtExceptionInfo *info) |
| 42 | { | 40 | { |
| 43 | (void) info; | 41 | (void) info; |
| 44 | - EXC_STUB_TRACE("not supported"); | 42 | + HCCL_VM_TRACE("not supported"); |
| 45 | return ACL_SUCCESS; | 43 | return ACL_SUCCESS; |
| 46 | } | 44 | } |
| 47 | 45 | ||
| 48 | uint32_t aclrtGetStreamIdFromExceptionInfo(const aclrtExceptionInfo *info) | 46 | uint32_t aclrtGetStreamIdFromExceptionInfo(const aclrtExceptionInfo *info) |
| 49 | { | 47 | { |
| 50 | (void) info; | 48 | (void) info; |
| 51 | - EXC_STUB_TRACE("not supported"); | 49 | + HCCL_VM_TRACE("not supported"); |
| 52 | return ACL_SUCCESS; | 50 | return ACL_SUCCESS; |
| 53 | } | 51 | } |
| 54 | 52 | ||
| 55 | uint32_t aclrtGetThreadIdFromExceptionInfo(const aclrtExceptionInfo *info) | 53 | uint32_t aclrtGetThreadIdFromExceptionInfo(const aclrtExceptionInfo *info) |
| 56 | { | 54 | { |
| 57 | (void) info; | 55 | (void) info; |
| 58 | - EXC_STUB_TRACE("not supported"); | 56 | + HCCL_VM_TRACE("not supported"); |
| 59 | return ACL_SUCCESS; | 57 | return ACL_SUCCESS; |
| 60 | } | 58 | } |
| 61 | 59 | ||
| 62 | uint32_t aclrtGetDeviceIdFromExceptionInfo(const aclrtExceptionInfo *info) | 60 | uint32_t aclrtGetDeviceIdFromExceptionInfo(const aclrtExceptionInfo *info) |
| 63 | { | 61 | { |
| 64 | (void) info; | 62 | (void) info; |
| 65 | - EXC_STUB_TRACE("not supported"); | 63 | + HCCL_VM_TRACE("not supported"); |
| 66 | return ACL_SUCCESS; | 64 | return ACL_SUCCESS; |
| 67 | } | 65 | } |
| 68 | 66 | ||
| 69 | uint32_t aclrtGetErrorCodeFromExceptionInfo(const aclrtExceptionInfo *info) | 67 | uint32_t aclrtGetErrorCodeFromExceptionInfo(const aclrtExceptionInfo *info) |
| 70 | { | 68 | { |
| 71 | (void) info; | 69 | (void) info; |
| 72 | - EXC_STUB_TRACE("not supported"); | 70 | + HCCL_VM_TRACE("not supported"); |
| 73 | return ACL_SUCCESS; | 71 | return ACL_SUCCESS; |
| 74 | } | 72 | } |
| 75 | 73 | ||
| 76 | aclError aclrtPeekAtLastError(aclrtLastErrLevel level) | 74 | aclError aclrtPeekAtLastError(aclrtLastErrLevel level) |
| 77 | { | 75 | { |
| 78 | (void) level; | 76 | (void) level; |
| 79 | - EXC_STUB_TRACE("not supported"); | 77 | + HCCL_VM_TRACE("not supported"); |
| 80 | return ACL_SUCCESS; | 78 | return ACL_SUCCESS; |
| 81 | } | 79 | } |
| 82 | 80 | ||
| 83 | aclError aclrtGetLastError(aclrtLastErrLevel level) | 81 | aclError aclrtGetLastError(aclrtLastErrLevel level) |
| 84 | { | 82 | { |
| 85 | (void) level; | 83 | (void) level; |
| 86 | - EXC_STUB_TRACE("not supported"); | 84 | + HCCL_VM_TRACE("not supported"); |
| 87 | return ACL_SUCCESS; | 85 | return ACL_SUCCESS; |
| 88 | } | 86 | } |
| 89 | 87 | ||
| @@ -93,7 +91,7 @@ aclError aclrtGetMemUceInfo(int32_t deviceId, aclrtMemUceInfo *memUceInfoArray, | |||
| 93 | (void) memUceInfoArray; | 91 | (void) memUceInfoArray; |
| 94 | (void) arraySize; | 92 | (void) arraySize; |
| 95 | (void) retSize; | 93 | (void) retSize; |
| 96 | - EXC_STUB_TRACE("not supported"); | 94 | + HCCL_VM_TRACE("not supported"); |
| 97 | return ACL_SUCCESS; | 95 | return ACL_SUCCESS; |
| 98 | } | 96 | } |
| 99 | 97 | ||
| @@ -102,7 +100,7 @@ aclError aclrtMemUceRepair(int32_t deviceId, aclrtMemUceInfo *memUceInfoArray, s | |||
| 102 | (void) deviceId; | 100 | (void) deviceId; |
| 103 | (void) memUceInfoArray; | 101 | (void) memUceInfoArray; |
| 104 | (void) arraySize; | 102 | (void) arraySize; |
| 105 | - EXC_STUB_TRACE("not supported"); | 103 | + HCCL_VM_TRACE("not supported"); |
| 106 | return ACL_SUCCESS; | 104 | return ACL_SUCCESS; |
| 107 | } | 105 | } |
| 108 | 106 | ||
| @@ -110,14 +108,14 @@ aclError aclrtDeviceTaskAbort(int32_t deviceId, uint32_t timeout) | |||
| 110 | { | 108 | { |
| 111 | (void) deviceId; | 109 | (void) deviceId; |
| 112 | (void) timeout; | 110 | (void) timeout; |
| 113 | - EXC_STUB_TRACE("not supported"); | 111 | + HCCL_VM_TRACE("not supported"); |
| 114 | return ACL_SUCCESS; | 112 | return ACL_SUCCESS; |
| 115 | } | 113 | } |
| 116 | 114 | ||
| 117 | aclError aclRecoverAllHcclTasks(int32_t deviceId) | 115 | aclError aclRecoverAllHcclTasks(int32_t deviceId) |
| 118 | { | 116 | { |
| 119 | (void) deviceId; | 117 | (void) deviceId; |
| 120 | - EXC_STUB_TRACE("not supported"); | 118 | + HCCL_VM_TRACE("not supported"); |
| 121 | return ACL_SUCCESS; | 119 | return ACL_SUCCESS; |
| 122 | } | 120 | } |
| 123 | 121 | ||
| @@ -125,7 +123,7 @@ aclError aclrtGetErrorVerbose(int32_t deviceId, aclrtErrorInfo *errorInfo) | |||
| 125 | { | 123 | { |
| 126 | (void) deviceId; | 124 | (void) deviceId; |
| 127 | (void) errorInfo; | 125 | (void) errorInfo; |
| 128 | - EXC_STUB_TRACE("not supported"); | 126 | + HCCL_VM_TRACE("not supported"); |
| 129 | return ACL_SUCCESS; | 127 | return ACL_SUCCESS; |
| 130 | } | 128 | } |
| 131 | 129 | ||
| @@ -133,7 +131,7 @@ aclError aclrtRepairError(int32_t deviceId, const aclrtErrorInfo *errorInfo) | |||
| 133 | { | 131 | { |
| 134 | (void) deviceId; | 132 | (void) deviceId; |
| 135 | (void) errorInfo; | 133 | (void) errorInfo; |
| 136 | - EXC_STUB_TRACE("not supported"); | 134 | + HCCL_VM_TRACE("not supported"); |
| 137 | return ACL_SUCCESS; | 135 | return ACL_SUCCESS; |
| 138 | } | 136 | } |
| 139 | 137 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -21,11 +24,6 @@ | |||
| 21 | 24 | ||
| 22 | 25 | ||
| 23 | 26 | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | 27 | ||
| 30 | 28 | ||
| 31 | extern "C" { | 29 | extern "C" { |
| @@ -37,7 +35,7 @@ aclError aclrtLaunchCallback(aclrtCallback fn, void *userData, aclrtCallbackBloc | |||
| 37 | (void) userData; | 35 | (void) userData; |
| 38 | (void) blockType; | 36 | (void) blockType; |
| 39 | (void) stream; | 37 | (void) stream; |
| 40 | - EXEC_STUB_TRACE("not supported"); | 38 | + HCCL_VM_TRACE("not supported"); |
| 41 | return ACL_SUCCESS; | 39 | return ACL_SUCCESS; |
| 42 | } | 40 | } |
| 43 | 41 | ||
| @@ -45,14 +43,14 @@ aclError aclrtSubscribeReport(uint64_t threadId, aclrtStream stream) | |||
| 45 | { | 43 | { |
| 46 | (void) threadId; | 44 | (void) threadId; |
| 47 | (void) stream; | 45 | (void) stream; |
| 48 | - EXEC_STUB_TRACE("not supported"); | 46 | + HCCL_VM_TRACE("not supported"); |
| 49 | return ACL_SUCCESS; | 47 | return ACL_SUCCESS; |
| 50 | } | 48 | } |
| 51 | 49 | ||
| 52 | aclError aclrtProcessReport(int32_t timeout) | 50 | aclError aclrtProcessReport(int32_t timeout) |
| 53 | { | 51 | { |
| 54 | (void) timeout; | 52 | (void) timeout; |
| 55 | - EXEC_STUB_TRACE("not supported"); | 53 | + HCCL_VM_TRACE("not supported"); |
| 56 | return ACL_SUCCESS; | 54 | return ACL_SUCCESS; |
| 57 | } | 55 | } |
| 58 | 56 | ||
| @@ -60,7 +58,7 @@ aclError aclrtUnSubscribeReport(uint64_t threadId, aclrtStream stream) | |||
| 60 | { | 58 | { |
| 61 | (void) threadId; | 59 | (void) threadId; |
| 62 | (void) stream; | 60 | (void) stream; |
| 63 | - EXEC_STUB_TRACE("not supported"); | 61 | + HCCL_VM_TRACE("not supported"); |
| 64 | return ACL_SUCCESS; | 62 | return ACL_SUCCESS; |
| 65 | } | 63 | } |
| 66 | 64 | ||
| @@ -68,14 +66,14 @@ aclError aclrtSubscribeHostFunc(uint64_t hostFuncThreadId, aclrtStream exeStream | |||
| 68 | { | 66 | { |
| 69 | (void) hostFuncThreadId; | 67 | (void) hostFuncThreadId; |
| 70 | (void) exeStream; | 68 | (void) exeStream; |
| 71 | - EXEC_STUB_TRACE("not supported"); | 69 | + HCCL_VM_TRACE("not supported"); |
| 72 | return ACL_SUCCESS; | 70 | return ACL_SUCCESS; |
| 73 | } | 71 | } |
| 74 | 72 | ||
| 75 | aclError aclrtProcessHostFunc(int32_t timeout) | 73 | aclError aclrtProcessHostFunc(int32_t timeout) |
| 76 | { | 74 | { |
| 77 | (void) timeout; | 75 | (void) timeout; |
| 78 | - EXEC_STUB_TRACE("not supported"); | 76 | + HCCL_VM_TRACE("not supported"); |
| 79 | return ACL_SUCCESS; | 77 | return ACL_SUCCESS; |
| 80 | } | 78 | } |
| 81 | 79 | ||
| @@ -83,21 +81,21 @@ aclError aclrtUnSubscribeHostFunc(uint64_t hostFuncThreadId, aclrtStream exeStre | |||
| 83 | { | 81 | { |
| 84 | (void) hostFuncThreadId; | 82 | (void) hostFuncThreadId; |
| 85 | (void) exeStream; | 83 | (void) exeStream; |
| 86 | - EXEC_STUB_TRACE("not supported"); | 84 | + HCCL_VM_TRACE("not supported"); |
| 87 | return ACL_SUCCESS; | 85 | return ACL_SUCCESS; |
| 88 | } | 86 | } |
| 89 | 87 | ||
| 90 | aclError aclrtGetOpTimeoutInterval(uint64_t *interval) | 88 | aclError aclrtGetOpTimeoutInterval(uint64_t *interval) |
| 91 | { | 89 | { |
| 92 | (void) interval; | 90 | (void) interval; |
| 93 | - EXEC_STUB_TRACE("not supported"); | 91 | + HCCL_VM_TRACE("not supported"); |
| 94 | return ACL_SUCCESS; | 92 | return ACL_SUCCESS; |
| 95 | } | 93 | } |
| 96 | 94 | ||
| 97 | aclError aclrtSetOpExecuteTimeOut(uint32_t timeout) | 95 | aclError aclrtSetOpExecuteTimeOut(uint32_t timeout) |
| 98 | { | 96 | { |
| 99 | (void) timeout; | 97 | (void) timeout; |
| 100 | - EXEC_STUB_TRACE("not supported"); | 98 | + HCCL_VM_TRACE("not supported"); |
| 101 | return ACL_SUCCESS; | 99 | return ACL_SUCCESS; |
| 102 | } | 100 | } |
| 103 | 101 | ||
| @@ -105,21 +103,21 @@ aclError aclrtSetOpExecuteTimeOutV2(uint64_t timeout, uint64_t *actualTimeout) | |||
| 105 | { | 103 | { |
| 106 | (void) timeout; | 104 | (void) timeout; |
| 107 | (void) actualTimeout; | 105 | (void) actualTimeout; |
| 108 | - EXEC_STUB_TRACE("not supported"); | 106 | + HCCL_VM_TRACE("not supported"); |
| 109 | return ACL_SUCCESS; | 107 | return ACL_SUCCESS; |
| 110 | } | 108 | } |
| 111 | 109 | ||
| 112 | aclError aclrtSetOpExecuteTimeOutWithMs(uint32_t timeout) | 110 | aclError aclrtSetOpExecuteTimeOutWithMs(uint32_t timeout) |
| 113 | { | 111 | { |
| 114 | (void) timeout; | 112 | (void) timeout; |
| 115 | - EXEC_STUB_TRACE("not supported"); | 113 | + HCCL_VM_TRACE("not supported"); |
| 116 | return ACL_SUCCESS; | 114 | return ACL_SUCCESS; |
| 117 | } | 115 | } |
| 118 | 116 | ||
| 119 | aclError aclrtGetThreadLastTaskId(uint32_t *taskId) | 117 | aclError aclrtGetThreadLastTaskId(uint32_t *taskId) |
| 120 | { | 118 | { |
| 121 | (void) taskId; | 119 | (void) taskId; |
| 122 | - EXEC_STUB_TRACE("not supported"); | 120 | + HCCL_VM_TRACE("not supported"); |
| 123 | return ACL_SUCCESS; | 121 | return ACL_SUCCESS; |
| 124 | } | 122 | } |
| 125 | 123 | ||
| @@ -128,7 +126,7 @@ aclError aclrtLaunchHostFunc(aclrtStream stream, aclrtHostFunc fn, void *args) | |||
| 128 | (void) stream; | 126 | (void) stream; |
| 129 | (void) fn; | 127 | (void) fn; |
| 130 | (void) args; | 128 | (void) args; |
| 131 | - EXEC_STUB_TRACE("not supported"); | 129 | + HCCL_VM_TRACE("not supported"); |
| 132 | return ACL_SUCCESS; | 130 | return ACL_SUCCESS; |
| 133 | } | 131 | } |
| 134 | 132 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -27,10 +30,6 @@ | |||
| 27 | 30 | ||
| 28 | 31 | ||
| 29 | 32 | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| 33 | - | ||
| 34 | 33 | ||
| 35 | // rank 进程加载本库时先于 main 预热仅校验模式缓存,使后续引流判定拿到确定值。 | 34 | // rank 进程加载本库时先于 main 预热仅校验模式缓存,使后续引流判定拿到确定值。 |
| 36 | // 只读 DB、不打日志(此刻日志组件可能尚未初始化)。 | 35 | // 只读 DB、不打日志(此刻日志组件可能尚未初始化)。 |
| @@ -73,14 +72,14 @@ bool GetPtrNameByVirPtr(const void* virPtr, uint32_t& offset, sim::PhyMemBlock & | |||
| 73 | (devPtr < (virMem.start_ptr + virMem.size)) && | 72 | (devPtr < (virMem.start_ptr + virMem.size)) && |
| 74 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); | 73 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); |
| 75 | if (!virMemRes.second) { | 74 | if (!virMemRes.second) { |
| 76 | - MEM_STUB_ERROR("can not find this buff offset ptr:{:p}", virPtr); | 75 | + HCCL_VM_ERROR("can not find this buff offset ptr:{:p}", virPtr); |
| 77 | return false; | 76 | return false; |
| 78 | } | 77 | } |
| 79 | 78 | ||
| 80 | auto phyMemId = virMemRes.first.phy_mem_id; | 79 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 81 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 80 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 82 | if (!phyMemRes.has_value()) { | 81 | if (!phyMemRes.has_value()) { |
| 83 | - MEM_STUB_ERROR("can not find phy Mem id:{:d}", phyMemId); | 82 | + HCCL_VM_ERROR("can not find phy Mem id:{:d}", phyMemId); |
| 84 | return false; | 83 | return false; |
| 85 | }; | 84 | }; |
| 86 | offset = (uint32_t)(devPtr - virMemRes.first.start_ptr); | 85 | offset = (uint32_t)(devPtr - virMemRes.first.start_ptr); |
| @@ -99,7 +98,7 @@ uint32_t GetRankIdByVirAddr(const void* virAddr) | |||
| 99 | uint32_t deviceId = phyMem.device_id; | 98 | uint32_t deviceId = phyMem.device_id; |
| 100 | auto rank = RunnerDB::GetOneByPred<sim::Rank>([deviceId](const sim::Rank &rank) { return rank.device_id == deviceId; }); | 99 | auto rank = RunnerDB::GetOneByPred<sim::Rank>([deviceId](const sim::Rank &rank) { return rank.device_id == deviceId; }); |
| 101 | if (!rank.second) { | 100 | if (!rank.second) { |
| 102 | - MEM_STUB_ERROR("cannot find rank by device id[{}]", deviceId); | 101 | + HCCL_VM_ERROR("cannot find rank by device id[{}]", deviceId); |
| 103 | return 0; | 102 | return 0; |
| 104 | } | 103 | } |
| 105 | 104 | ||
| @@ -115,14 +114,14 @@ void* GetRealPtrByAddr(const void* virPtr) | |||
| 115 | (devPtr < (virMem.start_ptr + virMem.size)) && | 114 | (devPtr < (virMem.start_ptr + virMem.size)) && |
| 116 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); | 115 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); |
| 117 | if (!virMemRes.second) { | 116 | if (!virMemRes.second) { |
| 118 | - MEM_STUB_ERROR("can not find this buff offset ptr:{:p}", virPtr); | 117 | + HCCL_VM_ERROR("can not find this buff offset ptr:{:p}", virPtr); |
| 119 | return nullptr; | 118 | return nullptr; |
| 120 | } | 119 | } |
| 121 | 120 | ||
| 122 | void* devStartPtr = (void*)(uintptr_t)virMemRes.first.start_ptr; | 121 | void* devStartPtr = (void*)(uintptr_t)virMemRes.first.start_ptr; |
| 123 | auto hostPtr = sim::DeviceMemoryManager::GetInstance().GetHostPtrByDevPtr(devStartPtr); | 122 | auto hostPtr = sim::DeviceMemoryManager::GetInstance().GetHostPtrByDevPtr(devStartPtr); |
| 124 | if (hostPtr == nullptr) { | 123 | if (hostPtr == nullptr) { |
| 125 | - MEM_STUB_ERROR("can not find host ptr by dev ptr:{:p}", devStartPtr); | 124 | + HCCL_VM_ERROR("can not find host ptr by dev ptr:{:p}", devStartPtr); |
| 126 | return nullptr; | 125 | return nullptr; |
| 127 | } | 126 | } |
| 128 | 127 | ||
| @@ -146,7 +145,7 @@ aclError aclrtMallocHost(void **hostPtr, size_t size) | |||
| 146 | bool checkOnlyMode = sim::IsCheckOnlyMode(); | 145 | bool checkOnlyMode = sim::IsCheckOnlyMode(); |
| 147 | // 超过复用区上界直接报错,不回退真实分配。 | 146 | // 超过复用区上界直接报错,不回退真实分配。 |
| 148 | if (sim::CommPoolPolicy::ExceedsCeiling(size, checkOnlyMode)) { | 147 | if (sim::CommPoolPolicy::ExceedsCeiling(size, checkOnlyMode)) { |
| 149 | - MEM_STUB_ERROR("malloc host size:{:d} exceeds pool ceiling, reject", size); | 148 | + HCCL_VM_ERROR("malloc host size:{:d} exceeds pool ceiling, reject", size); |
| 150 | return ACL_ERROR_INTERNAL_ERROR; | 149 | return ACL_ERROR_INTERNAL_ERROR; |
| 151 | } | 150 | } |
| 152 | // 仅校验模式下大块引流到共享复用区 HcclCommPool | 151 | // 仅校验模式下大块引流到共享复用区 HcclCommPool |
| @@ -155,16 +154,16 @@ aclError aclrtMallocHost(void **hostPtr, size_t size) | |||
| 155 | if (base == nullptr) { | 154 | if (base == nullptr) { |
| 156 | base = sim::MemoryManager::GetInstance().AcquireMemByName(sim::CommPoolPolicy::kPoolName); | 155 | base = sim::MemoryManager::GetInstance().AcquireMemByName(sim::CommPoolPolicy::kPoolName); |
| 157 | if (base == nullptr) { | 156 | if (base == nullptr) { |
| 158 | - MEM_STUB_ERROR("malloc host acquire HcclCommPool failed, size:{:d}", size); | 157 | + HCCL_VM_ERROR("malloc host acquire HcclCommPool failed, size:{:d}", size); |
| 159 | return ACL_ERROR_INTERNAL_ERROR; | 158 | return ACL_ERROR_INTERNAL_ERROR; |
| 160 | } | 159 | } |
| 161 | CacheCommPoolBase(base); | 160 | CacheCommPoolBase(base); |
| 162 | } | 161 | } |
| 163 | *hostPtr = base; | 162 | *hostPtr = base; |
| 164 | - MEM_STUB_INFO("malloc host (SHM) addr:{:p}, size:{:d}", *hostPtr, size); | 163 | + HCCL_VM_INFO("malloc host (SHM) addr:{:p}, size:{:d}", *hostPtr, size); |
| 165 | } else { | 164 | } else { |
| 166 | *hostPtr = malloc(size); | 165 | *hostPtr = malloc(size); |
| 167 | - MEM_STUB_INFO("malloc host addr:{:p}, size:{:d}", *hostPtr, size); | 166 | + HCCL_VM_INFO("malloc host addr:{:p}, size:{:d}", *hostPtr, size); |
| 168 | } | 167 | } |
| 169 | return ACL_SUCCESS; | 168 | return ACL_SUCCESS; |
| 170 | } | 169 | } |
| @@ -177,10 +176,10 @@ aclError aclrtMallocHostWithCfg(void **ptr, uint64_t size, aclrtMallocConfig *cf | |||
| 177 | 176 | ||
| 178 | aclError aclrtFreeHost(void *hostPtr) | 177 | aclError aclrtFreeHost(void *hostPtr) |
| 179 | { | 178 | { |
| 180 | - MEM_STUB_INFO("[MEM] free host addr:{:p}", hostPtr); | 179 | + HCCL_VM_INFO("free host addr:{:p}", hostPtr); |
| 181 | // 池内地址:空操作(复用区生命周期由主进程管理,不在此 free) | 180 | // 池内地址:空操作(复用区生命周期由主进程管理,不在此 free) |
| 182 | if (IsInCommPool(hostPtr)) { | 181 | if (IsInCommPool(hostPtr)) { |
| 183 | - MEM_STUB_INFO("free host (SHM noop) addr:{:p}", hostPtr); | 182 | + HCCL_VM_INFO("free host (SHM noop) addr:{:p}", hostPtr); |
| 184 | return ACL_SUCCESS; | 183 | return ACL_SUCCESS; |
| 185 | } | 184 | } |
| 186 | free(hostPtr); | 185 | free(hostPtr); |
| @@ -195,7 +194,7 @@ aclError aclrtMalloc(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | |||
| 195 | } | 194 | } |
| 196 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 195 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 197 | if (!currCtx.has_value()) { | 196 | if (!currCtx.has_value()) { |
| 198 | - MEM_STUB_ERROR("can not find current context:{:d}", runner.current_ctx_id); | 197 | + HCCL_VM_ERROR("can not find current context:{:d}", runner.current_ctx_id); |
| 199 | return ACL_ERROR_INTERNAL_ERROR; | 198 | return ACL_ERROR_INTERNAL_ERROR; |
| 200 | } | 199 | } |
| 201 | 200 | ||
| @@ -203,7 +202,7 @@ aclError aclrtMalloc(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | |||
| 203 | 202 | ||
| 204 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); | 203 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); |
| 205 | if (!dev.has_value()) { | 204 | if (!dev.has_value()) { |
| 206 | - MEM_STUB_ERROR("can not find device id:{:d}", deviceId); | 205 | + HCCL_VM_ERROR("can not find device id:{:d}", deviceId); |
| 207 | return ACL_ERROR_INTERNAL_ERROR; | 206 | return ACL_ERROR_INTERNAL_ERROR; |
| 208 | } | 207 | } |
| 209 | 208 | ||
| @@ -211,7 +210,7 @@ aclError aclrtMalloc(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | |||
| 211 | 210 | ||
| 212 | auto virPtr = sim::DeviceMemoryManager::GetInstance().AllocVirMem(rankId, size); | 211 | auto virPtr = sim::DeviceMemoryManager::GetInstance().AllocVirMem(rankId, size); |
| 213 | if (virPtr == nullptr) { | 212 | if (virPtr == nullptr) { |
| 214 | - MEM_STUB_ERROR("can not alloc vir mem deviceId:{:d}, phyMemId:{:d}, size:{:d}", deviceId, rankId, size); | 213 | + HCCL_VM_ERROR("can not alloc vir mem deviceId:{:d}, phyMemId:{:d}, size:{:d}", deviceId, rankId, size); |
| 215 | return ACL_ERROR_INTERNAL_ERROR; | 214 | return ACL_ERROR_INTERNAL_ERROR; |
| 216 | } | 215 | } |
| 217 | 216 | ||
| @@ -219,7 +218,7 @@ aclError aclrtMalloc(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | |||
| 219 | auto hostPtr = sim::DeviceMemoryManager::GetInstance().AllocPhyMem(memName.c_str(), deviceId, size); | 218 | auto hostPtr = sim::DeviceMemoryManager::GetInstance().AllocPhyMem(memName.c_str(), deviceId, size); |
| 220 | if (hostPtr == nullptr) { | 219 | if (hostPtr == nullptr) { |
| 221 | sim::DeviceMemoryManager::GetInstance().FreeVirMem(rankId, devPtr); | 220 | sim::DeviceMemoryManager::GetInstance().FreeVirMem(rankId, devPtr); |
| 222 | - MEM_STUB_ERROR("can not alloc phy mem deviceId:{:d}, size:{:d}", deviceId, size); | 221 | + HCCL_VM_ERROR("can not alloc phy mem deviceId:{:d}, size:{:d}", deviceId, size); |
| 223 | return ACL_ERROR_INTERNAL_ERROR; | 222 | return ACL_ERROR_INTERNAL_ERROR; |
| 224 | } | 223 | } |
| 225 | // 大块引流到复用区时缓存池基址,供 IsInCommPool 判定(与主机侧共用同一基址)。 | 224 | // 大块引流到复用区时缓存池基址,供 IsInCommPool 判定(与主机侧共用同一基址)。 |
| @@ -249,20 +248,20 @@ aclError aclrtMalloc(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | |||
| 249 | *devPtr = virPtr; | 248 | *devPtr = virPtr; |
| 250 | // 缓存设备地址到host地址 | 249 | // 缓存设备地址到host地址 |
| 251 | sim::DeviceMemoryManager::GetInstance().MapDevPtrHostPtr(virPtr, hostPtr); | 250 | sim::DeviceMemoryManager::GetInstance().MapDevPtrHostPtr(virPtr, hostPtr); |
| 252 | - MEM_STUB_INFO("malloc dev addr:{:p}, memName:{}, size:{:d}", virPtr, memName, size); | 251 | + HCCL_VM_INFO("malloc dev addr:{:p}, memName:{}, size:{:d}", virPtr, memName, size); |
| 253 | return ACL_SUCCESS; | 252 | return ACL_SUCCESS; |
| 254 | } | 253 | } |
| 255 | 254 | ||
| 256 | aclError aclrtMallocAlign32(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | 255 | aclError aclrtMallocAlign32(void **devPtr, size_t size, aclrtMemMallocPolicy policy) |
| 257 | { | 256 | { |
| 258 | - MEM_STUB_INFO(".."); | 257 | + HCCL_VM_INFO(".."); |
| 259 | size_t realSize = ((size + 31) / 32) * 32; | 258 | size_t realSize = ((size + 31) / 32) * 32; |
| 260 | return aclrtMalloc(devPtr, realSize, policy); | 259 | return aclrtMalloc(devPtr, realSize, policy); |
| 261 | } | 260 | } |
| 262 | 261 | ||
| 263 | aclError aclrtMallocCached(void **devPtr, size_t size, aclrtMemMallocPolicy policy) | 262 | aclError aclrtMallocCached(void **devPtr, size_t size, aclrtMemMallocPolicy policy) |
| 264 | { | 263 | { |
| 265 | - MEM_STUB_INFO(".."); | 264 | + HCCL_VM_INFO(".."); |
| 266 | return aclrtMalloc(devPtr, size, policy); | 265 | return aclrtMalloc(devPtr, size, policy); |
| 267 | } | 266 | } |
| 268 | 267 | ||
| @@ -270,7 +269,7 @@ aclError aclrtMemFlush(void *devPtr, size_t size) | |||
| 270 | { | 269 | { |
| 271 | (void) devPtr; | 270 | (void) devPtr; |
| 272 | (void) size; | 271 | (void) size; |
| 273 | - MEM_STUB_INFO(".."); | 272 | + HCCL_VM_INFO(".."); |
| 274 | return ACL_SUCCESS; | 273 | return ACL_SUCCESS; |
| 275 | } | 274 | } |
| 276 | 275 | ||
| @@ -278,21 +277,21 @@ aclError aclrtMemInvalidate(void *devPtr, size_t size) | |||
| 278 | { | 277 | { |
| 279 | (void) devPtr; | 278 | (void) devPtr; |
| 280 | (void) size; | 279 | (void) size; |
| 281 | - MEM_STUB_INFO(".."); | 280 | + HCCL_VM_INFO(".."); |
| 282 | return ACL_SUCCESS; | 281 | return ACL_SUCCESS; |
| 283 | } | 282 | } |
| 284 | 283 | ||
| 285 | aclError aclrtMallocWithCfg(void **devPtr, size_t size, aclrtMemMallocPolicy policy, aclrtMallocConfig *cfg) | 284 | aclError aclrtMallocWithCfg(void **devPtr, size_t size, aclrtMemMallocPolicy policy, aclrtMallocConfig *cfg) |
| 286 | { | 285 | { |
| 287 | (void) cfg; | 286 | (void) cfg; |
| 288 | - MEM_STUB_INFO(".."); | 287 | + HCCL_VM_INFO(".."); |
| 289 | return aclrtMalloc(devPtr, size, policy); | 288 | return aclrtMalloc(devPtr, size, policy); |
| 290 | } | 289 | } |
| 291 | 290 | ||
| 292 | aclError aclrtMallocForTaskScheduler(void **devPtr, size_t size, aclrtMemMallocPolicy policy, aclrtMallocConfig *cfg) | 291 | aclError aclrtMallocForTaskScheduler(void **devPtr, size_t size, aclrtMemMallocPolicy policy, aclrtMallocConfig *cfg) |
| 293 | { | 292 | { |
| 294 | (void) cfg; | 293 | (void) cfg; |
| 295 | - MEM_STUB_INFO(".."); | 294 | + HCCL_VM_INFO(".."); |
| 296 | return aclrtMalloc(devPtr, size, policy); | 295 | return aclrtMalloc(devPtr, size, policy); |
| 297 | } | 296 | } |
| 298 | 297 | ||
| @@ -303,21 +302,21 @@ aclError aclrtFree(void* devPtr) | |||
| 303 | [startPtr](const sim::VirtualMemBlock &virMem) | 302 | [startPtr](const sim::VirtualMemBlock &virMem) |
| 304 | { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); | 303 | { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); |
| 305 | if (!virMemRes.second) { | 304 | if (!virMemRes.second) { |
| 306 | - MEM_STUB_ERROR("can not find this buff offset ptr:0x{:x}", startPtr); | 305 | + HCCL_VM_ERROR("can not find this buff offset ptr:0x{:x}", startPtr); |
| 307 | return ACL_ERROR_INTERNAL_ERROR; | 306 | return ACL_ERROR_INTERNAL_ERROR; |
| 308 | } | 307 | } |
| 309 | 308 | ||
| 310 | auto phyMemId = virMemRes.first.phy_mem_id; | 309 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 311 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 310 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 312 | if (!phyMemRes.has_value()) { | 311 | if (!phyMemRes.has_value()) { |
| 313 | - MEM_STUB_ERROR("can not find phy Mem id:{:d}", phyMemId); | 312 | + HCCL_VM_ERROR("can not find phy Mem id:{:d}", phyMemId); |
| 314 | return ACL_ERROR_INTERNAL_ERROR; | 313 | return ACL_ERROR_INTERNAL_ERROR; |
| 315 | }; | 314 | }; |
| 316 | 315 | ||
| 317 | auto deviceId = phyMemRes->device_id; | 316 | auto deviceId = phyMemRes->device_id; |
| 318 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); | 317 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); |
| 319 | if (!dev.has_value()) { | 318 | if (!dev.has_value()) { |
| 320 | - MEM_STUB_ERROR("can not find device id:{:d}", deviceId); | 319 | + HCCL_VM_ERROR("can not find device id:{:d}", deviceId); |
| 321 | return ACL_ERROR_INTERNAL_ERROR; | 320 | return ACL_ERROR_INTERNAL_ERROR; |
| 322 | } | 321 | } |
| 323 | 322 | ||
| @@ -342,7 +341,7 @@ aclError aclrtFree(void* devPtr) | |||
| 342 | RunnerDB::Update<sim::PhyMemBlock>(phyMemId, [](sim::PhyMemBlock &memBlock) { memBlock.is_freed = 1; }); | 341 | RunnerDB::Update<sim::PhyMemBlock>(phyMemId, [](sim::PhyMemBlock &memBlock) { memBlock.is_freed = 1; }); |
| 343 | RunnerDB::Update<sim::VirtualMemBlock>(virMemRes.first.id, [](sim::VirtualMemBlock &virtMemBlock) { virtMemBlock.is_freed = 1; }); | 342 | RunnerDB::Update<sim::VirtualMemBlock>(virMemRes.first.id, [](sim::VirtualMemBlock &virtMemBlock) { virtMemBlock.is_freed = 1; }); |
| 344 | 343 | ||
| 345 | - MEM_STUB_INFO("free dev addr:{:p}, memName:{}, size:{:d}", devPtr, phyMemRes->name, virMemRes.first.size); | 344 | + HCCL_VM_INFO("free dev addr:{:p}, memName:{}, size:{:d}", devPtr, phyMemRes->name, virMemRes.first.size); |
| 346 | return ACL_SUCCESS; | 345 | return ACL_SUCCESS; |
| 347 | } | 346 | } |
| 348 | 347 | ||
| @@ -352,20 +351,20 @@ aclError aclrtMemset(void *devPtr, size_t maxCount, int32_t value, size_t count) | |||
| 352 | uint32_t offset = 0; | 351 | uint32_t offset = 0; |
| 353 | if (!GetPtrNameByVirPtr(devPtr, offset, phyMem)) { | 352 | if (!GetPtrNameByVirPtr(devPtr, offset, phyMem)) { |
| 354 | memset(devPtr, value, count); | 353 | memset(devPtr, value, count); |
| 355 | - MEM_STUB_INFO("sys mem memset ptr:{:p}, maxCount: {:d}, value: {:d}, count: {:d}", devPtr, maxCount, value, count); | 354 | + HCCL_VM_INFO("sys mem memset ptr:{:p}, maxCount: {:d}, value: {:d}, count: {:d}", devPtr, maxCount, value, count); |
| 356 | return ACL_SUCCESS; | 355 | return ACL_SUCCESS; |
| 357 | } | 356 | } |
| 358 | 357 | ||
| 359 | char* hostPtr = (char*)sim::DeviceMemoryManager::GetInstance().AcquirePhyMem(phyMem.name, phyMem.device_id, phyMem.size); | 358 | char* hostPtr = (char*)sim::DeviceMemoryManager::GetInstance().AcquirePhyMem(phyMem.name, phyMem.device_id, phyMem.size); |
| 360 | if (hostPtr == nullptr) { | 359 | if (hostPtr == nullptr) { |
| 361 | - MEM_STUB_INFO("dev mem memset ptr:{:p} name:{} can not get host ptr", devPtr, phyMem.name); | 360 | + HCCL_VM_INFO("dev mem memset ptr:{:p} name:{} can not get host ptr", devPtr, phyMem.name); |
| 362 | return ACL_ERROR_INTERNAL_ERROR; | 361 | return ACL_ERROR_INTERNAL_ERROR; |
| 363 | } | 362 | } |
| 364 | 363 | ||
| 365 | // 目的设备块按 size 判是否在复用区,与申请一致,命中则跳过实际写入并按池释放。 | 364 | // 目的设备块按 size 判是否在复用区,与申请一致,命中则跳过实际写入并按池释放。 |
| 366 | bool pooled = sim::CommPoolPolicy::ShouldRedirect(phyMem.size, sim::IsCheckOnlyMode()); | 365 | bool pooled = sim::CommPoolPolicy::ShouldRedirect(phyMem.size, sim::IsCheckOnlyMode()); |
| 367 | if (pooled) { | 366 | if (pooled) { |
| 368 | - MEM_STUB_INFO("memset skip (pool) ptr:{:p}, size:{:d}, count:{:d}", devPtr, phyMem.size, count); | 367 | + HCCL_VM_INFO("memset skip (pool) ptr:{:p}, size:{:d}, count:{:d}", devPtr, phyMem.size, count); |
| 369 | } else { | 368 | } else { |
| 370 | memset(hostPtr + offset, value, count); | 369 | memset(hostPtr + offset, value, count); |
| 371 | } | 370 | } |
| @@ -374,14 +373,14 @@ aclError aclrtMemset(void *devPtr, size_t maxCount, int32_t value, size_t count) | |||
| 374 | } else { | 373 | } else { |
| 375 | sim::DeviceMemoryManager::GetInstance().ReleasePhyMem(phyMem.name, phyMem.device_id); | 374 | sim::DeviceMemoryManager::GetInstance().ReleasePhyMem(phyMem.name, phyMem.device_id); |
| 376 | } | 375 | } |
| 377 | - MEM_STUB_INFO("dev mem memset ptr:{:p}, memName:{}, count: {:d}", devPtr, phyMem.name, count); | 376 | + HCCL_VM_INFO("dev mem memset ptr:{:p}, memName:{}, count: {:d}", devPtr, phyMem.name, count); |
| 378 | return ACL_SUCCESS; | 377 | return ACL_SUCCESS; |
| 379 | } | 378 | } |
| 380 | 379 | ||
| 381 | aclError aclrtMemsetAsync(void *devPtr, size_t maxCount, int32_t value, size_t count, aclrtStream stream) | 380 | aclError aclrtMemsetAsync(void *devPtr, size_t maxCount, int32_t value, size_t count, aclrtStream stream) |
| 382 | { | 381 | { |
| 383 | (void) stream; | 382 | (void) stream; |
| 384 | - MEM_STUB_INFO(".."); | 383 | + HCCL_VM_INFO(".."); |
| 385 | // 当前先不生成任务,后续有需要再根据实际情况生成任务 | 384 | // 当前先不生成任务,后续有需要再根据实际情况生成任务 |
| 386 | return aclrtMemset(devPtr, maxCount, value, count); | 385 | return aclrtMemset(devPtr, maxCount, value, count); |
| 387 | } | 386 | } |
| @@ -403,37 +402,37 @@ aclError aclrtMemcpy(void *dst, size_t destMax, const void *src, size_t count, a | |||
| 403 | } else if (kind == ACL_MEMCPY_DEVICE_TO_DEVICE) { | 402 | } else if (kind == ACL_MEMCPY_DEVICE_TO_DEVICE) { |
| 404 | auto dstAddr = GetRealPtrByAddr(dst); | 403 | auto dstAddr = GetRealPtrByAddr(dst); |
| 405 | if (dstAddr == nullptr) { | 404 | if (dstAddr == nullptr) { |
| 406 | - MEM_STUB_WARN(" only support self D2D memcpy"); | 405 | + HCCL_VM_WARN(" only support self D2D memcpy"); |
| 407 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; | 406 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; |
| 408 | } | 407 | } |
| 409 | auto srcAddr = GetRealPtrByAddr(src); | 408 | auto srcAddr = GetRealPtrByAddr(src); |
| 410 | if (srcAddr == nullptr) { | 409 | if (srcAddr == nullptr) { |
| 411 | - MEM_STUB_WARN("only support self D2D memcpy"); | 410 | + HCCL_VM_WARN("only support self D2D memcpy"); |
| 412 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; | 411 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; |
| 413 | } | 412 | } |
| 414 | // 目的设备块按 size 判是否在复用区,与申请一致,命中则跳过实际拷贝。 | 413 | // 目的设备块按 size 判是否在复用区,与申请一致,命中则跳过实际拷贝。 |
| 415 | if (IsInCommPool(dstAddr)) { | 414 | if (IsInCommPool(dstAddr)) { |
| 416 | - MEM_STUB_INFO("memcpy D2D skip (dst big->pool) count:{:d}", count); | 415 | + HCCL_VM_INFO("memcpy D2D skip (dst big->pool) count:{:d}", count); |
| 417 | } else { | 416 | } else { |
| 418 | memcpy(dstAddr, srcAddr, count); | 417 | memcpy(dstAddr, srcAddr, count); |
| 419 | } | 418 | } |
| 420 | return ACL_SUCCESS; | 419 | return ACL_SUCCESS; |
| 421 | } else { | 420 | } else { |
| 422 | - MEM_STUB_INFO("src:{:p} to dst:{:p}, size:{:d} type: {:d} not support", src, dst, count, (int)kind); | 421 | + HCCL_VM_INFO("src:{:p} to dst:{:p}, size:{:d} type: {:d} not support", src, dst, count, (int)kind); |
| 423 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; | 422 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; |
| 424 | } | 423 | } |
| 425 | 424 | ||
| 426 | sim::PhyMemBlock phyMem{}; | 425 | sim::PhyMemBlock phyMem{}; |
| 427 | uint32_t offset = 0; | 426 | uint32_t offset = 0; |
| 428 | if (!GetPtrNameByVirPtr(devPtr, offset, phyMem)) { | 427 | if (!GetPtrNameByVirPtr(devPtr, offset, phyMem)) { |
| 429 | - MEM_STUB_INFO("dev mem memcpy ptr:{:p}, count: {:d}", devPtr, count); | 428 | + HCCL_VM_INFO("dev mem memcpy ptr:{:p}, count: {:d}", devPtr, count); |
| 430 | return ACL_ERROR_INTERNAL_ERROR; | 429 | return ACL_ERROR_INTERNAL_ERROR; |
| 431 | } | 430 | } |
| 432 | 431 | ||
| 433 | - MEM_STUB_INFO("host:{:p} dev:{:p} memName:{} size:{:d} type:{:d}", hostPtr, devPtr, phyMem.name, count, (int)kind); | 432 | + HCCL_VM_INFO("host:{:p} dev:{:p} memName:{} size:{:d} type:{:d}", hostPtr, devPtr, phyMem.name, count, (int)kind); |
| 434 | char* hostDevPtr = (char*)sim::DeviceMemoryManager::GetInstance().AcquirePhyMem(phyMem.name, phyMem.device_id, phyMem.size); | 433 | char* hostDevPtr = (char*)sim::DeviceMemoryManager::GetInstance().AcquirePhyMem(phyMem.name, phyMem.device_id, phyMem.size); |
| 435 | if (hostDevPtr == nullptr) { | 434 | if (hostDevPtr == nullptr) { |
| 436 | - MEM_STUB_INFO(" dev mem memcpy ptr:{:p} name:{} can not get host ptr", devPtr, phyMem.name); | 435 | + HCCL_VM_INFO(" dev mem memcpy ptr:{:p} name:{} can not get host ptr", devPtr, phyMem.name); |
| 437 | return ACL_ERROR_INTERNAL_ERROR; | 436 | return ACL_ERROR_INTERNAL_ERROR; |
| 438 | } | 437 | } |
| 439 | 438 | ||
| @@ -442,13 +441,13 @@ aclError aclrtMemcpy(void *dst, size_t destMax, const void *src, size_t count, a | |||
| 442 | bool devPooled = sim::CommPoolPolicy::ShouldRedirect(phyMem.size, sim::IsCheckOnlyMode()); | 441 | bool devPooled = sim::CommPoolPolicy::ShouldRedirect(phyMem.size, sim::IsCheckOnlyMode()); |
| 443 | if (kind == ACL_MEMCPY_HOST_TO_DEVICE) { | 442 | if (kind == ACL_MEMCPY_HOST_TO_DEVICE) { |
| 444 | if (devPooled) { | 443 | if (devPooled) { |
| 445 | - MEM_STUB_INFO("memcpy H2D skip (dst big->pool) count:{:d}", count); | 444 | + HCCL_VM_INFO("memcpy H2D skip (dst big->pool) count:{:d}", count); |
| 446 | } else { | 445 | } else { |
| 447 | memcpy(hostDevPtr + offset, hostPtr, count); | 446 | memcpy(hostDevPtr + offset, hostPtr, count); |
| 448 | } | 447 | } |
| 449 | } else if (kind == ACL_MEMCPY_DEVICE_TO_HOST) { | 448 | } else if (kind == ACL_MEMCPY_DEVICE_TO_HOST) { |
| 450 | if (IsInCommPool(hostPtr)) { | 449 | if (IsInCommPool(hostPtr)) { |
| 451 | - MEM_STUB_INFO("memcpy D2H skip (dst host pool) count:{:d}", count); | 450 | + HCCL_VM_INFO("memcpy D2H skip (dst host pool) count:{:d}", count); |
| 452 | } else { | 451 | } else { |
| 453 | memcpy(hostPtr, hostDevPtr + offset, count); | 452 | memcpy(hostPtr, hostDevPtr + offset, count); |
| 454 | } | 453 | } |
| @@ -493,7 +492,7 @@ aclError aclrtMemcpyAsync(void *dst, size_t destMax, const void *src, size_t cou | |||
| 493 | uint32_t index{0}; | 492 | uint32_t index{0}; |
| 494 | auto ret = InsertTaskToCollection(&taskMetaData, &index); | 493 | auto ret = InsertTaskToCollection(&taskMetaData, &index); |
| 495 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { | 494 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { |
| 496 | - MEM_STUB_ERROR("InsertTaskToCollection fail"); | 495 | + HCCL_VM_ERROR("InsertTaskToCollection fail"); |
| 497 | return ACL_ERROR_INTERNAL_ERROR; | 496 | return ACL_ERROR_INTERNAL_ERROR; |
| 498 | } | 497 | } |
| 499 | 498 | ||
| @@ -507,22 +506,22 @@ aclError aclrtMemcpyAsync(void *dst, size_t destMax, const void *src, size_t cou | |||
| 507 | auto taskId = RunnerDB::Add<sim::Task>(task); | 506 | auto taskId = RunnerDB::Add<sim::Task>(task); |
| 508 | return ACL_SUCCESS; | 507 | return ACL_SUCCESS; |
| 509 | } else { | 508 | } else { |
| 510 | - MEM_STUB_INFO("src:{:p} to dst:{:p}, size:{:d} type: {:d} not support", src, dst, count, (int)kind); | 509 | + HCCL_VM_INFO("src:{:p} to dst:{:p}, size:{:d} type: {:d} not support", src, dst, count, (int)kind); |
| 511 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; | 510 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; |
| 512 | } | 511 | } |
| 513 | 512 | ||
| 514 | sim::PhyMemBlock phyMem{}; | 513 | sim::PhyMemBlock phyMem{}; |
| 515 | uint32_t offset = 0; | 514 | uint32_t offset = 0; |
| 516 | if (!GetPtrNameByVirPtr(devPtr, offset, phyMem)) { | 515 | if (!GetPtrNameByVirPtr(devPtr, offset, phyMem)) { |
| 517 | - MEM_STUB_INFO("dev mem memcpy ptr:{:p}, count: {:d}", devPtr, count); | 516 | + HCCL_VM_INFO("dev mem memcpy ptr:{:p}, count: {:d}", devPtr, count); |
| 518 | return ACL_ERROR_INTERNAL_ERROR; | 517 | return ACL_ERROR_INTERNAL_ERROR; |
| 519 | } | 518 | } |
| 520 | 519 | ||
| 521 | - MEM_STUB_INFO("host:{:p} dev:{:p} memName:{} size:{:d} type:{:d}", hostPtr, devPtr, phyMem.name, count, (int)kind); | 520 | + HCCL_VM_INFO("host:{:p} dev:{:p} memName:{} size:{:d} type:{:d}", hostPtr, devPtr, phyMem.name, count, (int)kind); |
| 522 | 521 | ||
| 523 | char* hostDevPtr = (char*)sim::DeviceMemoryManager::GetInstance().AcquirePhyMem(phyMem.name, phyMem.device_id, phyMem.size); | 522 | char* hostDevPtr = (char*)sim::DeviceMemoryManager::GetInstance().AcquirePhyMem(phyMem.name, phyMem.device_id, phyMem.size); |
| 524 | if (hostDevPtr == nullptr) { | 523 | if (hostDevPtr == nullptr) { |
| 525 | - MEM_STUB_INFO("dev mem memcpy ptr:{:p} name:{} can not get host ptr", devPtr, phyMem.name); | 524 | + HCCL_VM_INFO("dev mem memcpy ptr:{:p} name:{} can not get host ptr", devPtr, phyMem.name); |
| 526 | return ACL_ERROR_INTERNAL_ERROR; | 525 | return ACL_ERROR_INTERNAL_ERROR; |
| 527 | } | 526 | } |
| 528 | 527 | ||
| @@ -531,13 +530,13 @@ aclError aclrtMemcpyAsync(void *dst, size_t destMax, const void *src, size_t cou | |||
| 531 | bool devPooled = sim::CommPoolPolicy::ShouldRedirect(phyMem.size, sim::IsCheckOnlyMode()); | 530 | bool devPooled = sim::CommPoolPolicy::ShouldRedirect(phyMem.size, sim::IsCheckOnlyMode()); |
| 532 | if (kind == ACL_MEMCPY_HOST_TO_DEVICE) { | 531 | if (kind == ACL_MEMCPY_HOST_TO_DEVICE) { |
| 533 | if (devPooled) { | 532 | if (devPooled) { |
| 534 | - MEM_STUB_INFO("memcpy H2D skip (dst big->pool) count:{:d}", count); | 533 | + HCCL_VM_INFO("memcpy H2D skip (dst big->pool) count:{:d}", count); |
| 535 | } else { | 534 | } else { |
| 536 | memcpy(hostDevPtr + offset, hostPtr, count); | 535 | memcpy(hostDevPtr + offset, hostPtr, count); |
| 537 | } | 536 | } |
| 538 | } else if (kind == ACL_MEMCPY_DEVICE_TO_HOST) { | 537 | } else if (kind == ACL_MEMCPY_DEVICE_TO_HOST) { |
| 539 | if (IsInCommPool(hostPtr)) { | 538 | if (IsInCommPool(hostPtr)) { |
| 540 | - MEM_STUB_INFO("memcpy D2H skip (dst host pool) count:{:d}", count); | 539 | + HCCL_VM_INFO("memcpy D2H skip (dst host pool) count:{:d}", count); |
| 541 | } else { | 540 | } else { |
| 542 | memcpy(hostPtr, hostDevPtr + offset, count); | 541 | memcpy(hostPtr, hostDevPtr + offset, count); |
| 543 | } | 542 | } |
| @@ -647,7 +646,7 @@ aclError aclrtMallocPhysical(aclrtDrvMemHandle *handle, size_t size, const aclrt | |||
| 647 | } | 646 | } |
| 648 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 647 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 649 | if (!currCtx.has_value()) { | 648 | if (!currCtx.has_value()) { |
| 650 | - MEM_STUB_ERROR("can not find current context:{:d}", runner.current_ctx_id); | 649 | + HCCL_VM_ERROR("can not find current context:{:d}", runner.current_ctx_id); |
| 651 | return ACL_ERROR_INTERNAL_ERROR; | 650 | return ACL_ERROR_INTERNAL_ERROR; |
| 652 | } | 651 | } |
| 653 | 652 | ||
| @@ -656,7 +655,7 @@ aclError aclrtMallocPhysical(aclrtDrvMemHandle *handle, size_t size, const aclrt | |||
| 656 | std::string memName = GenDevMemName(deviceId); | 655 | std::string memName = GenDevMemName(deviceId); |
| 657 | auto hostPtr = sim::DeviceMemoryManager::GetInstance().AllocPhyMem(memName.c_str(), deviceId, size); | 656 | auto hostPtr = sim::DeviceMemoryManager::GetInstance().AllocPhyMem(memName.c_str(), deviceId, size); |
| 658 | if (hostPtr == nullptr) { | 657 | if (hostPtr == nullptr) { |
| 659 | - MEM_STUB_ERROR("can not alloc phy mem deviceId:{:d}, size:{:d}", deviceId, size); | 658 | + HCCL_VM_ERROR("can not alloc phy mem deviceId:{:d}, size:{:d}", deviceId, size); |
| 660 | return ACL_ERROR_INTERNAL_ERROR; | 659 | return ACL_ERROR_INTERNAL_ERROR; |
| 661 | } | 660 | } |
| 662 | // 大块物理内存同样引流到复用区,缓存池基址,使其被 map 后的 memcpy/memset 能按地址判定。 | 661 | // 大块物理内存同样引流到复用区,缓存池基址,使其被 map 后的 memcpy/memset 能按地址判定。 |
| @@ -672,7 +671,7 @@ aclError aclrtMallocPhysical(aclrtDrvMemHandle *handle, size_t size, const aclrt | |||
| 672 | phyMem.ref_count = 1; | 671 | phyMem.ref_count = 1; |
| 673 | auto phyMemId = RunnerDB::Add<sim::PhyMemBlock>(phyMem); | 672 | auto phyMemId = RunnerDB::Add<sim::PhyMemBlock>(phyMem); |
| 674 | 673 | ||
| 675 | - MEM_STUB_INFO("malloc phy mem name:{}, size:{:d}, id:{:d}", memName, size, phyMemId); | 674 | + HCCL_VM_INFO("malloc phy mem name:{}, size:{:d}, id:{:d}", memName, size, phyMemId); |
| 676 | *handle = (aclrtDrvMemHandle)phyMemId; | 675 | *handle = (aclrtDrvMemHandle)phyMemId; |
| 677 | return ACL_SUCCESS; | 676 | return ACL_SUCCESS; |
| 678 | } | 677 | } |
| @@ -682,7 +681,7 @@ aclError aclrtFreePhysical(aclrtDrvMemHandle handle) | |||
| 682 | auto phyMemId = (uint64_t)(uintptr_t)handle; | 681 | auto phyMemId = (uint64_t)(uintptr_t)handle; |
| 683 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 682 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 684 | if (!phyMemRes.has_value()) { | 683 | if (!phyMemRes.has_value()) { |
| 685 | - MEM_STUB_ERROR("can not find phy Mem id:{:d}", phyMemId); | 684 | + HCCL_VM_ERROR("can not find phy Mem id:{:d}", phyMemId); |
| 686 | return ACL_ERROR_INTERNAL_ERROR; | 685 | return ACL_ERROR_INTERNAL_ERROR; |
| 687 | }; | 686 | }; |
| 688 | 687 | ||
| @@ -694,7 +693,7 @@ aclError aclrtFreePhysical(aclrtDrvMemHandle handle) | |||
| 694 | } | 693 | } |
| 695 | // 删除数据库记录 | 694 | // 删除数据库记录 |
| 696 | RunnerDB::Delete<sim::PhyMemBlock>(phyMemId); | 695 | RunnerDB::Delete<sim::PhyMemBlock>(phyMemId); |
| 697 | - MEM_STUB_INFO("free phy mem, id:{:d}", phyMemId); | 696 | + HCCL_VM_INFO("free phy mem, id:{:d}", phyMemId); |
| 698 | 697 | ||
| 699 | return ACL_SUCCESS; | 698 | return ACL_SUCCESS; |
| 700 | } | 699 | } |
| @@ -710,7 +709,7 @@ aclError aclrtReserveMemAddress(void **virPtr, size_t size, size_t alignment, vo | |||
| 710 | } | 709 | } |
| 711 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 710 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 712 | if (!currCtx.has_value()) { | 711 | if (!currCtx.has_value()) { |
| 713 | - MEM_STUB_ERROR("can not find current context:{:d}", runner.current_ctx_id); | 712 | + HCCL_VM_ERROR("can not find current context:{:d}", runner.current_ctx_id); |
| 714 | return ACL_ERROR_INTERNAL_ERROR; | 713 | return ACL_ERROR_INTERNAL_ERROR; |
| 715 | } | 714 | } |
| 716 | 715 | ||
| @@ -718,7 +717,7 @@ aclError aclrtReserveMemAddress(void **virPtr, size_t size, size_t alignment, vo | |||
| 718 | 717 | ||
| 719 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); | 718 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); |
| 720 | if (!dev.has_value()) { | 719 | if (!dev.has_value()) { |
| 721 | - MEM_STUB_ERROR("can not find device id:{:d}", deviceId); | 720 | + HCCL_VM_ERROR("can not find device id:{:d}", deviceId); |
| 722 | return ACL_ERROR_INTERNAL_ERROR; | 721 | return ACL_ERROR_INTERNAL_ERROR; |
| 723 | } | 722 | } |
| 724 | 723 | ||
| @@ -727,7 +726,7 @@ aclError aclrtReserveMemAddress(void **virPtr, size_t size, size_t alignment, vo | |||
| 727 | std::string memName = GenDevMemName(deviceId); | 726 | std::string memName = GenDevMemName(deviceId); |
| 728 | auto devPtr = sim::DeviceMemoryManager::GetInstance().AllocVirMem( devPhyId, size); | 727 | auto devPtr = sim::DeviceMemoryManager::GetInstance().AllocVirMem( devPhyId, size); |
| 729 | if (devPtr == nullptr) { | 728 | if (devPtr == nullptr) { |
| 730 | - MEM_STUB_ERROR("can not alloc vir mem deviceId:{:d}, phyMemId:{:d}, size:{:d}", deviceId, devPhyId, size); | 729 | + HCCL_VM_ERROR("can not alloc vir mem deviceId:{:d}, phyMemId:{:d}, size:{:d}", deviceId, devPhyId, size); |
| 731 | return ACL_ERROR_INTERNAL_ERROR; | 730 | return ACL_ERROR_INTERNAL_ERROR; |
| 732 | } | 731 | } |
| 733 | 732 | ||
| @@ -741,7 +740,7 @@ aclError aclrtReserveMemAddress(void **virPtr, size_t size, size_t alignment, vo | |||
| 741 | virMem.src_type = (uint8_t)sim::VIR_MEM_TYPE_DEV; | 740 | virMem.src_type = (uint8_t)sim::VIR_MEM_TYPE_DEV; |
| 742 | virMem.policy = 0; | 741 | virMem.policy = 0; |
| 743 | RunnerDB::Add<sim::VirtualMemBlock>(virMem); | 742 | RunnerDB::Add<sim::VirtualMemBlock>(virMem); |
| 744 | - MEM_STUB_INFO("reserve vir mem, size:{:d}, devPtr:{:p}", size, devPtr); | 743 | + HCCL_VM_INFO("reserve vir mem, size:{:d}, devPtr:{:p}", size, devPtr); |
| 745 | *virPtr = devPtr; | 744 | *virPtr = devPtr; |
| 746 | return ACL_SUCCESS; | 745 | return ACL_SUCCESS; |
| 747 | } | 746 | } |
| @@ -753,13 +752,13 @@ aclError aclrtReleaseMemAddress(void *virPtr) | |||
| 753 | [devPtr](const sim::VirtualMemBlock &virMem) | 752 | [devPtr](const sim::VirtualMemBlock &virMem) |
| 754 | { return virMem.start_ptr == devPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); | 753 | { return virMem.start_ptr == devPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); |
| 755 | if (!virMemRes.second) { | 754 | if (!virMemRes.second) { |
| 756 | - MEM_STUB_ERROR("can not find this buff offset ptr:{:p}", devPtr); | 755 | + HCCL_VM_ERROR("can not find this buff offset ptr:{:p}", devPtr); |
| 757 | return ACL_ERROR_INTERNAL_ERROR; | 756 | return ACL_ERROR_INTERNAL_ERROR; |
| 758 | } | 757 | } |
| 759 | auto ctxId = virMemRes.first.ctx_id; | 758 | auto ctxId = virMemRes.first.ctx_id; |
| 760 | auto currCtx = RunnerDB::GetById<sim::Context>(ctxId); | 759 | auto currCtx = RunnerDB::GetById<sim::Context>(ctxId); |
| 761 | if (!currCtx.has_value()) { | 760 | if (!currCtx.has_value()) { |
| 762 | - MEM_STUB_ERROR("can not find current context:{:d}", ctxId); | 761 | + HCCL_VM_ERROR("can not find current context:{:d}", ctxId); |
| 763 | return ACL_ERROR_INTERNAL_ERROR; | 762 | return ACL_ERROR_INTERNAL_ERROR; |
| 764 | } | 763 | } |
| 765 | 764 | ||
| @@ -767,7 +766,7 @@ aclError aclrtReleaseMemAddress(void *virPtr) | |||
| 767 | 766 | ||
| 768 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); | 767 | auto dev = RunnerDB::GetById<sim::Device>(deviceId); |
| 769 | if (!dev.has_value()) { | 768 | if (!dev.has_value()) { |
| 770 | - MEM_STUB_ERROR("can not find device id:{:d}", deviceId); | 769 | + HCCL_VM_ERROR("can not find device id:{:d}", deviceId); |
| 771 | return ACL_ERROR_INTERNAL_ERROR; | 770 | return ACL_ERROR_INTERNAL_ERROR; |
| 772 | } | 771 | } |
| 773 | 772 | ||
| @@ -775,7 +774,7 @@ aclError aclrtReleaseMemAddress(void *virPtr) | |||
| 775 | 774 | ||
| 776 | sim::DeviceMemoryManager::GetInstance().FreeVirMem(devPhyId, virPtr); | 775 | sim::DeviceMemoryManager::GetInstance().FreeVirMem(devPhyId, virPtr); |
| 777 | RunnerDB::Delete<sim::VirtualMemBlock>(virMemRes.first.id); | 776 | RunnerDB::Delete<sim::VirtualMemBlock>(virMemRes.first.id); |
| 778 | - MEM_STUB_INFO("release vir mem, devPtr:{:p}", virPtr); | 777 | + HCCL_VM_INFO("release vir mem, devPtr:{:p}", virPtr); |
| 779 | 778 | ||
| 780 | return ACL_SUCCESS; | 779 | return ACL_SUCCESS; |
| 781 | } | 780 | } |
| @@ -791,13 +790,13 @@ aclError aclrtMapMem(void *virPtr, size_t size, size_t offset, aclrtDrvMemHandle | |||
| 791 | [devPtr](const sim::VirtualMemBlock &virMem) | 790 | [devPtr](const sim::VirtualMemBlock &virMem) |
| 792 | { return virMem.start_ptr == devPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); | 791 | { return virMem.start_ptr == devPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); |
| 793 | if (!virMemRes.second) { | 792 | if (!virMemRes.second) { |
| 794 | - MEM_STUB_ERROR("can not find this buff offset ptr:{:p}", devPtr); | 793 | + HCCL_VM_ERROR("can not find this buff offset ptr:{:p}", devPtr); |
| 795 | return ACL_ERROR_INTERNAL_ERROR; | 794 | return ACL_ERROR_INTERNAL_ERROR; |
| 796 | } | 795 | } |
| 797 | 796 | ||
| 798 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 797 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 799 | if (!phyMemRes.has_value()) { | 798 | if (!phyMemRes.has_value()) { |
| 800 | - MEM_STUB_ERROR("can not find phy Mem id:{:d}", phyMemId); | 799 | + HCCL_VM_ERROR("can not find phy Mem id:{:d}", phyMemId); |
| 801 | return ACL_ERROR_INTERNAL_ERROR; | 800 | return ACL_ERROR_INTERNAL_ERROR; |
| 802 | }; | 801 | }; |
| 803 | 802 | ||
| @@ -816,14 +815,14 @@ aclError aclrtUnmapMem(void *virPtr) | |||
| 816 | [devPtr](const sim::VirtualMemBlock &virMem) | 815 | [devPtr](const sim::VirtualMemBlock &virMem) |
| 817 | { return virMem.start_ptr == devPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); | 816 | { return virMem.start_ptr == devPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV; }); |
| 818 | if (!virMemRes.second) { | 817 | if (!virMemRes.second) { |
| 819 | - MEM_STUB_ERROR("can not find this buff offset ptr:{:p}", devPtr); | 818 | + HCCL_VM_ERROR("can not find this buff offset ptr:{:p}", devPtr); |
| 820 | return ACL_ERROR_INTERNAL_ERROR; | 819 | return ACL_ERROR_INTERNAL_ERROR; |
| 821 | } | 820 | } |
| 822 | 821 | ||
| 823 | auto phyMemId = virMemRes.first.phy_mem_id; | 822 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 824 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 823 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 825 | if (!phyMemRes.has_value()) { | 824 | if (!phyMemRes.has_value()) { |
| 826 | - MEM_STUB_ERROR("can not find phy Mem id:{:d}", phyMemId); | 825 | + HCCL_VM_ERROR("can not find phy Mem id:{:d}", phyMemId); |
| 827 | return ACL_ERROR_INTERNAL_ERROR; | 826 | return ACL_ERROR_INTERNAL_ERROR; |
| 828 | }; | 827 | }; |
| 829 | 828 | ||
| @@ -842,7 +841,7 @@ aclError aclrtMemExportToShareableHandle(aclrtDrvMemHandle handle, aclrtMemHandl | |||
| 842 | 841 | ||
| 843 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 842 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 844 | if (!phyMemRes.has_value()) { | 843 | if (!phyMemRes.has_value()) { |
| 845 | - MEM_STUB_ERROR("cannot find phy Mem offset: {:d}", phyMemId); | 844 | + HCCL_VM_ERROR("cannot find phy Mem offset: {:d}", phyMemId); |
| 846 | return ACL_ERROR_INVALID_PARAM; | 845 | return ACL_ERROR_INVALID_PARAM; |
| 847 | } | 846 | } |
| 848 | 847 | ||
| @@ -884,7 +883,7 @@ aclError aclrtMemImportFromShareableHandle(uint64_t shareableHandle, int32_t dev | |||
| 884 | 883 | ||
| 885 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 884 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 886 | if (!phyMemRes.has_value()) { | 885 | if (!phyMemRes.has_value()) { |
| 887 | - MEM_STUB_ERROR("cannot find phy Mem offset: {:d}", phyMemId); | 886 | + HCCL_VM_ERROR("cannot find phy Mem offset: {:d}", phyMemId); |
| 888 | return ACL_ERROR_INVALID_PARAM; | 887 | return ACL_ERROR_INVALID_PARAM; |
| 889 | } | 888 | } |
| 890 | 889 | ||
| @@ -934,20 +933,20 @@ aclError aclrtPointerGetAttributes(const void *ptr, aclrtPtrAttributes *attribut | |||
| 934 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( | 933 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( |
| 935 | [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV;}); | 934 | [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV;}); |
| 936 | if (!virMemRes.second) { | 935 | if (!virMemRes.second) { |
| 937 | - MEM_STUB_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); | 936 | + HCCL_VM_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); |
| 938 | return ACL_ERROR_INVALID_PARAM; | 937 | return ACL_ERROR_INVALID_PARAM; |
| 939 | } | 938 | } |
| 940 | 939 | ||
| 941 | auto phyMemId = virMemRes.first.phy_mem_id; | 940 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 942 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 941 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 943 | if (!phyMemRes.has_value()) { | 942 | if (!phyMemRes.has_value()) { |
| 944 | - MEM_STUB_ERROR("cannot find phy Mem offset: {:d}", phyMemId); | 943 | + HCCL_VM_ERROR("cannot find phy Mem offset: {:d}", phyMemId); |
| 945 | return ACL_ERROR_INVALID_PARAM; | 944 | return ACL_ERROR_INVALID_PARAM; |
| 946 | } | 945 | } |
| 947 | 946 | ||
| 948 | auto devRes = RunnerDB::GetById<sim::Device>(phyMemRes->device_id); | 947 | auto devRes = RunnerDB::GetById<sim::Device>(phyMemRes->device_id); |
| 949 | if (!devRes.has_value()) { | 948 | if (!devRes.has_value()) { |
| 950 | - MEM_STUB_ERROR("cannot find phy Mem device: {:d}", phyMemRes->device_id); | 949 | + HCCL_VM_ERROR("cannot find phy Mem device: {:d}", phyMemRes->device_id); |
| 951 | return ACL_ERROR_INVALID_PARAM; | 950 | return ACL_ERROR_INVALID_PARAM; |
| 952 | } | 951 | } |
| 953 | 952 | ||
| @@ -964,7 +963,7 @@ aclError aclrtHostRegister(void *ptr, uint64_t size, aclrtHostRegisterType type, | |||
| 964 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( | 963 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( |
| 965 | [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_HOST;}); | 964 | [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_HOST;}); |
| 966 | if (!virMemRes.second) { | 965 | if (!virMemRes.second) { |
| 967 | - MEM_STUB_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); | 966 | + HCCL_VM_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); |
| 968 | return ACL_ERROR_INVALID_PARAM; | 967 | return ACL_ERROR_INVALID_PARAM; |
| 969 | } | 968 | } |
| 970 | *devPtr = ptr; | 969 | *devPtr = ptr; |
| @@ -977,7 +976,7 @@ aclError aclrtHostUnregister(void *ptr) | |||
| 977 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( | 976 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( |
| 978 | [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_HOST;}); | 977 | [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr && virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_HOST;}); |
| 979 | if (!virMemRes.second) { | 978 | if (!virMemRes.second) { |
| 980 | - MEM_STUB_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); | 979 | + HCCL_VM_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); |
| 981 | return ACL_ERROR_INVALID_PARAM; | 980 | return ACL_ERROR_INVALID_PARAM; |
| 982 | } | 981 | } |
| 983 | return ACL_SUCCESS; | 982 | return ACL_SUCCESS; |
| @@ -1008,15 +1007,15 @@ aclError aclrtIpcMemGetExportKey(void *devPtr, size_t size, char *key, size_t le | |||
| 1008 | (void) flags; | 1007 | (void) flags; |
| 1009 | uint64_t startPtr = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(devPtr)); | 1008 | uint64_t startPtr = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(devPtr)); |
| 1010 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( | 1009 | auto virMemRes = RunnerDB::GetOneByPred<sim::VirtualMemBlock>( |
| 1011 | - [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr;}); | 1010 | + [startPtr](const sim::VirtualMemBlock &virMem) { return virMem.start_ptr == startPtr; }); |
| 1012 | if (!virMemRes.second) { | 1011 | if (!virMemRes.second) { |
| 1013 | - MEM_STUB_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); | 1012 | + HCCL_VM_ERROR("can not find this buff offset ptr: 0x{:x}", startPtr); |
| 1014 | return ACL_ERROR_INVALID_PARAM; | 1013 | return ACL_ERROR_INVALID_PARAM; |
| 1015 | } | 1014 | } |
| 1016 | auto phyMemId = virMemRes.first.phy_mem_id; | 1015 | auto phyMemId = virMemRes.first.phy_mem_id; |
| 1017 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); | 1016 | auto phyMemRes = RunnerDB::GetById<sim::PhyMemBlock>(phyMemId); |
| 1018 | if (!phyMemRes.has_value()) { | 1017 | if (!phyMemRes.has_value()) { |
| 1019 | - MEM_STUB_ERROR("cannot find phy Mem offset: {:d}", phyMemId); | 1018 | + HCCL_VM_ERROR("cannot find phy Mem offset: {:d}", phyMemId); |
| 1020 | return ACL_ERROR_INVALID_PARAM; | 1019 | return ACL_ERROR_INVALID_PARAM; |
| 1021 | } | 1020 | } |
| 1022 | 1021 | ||
| @@ -1056,13 +1055,13 @@ aclError aclrtIpcMemImportByKey(void **devPtr, const char *key, uint64_t flags) | |||
| 1056 | uint64_t ipcRecordIdx = *(const uint64_t *)key; | 1055 | uint64_t ipcRecordIdx = *(const uint64_t *)key; |
| 1057 | auto recordRes = RunnerDB::GetById<sim::IpcMemRecord>(ipcRecordIdx); | 1056 | auto recordRes = RunnerDB::GetById<sim::IpcMemRecord>(ipcRecordIdx); |
| 1058 | if (!recordRes.has_value()) { | 1057 | if (!recordRes.has_value()) { |
| 1059 | - MEM_STUB_ERROR("cannot find ipc record: {:d}", ipcRecordIdx); | 1058 | + HCCL_VM_ERROR("cannot find ipc record: {:d}", ipcRecordIdx); |
| 1060 | return ACL_ERROR_INVALID_PARAM; | 1059 | return ACL_ERROR_INVALID_PARAM; |
| 1061 | } | 1060 | } |
| 1062 | 1061 | ||
| 1063 | auto virMemRes = RunnerDB::GetById<sim::VirtualMemBlock>(recordRes->vir_mem_id); | 1062 | auto virMemRes = RunnerDB::GetById<sim::VirtualMemBlock>(recordRes->vir_mem_id); |
| 1064 | if (!virMemRes.has_value()) { | 1063 | if (!virMemRes.has_value()) { |
| 1065 | - MEM_STUB_ERROR("cannot find vir mem: {:d}", recordRes->vir_mem_id); | 1064 | + HCCL_VM_ERROR("cannot find vir mem: {:d}", recordRes->vir_mem_id); |
| 1066 | return ACL_ERROR_INVALID_PARAM; | 1065 | return ACL_ERROR_INVALID_PARAM; |
| 1067 | } | 1066 | } |
| 1068 | *devPtr = (void *)virMemRes->start_ptr; | 1067 | *devPtr = (void *)virMemRes->start_ptr; |
| @@ -1075,7 +1074,7 @@ aclError aclrtIpcMemClose(const char *key) | |||
| 1075 | uint64_t ipcRecordIdx = *reinterpret_cast<const uint64_t*>(key); | 1074 | uint64_t ipcRecordIdx = *reinterpret_cast<const uint64_t*>(key); |
| 1076 | auto recordRes = RunnerDB::GetById<sim::IpcMemRecord>(ipcRecordIdx); | 1075 | auto recordRes = RunnerDB::GetById<sim::IpcMemRecord>(ipcRecordIdx); |
| 1077 | if (!recordRes.has_value()) { | 1076 | if (!recordRes.has_value()) { |
| 1078 | - MEM_STUB_ERROR("cannot find ipc record: {:d}", ipcRecordIdx); | 1077 | + HCCL_VM_ERROR("cannot find ipc record: {:d}", ipcRecordIdx); |
| 1079 | return ACL_ERROR_INVALID_PARAM; | 1078 | return ACL_ERROR_INVALID_PARAM; |
| 1080 | } | 1079 | } |
| 1081 | 1080 | ||
| @@ -1156,14 +1155,14 @@ rtError_t rtMalloc(void **devPtr, uint64_t size, rtMemType_t type, const uint16_ | |||
| 1156 | (void) type; | 1155 | (void) type; |
| 1157 | (void) moduleId; | 1156 | (void) moduleId; |
| 1158 | int ret = aclrtMalloc(devPtr, size, aclrtMemMallocPolicy::ACL_MEM_MALLOC_HUGE_FIRST); | 1157 | int ret = aclrtMalloc(devPtr, size, aclrtMemMallocPolicy::ACL_MEM_MALLOC_HUGE_FIRST); |
| 1159 | - MEM_STUB_INFO("enter into stub rtMalloc.... addr:{:p}", *devPtr); | 1158 | + HCCL_VM_INFO("enter into stub rtMalloc.... addr:{:p}", *devPtr); |
| 1160 | return ret; | 1159 | return ret; |
| 1161 | } | 1160 | } |
| 1162 | 1161 | ||
| 1163 | aclError rtFree(void* devPtr) | 1162 | aclError rtFree(void* devPtr) |
| 1164 | { | 1163 | { |
| 1165 | (void) devPtr; | 1164 | (void) devPtr; |
| 1166 | - MEM_STUB_ERROR("enter into rtFree not support"); | 1165 | + HCCL_VM_ERROR("enter into rtFree not support"); |
| 1167 | return ACL_ERROR_INTERNAL_ERROR; | 1166 | return ACL_ERROR_INTERNAL_ERROR; |
| 1168 | } | 1167 | } |
| 1169 | 1168 | ||
| @@ -1190,7 +1189,7 @@ aclError aclrtReduceAsync(void *dst, const void *src, uint64_t count, aclrtReduc | |||
| 1190 | 1189 | ||
| 1191 | auto iter = rtDataType2CheckerDataType.find(type); | 1190 | auto iter = rtDataType2CheckerDataType.find(type); |
| 1192 | if (iter == rtDataType2CheckerDataType.end()) { | 1191 | if (iter == rtDataType2CheckerDataType.end()) { |
| 1193 | - MEM_STUB_ERROR("not support data type: {}", static_cast<uint32_t>(type)); | 1192 | + HCCL_VM_ERROR("not support data type: {}", static_cast<uint32_t>(type)); |
| 1194 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; | 1193 | return ACL_ERROR_RT_FEATURE_NOT_SUPPORT; |
| 1195 | } | 1194 | } |
| 1196 | 1195 | ||
| @@ -1218,10 +1217,10 @@ aclError aclrtReduceAsync(void *dst, const void *src, uint64_t count, aclrtReduc | |||
| 1218 | taskMetaData.taskData.reduce.reduceOp = static_cast<uint8_t>(kind); | 1217 | taskMetaData.taskData.reduce.reduceOp = static_cast<uint8_t>(kind); |
| 1219 | 1218 | ||
| 1220 | uint32_t index{0}; | 1219 | uint32_t index{0}; |
| 1221 | - MEM_STUB_DEBUG("Get reduce task, streamId={:d}", streamId); | 1220 | + HCCL_VM_DEBUG("Get reduce task, streamId={:d}", streamId); |
| 1222 | auto ret = InsertTaskToCollection(&taskMetaData, &index); | 1221 | auto ret = InsertTaskToCollection(&taskMetaData, &index); |
| 1223 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { | 1222 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { |
| 1224 | - MEM_STUB_ERROR("InsertTaskToCollection fail"); | 1223 | + HCCL_VM_ERROR("InsertTaskToCollection fail"); |
| 1225 | return ACL_ERROR_INTERNAL_ERROR; | 1224 | return ACL_ERROR_INTERNAL_ERROR; |
| 1226 | } | 1225 | } |
| 1227 | 1226 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -27,11 +30,6 @@ | |||
| 27 | 30 | ||
| 28 | 31 | ||
| 29 | 32 | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| 33 | - | ||
| 34 | - | ||
| 35 | 33 | ||
| 36 | 34 | ||
| 37 | extern "C" { | 35 | extern "C" { |
| @@ -43,16 +41,16 @@ HcclResult hrtGetDeviceType(DevType &devType) | |||
| 43 | auto device = RunnerDB::GetById<sim::Device>(devKey); | 41 | auto device = RunnerDB::GetById<sim::Device>(devKey); |
| 44 | if (!device.has_value()) { | 42 | if (!device.has_value()) { |
| 45 | // not find | 43 | // not find |
| 46 | - NEW_STUB_ERROR("current device type not found: {:d}", devKey); | 44 | + HCCL_VM_ERROR("current device type not found: {:d}", devKey); |
| 47 | return HCCL_E_NOT_FOUND; | 45 | return HCCL_E_NOT_FOUND; |
| 48 | } | 46 | } |
| 49 | if (strcmp(device->soc_version, "Ascend950") == 0) { | 47 | if (strcmp(device->soc_version, "Ascend950") == 0) { |
| 50 | devType = DevType::DEV_TYPE_950; | 48 | devType = DevType::DEV_TYPE_950; |
| 51 | } else { | 49 | } else { |
| 52 | - NEW_STUB_ERROR("not support device soc version: {:s}", device->soc_version); | 50 | + HCCL_VM_ERROR("not support device soc version: {:s}", device->soc_version); |
| 53 | return HCCL_E_NOT_SUPPORT; | 51 | return HCCL_E_NOT_SUPPORT; |
| 54 | } | 52 | } |
| 55 | - NEW_STUB_TRACE("Get current device type: {}", static_cast<int>(devType)); | 53 | + HCCL_VM_TRACE("Get current device type: {}", static_cast<int>(devType)); |
| 56 | return HCCL_SUCCESS; | 54 | return HCCL_SUCCESS; |
| 57 | } | 55 | } |
| 58 | 56 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -23,11 +26,6 @@ | |||
| 23 | 26 | ||
| 24 | 27 | ||
| 25 | 28 | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | - | ||
| 31 | 29 | ||
| 32 | 30 | ||
| 33 | extern "C" { | 31 | extern "C" { |
| @@ -38,21 +36,21 @@ void PrintTaskMetaData(const HcclTaskMetaData &taskMeta) | |||
| 38 | pid_t pid = getpid(); | 36 | pid_t pid = getpid(); |
| 39 | switch (taskMeta.taskType) { | 37 | switch (taskMeta.taskType) { |
| 40 | case HccLTaskMetaType::MEM_CPY: | 38 | case HccLTaskMetaType::MEM_CPY: |
| 41 | - NOTIFY_STUB_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[MEM_CPY], srcOffset[{}], dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}]\n", | 39 | + HCCL_VM_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[MEM_CPY], srcOffset[{}], dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}]", |
| 42 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.transMem.srcOffset, taskMeta.taskData.transMem.dstOffset, taskMeta.taskData.transMem.len, | 40 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.transMem.srcOffset, taskMeta.taskData.transMem.dstOffset, taskMeta.taskData.transMem.len, |
| 43 | taskMeta.taskData.transMem.srcRankId, taskMeta.taskData.transMem.dstRankId); | 41 | taskMeta.taskData.transMem.srcRankId, taskMeta.taskData.transMem.dstRankId); |
| 44 | break; | 42 | break; |
| 45 | - case HccLTaskMetaType::REDUCE: | 43 | + case HccLTaskMetaType::REDUCE: |
| 46 | - NOTIFY_STUB_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[REDUCE], srcOffset[{}], dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}], reduceOp[{}], dataType[{}]\n", | 44 | + HCCL_VM_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[REDUCE], srcOffset[{}], dstOffset[{}], len[{}], srcRankId[{}], dstRankId[{}], reduceOp[{}], dataType[{}]", |
| 47 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.reduce.srcOffset, taskMeta.taskData.reduce.dstOffset, taskMeta.taskData.reduce.dataCount, | 45 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.reduce.srcOffset, taskMeta.taskData.reduce.dstOffset, taskMeta.taskData.reduce.dataCount, |
| 48 | taskMeta.taskData.reduce.srcRankId, taskMeta.taskData.reduce.dstRankId, taskMeta.taskData.reduce.reduceOp, taskMeta.taskData.reduce.dataType); | 46 | taskMeta.taskData.reduce.srcRankId, taskMeta.taskData.reduce.dstRankId, taskMeta.taskData.reduce.reduceOp, taskMeta.taskData.reduce.dataType); |
| 49 | break; | 47 | break; |
| 50 | case HccLTaskMetaType::NOTIFY_WAIT: | 48 | case HccLTaskMetaType::NOTIFY_WAIT: |
| 51 | - NOTIFY_STUB_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[NOTIFY_WAIT], notifyId[{}], srcRankId[{}], dstRankId[{}]\n", | 49 | + HCCL_VM_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[NOTIFY_WAIT], notifyId[{}], srcRankId[{}], dstRankId[{}]", |
| 52 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); | 50 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); |
| 53 | break; | 51 | break; |
| 54 | case HccLTaskMetaType::NOTIFY_RECORD: | 52 | case HccLTaskMetaType::NOTIFY_RECORD: |
| 55 | - NOTIFY_STUB_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[NOTIFY_RECORD], notifyId[{}], srcRankId[{}], dstRankId[{}]\n", | 53 | + HCCL_VM_INFO("pid[{}]: rankId[{}], streamId[{}], taskType[NOTIFY_RECORD], notifyId[{}], srcRankId[{}], dstRankId[{}]", |
| 56 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); | 54 | pid, taskMeta.rankId, taskMeta.streamId, taskMeta.taskData.notify.notifyId, taskMeta.taskData.notify.srcRankId, taskMeta.taskData.notify.dstRankId); |
| 57 | break; | 55 | break; |
| 58 | default: | 56 | default: |
| @@ -68,7 +66,7 @@ aclError aclrtCreateNotify(aclrtNotify *notify, uint64_t flag) | |||
| 68 | return ACL_ERROR_INVALID_PARAM; | 66 | return ACL_ERROR_INVALID_PARAM; |
| 69 | } | 67 | } |
| 70 | if (runner.current_ctx_id == 0) { | 68 | if (runner.current_ctx_id == 0) { |
| 71 | - HCCL_VM_ERROR("[aclrtCreateNotify] invalid param"); | 69 | + HCCL_VM_ERROR("invalid param"); |
| 72 | return ACL_ERROR_INVALID_PARAM; | 70 | return ACL_ERROR_INVALID_PARAM; |
| 73 | } | 71 | } |
| 74 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 72 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| @@ -84,14 +82,14 @@ aclError aclrtCreateNotify(aclrtNotify *notify, uint64_t flag) | |||
| 84 | auto res = RunnerDB::Add<sim::Notify>(tmp); | 82 | auto res = RunnerDB::Add<sim::Notify>(tmp); |
| 85 | 83 | ||
| 86 | *notify = (aclrtNotify)res; | 84 | *notify = (aclrtNotify)res; |
| 87 | - HCCL_VM_DEBUG("[aclstub][aclrtCreateNotify] notify: {:d}", res); | 85 | + HCCL_VM_DEBUG("notify: {:d}", res); |
| 88 | return ACL_SUCCESS; | 86 | return ACL_SUCCESS; |
| 89 | } | 87 | } |
| 90 | 88 | ||
| 91 | aclError aclrtDestroyNotify(aclrtNotify notify) | 89 | aclError aclrtDestroyNotify(aclrtNotify notify) |
| 92 | { | 90 | { |
| 93 | uint64_t notifyId = (uint32_t)(uintptr_t)notify; | 91 | uint64_t notifyId = (uint32_t)(uintptr_t)notify; |
| 94 | - HCCL_VM_DEBUG("[aclstub][aclrtDestroyNotify] notify: {:d}", notifyId); | 92 | + HCCL_VM_DEBUG("notify: {:d}", notifyId); |
| 95 | RunnerDB::Delete<sim::Notify>(notifyId); | 93 | RunnerDB::Delete<sim::Notify>(notifyId); |
| 96 | return ACL_SUCCESS; | 94 | return ACL_SUCCESS; |
| 97 | } | 95 | } |
| @@ -99,7 +97,7 @@ aclError aclrtDestroyNotify(aclrtNotify notify) | |||
| 99 | aclError aclrtGetNotifyId(aclrtNotify notify, uint32_t *notifyId) | 97 | aclError aclrtGetNotifyId(aclrtNotify notify, uint32_t *notifyId) |
| 100 | { | 98 | { |
| 101 | *notifyId = (uint32_t)(uintptr_t)notify; | 99 | *notifyId = (uint32_t)(uintptr_t)notify; |
| 102 | - HCCL_VM_DEBUG("[aclstub][aclrtGetNotifyId] notifyId: {:d}", *notifyId); | 100 | + HCCL_VM_DEBUG("notifyId: {:d}", *notifyId); |
| 103 | return ACL_SUCCESS; | 101 | return ACL_SUCCESS; |
| 104 | } | 102 | } |
| 105 | 103 | ||
| @@ -122,10 +120,10 @@ aclError aclrtRecordNotify(aclrtNotify notify, aclrtStream stream) | |||
| 122 | PrintTaskMetaData(taskMetaData); | 120 | PrintTaskMetaData(taskMetaData); |
| 123 | 121 | ||
| 124 | uint32_t index{0}; | 122 | uint32_t index{0}; |
| 125 | - HCCL_VM_DEBUG("[aclstub][aclrtRecordNotify] Get notify task, id={:d}, streamId={:d}", notifyId, streamId); | 123 | + HCCL_VM_DEBUG("Get notify task, id={:d}, streamId={:d}", notifyId, streamId); |
| 126 | auto ret = InsertTaskToCollection(&taskMetaData, &index); | 124 | auto ret = InsertTaskToCollection(&taskMetaData, &index); |
| 127 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { | 125 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { |
| 128 | - HCCL_VM_ERROR("[aclstub] InsertTaskToCollection fail"); | 126 | + HCCL_VM_ERROR("InsertTaskToCollection fail"); |
| 129 | return ACL_ERROR_INTERNAL_ERROR; | 127 | return ACL_ERROR_INTERNAL_ERROR; |
| 130 | } | 128 | } |
| 131 | 129 | ||
| @@ -159,10 +157,10 @@ aclError aclrtWaitAndResetNotify(aclrtNotify notify, aclrtStream stream, uint32_ | |||
| 159 | PrintTaskMetaData(taskMetaData); | 157 | PrintTaskMetaData(taskMetaData); |
| 160 | 158 | ||
| 161 | uint32_t index{0}; | 159 | uint32_t index{0}; |
| 162 | - HCCL_VM_DEBUG("[aclstub][aclrtWaitAndResetNotify] Get notify task, id={:d}, streamId={:d}", notifyId, streamId); | 160 | + HCCL_VM_DEBUG("Get notify task, id={:d}, streamId={:d}", notifyId, streamId); |
| 163 | auto ret = InsertTaskToCollection(&taskMetaData, &index); | 161 | auto ret = InsertTaskToCollection(&taskMetaData, &index); |
| 164 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { | 162 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { |
| 165 | - HCCL_VM_ERROR("[aclstub] InsertTaskToCollection fail"); | 163 | + HCCL_VM_ERROR("InsertTaskToCollection fail"); |
| 166 | return ACL_ERROR_INTERNAL_ERROR; | 164 | return ACL_ERROR_INTERNAL_ERROR; |
| 167 | } | 165 | } |
| 168 | 166 | ||
| @@ -199,14 +197,14 @@ aclError aclrtNotifyGetExportKey(aclrtNotify notify, char *key, size_t len, uint | |||
| 199 | memcpy(key, notifyKeyStr.data(), notifyKeyStr.length()); | 197 | memcpy(key, notifyKeyStr.data(), notifyKeyStr.length()); |
| 200 | tmp.create_pid = getpid(); | 198 | tmp.create_pid = getpid(); |
| 201 | auto res = RunnerDB::Add<sim::IpcNotify>(tmp); | 199 | auto res = RunnerDB::Add<sim::IpcNotify>(tmp); |
| 202 | - HCCL_VM_DEBUG("[aclstub][aclrtNotifyGetExportKey] notify: {:d}", notifyId); | 200 | + HCCL_VM_DEBUG("notify: {:d}", notifyId); |
| 203 | return ACL_SUCCESS; | 201 | return ACL_SUCCESS; |
| 204 | } | 202 | } |
| 205 | 203 | ||
| 206 | aclError aclrtNotifySetImportPid(aclrtNotify notify, int32_t *pid, size_t num) | 204 | aclError aclrtNotifySetImportPid(aclrtNotify notify, int32_t *pid, size_t num) |
| 207 | { | 205 | { |
| 208 | uint64_t notifyId = (uint32_t)(uintptr_t)notify; | 206 | uint64_t notifyId = (uint32_t)(uintptr_t)notify; |
| 209 | - HCCL_VM_DEBUG("[aclstub][aclrtNotifySetImportPid] notify: {:d}", notifyId); | 207 | + HCCL_VM_DEBUG("notify: {:d}", notifyId); |
| 210 | 208 | ||
| 211 | auto ipcNotify = RunnerDB::GetOneByPred<sim::IpcNotify>([notifyId](const sim::IpcNotify& ipc) { | 209 | auto ipcNotify = RunnerDB::GetOneByPred<sim::IpcNotify>([notifyId](const sim::IpcNotify& ipc) { |
| 212 | return ipc.notify_id == notifyId; | 210 | return ipc.notify_id == notifyId; |
| @@ -237,7 +235,7 @@ aclError aclrtNotifyImportByKey(aclrtNotify *notify, const char *key, uint64_t f | |||
| 237 | return ACL_ERROR_INVALID_PARAM; | 235 | return ACL_ERROR_INVALID_PARAM; |
| 238 | } | 236 | } |
| 239 | *notify = (aclrtNotify)ipcNotify.first.notify_id; | 237 | *notify = (aclrtNotify)ipcNotify.first.notify_id; |
| 240 | - HCCL_VM_DEBUG("[aclstub][aclrtNotifyImportByKey] notify: {:d}", ipcNotify.first.notify_id); | 238 | + HCCL_VM_DEBUG("notify: {:d}", ipcNotify.first.notify_id); |
| 241 | return ACL_SUCCESS; | 239 | return ACL_SUCCESS; |
| 242 | } | 240 | } |
| 243 | 241 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -22,11 +25,6 @@ | |||
| 22 | 25 | ||
| 23 | 26 | ||
| 24 | 27 | ||
| 25 | - | ||
| 26 | - | ||
| 27 | - | ||
| 28 | - | ||
| 29 | - | ||
| 30 | 28 | ||
| 31 | 29 | ||
| 32 | extern "C" { | 30 | extern "C" { |
| @@ -36,7 +34,7 @@ aclError aclrtSetSysParamOpt(aclSysParamOpt opt, int64_t value) | |||
| 36 | { | 34 | { |
| 37 | (void) opt; | 35 | (void) opt; |
| 38 | (void) value; | 36 | (void) value; |
| 39 | - CONFIG_STUB_WARN("not support"); | 37 | + HCCL_VM_WARN("not support"); |
| 40 | return ACL_SUCCESS; | 38 | return ACL_SUCCESS; |
| 41 | } | 39 | } |
| 42 | 40 | ||
| @@ -44,7 +42,7 @@ aclError aclrtGetSysParamOpt(aclSysParamOpt opt, int64_t *value) | |||
| 44 | { | 42 | { |
| 45 | (void) opt; | 43 | (void) opt; |
| 46 | (void) value; | 44 | (void) value; |
| 47 | - CONFIG_STUB_WARN("not support"); | 45 | + HCCL_VM_WARN("not support"); |
| 48 | return ACL_SUCCESS; | 46 | return ACL_SUCCESS; |
| 49 | } | 47 | } |
| 50 | 48 | ||
| @@ -53,7 +51,7 @@ aclError aclrtGetDeviceResLimit(int32_t deviceId, aclrtDevResLimitType type, uin | |||
| 53 | (void) deviceId; | 51 | (void) deviceId; |
| 54 | (void) type; | 52 | (void) type; |
| 55 | (void) value; | 53 | (void) value; |
| 56 | - CONFIG_STUB_WARN("not support"); | 54 | + HCCL_VM_WARN("not support"); |
| 57 | return ACL_SUCCESS; | 55 | return ACL_SUCCESS; |
| 58 | } | 56 | } |
| 59 | 57 | ||
| @@ -62,14 +60,14 @@ aclError aclrtSetDeviceResLimit(int32_t deviceId, aclrtDevResLimitType type, uin | |||
| 62 | (void) deviceId; | 60 | (void) deviceId; |
| 63 | (void) type; | 61 | (void) type; |
| 64 | (void) value; | 62 | (void) value; |
| 65 | - CONFIG_STUB_WARN("not support"); | 63 | + HCCL_VM_WARN("not support"); |
| 66 | return ACL_SUCCESS; | 64 | return ACL_SUCCESS; |
| 67 | } | 65 | } |
| 68 | 66 | ||
| 69 | aclError aclrtResetDeviceResLimit(int32_t deviceId) | 67 | aclError aclrtResetDeviceResLimit(int32_t deviceId) |
| 70 | { | 68 | { |
| 71 | (void) deviceId; | 69 | (void) deviceId; |
| 72 | - CONFIG_STUB_WARN("not support"); | 70 | + HCCL_VM_WARN("not support"); |
| 73 | return ACL_SUCCESS; | 71 | return ACL_SUCCESS; |
| 74 | } | 72 | } |
| 75 | 73 | ||
| @@ -78,7 +76,7 @@ aclError aclrtGetStreamResLimit(aclrtStream stream, aclrtDevResLimitType type, u | |||
| 78 | (void) stream; | 76 | (void) stream; |
| 79 | (void) type; | 77 | (void) type; |
| 80 | (void) value; | 78 | (void) value; |
| 81 | - CONFIG_STUB_WARN("not support"); | 79 | + HCCL_VM_WARN("not support"); |
| 82 | return ACL_SUCCESS; | 80 | return ACL_SUCCESS; |
| 83 | } | 81 | } |
| 84 | 82 | ||
| @@ -87,28 +85,28 @@ aclError aclrtSetStreamResLimit(aclrtStream stream, aclrtDevResLimitType type, u | |||
| 87 | (void) stream; | 85 | (void) stream; |
| 88 | (void) type; | 86 | (void) type; |
| 89 | (void) value; | 87 | (void) value; |
| 90 | - CONFIG_STUB_WARN("not support"); | 88 | + HCCL_VM_WARN("not support"); |
| 91 | return ACL_SUCCESS; | 89 | return ACL_SUCCESS; |
| 92 | } | 90 | } |
| 93 | 91 | ||
| 94 | aclError aclrtResetStreamResLimit(aclrtStream stream) | 92 | aclError aclrtResetStreamResLimit(aclrtStream stream) |
| 95 | { | 93 | { |
| 96 | (void) stream; | 94 | (void) stream; |
| 97 | - CONFIG_STUB_WARN("not support"); | 95 | + HCCL_VM_WARN("not support"); |
| 98 | return ACL_SUCCESS; | 96 | return ACL_SUCCESS; |
| 99 | } | 97 | } |
| 100 | 98 | ||
| 101 | aclError aclrtUseStreamResInCurrentThread(aclrtStream stream) | 99 | aclError aclrtUseStreamResInCurrentThread(aclrtStream stream) |
| 102 | { | 100 | { |
| 103 | (void) stream; | 101 | (void) stream; |
| 104 | - CONFIG_STUB_WARN("not support"); | 102 | + HCCL_VM_WARN("not support"); |
| 105 | return ACL_SUCCESS; | 103 | return ACL_SUCCESS; |
| 106 | } | 104 | } |
| 107 | 105 | ||
| 108 | aclError aclrtUnuseStreamResInCurrentThread(aclrtStream stream) | 106 | aclError aclrtUnuseStreamResInCurrentThread(aclrtStream stream) |
| 109 | { | 107 | { |
| 110 | (void) stream; | 108 | (void) stream; |
| 111 | - CONFIG_STUB_WARN("not support"); | 109 | + HCCL_VM_WARN("not support"); |
| 112 | return ACL_SUCCESS; | 110 | return ACL_SUCCESS; |
| 113 | } | 111 | } |
| 114 | 112 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -28,11 +31,6 @@ | |||
| 28 | 31 | ||
| 29 | static const uint32_t WAIT_FALG_TIMEOUT = 600; | 32 | static const uint32_t WAIT_FALG_TIMEOUT = 600; |
| 30 | 33 | ||
| 31 | - | ||
| 32 | - | ||
| 33 | - | ||
| 34 | - | ||
| 35 | - | ||
| 36 | 34 | ||
| 37 | 35 | ||
| 38 | extern "C" { | 36 | extern "C" { |
| @@ -47,12 +45,12 @@ aclError aclrtCreateStreamWithConfig(aclrtStream *stream, uint32_t priority, uin | |||
| 47 | return ACL_ERROR_INVALID_PARAM; | 45 | return ACL_ERROR_INVALID_PARAM; |
| 48 | } | 46 | } |
| 49 | if (runner.current_ctx_id == 0) { | 47 | if (runner.current_ctx_id == 0) { |
| 50 | - STREAM_STUB_ERROR("invalid param"); | 48 | + HCCL_VM_ERROR("invalid param"); |
| 51 | return ACL_ERROR_INVALID_PARAM; | 49 | return ACL_ERROR_INVALID_PARAM; |
| 52 | } | 50 | } |
| 53 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 51 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 54 | if (!currCtx.has_value()) { | 52 | if (!currCtx.has_value()) { |
| 55 | - STREAM_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 53 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 56 | return ACL_ERROR_INVALID_PARAM; | 54 | return ACL_ERROR_INVALID_PARAM; |
| 57 | } | 55 | } |
| 58 | 56 | ||
| @@ -66,7 +64,7 @@ aclError aclrtCreateStreamWithConfig(aclrtStream *stream, uint32_t priority, uin | |||
| 66 | auto res = RunnerDB::Add<sim::Stream>(streamTmp); | 64 | auto res = RunnerDB::Add<sim::Stream>(streamTmp); |
| 67 | 65 | ||
| 68 | *stream = (aclrtStream)res; | 66 | *stream = (aclrtStream)res; |
| 69 | - STREAM_STUB_DEBUG("id:{:d}", res); | 67 | + HCCL_VM_DEBUG("id:{:d}", res); |
| 70 | sim::SetLastStreamIdTls(res); | 68 | sim::SetLastStreamIdTls(res); |
| 71 | return ACL_SUCCESS; | 69 | return ACL_SUCCESS; |
| 72 | } | 70 | } |
| @@ -79,7 +77,7 @@ aclError aclrtCreateStream(aclrtStream *stream) | |||
| 79 | aclError aclrtDestroyStream(aclrtStream stream) | 77 | aclError aclrtDestroyStream(aclrtStream stream) |
| 80 | { | 78 | { |
| 81 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 79 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 82 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 80 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 83 | RunnerDB::Delete<sim::Stream>(streamId); | 81 | RunnerDB::Delete<sim::Stream>(streamId); |
| 84 | return ACL_SUCCESS; | 82 | return ACL_SUCCESS; |
| 85 | } | 83 | } |
| @@ -93,10 +91,10 @@ aclError aclrtActiveStream(aclrtStream activeStream, aclrtStream stream) | |||
| 93 | { | 91 | { |
| 94 | (void) stream; | 92 | (void) stream; |
| 95 | uint64_t activStreamId = (uint64_t)(uintptr_t)activeStream; | 93 | uint64_t activStreamId = (uint64_t)(uintptr_t)activeStream; |
| 96 | - STREAM_STUB_DEBUG("id:{:d}", activStreamId); | 94 | + HCCL_VM_DEBUG("id:{:d}", activStreamId); |
| 97 | auto res = RunnerDB::Update<sim::Stream>(activStreamId, [](sim::Stream &stm) { stm.activated = 1; }); | 95 | auto res = RunnerDB::Update<sim::Stream>(activStreamId, [](sim::Stream &stm) { stm.activated = 1; }); |
| 98 | if (!res) { | 96 | if (!res) { |
| 99 | - STREAM_STUB_ERROR("stream not found:{:d}", activStreamId); | 97 | + HCCL_VM_ERROR("stream not found:{:d}", activStreamId); |
| 100 | return ACL_ERROR_INVALID_PARAM; | 98 | return ACL_ERROR_INVALID_PARAM; |
| 101 | } | 99 | } |
| 102 | return ACL_SUCCESS; | 100 | return ACL_SUCCESS; |
| @@ -105,7 +103,7 @@ aclError aclrtActiveStream(aclrtStream activeStream, aclrtStream stream) | |||
| 105 | aclError aclrtSetStreamFailureMode(aclrtStream stream, uint64_t mode) | 103 | aclError aclrtSetStreamFailureMode(aclrtStream stream, uint64_t mode) |
| 106 | { | 104 | { |
| 107 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 105 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 108 | - STREAM_STUB_DEBUG("id:{:d} mode:{:d}", streamId, mode); | 106 | + HCCL_VM_DEBUG("id:{:d} mode:{:d}", streamId, mode); |
| 109 | RunnerDB::Update<sim::Stream>(streamId, [streamId, mode](sim::Stream &stm) { stm.failure_mode = mode; }); | 107 | RunnerDB::Update<sim::Stream>(streamId, [streamId, mode](sim::Stream &stm) { stm.failure_mode = mode; }); |
| 110 | 108 | ||
| 111 | return ACL_SUCCESS; | 109 | return ACL_SUCCESS; |
| @@ -118,7 +116,7 @@ aclError aclrtSynchronizeStreamWithTimeout(aclrtStream stream, int32_t timeout) | |||
| 118 | // GetMode(); | 116 | // GetMode(); |
| 119 | std::string mode = "checker"; | 117 | std::string mode = "checker"; |
| 120 | const int WAIT_COUNTDOWN = 10; // 等待20s | 118 | const int WAIT_COUNTDOWN = 10; // 等待20s |
| 121 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 119 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 122 | return ACL_SUCCESS; | 120 | return ACL_SUCCESS; |
| 123 | } | 121 | } |
| 124 | 122 | ||
| @@ -157,21 +155,21 @@ aclError aclrtSynchronizeStream(aclrtStream stream) | |||
| 157 | syncRecord.streamId = (uint64_t)(uintptr_t)stream; | 155 | syncRecord.streamId = (uint64_t)(uintptr_t)stream; |
| 158 | syncRecord.status = 0; | 156 | syncRecord.status = 0; |
| 159 | if (sim::InsertSyncRecord(syncRecord) != 0) { | 157 | if (sim::InsertSyncRecord(syncRecord) != 0) { |
| 160 | - STREAM_STUB_ERROR("insert sync record fail"); | 158 | + HCCL_VM_ERROR("insert sync record fail"); |
| 161 | return ACL_ERROR_INTERNAL_ERROR; | 159 | return ACL_ERROR_INTERNAL_ERROR; |
| 162 | } | 160 | } |
| 163 | 161 | ||
| 164 | // 4. 阻塞等待本轮同步完成(runner 模式等 runner;no-runner 模式等 host barrier-only) | 162 | // 4. 阻塞等待本轮同步完成(runner 模式等 runner;no-runner 模式等 host barrier-only) |
| 165 | bool done = syncer.tryBlockProxyUntilRunnerDone(targetRound, WAIT_FALG_TIMEOUT * 1000); | 163 | bool done = syncer.tryBlockProxyUntilRunnerDone(targetRound, WAIT_FALG_TIMEOUT * 1000); |
| 166 | if (!done) { | 164 | if (!done) { |
| 167 | - STREAM_STUB_ERROR("sync timeout round:{}", targetRound); | 165 | + HCCL_VM_ERROR("sync timeout round:{}", targetRound); |
| 168 | return ACL_ERROR_INTERNAL_ERROR; | 166 | return ACL_ERROR_INTERNAL_ERROR; |
| 169 | } | 167 | } |
| 170 | 168 | ||
| 171 | syncRecord.status = 1; | 169 | syncRecord.status = 1; |
| 172 | std::vector<sim::SyncRecordTab> syncRecords = {syncRecord}; | 170 | std::vector<sim::SyncRecordTab> syncRecords = {syncRecord}; |
| 173 | if (sim::UpdateSyncRecordStatus(syncRecords) != 0) { | 171 | if (sim::UpdateSyncRecordStatus(syncRecords) != 0) { |
| 174 | - STREAM_STUB_ERROR("update sync record fail"); | 172 | + HCCL_VM_ERROR("update sync record fail"); |
| 175 | return ACL_ERROR_INTERNAL_ERROR; | 173 | return ACL_ERROR_INTERNAL_ERROR; |
| 176 | } | 174 | } |
| 177 | 175 | ||
| @@ -179,7 +177,7 @@ aclError aclrtSynchronizeStream(aclrtStream stream) | |||
| 179 | clock_gettime(CLOCK_MONOTONIC, &tsEnd); | 177 | clock_gettime(CLOCK_MONOTONIC, &tsEnd); |
| 180 | double elapsedMs = (tsEnd.tv_sec - tsStart.tv_sec) * 1000.0 | 178 | double elapsedMs = (tsEnd.tv_sec - tsStart.tv_sec) * 1000.0 |
| 181 | + (tsEnd.tv_nsec - tsStart.tv_nsec) / 1e6; | 179 | + (tsEnd.tv_nsec - tsStart.tv_nsec) / 1e6; |
| 182 | - STREAM_STUB_INFO("sync round:{:d} done {:.3f}ms", targetRound, elapsedMs); | 180 | + HCCL_VM_INFO("sync round:{:d} done {:.3f}ms", targetRound, elapsedMs); |
| 183 | 181 | ||
| 184 | return ACL_SUCCESS; | 182 | return ACL_SUCCESS; |
| 185 | } | 183 | } |
| @@ -187,7 +185,7 @@ aclError aclrtSynchronizeStream(aclrtStream stream) | |||
| 187 | aclError aclrtStreamAbort(aclrtStream stream) | 185 | aclError aclrtStreamAbort(aclrtStream stream) |
| 188 | { | 186 | { |
| 189 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 187 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 190 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 188 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 191 | RunnerDB::Update<sim::Stream>(streamId, [streamId](sim::Stream &stm) { stm.activated = 0; }); | 189 | RunnerDB::Update<sim::Stream>(streamId, [streamId](sim::Stream &stm) { stm.activated = 0; }); |
| 192 | 190 | ||
| 193 | return ACL_SUCCESS; | 191 | return ACL_SUCCESS; |
| @@ -196,10 +194,10 @@ aclError aclrtStreamAbort(aclrtStream stream) | |||
| 196 | aclError aclrtStreamQuery(aclrtStream stream, aclrtStreamStatus *status) | 194 | aclError aclrtStreamQuery(aclrtStream stream, aclrtStreamStatus *status) |
| 197 | { | 195 | { |
| 198 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 196 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 199 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 197 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 200 | auto res = RunnerDB::GetById<sim::Stream>(streamId); | 198 | auto res = RunnerDB::GetById<sim::Stream>(streamId); |
| 201 | if (!res.has_value()) { | 199 | if (!res.has_value()) { |
| 202 | - STREAM_STUB_ERROR("stream not found:{:d}", streamId); | 200 | + HCCL_VM_ERROR("stream not found:{:d}", streamId); |
| 203 | return ACL_ERROR_INVALID_PARAM; | 201 | return ACL_ERROR_INVALID_PARAM; |
| 204 | } | 202 | } |
| 205 | *status = (aclrtStreamStatus)res->task_complete_status; | 203 | *status = (aclrtStreamStatus)res->task_complete_status; |
| @@ -214,12 +212,12 @@ aclError aclrtGetStreamAvailableNum(uint32_t *streamCount) | |||
| 214 | return ACL_ERROR_INVALID_PARAM; | 212 | return ACL_ERROR_INVALID_PARAM; |
| 215 | } | 213 | } |
| 216 | if (runner.current_ctx_id == 0) { | 214 | if (runner.current_ctx_id == 0) { |
| 217 | - STREAM_STUB_ERROR("invalid param"); | 215 | + HCCL_VM_ERROR("invalid param"); |
| 218 | return ACL_ERROR_INVALID_PARAM; | 216 | return ACL_ERROR_INVALID_PARAM; |
| 219 | } | 217 | } |
| 220 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); | 218 | auto currCtx = RunnerDB::GetById<sim::Context>(runner.current_ctx_id); |
| 221 | if (!currCtx.has_value()) { | 219 | if (!currCtx.has_value()) { |
| 222 | - STREAM_STUB_ERROR("ctx not found:{:d}", runner.current_ctx_id); | 220 | + HCCL_VM_ERROR("ctx not found:{:d}", runner.current_ctx_id); |
| 223 | return ACL_ERROR_INVALID_PARAM; | 221 | return ACL_ERROR_INVALID_PARAM; |
| 224 | } | 222 | } |
| 225 | 223 | ||
| @@ -227,7 +225,7 @@ aclError aclrtGetStreamAvailableNum(uint32_t *streamCount) | |||
| 227 | 225 | ||
| 228 | auto device = RunnerDB::GetById<sim::Device>(devId); | 226 | auto device = RunnerDB::GetById<sim::Device>(devId); |
| 229 | if (!device.has_value()) { | 227 | if (!device.has_value()) { |
| 230 | - STREAM_STUB_ERROR("device not found:{:d}", devId); | 228 | + HCCL_VM_ERROR("device not found:{:d}", devId); |
| 231 | return ACL_ERROR_INVALID_PARAM; | 229 | return ACL_ERROR_INVALID_PARAM; |
| 232 | } | 230 | } |
| 233 | 231 | ||
| @@ -254,14 +252,14 @@ aclError aclrtGetStreamAvailableNum(uint32_t *streamCount) | |||
| 254 | aclError aclrtStreamGetId(aclrtStream stream, int32_t *streamId) | 252 | aclError aclrtStreamGetId(aclrtStream stream, int32_t *streamId) |
| 255 | { | 253 | { |
| 256 | *streamId = (uint32_t)(uintptr_t)stream; | 254 | *streamId = (uint32_t)(uintptr_t)stream; |
| 257 | - STREAM_STUB_DEBUG("id:{:d}", *streamId); | 255 | + HCCL_VM_DEBUG("id:{:d}", *streamId); |
| 258 | return ACL_SUCCESS; | 256 | return ACL_SUCCESS; |
| 259 | } | 257 | } |
| 260 | 258 | ||
| 261 | aclError aclrtSetStreamOverflowSwitch(aclrtStream stream, uint32_t flag) | 259 | aclError aclrtSetStreamOverflowSwitch(aclrtStream stream, uint32_t flag) |
| 262 | { | 260 | { |
| 263 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 261 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 264 | - STREAM_STUB_DEBUG("id:{:d} flag:{:d}", streamId, flag); | 262 | + HCCL_VM_DEBUG("id:{:d} flag:{:d}", streamId, flag); |
| 265 | RunnerDB::Update<sim::Stream>(streamId, [streamId, flag](sim::Stream &stm) { stm.overflow_switch = flag; }); | 263 | RunnerDB::Update<sim::Stream>(streamId, [streamId, flag](sim::Stream &stm) { stm.overflow_switch = flag; }); |
| 266 | return ACL_SUCCESS; | 264 | return ACL_SUCCESS; |
| 267 | } | 265 | } |
| @@ -269,10 +267,10 @@ aclError aclrtSetStreamOverflowSwitch(aclrtStream stream, uint32_t flag) | |||
| 269 | aclError aclrtGetStreamOverflowSwitch(aclrtStream stream, uint32_t *flag) | 267 | aclError aclrtGetStreamOverflowSwitch(aclrtStream stream, uint32_t *flag) |
| 270 | { | 268 | { |
| 271 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 269 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 272 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 270 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 273 | auto res = RunnerDB::GetById<sim::Stream>(streamId); | 271 | auto res = RunnerDB::GetById<sim::Stream>(streamId); |
| 274 | if (!res.has_value()) { | 272 | if (!res.has_value()) { |
| 275 | - STREAM_STUB_ERROR("stream not found:{:d}", streamId); | 273 | + HCCL_VM_ERROR("stream not found:{:d}", streamId); |
| 276 | return ACL_ERROR_INVALID_PARAM; | 274 | return ACL_ERROR_INVALID_PARAM; |
| 277 | } | 275 | } |
| 278 | *flag = res->overflow_switch; | 276 | *flag = res->overflow_switch; |
| @@ -282,10 +280,10 @@ aclError aclrtGetStreamOverflowSwitch(aclrtStream stream, uint32_t *flag) | |||
| 282 | aclError aclrtSetStreamAttribute(aclrtStream stream, aclrtStreamAttr stmAttrType, aclrtStreamAttrValue *value) | 280 | aclError aclrtSetStreamAttribute(aclrtStream stream, aclrtStreamAttr stmAttrType, aclrtStreamAttrValue *value) |
| 283 | { | 281 | { |
| 284 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 282 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 285 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 283 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 286 | auto res = RunnerDB::GetById<sim::Stream>(streamId); | 284 | auto res = RunnerDB::GetById<sim::Stream>(streamId); |
| 287 | if (!res.has_value()) { | 285 | if (!res.has_value()) { |
| 288 | - STREAM_STUB_ERROR("stream not found:{:d}", streamId); | 286 | + HCCL_VM_ERROR("stream not found:{:d}", streamId); |
| 289 | return ACL_ERROR_INVALID_PARAM; | 287 | return ACL_ERROR_INVALID_PARAM; |
| 290 | } | 288 | } |
| 291 | 289 | ||
| @@ -306,10 +304,10 @@ aclError aclrtSetStreamAttribute(aclrtStream stream, aclrtStreamAttr stmAttrType | |||
| 306 | aclError aclrtGetStreamAttribute(aclrtStream stream, aclrtStreamAttr stmAttrType, aclrtStreamAttrValue *value) | 304 | aclError aclrtGetStreamAttribute(aclrtStream stream, aclrtStreamAttr stmAttrType, aclrtStreamAttrValue *value) |
| 307 | { | 305 | { |
| 308 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 306 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 309 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 307 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 310 | auto res = RunnerDB::GetById<sim::Stream>(streamId); | 308 | auto res = RunnerDB::GetById<sim::Stream>(streamId); |
| 311 | if (!res.has_value()) { | 309 | if (!res.has_value()) { |
| 312 | - STREAM_STUB_ERROR("stream not found:{:d}", streamId); | 310 | + HCCL_VM_ERROR("stream not found:{:d}", streamId); |
| 313 | return ACL_ERROR_INVALID_PARAM; | 311 | return ACL_ERROR_INVALID_PARAM; |
| 314 | } | 312 | } |
| 315 | 313 | ||
| @@ -332,7 +330,7 @@ aclError aclrtStreamStop(aclrtStream stream) | |||
| 332 | rtError_t rtStreamSynchronize(rtStream_t stream) | 330 | rtError_t rtStreamSynchronize(rtStream_t stream) |
| 333 | { | 331 | { |
| 334 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 332 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 335 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 333 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 336 | return aclrtSynchronizeStream((aclrtStream)stream); | 334 | return aclrtSynchronizeStream((aclrtStream)stream); |
| 337 | } | 335 | } |
| 338 | 336 | ||
| @@ -345,7 +343,7 @@ rtError_t rtStreamGetSqid(const rtStream_t stream, uint32_t *sqId) | |||
| 345 | { | 343 | { |
| 346 | uint64_t streamId = (uint64_t)(uintptr_t)stream; | 344 | uint64_t streamId = (uint64_t)(uintptr_t)stream; |
| 347 | *sqId = streamId; | 345 | *sqId = streamId; |
| 348 | - STREAM_STUB_DEBUG("id:{:d}", streamId); | 346 | + HCCL_VM_DEBUG("id:{:d}", streamId); |
| 349 | return ACL_SUCCESS; | 347 | return ACL_SUCCESS; |
| 350 | } | 348 | } |
| 351 | 349 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -19,11 +22,6 @@ | |||
| 19 | 22 | ||
| 20 | 23 | ||
| 21 | 24 | ||
| 22 | - | ||
| 23 | - | ||
| 24 | - | ||
| 25 | - | ||
| 26 | - | ||
| 27 | 25 | ||
| 28 | 26 | ||
| 29 | extern "C" { | 27 | extern "C" { |
| @@ -33,7 +31,7 @@ aclError aclsysGetVersionStr(char *pkgName, char *versionStr) | |||
| 33 | { | 31 | { |
| 34 | (void) pkgName; | 32 | (void) pkgName; |
| 35 | memcpy(versionStr, "9.0.0", sizeof("9.0.0")); | 33 | memcpy(versionStr, "9.0.0", sizeof("9.0.0")); |
| 36 | - ACLRT_STUB_DEBUG("get version:{}", versionStr); | 34 | + HCCL_VM_DEBUG("get version:{}", versionStr); |
| 37 | return ACL_SUCCESS; | 35 | return ACL_SUCCESS; |
| 38 | } | 36 | } |
| 39 | 37 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -44,14 +47,14 @@ extern "C" { | |||
| 44 | int RaIsFirstUsed(int insId) | 47 | int RaIsFirstUsed(int insId) |
| 45 | { | 48 | { |
| 46 | (void) insId; | 49 | (void) insId; |
| 47 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 50 | + HCCL_VM_WARN("is empty"); |
| 48 | return 0; | 51 | return 0; |
| 49 | } | 52 | } |
| 50 | 53 | ||
| 51 | int RaIsLastUsed(int insId) | 54 | int RaIsLastUsed(int insId) |
| 52 | { | 55 | { |
| 53 | (void) insId; | 56 | (void) insId; |
| 54 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 57 | + HCCL_VM_WARN("is empty"); |
| 55 | return 0; | 58 | return 0; |
| 56 | } | 59 | } |
| 57 | 60 | ||
| @@ -59,8 +62,9 @@ int ibv_get_cq_event_stub(struct ibv_comp_channel *channel, struct ibv_cq **cq, | |||
| 59 | { | 62 | { |
| 60 | (void) cq; | 63 | (void) cq; |
| 61 | (void) cq_context; | 64 | (void) cq_context; |
| 62 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 65 | + HCCL_VM_WARN("is empty"); |
| 63 | if (!channel) { | 66 | if (!channel) { |
| 67 | + HCCL_VM_ERROR("channel is null"); | ||
| 64 | return -1; | 68 | return -1; |
| 65 | } | 69 | } |
| 66 | return 0; | 70 | return 0; |
| @@ -70,7 +74,7 @@ void ibv_ack_cq_events_stub(struct ibv_cq *cq, unsigned int nevents) | |||
| 70 | { | 74 | { |
| 71 | (void) cq; | 75 | (void) cq; |
| 72 | (void) nevents; | 76 | (void) nevents; |
| 73 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 77 | + HCCL_VM_WARN("is empty"); |
| 74 | return; | 78 | return; |
| 75 | } | 79 | } |
| 76 | 80 | ||
| @@ -80,7 +84,7 @@ void ibv_query_qp_stub(struct ibv_qp *qp, struct ibv_qp_attr *attr, int attr_mas | |||
| 80 | (void) attr; | 84 | (void) attr; |
| 81 | (void) attr_mask; | 85 | (void) attr_mask; |
| 82 | (void) init_attr; | 86 | (void) init_attr; |
| 83 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 87 | + HCCL_VM_WARN("is empty"); |
| 84 | return; | 88 | return; |
| 85 | } | 89 | } |
| 86 | 90 | ||
| @@ -89,7 +93,7 @@ int32_t UtraceCreateWithAttr(int32_t tracerType, const char *objName, const Trac | |||
| 89 | (void)(tracerType); | 93 | (void)(tracerType); |
| 90 | (void)(objName); | 94 | (void)(objName); |
| 91 | (void)(attr); | 95 | (void)(attr); |
| 92 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 96 | + HCCL_VM_WARN("is empty"); |
| 93 | return 0; | 97 | return 0; |
| 94 | } | 98 | } |
| 95 | 99 | ||
| @@ -98,14 +102,14 @@ HcclResult UtraceSubmit(int32_t handle, const void *buffer, uint32_t bufSize) | |||
| 98 | (void)(handle); | 102 | (void)(handle); |
| 99 | (void)(buffer); | 103 | (void)(buffer); |
| 100 | (void)(bufSize); | 104 | (void)(bufSize); |
| 101 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 105 | + HCCL_VM_WARN("is empty"); |
| 102 | return HCCL_SUCCESS; | 106 | return HCCL_SUCCESS; |
| 103 | } | 107 | } |
| 104 | 108 | ||
| 105 | int32_t UtraceSetGlobalAttr(const TraceGlobalAttr *attr) | 109 | int32_t UtraceSetGlobalAttr(const TraceGlobalAttr *attr) |
| 106 | { | 110 | { |
| 107 | (void)(attr); | 111 | (void)(attr); |
| 108 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 112 | + HCCL_VM_WARN("is empty"); |
| 109 | return 0; | 113 | return 0; |
| 110 | } | 114 | } |
| 111 | 115 | ||
| @@ -113,14 +117,14 @@ int32_t UtraceSave(TracerType tracerType, bool syncFlag) | |||
| 113 | { | 117 | { |
| 114 | (void)(tracerType); | 118 | (void)(tracerType); |
| 115 | (void)(syncFlag); | 119 | (void)(syncFlag); |
| 116 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 120 | + HCCL_VM_WARN("is empty"); |
| 117 | return 0; | 121 | return 0; |
| 118 | } | 122 | } |
| 119 | 123 | ||
| 120 | void UtraceDestroy(int32_t handle) | 124 | void UtraceDestroy(int32_t handle) |
| 121 | { | 125 | { |
| 122 | (void)(handle); | 126 | (void)(handle); |
| 123 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 127 | + HCCL_VM_WARN("is empty"); |
| 124 | return; | 128 | return; |
| 125 | } | 129 | } |
| 126 | 130 | ||
| @@ -132,7 +136,7 @@ int ibv_exp_post_send_stub(struct ibv_qp *qp, struct ibv_send_wr *wr, struct ibv | |||
| 132 | (void) bad_wr; | 136 | (void) bad_wr; |
| 133 | (void) ext_attr; | 137 | (void) ext_attr; |
| 134 | (void) ext_resp; | 138 | (void) ext_resp; |
| 135 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 139 | + HCCL_VM_WARN("is empty"); |
| 136 | return 0; | 140 | return 0; |
| 137 | } | 141 | } |
| 138 | 142 | ||
| @@ -144,20 +148,20 @@ int ibv_ext_post_send_stub(struct ibv_qp *qp, struct ibv_send_wr *wr, struct ibv | |||
| 144 | (void) bad_wr; | 148 | (void) bad_wr; |
| 145 | (void) ext_attr; | 149 | (void) ext_attr; |
| 146 | (void) ext_resp; | 150 | (void) ext_resp; |
| 147 | - HCCL_VM_INFO("[RTS_STUB] stub is empty"); | 151 | + HCCL_VM_WARN("is empty"); |
| 148 | return 0; | 152 | return 0; |
| 149 | } | 153 | } |
| 150 | 154 | ||
| 151 | HcclResult hrtTsdCapabilityGet(uint32_t deviceLogicId, int32_t type, uint64_t ptr) | 155 | HcclResult hrtTsdCapabilityGet(uint32_t deviceLogicId, int32_t type, uint64_t ptr) |
| 152 | { | 156 | { |
| 153 | - HCCL_VM_INFO("[RTS_STUB] deviceLogicId:[{}] type:[{}]], ptr:[{}]\n", deviceLogicId, type, ptr); | 157 | + HCCL_VM_INFO("deviceLogicId:[{}] type:[{}]], ptr:[{}]\n", deviceLogicId, type, ptr); |
| 154 | return (HcclResult)0; | 158 | return (HcclResult)0; |
| 155 | } | 159 | } |
| 156 | 160 | ||
| 157 | HcclResult hrtGetDeviceInfo(u32 deviceId, HcclRtDeviceModuleType hcclModuleType, HcclRtDeviceInfoType hcclInfoType, | 161 | HcclResult hrtGetDeviceInfo(u32 deviceId, HcclRtDeviceModuleType hcclModuleType, HcclRtDeviceInfoType hcclInfoType, |
| 158 | s64 &val) | 162 | s64 &val) |
| 159 | { | 163 | { |
| 160 | - HCCL_VM_INFO("[RTS_STUB] deviceId:[{}] hcclModuleType:[{}], HcclRtDeviceInfoType:[{}]\n", deviceId, | 164 | + HCCL_VM_INFO("deviceId:[{}] hcclModuleType:[{}], HcclRtDeviceInfoType:[{}]\n", deviceId, |
| 161 | (int)hcclModuleType, (int)hcclInfoType); | 165 | (int)hcclModuleType, (int)hcclInfoType); |
| 162 | return (HcclResult)0; | 166 | return (HcclResult)0; |
| 163 | } | 167 | } |
| @@ -177,10 +181,9 @@ void *__HcclDlsymSub(void *handle, const char *funcName) | |||
| 177 | (void) handle; | 181 | (void) handle; |
| 178 | void *addr = dlsym(RTLD_DEFAULT, funcName); | 182 | void *addr = dlsym(RTLD_DEFAULT, funcName); |
| 179 | if (addr == nullptr) { | 183 | if (addr == nullptr) { |
| 180 | - HCCL_VM_INFO("[RTS_STUB] not find {} Error: [{}]\n", funcName, dlerror()); | 184 | + HCCL_VM_ERROR("not find {} Error: [{}]\n", funcName, dlerror()); |
| 181 | return nullptr; | 185 | return nullptr; |
| 182 | } | 186 | } |
| 183 | - // CCU_STUB_DEBUG("[HcclDlsymSub] [{}]\n", funcName); | ||
| 184 | return addr; | 187 | return addr; |
| 185 | } | 188 | } |
| 186 | 189 | ||
| @@ -6,6 +6,19 @@ | |||
| 6 | # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | 6 | # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. |
| 7 | # See LICENSE in the root of the software repository for the full text of the License. | 7 | # See LICENSE in the root of the software repository for the full text of the License. |
| 8 | 8 | ||
| 9 | +add_compile_options( | ||
| 10 | + -fno-common | ||
| 11 | + -fno-strict-aliasing | ||
| 12 | + -pipe | ||
| 13 | + -O0 | ||
| 14 | + -g | ||
| 15 | +) | ||
| 16 | +# 日志染色: aiv_kernel 整片统一 [AIV] | ||
| 17 | +add_compile_definitions( | ||
| 18 | + HCCL_VM_MODULE="AIV" | ||
| 19 | + _GLIBCXX_USE_CXX11_ABI=0 | ||
| 20 | +) | ||
| 21 | + | ||
| 9 | if(NOT DEFINED HCCL_CODE_ROOT) | 22 | if(NOT DEFINED HCCL_CODE_ROOT) |
| 10 | if(NOT DEFINED ENV{HCCL_CODE_HOME}) | 23 | if(NOT DEFINED ENV{HCCL_CODE_HOME}) |
| 11 | message(FATAL_ERROR "Error: BUILD_AIV is ON, but HCCL_CODE_HOME environment variable is not set. Please set it to the HCCL source path.") | 24 | message(FATAL_ERROR "Error: BUILD_AIV is ON, but HCCL_CODE_HOME environment variable is not set. Please set it to the HCCL source path.") |
| @@ -41,6 +54,8 @@ add_library(hccl_aiv_kernel SHARED | |||
| 41 | $<TARGET_OBJECTS:hccl_aiv_reduce_op> | 54 | $<TARGET_OBJECTS:hccl_aiv_reduce_op> |
| 42 | $<TARGET_OBJECTS:hccl_aiv_reduce_scatter_op> | 55 | $<TARGET_OBJECTS:hccl_aiv_reduce_scatter_op> |
| 43 | $<TARGET_OBJECTS:hccl_aiv_scatter_op> | 56 | $<TARGET_OBJECTS:hccl_aiv_scatter_op> |
| 57 | + | ||
| 58 | + ${HCCL_VM_SRC}/utils/sim_common_api.cc | ||
| 44 | ) | 59 | ) |
| 45 | 60 | ||
| 46 | target_link_libraries(hccl_aiv_kernel PRIVATE log) | 61 | target_link_libraries(hccl_aiv_kernel PRIVATE log) |
| @@ -140,7 +140,7 @@ AivDataSlice AivKernelExecutor::ResolveGlobalDataSlice(uint64_t addr, uint64_t s | |||
| 140 | } | 140 | } |
| 141 | 141 | ||
| 142 | void AivKernelExecutor::DumpAllTasks() const { | 142 | void AivKernelExecutor::DumpAllTasks() const { |
| 143 | - HCCL_VM_DEBUG("[DumpAivTasks]"); | 143 | + HCCL_VM_DEBUG(""); |
| 144 | HCCL_VM_DEBUG("[rankId={}]", GetRankId()); | 144 | HCCL_VM_DEBUG("[rankId={}]", GetRankId()); |
| 145 | 145 | ||
| 146 | for (const auto& core : aivCores_) { | 146 | for (const auto& core : aivCores_) { |
| @@ -175,17 +175,17 @@ void AivCore::AppendMTE3(const std::shared_ptr<AivTask>& task) { | |||
| 175 | void AivCore::DumpAllTasks() const { | 175 | void AivCore::DumpAllTasks() const { |
| 176 | HCCL_VM_DEBUG("[blockIdx={:d}]", blockIdx_); | 176 | HCCL_VM_DEBUG("[blockIdx={:d}]", blockIdx_); |
| 177 | 177 | ||
| 178 | - HCCL_VM_DEBUG("[Scalar]"); | 178 | + HCCL_VM_DEBUG(""); |
| 179 | for (const auto& task : pipeScalar_) { | 179 | for (const auto& task : pipeScalar_) { |
| 180 | HCCL_VM_DEBUG(" {:s}", task->Describe()); | 180 | HCCL_VM_DEBUG(" {:s}", task->Describe()); |
| 181 | } | 181 | } |
| 182 | 182 | ||
| 183 | - HCCL_VM_DEBUG("[MTE2]"); | 183 | + HCCL_VM_DEBUG("MTE2 "); |
| 184 | for (const auto& task : pipeMTE2_) { | 184 | for (const auto& task : pipeMTE2_) { |
| 185 | HCCL_VM_DEBUG(" {:s}", task->Describe()); | 185 | HCCL_VM_DEBUG(" {:s}", task->Describe()); |
| 186 | } | 186 | } |
| 187 | 187 | ||
| 188 | - HCCL_VM_DEBUG("[MTE3]"); | 188 | + HCCL_VM_DEBUG("MTE3 "); |
| 189 | for (const auto& task : pipeMTE3_) { | 189 | for (const auto& task : pipeMTE3_) { |
| 190 | HCCL_VM_DEBUG(" {:s}", task->Describe()); | 190 | HCCL_VM_DEBUG(" {:s}", task->Describe()); |
| 191 | } | 191 | } |
| @@ -9,6 +9,7 @@ | |||
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | 11 | ||
| 12 | + | ||
| 12 | 13 | ||
| 13 | 14 | ||
| 14 | 15 | ||
| @@ -210,15 +211,9 @@ bool WriteJsonAtomically(const json &content, const std::string &filePath) | |||
| 210 | 211 | ||
| 211 | bool ResolveExecutorJsonFilePath(const AivKernelExecutor &executor, uint32_t launchIndex, std::string &filePath) | 212 | bool ResolveExecutorJsonFilePath(const AivKernelExecutor &executor, uint32_t launchIndex, std::string &filePath) |
| 212 | { | 213 | { |
| 213 | - const char *installDirEnv = std::getenv("HCCL_VM_INSTALL_DIR"); | 214 | + const std::string dataRelPath = "data/hcclvm_aiv_rank" + std::to_string(executor.GetRankId()) + |
| 214 | - if (installDirEnv == nullptr || installDirEnv[0] == '\0') { | 215 | + "_launch" + std::to_string(launchIndex) + "_task.json"; |
| 215 | - return false; | 216 | + filePath = InstallPath::ResolveToInstallRoot(dataRelPath); |
| 216 | - } | ||
| 217 | - | ||
| 218 | - const fs::path outputPath = fs::path(installDirEnv) / "data" / | ||
| 219 | - ("hcclvm_aiv_rank" + std::to_string(executor.GetRankId()) + | ||
| 220 | - "_launch" + std::to_string(launchIndex) + "_task.json"); | ||
| 221 | - filePath = outputPath.string(); | ||
| 222 | return true; | 217 | return true; |
| 223 | } | 218 | } |
| 224 | 219 | ||
| @@ -38,15 +38,15 @@ void aiv_env_init(uint32_t rankId, | |||
| 38 | executor.SetCurOp(opParam); | 38 | executor.SetCurOp(opParam); |
| 39 | const std::string inputMemDesc = AivSim::Mem{input, inputSize}.Describe(); | 39 | const std::string inputMemDesc = AivSim::Mem{input, inputSize}.Describe(); |
| 40 | const std::string outputMemDesc = AivSim::Mem{output, outputSize}.Describe(); | 40 | const std::string outputMemDesc = AivSim::Mem{output, outputSize}.Describe(); |
| 41 | - HCCL_VM_DEBUG("[aiv_env_init] begin"); | 41 | + HCCL_VM_DEBUG("begin"); |
| 42 | HCCL_VM_DEBUG( | 42 | HCCL_VM_DEBUG( |
| 43 | - "[aiv_env_init] rankId={}, blockNum={}, buffIn={:p}, rankSize={}", | 43 | + "rankId={}, blockNum={}, buffIn={:p}, rankSize={}", |
| 44 | rankId, | 44 | rankId, |
| 45 | blockNum, | 45 | blockNum, |
| 46 | buffIn, | 46 | buffIn, |
| 47 | rankSize); | 47 | rankSize); |
| 48 | HCCL_VM_DEBUG( | 48 | HCCL_VM_DEBUG( |
| 49 | - "[aiv_env_init] input={}, output={}, " | 49 | + "input={}, output={}, " |
| 50 | "inputGlobalOffsetBase={}, outputGlobalOffsetBase={}, cclBufferSize={}, flagBufferSize={}", | 50 | "inputGlobalOffsetBase={}, outputGlobalOffsetBase={}, cclBufferSize={}, flagBufferSize={}", |
| 51 | inputMemDesc, | 51 | inputMemDesc, |
| 52 | outputMemDesc, | 52 | outputMemDesc, |
| @@ -55,7 +55,7 @@ void aiv_env_init(uint32_t rankId, | |||
| 55 | cclBufferSize, | 55 | cclBufferSize, |
| 56 | flagBufferSize); | 56 | flagBufferSize); |
| 57 | HCCL_VM_DEBUG( | 57 | HCCL_VM_DEBUG( |
| 58 | - "[aiv_env_init] curOp: dataType={}, len={}, reduceOp={}, root={}, sliceId={}, inputStride={}, " | 58 | + "curOp: dataType={}, len={}, reduceOp={}, root={}, sliceId={}, inputStride={}, " |
| 59 | "outputStride={}, kernelName={}", | 59 | "outputStride={}, kernelName={}", |
| 60 | opParam.dataType, | 60 | opParam.dataType, |
| 61 | opParam.len, | 61 | opParam.len, |
| @@ -66,7 +66,7 @@ void aiv_env_init(uint32_t rankId, | |||
| 66 | opParam.outputStride, | 66 | opParam.outputStride, |
| 67 | opParam.kernelName); | 67 | opParam.kernelName); |
| 68 | 68 | ||
| 69 | - HCCL_VM_DEBUG("[aiv_env_init] SetIoBuffer"); | 69 | + HCCL_VM_DEBUG("SetIoBuffer"); |
| 70 | executor.SetIoBuffer( | 70 | executor.SetIoBuffer( |
| 71 | input, | 71 | input, |
| 72 | inputSize, | 72 | inputSize, |
| @@ -76,7 +76,7 @@ void aiv_env_init(uint32_t rankId, | |||
| 76 | outputGlobalOffsetBase); | 76 | outputGlobalOffsetBase); |
| 77 | 77 | ||
| 78 | if (buffIn == nullptr) { | 78 | if (buffIn == nullptr) { |
| 79 | - HCCL_VM_DEBUG("[aiv_env_init] buffIn is null, skip SetCommBuffer initialization"); | 79 | + HCCL_VM_DEBUG("buffIn is null, skip SetCommBuffer initialization"); |
| 80 | return; | 80 | return; |
| 81 | } | 81 | } |
| 82 | 82 | ||
| @@ -89,7 +89,7 @@ void aiv_env_init(uint32_t rankId, | |||
| 89 | const std::string cclMemDesc = AivSim::Mem{cclBuffer, cclBufferSize}.Describe(); | 89 | const std::string cclMemDesc = AivSim::Mem{cclBuffer, cclBufferSize}.Describe(); |
| 90 | const std::string flagMemDesc = AivSim::Mem{flagBuffer, flagBufferSize}.Describe(); | 90 | const std::string flagMemDesc = AivSim::Mem{flagBuffer, flagBufferSize}.Describe(); |
| 91 | HCCL_VM_DEBUG( | 91 | HCCL_VM_DEBUG( |
| 92 | - "[aiv_env_init] SetCommBuffer rank={}, cclBuffer={}, flagBuffer={}", | 92 | + "SetCommBuffer rank={}, cclBuffer={}, flagBuffer={}", |
| 93 | i, | 93 | i, |
| 94 | cclMemDesc, | 94 | cclMemDesc, |
| 95 | flagMemDesc); | 95 | flagMemDesc); |
| @@ -78,6 +78,48 @@ int32_t MsprofReportCompactInfo(uint32_t agingFlag, const VOID_PTR data, uint32_ | |||
| 78 | return 0; | 78 | return 0; |
| 79 | } | 79 | } |
| 80 | 80 | ||
| 81 | +int32_t MsprofReportBatchAdditionalInfo(uint32_t nonPersistantFlag, const VOID_PTR data, uint32_t length) | ||
| 82 | +{ | ||
| 83 | + (void) nonPersistantFlag; | ||
| 84 | + (void) data; | ||
| 85 | + (void) length; | ||
| 86 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 87 | + return 0; | ||
| 88 | +} | ||
| 89 | + | ||
| 90 | +int32_t AdprofReportAdditionalInfo(uint32_t agingFlag, const void *data, uint32_t length) | ||
| 91 | +{ | ||
| 92 | + (void) agingFlag; | ||
| 93 | + (void) data; | ||
| 94 | + (void) length; | ||
| 95 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 96 | + return 0; | ||
| 97 | +} | ||
| 98 | + | ||
| 99 | +int32_t AdprofReportBatchAdditionalInfo(uint32_t nonPersistantFlag, const void *data, uint32_t length) | ||
| 100 | +{ | ||
| 101 | + (void) nonPersistantFlag; | ||
| 102 | + (void) data; | ||
| 103 | + (void) length; | ||
| 104 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 105 | + return 0; | ||
| 106 | +} | ||
| 107 | + | ||
| 108 | +uint64_t AdprofGetHashId(const char *hashInfo, size_t length) | ||
| 109 | +{ | ||
| 110 | + (void) hashInfo; | ||
| 111 | + (void) length; | ||
| 112 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 113 | + return 1; | ||
| 114 | +} | ||
| 115 | + | ||
| 116 | +int32_t AdprofCheckFeatureIsOn(uint64_t feature) | ||
| 117 | +{ | ||
| 118 | + (void) feature; | ||
| 119 | + HCCL_VM_INFO("[APROF] stub"); | ||
| 120 | + return 0; | ||
| 121 | +} | ||
| 122 | + | ||
| 81 | 123 | ||
| 82 | } | 124 | } |
| 83 | 125 | ||
| @@ -7,6 +7,8 @@ | |||
| 7 | * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | 7 | * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. |
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 11 | + | ||
| 10 | 12 | ||
| 11 | 13 | ||
| 12 | 14 | ||
| @@ -25,11 +27,6 @@ | |||
| 25 | 27 | ||
| 26 | 28 | ||
| 27 | 29 | ||
| 28 | - | ||
| 29 | - | ||
| 30 | - | ||
| 31 | - | ||
| 32 | - | ||
| 33 | 30 | ||
| 34 | 31 | ||
| 35 | extern "C" { | 32 | extern "C" { |
| @@ -55,7 +52,7 @@ HcclResult HcclCommInitRootInfo(uint32_t nRanks, const HcclRootInfo *rootInfo, u | |||
| 55 | // 解析ranktable文件,初始化通信域相关数据库表项 | 52 | // 解析ranktable文件,初始化通信域相关数据库表项 |
| 56 | const char* clusterInfo = std::getenv("RANK_TABLE_FILE"); | 53 | const char* clusterInfo = std::getenv("RANK_TABLE_FILE"); |
| 57 | if (!clusterInfo) { | 54 | if (!clusterInfo) { |
| 58 | - COMM_STUB_ERROR("RANK_TABLE_FILE env not set, please check your config."); | 55 | + HCCL_VM_ERROR("RANK_TABLE_FILE env not set, please check your config."); |
| 59 | return HcclResult::HCCL_E_INTERNAL; | 56 | return HcclResult::HCCL_E_INTERNAL; |
| 60 | } | 57 | } |
| 61 | 58 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -16,11 +19,6 @@ | |||
| 16 | 19 | ||
| 17 | 20 | ||
| 18 | 21 | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | 22 | ||
| 25 | namespace sim { | 23 | namespace sim { |
| 26 | const std::map<HcclDataType, u32> DATA_TYPE_SIZE_MAP = { | 24 | const std::map<HcclDataType, u32> DATA_TYPE_SIZE_MAP = { |
| @@ -47,7 +45,7 @@ int GetDataTypeSize(HcclDataType dataType, uint32_t &size) | |||
| 47 | { | 45 | { |
| 48 | auto iter = DATA_TYPE_SIZE_MAP.find(dataType); | 46 | auto iter = DATA_TYPE_SIZE_MAP.find(dataType); |
| 49 | if (iter == DATA_TYPE_SIZE_MAP.end()) { | 47 | if (iter == DATA_TYPE_SIZE_MAP.end()) { |
| 50 | - PROXY_STUB_ERROR("not support data type: {}", static_cast<uint32_t>(dataType)); | 48 | + HCCL_VM_ERROR("not support data type: {}", static_cast<uint32_t>(dataType)); |
| 51 | return 1; | 49 | return 1; |
| 52 | } | 50 | } |
| 53 | size = iter->second; | 51 | size = iter->second; |
| @@ -63,7 +61,7 @@ bool IsDeviceAddress(void *addr) | |||
| 63 | (devPtr < (virMem.start_ptr + virMem.size)) && | 61 | (devPtr < (virMem.start_ptr + virMem.size)) && |
| 64 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); | 62 | (virMem.src_type == (uint8_t)sim::VIR_MEM_TYPE_DEV)); }); |
| 65 | if (!virMemRes.second) { | 63 | if (!virMemRes.second) { |
| 66 | - PROXY_STUB_INFO("this buff offset ptr not found:{:p} in VirtualMemBlock.", addr); | 64 | + HCCL_VM_INFO("this buff offset ptr not found:{:p} in VirtualMemBlock.", addr); |
| 67 | return false; | 65 | return false; |
| 68 | } | 66 | } |
| 69 | 67 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -34,11 +37,6 @@ | |||
| 34 | 37 | ||
| 35 | 38 | ||
| 36 | 39 | ||
| 37 | - | ||
| 38 | - | ||
| 39 | - | ||
| 40 | - | ||
| 41 | - | ||
| 42 | 40 | ||
| 43 | extern uint64_t g_cur_server_key; | 41 | extern uint64_t g_cur_server_key; |
| 44 | extern thread_local std::string g_currentOpFastLaunchTag; | 42 | extern thread_local std::string g_currentOpFastLaunchTag; |
| @@ -51,7 +49,7 @@ extern "C" { | |||
| 51 | 49 | ||
| 52 | int GetEnableCcuDie(channel_info_out *output, uint8_t dieId) | 50 | int GetEnableCcuDie(channel_info_out *output, uint8_t dieId) |
| 53 | { | 51 | { |
| 54 | - CCU_STUB_INFO("dieId:{}.", dieId); | 52 | + HCCL_VM_INFO("dieId:{}.", dieId); |
| 55 | output->data.data_info.data_array[0].dieinfo.enable_flag = 1; | 53 | output->data.data_info.data_array[0].dieinfo.enable_flag = 1; |
| 56 | return 0; | 54 | return 0; |
| 57 | } | 55 | } |
| @@ -90,13 +88,13 @@ int SetCcuResourceBasicInfo(channel_info_out* output, uint8_t dieId, uint32_t de | |||
| 90 | { | 88 | { |
| 91 | sim::Device device{}; | 89 | sim::Device device{}; |
| 92 | if (GetDeviceByPhysicalId(devId, device) != ACL_SUCCESS) { | 90 | if (GetDeviceByPhysicalId(devId, device) != ACL_SUCCESS) { |
| 93 | - CCU_STUB_ERROR("get device by logic id {} failed.", devId); | 91 | + HCCL_VM_ERROR("get device by logic id {} failed.", devId); |
| 94 | return -1; | 92 | return -1; |
| 95 | } | 93 | } |
| 96 | if (strcmp(device.soc_version, "Ascend950") == 0) { | 94 | if (strcmp(device.soc_version, "Ascend950") == 0) { |
| 97 | SetCcuV1ResourceBasicInfo(output, dieId, devId); | 95 | SetCcuV1ResourceBasicInfo(output, dieId, devId); |
| 98 | } else { | 96 | } else { |
| 99 | - CCU_STUB_ERROR("unknown device version: {} for devId: {}", device.soc_version, devId); | 97 | + HCCL_VM_ERROR("unknown device version: {} for devId: {}", device.soc_version, devId); |
| 100 | return -1; | 98 | return -1; |
| 101 | } | 99 | } |
| 102 | return 0; | 100 | return 0; |
| @@ -160,25 +158,25 @@ bool write_or_overwrite_in_cwd(const std::string& filename, const std::string &d | |||
| 160 | // input->op == Hccl::CcuOpcodeType::CCU_U_OP_SET_INSTRUCTION | 158 | // input->op == Hccl::CcuOpcodeType::CCU_U_OP_SET_INSTRUCTION |
| 161 | int LoadMicrocodeInstructionStub(uint32_t devId, uint8_t dieId, const channel_info_in *input) | 159 | int LoadMicrocodeInstructionStub(uint32_t devId, uint8_t dieId, const channel_info_in *input) |
| 162 | { | 160 | { |
| 163 | - CCU_STUB_INFO("enter LoadMicrocodeInstruction ....."); | 161 | + HCCL_VM_INFO("enter LoadMicrocodeInstruction ....."); |
| 164 | if (dieId >= DIE_NUM) { | 162 | if (dieId >= DIE_NUM) { |
| 165 | - CCU_STUB_ERROR("wrong param of die id: {}", dieId); | 163 | + HCCL_VM_ERROR("wrong param of die id: {}", dieId); |
| 166 | return -1; | 164 | return -1; |
| 167 | } | 165 | } |
| 168 | 166 | ||
| 169 | sim::Device device{}; | 167 | sim::Device device{}; |
| 170 | if (GetDeviceByPhysicalId(devId, device) != ACL_SUCCESS) { | 168 | if (GetDeviceByPhysicalId(devId, device) != ACL_SUCCESS) { |
| 171 | - CCU_STUB_ERROR("get device by logic id {} failed.", devId); | 169 | + HCCL_VM_ERROR("get device by logic id {} failed.", devId); |
| 172 | return -1; | 170 | return -1; |
| 173 | } | 171 | } |
| 174 | sim::Ccu ccu{}; | 172 | sim::Ccu ccu{}; |
| 175 | if (GetCcuFromDeviceByDieId(device.id, dieId, ccu) != ACL_SUCCESS) { | 173 | if (GetCcuFromDeviceByDieId(device.id, dieId, ccu) != ACL_SUCCESS) { |
| 176 | - CCU_STUB_ERROR("get ccu from device by die id {} failed.", dieId); | 174 | + HCCL_VM_ERROR("get ccu from device by die id {} failed.", dieId); |
| 177 | return -1; | 175 | return -1; |
| 178 | } | 176 | } |
| 179 | sim::CcuResource ccuRes; | 177 | sim::CcuResource ccuRes; |
| 180 | if (GetCcuResourceByCcu(ccu.id, ccuRes) != ACL_SUCCESS) { | 178 | if (GetCcuResourceByCcu(ccu.id, ccuRes) != ACL_SUCCESS) { |
| 181 | - CCU_STUB_ERROR("get ccu resource by ccu {} failed.", ccu.id); | 179 | + HCCL_VM_ERROR("get ccu resource by ccu {} failed.", ccu.id); |
| 182 | return -1; | 180 | return -1; |
| 183 | } | 181 | } |
| 184 | 182 | ||
| @@ -187,7 +185,7 @@ int LoadMicrocodeInstructionStub(uint32_t devId, uint8_t dieId, const channel_in | |||
| 187 | return r.device_id == devKey; | 185 | return r.device_id == devKey; |
| 188 | }); | 186 | }); |
| 189 | if (!rank.second) { | 187 | if (!rank.second) { |
| 190 | - CCU_STUB_ERROR("can not find any rank"); | 188 | + HCCL_VM_ERROR("can not find any rank"); |
| 191 | return -1; | 189 | return -1; |
| 192 | } | 190 | } |
| 193 | auto rankId = rank.first.rank_id; | 191 | auto rankId = rank.first.rank_id; |
| @@ -200,11 +198,11 @@ int LoadMicrocodeInstructionStub(uint32_t devId, uint8_t dieId, const channel_in | |||
| 200 | auto ccuDataTmp = (ccu_data_type_union)(input->data.data_info.data_array[0]); | 198 | auto ccuDataTmp = (ccu_data_type_union)(input->data.data_info.data_array[0]); |
| 201 | auto instructionData = reinterpret_cast<hcomm::CcuRep::CcuInstr*>(GetRealPtrByAddr((void *)ccuDataTmp.insinfo.resourceAddr)); | 199 | auto instructionData = reinterpret_cast<hcomm::CcuRep::CcuInstr*>(GetRealPtrByAddr((void *)ccuDataTmp.insinfo.resourceAddr)); |
| 202 | if (instructionData == nullptr) { | 200 | if (instructionData == nullptr) { |
| 203 | - CCU_STUB_ERROR("get ccu instruction data by resourceAddr failed addr:0x{:x}", ccuDataTmp.insinfo.resourceAddr); | 201 | + HCCL_VM_ERROR("get ccu instruction data by resourceAddr failed addr:0x{:x}", ccuDataTmp.insinfo.resourceAddr); |
| 204 | return -1; | 202 | return -1; |
| 205 | } | 203 | } |
| 206 | if (sim::UpdateAndInsertByCcuId(ccuId, devKey, rankId, dieId, instrCnt, instrOffset, instrInfoSize, instructionData) != 0) { | 204 | if (sim::UpdateAndInsertByCcuId(ccuId, devKey, rankId, dieId, instrCnt, instrOffset, instrInfoSize, instructionData) != 0) { |
| 207 | - CCU_STUB_ERROR("update ccu failed"); | 205 | + HCCL_VM_ERROR("update ccu failed"); |
| 208 | return -1; | 206 | return -1; |
| 209 | } | 207 | } |
| 210 | 208 | ||
| @@ -218,7 +216,7 @@ int LoadMicrocodeInstructionStub(uint32_t devId, uint8_t dieId, const channel_in | |||
| 218 | mcData << "[InstrData][ " + std::to_string(startId + idx) + "]" + hcomm::CcuRep::ParseInstr(&instructionData[idx]) + "\n"; | 216 | mcData << "[InstrData][ " + std::to_string(startId + idx) + "]" + hcomm::CcuRep::ParseInstr(&instructionData[idx]) + "\n"; |
| 219 | } | 217 | } |
| 220 | } else { | 218 | } else { |
| 221 | - CCU_STUB_ERROR("not support device soc version: {:s}", device.soc_version); | 219 | + HCCL_VM_ERROR("not support device soc version: {:s}", device.soc_version); |
| 222 | return HCCL_E_NOT_SUPPORT; | 220 | return HCCL_E_NOT_SUPPORT; |
| 223 | } | 221 | } |
| 224 | 222 | ||
| @@ -231,7 +229,7 @@ int LoadMicrocodeInstructionStub(uint32_t devId, uint8_t dieId, const channel_in | |||
| 231 | instr.startId = startId; | 229 | instr.startId = startId; |
| 232 | instr.instrInfoSize = instrInfoSize; | 230 | instr.instrInfoSize = instrInfoSize; |
| 233 | if(sim::InsertCcuInstr(instr) != 0) { | 231 | if(sim::InsertCcuInstr(instr) != 0) { |
| 234 | - CCU_STUB_ERROR("insert instr failed"); | 232 | + HCCL_VM_ERROR("insert instr failed"); |
| 235 | return -1; | 233 | return -1; |
| 236 | } | 234 | } |
| 237 | return 0; | 235 | return 0; |
| @@ -254,20 +252,20 @@ int GetLocalEndPointByJetty(uint64_t jettyId, uint16_t dieId, sim::EndPoint &end | |||
| 254 | return jetty.jetty_id == jettyId && jetty.pid == getpid() && jetty.dieId == dieId; | 252 | return jetty.jetty_id == jettyId && jetty.pid == getpid() && jetty.dieId == dieId; |
| 255 | }); | 253 | }); |
| 256 | if (!localJetty.second) { | 254 | if (!localJetty.second) { |
| 257 | - CCU_STUB_ERROR("can not find jetty {} die:{} in local jetty map", jettyId); | 255 | + HCCL_VM_ERROR("can not find jetty {} die:{} in local jetty map", jettyId, dieId); |
| 258 | return -1; | 256 | return -1; |
| 259 | } | 257 | } |
| 260 | 258 | ||
| 261 | auto ctxRes = RunnerDB::GetById<sim::RaContext>(localJetty.first.ctx_handle); | 259 | auto ctxRes = RunnerDB::GetById<sim::RaContext>(localJetty.first.ctx_handle); |
| 262 | if (!ctxRes.has_value()) { | 260 | if (!ctxRes.has_value()) { |
| 263 | - CCU_STUB_ERROR("can not find context {} in local context map", localJetty.first.ctx_handle); | 261 | + HCCL_VM_ERROR("can not find context {} in local context map", localJetty.first.ctx_handle); |
| 264 | return -1; | 262 | return -1; |
| 265 | } | 263 | } |
| 266 | 264 | ||
| 267 | auto localEp = ctxRes->endpoint_id; | 265 | auto localEp = ctxRes->endpoint_id; |
| 268 | auto endPointOpt = RunnerDB::GetById<sim::EndPoint>(localEp); | 266 | auto endPointOpt = RunnerDB::GetById<sim::EndPoint>(localEp); |
| 269 | if (!endPointOpt.has_value()) { | 267 | if (!endPointOpt.has_value()) { |
| 270 | - CCU_STUB_ERROR("can not find endpoint:{:d}", localEp); | 268 | + HCCL_VM_ERROR("can not find endpoint:{:d}", localEp); |
| 271 | return -1; | 269 | return -1; |
| 272 | } | 270 | } |
| 273 | endPoint = *endPointOpt; | 271 | endPoint = *endPointOpt; |
| @@ -279,7 +277,7 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 279 | { | 277 | { |
| 280 | sim::Device locDevice{}; | 278 | sim::Device locDevice{}; |
| 281 | if (GetDeviceByPhysicalId(deviceId, locDevice) != ACL_SUCCESS) { | 279 | if (GetDeviceByPhysicalId(deviceId, locDevice) != ACL_SUCCESS) { |
| 282 | - CCU_STUB_ERROR("get device by physic id {} failed.", deviceId); | 280 | + HCCL_VM_ERROR("get device by physic id {} failed.", deviceId); |
| 283 | return -1; | 281 | return -1; |
| 284 | } | 282 | } |
| 285 | uint8_t dieId = input->data.data_info.udie_idx; | 283 | uint8_t dieId = input->data.data_info.udie_idx; |
| @@ -300,11 +298,11 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 300 | 298 | ||
| 301 | sim::EndPoint rmtEndPoint{}; | 299 | sim::EndPoint rmtEndPoint{}; |
| 302 | if (GetEndPointByIpAddr(ipAddr, rmtEndPoint) != 0) { | 300 | if (GetEndPointByIpAddr(ipAddr, rmtEndPoint) != 0) { |
| 303 | - CCU_STUB_ERROR("Get remote endpoint failed. ip:{}, eid:{}", ipAddr.c_str(), addr.EidToHexString()); | 301 | + HCCL_VM_ERROR("Get remote endpoint failed. ip:{}, eid:{}", ipAddr.c_str(), addr.EidToHexString()); |
| 304 | return -1; | 302 | return -1; |
| 305 | } | 303 | } |
| 306 | 304 | ||
| 307 | - CCU_STUB_INFO( | 305 | + HCCL_VM_INFO( |
| 308 | "channel info: loc phyId: {:d}, loc devKey: {:d}, loc dieId: {:d}, chId: {:d}, rmt devKey: {:d}, rmt dieId: {:d}, rmt ip: {}", | 306 | "channel info: loc phyId: {:d}, loc devKey: {:d}, loc dieId: {:d}, chId: {:d}, rmt devKey: {:d}, rmt dieId: {:d}, rmt ip: {}", |
| 309 | deviceId, | 307 | deviceId, |
| 310 | locDevice.id, | 308 | locDevice.id, |
| @@ -319,7 +317,7 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 319 | return r.device_id == locDevKey; | 317 | return r.device_id == locDevKey; |
| 320 | }); | 318 | }); |
| 321 | if (!locRank.second) { | 319 | if (!locRank.second) { |
| 322 | - CCU_STUB_ERROR("can not find loc rank by device key: {}", locDevKey); | 320 | + HCCL_VM_ERROR("can not find loc rank by device key: {}", locDevKey); |
| 323 | return -1; | 321 | return -1; |
| 324 | } | 322 | } |
| 325 | 323 | ||
| @@ -328,7 +326,7 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 328 | return r.device_id == rmtDevKey; | 326 | return r.device_id == rmtDevKey; |
| 329 | }); | 327 | }); |
| 330 | if (!rmtRank.second) { | 328 | if (!rmtRank.second) { |
| 331 | - CCU_STUB_ERROR("can not find rmt rank by device key: {}", rmtDevKey); | 329 | + HCCL_VM_ERROR("can not find rmt rank by device key: {}", rmtDevKey); |
| 332 | return -1; | 330 | return -1; |
| 333 | } | 331 | } |
| 334 | uint16_t srcJettyId {0}; | 332 | uint16_t srcJettyId {0}; |
| @@ -343,11 +341,11 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 343 | // 根据endPointPair获取src eid | 341 | // 根据endPointPair获取src eid |
| 344 | sim::EndPoint localEndPoint{}; | 342 | sim::EndPoint localEndPoint{}; |
| 345 | if (GetLocalEndPointByJetty(srcJettyId, srcDieId, localEndPoint) != 0) { | 343 | if (GetLocalEndPointByJetty(srcJettyId, srcDieId, localEndPoint) != 0) { |
| 346 | - CCU_STUB_ERROR("can not find local endPoint by jettyId: {}", srcJettyId); | 344 | + HCCL_VM_ERROR("can not find local endPoint by jettyId: {}", srcJettyId); |
| 347 | return -1; | 345 | return -1; |
| 348 | } | 346 | } |
| 349 | 347 | ||
| 350 | - CCU_STUB_INFO("[CHANNEL] add chn:{:d}, srcJetty:{:d},jettyNum:{:d}, srcAddr:{}, dstAddr:{}, {:d}<-->{:d}", | 348 | + HCCL_VM_INFO("add chn:{:d}, srcJetty:{:d},jettyNum:{:d}, srcAddr:{}, dstAddr:{}, {:d}<-->{:d}", |
| 351 | chId, srcJettyId, jettyNum, localEndPoint.ip_addr, rmtEndPoint.ip_addr, localEndPoint.id, rmtEndPoint.id); | 349 | chId, srcJettyId, jettyNum, localEndPoint.ip_addr, rmtEndPoint.ip_addr, localEndPoint.id, rmtEndPoint.id); |
| 352 | 350 | ||
| 353 | sim::CcuChannelTab ccuChannelTab{}; | 351 | sim::CcuChannelTab ccuChannelTab{}; |
| @@ -366,7 +364,7 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 366 | } | 364 | } |
| 367 | auto ret = sim::InsertCcuChannel(ccuChannelTab); | 365 | auto ret = sim::InsertCcuChannel(ccuChannelTab); |
| 368 | if (ret != 0) { | 366 | if (ret != 0) { |
| 369 | - CCU_STUB_ERROR("insert ccu channel table failed for channel id: {}", chId); | 367 | + HCCL_VM_ERROR("insert ccu channel table failed for channel id: {}", chId); |
| 370 | return -1; | 368 | return -1; |
| 371 | } | 369 | } |
| 372 | 370 | ||
| @@ -375,7 +373,7 @@ int ConfigChannelInfo(channel_info_in *input, uint32_t deviceId) | |||
| 375 | 373 | ||
| 376 | int ConfigJettyInfo(channel_info_in *input, uint32_t deviceId) | 374 | int ConfigJettyInfo(channel_info_in *input, uint32_t deviceId) |
| 377 | { | 375 | { |
| 378 | - CCU_STUB_INFO("Enter into config jetty info..."); | 376 | + HCCL_VM_INFO("Enter into config jetty info..."); |
| 379 | uint8_t dieId = input->data.data_info.udie_idx; | 377 | uint8_t dieId = input->data.data_info.udie_idx; |
| 380 | uint32_t jettyNum = input->data.data_info.data_array_size; | 378 | uint32_t jettyNum = input->data.data_info.data_array_size; |
| 381 | uint32_t startJettyCtxId = input->offset_start; | 379 | uint32_t startJettyCtxId = input->offset_start; |
| @@ -388,20 +386,20 @@ int ConfigJettyInfo(channel_info_in *input, uint32_t deviceId) | |||
| 388 | } | 386 | } |
| 389 | 387 | ||
| 390 | for (auto &tmp : jettyCtxData) { | 388 | for (auto &tmp : jettyCtxData) { |
| 391 | - CCU_STUB_DEBUG("doorbellAddr: [3]0x{:04x}, [2]0x{:04x}, [1]0x{:04x}, [0]0x{:04x}", | 389 | + HCCL_VM_DEBUG("doorbellAddr: [3]0x{:04x}, [2]0x{:04x}, [1]0x{:04x}, [0]0x{:04x}", |
| 392 | tmp.doorbellAddr[3], // 3: doorbell ��ַ���� | 390 | tmp.doorbellAddr[3], // 3: doorbell ��ַ���� |
| 393 | tmp.doorbellAddr[2], // 2: doorbell ��ַ���� | 391 | tmp.doorbellAddr[2], // 2: doorbell ��ַ���� |
| 394 | tmp.doorbellAddr[1], | 392 | tmp.doorbellAddr[1], |
| 395 | tmp.doorbellAddr[0]); | 393 | tmp.doorbellAddr[0]); |
| 396 | 394 | ||
| 397 | // ��ȫ���⣺��ֹ��ӡtoken�����Ϣ | 395 | // ��ȫ���⣺��ֹ��ӡtoken�����Ϣ |
| 398 | - CCU_STUB_DEBUG("pfeIdx: 0x{:04x}, ioDieId: 0x{:04x}, doorbellAddrType: 0x{:04x}, tokenValueIsValid: 0x{:04x}", | 396 | + HCCL_VM_DEBUG("pfeIdx: 0x{:04x}, ioDieId: 0x{:04x}, doorbellAddrType: 0x{:04x}, tokenValueIsValid: 0x{:04x}", |
| 399 | static_cast<uint16_t>(tmp.pfeIdx), | 397 | static_cast<uint16_t>(tmp.pfeIdx), |
| 400 | static_cast<uint16_t>(tmp.ioDieId), | 398 | static_cast<uint16_t>(tmp.ioDieId), |
| 401 | static_cast<uint16_t>(tmp.doorbellAddrType), | 399 | static_cast<uint16_t>(tmp.doorbellAddrType), |
| 402 | static_cast<uint16_t>(tmp.tokenValueIsValid)); | 400 | static_cast<uint16_t>(tmp.tokenValueIsValid)); |
| 403 | 401 | ||
| 404 | - CCU_STUB_DEBUG("sqeBasicBlockLeftShifts: 0x{:04x}, pi: 0x{:04x}, ci: 0x{:04x}, " | 402 | + HCCL_VM_DEBUG("sqeBasicBlockLeftShifts: 0x{:04x}, pi: 0x{:04x}, ci: 0x{:04x}, " |
| 405 | "maxCi: 0x{:04x}, oooCqeCnt: 0x{:04x}, startWqeBasicBlockIdxLow: 0x{:04x}, " | 403 | "maxCi: 0x{:04x}, oooCqeCnt: 0x{:04x}, startWqeBasicBlockIdxLow: 0x{:04x}, " |
| 406 | "startWqeBasicBlockIdxHigh: 0x{:04x}, doorbellSendState: 0x{:04x}", | 404 | "startWqeBasicBlockIdxHigh: 0x{:04x}, doorbellSendState: 0x{:04x}", |
| 407 | static_cast<uint16_t>(tmp.sqeBasicBlockLeftShifts), | 405 | static_cast<uint16_t>(tmp.sqeBasicBlockLeftShifts), |
| @@ -421,13 +419,13 @@ int GetCcuVersion(channel_info_out* output, uint32_t deviceId) | |||
| 421 | { | 419 | { |
| 422 | sim::Device locDevice{}; | 420 | sim::Device locDevice{}; |
| 423 | if (GetDeviceByPhysicalId(deviceId, locDevice) != ACL_SUCCESS) { | 421 | if (GetDeviceByPhysicalId(deviceId, locDevice) != ACL_SUCCESS) { |
| 424 | - CCU_STUB_ERROR("get device by physic id {} failed.", deviceId); | 422 | + HCCL_VM_ERROR("get device by physic id {} failed.", deviceId); |
| 425 | return -1; | 423 | return -1; |
| 426 | } | 424 | } |
| 427 | if (strcmp(locDevice.soc_version, "Ascend950") == 0) { | 425 | if (strcmp(locDevice.soc_version, "Ascend950") == 0) { |
| 428 | output->data.data_info.data_array[0].version = static_cast<ccu_version_e>(CcuVersion::CCU_V1); | 426 | output->data.data_info.data_array[0].version = static_cast<ccu_version_e>(CcuVersion::CCU_V1); |
| 429 | } else { | 427 | } else { |
| 430 | - CCU_STUB_ERROR("unknown soc version: {}", locDevice.soc_version); | 428 | + HCCL_VM_ERROR("unknown soc version: {}", locDevice.soc_version); |
| 431 | return -1; | 429 | return -1; |
| 432 | } | 430 | } |
| 433 | return 0; | 431 | return 0; |
| @@ -469,7 +467,7 @@ int RaCtxGetAsyncEvents(void *ctxHandle, struct AsyncEvent events[], unsigned in | |||
| 469 | 467 | ||
| 470 | int RaGetEidByIp(void *ctxHandle, struct IpInfo ip[], union HccpEid eid[], unsigned int *num) | 468 | int RaGetEidByIp(void *ctxHandle, struct IpInfo ip[], union HccpEid eid[], unsigned int *num) |
| 471 | { | 469 | { |
| 472 | - CCU_STUB_ERROR("not support for now."); | 470 | + HCCL_VM_ERROR("not support for now."); |
| 473 | return -1; | 471 | return -1; |
| 474 | } | 472 | } |
| 475 | 473 | ||
| @@ -477,7 +475,7 @@ int GetAllUsedEndPoint(uint32_t phyDevId, std::vector<sim::EndPoint> &endPoints) | |||
| 477 | { | 475 | { |
| 478 | sim::Device device{}; | 476 | sim::Device device{}; |
| 479 | if (GetDeviceByPhysicalId(phyDevId, device) != 0) { | 477 | if (GetDeviceByPhysicalId(phyDevId, device) != 0) { |
| 480 | - CCU_STUB_ERROR("can not find device by id:{:d}", phyDevId); | 478 | + HCCL_VM_ERROR("can not find device by id:{:d}", phyDevId); |
| 481 | return -1; | 479 | return -1; |
| 482 | } | 480 | } |
| 483 | 481 | ||
| @@ -494,7 +492,7 @@ int RaGetDevEidInfoNum(struct RaInfo info, unsigned int *num) | |||
| 494 | if (GetAllUsedEndPoint(info.phyId, endPoints) != 0) { | 492 | if (GetAllUsedEndPoint(info.phyId, endPoints) != 0) { |
| 495 | return -1; | 493 | return -1; |
| 496 | } | 494 | } |
| 497 | - CCU_STUB_INFO("return success {:d}", endPoints.size()); | 495 | + HCCL_VM_INFO("return success {:d}", endPoints.size()); |
| 498 | *num = endPoints.size(); | 496 | *num = endPoints.size(); |
| 499 | return 0; | 497 | return 0; |
| 500 | } | 498 | } |
| @@ -503,7 +501,7 @@ int RaGetDevEidInfoList(struct RaInfo info, struct HccpDevEidInfo infoList[], un | |||
| 503 | { | 501 | { |
| 504 | std::vector<sim::EndPoint> endPoints; | 502 | std::vector<sim::EndPoint> endPoints; |
| 505 | if (GetAllUsedEndPoint(info.phyId, endPoints) != 0) { | 503 | if (GetAllUsedEndPoint(info.phyId, endPoints) != 0) { |
| 506 | - CCU_STUB_ERROR("get all EndPoint failed phyId:{:d}", info.phyId); | 504 | + HCCL_VM_ERROR("get all EndPoint failed phyId:{:d}", info.phyId); |
| 507 | return -1; | 505 | return -1; |
| 508 | } | 506 | } |
| 509 | 507 | ||
| @@ -547,7 +545,7 @@ int rtCCULaunch(rtCcuTaskInfo_t *taskInfo, rtStream_t const stream) | |||
| 547 | uint32_t index{0}; | 545 | uint32_t index{0}; |
| 548 | auto ret = InsertTaskToCollection(&taskMetaData, &index); | 546 | auto ret = InsertTaskToCollection(&taskMetaData, &index); |
| 549 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { | 547 | if (ret != HcclSim::HcclVmResult::HCCL_SIM_SUCCESS) { |
| 550 | - CCU_STUB_ERROR("insert task fail"); | 548 | + HCCL_VM_ERROR("insert task fail"); |
| 551 | return ACL_ERROR_INTERNAL_ERROR; | 549 | return ACL_ERROR_INTERNAL_ERROR; |
| 552 | } | 550 | } |
| 553 | 551 | ||
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -33,10 +36,6 @@ constexpr uint32_t RA_SOCKET_BUF_SIZE = (64 * 1024); | |||
| 33 | 36 | ||
| 34 | constexpr const char* RA_SOCK_BUF_PREFIX = "ra_sock_"; | 37 | constexpr const char* RA_SOCK_BUF_PREFIX = "ra_sock_"; |
| 35 | 38 | ||
| 36 | - | ||
| 37 | - | ||
| 38 | - | ||
| 39 | - | ||
| 40 | 39 | ||
| 41 | 40 | ||
| 42 | 41 | ||
| @@ -65,7 +64,7 @@ static bool GenRaSocketBufKeyByPairId(uint64_t pairId) | |||
| 65 | std::string c2s = RA_SOCK_BUF_PREFIX + std::to_string(pairId) + "_c2s"; | 64 | std::string c2s = RA_SOCK_BUF_PREFIX + std::to_string(pairId) + "_c2s"; |
| 66 | void *ptrC2s = sim::CommunicationMemoryManager::GetInstance().AllocCommMem(c2s.data()); | 65 | void *ptrC2s = sim::CommunicationMemoryManager::GetInstance().AllocCommMem(c2s.data()); |
| 67 | if (ptrC2s == nullptr) { | 66 | if (ptrC2s == nullptr) { |
| 68 | - RA_SOCK_STUB_ERROR(" alloc sock name {} mem failed", c2s.data()); | 67 | + HCCL_VM_ERROR(" alloc sock name {} mem failed", c2s.data()); |
| 69 | return false; | 68 | return false; |
| 70 | } | 69 | } |
| 71 | 70 | ||
| @@ -73,7 +72,7 @@ static bool GenRaSocketBufKeyByPairId(uint64_t pairId) | |||
| 73 | void *ptrS2c = sim::CommunicationMemoryManager::GetInstance().AllocCommMem(s2c.data()); | 72 | void *ptrS2c = sim::CommunicationMemoryManager::GetInstance().AllocCommMem(s2c.data()); |
| 74 | if (ptrS2c == nullptr) { | 73 | if (ptrS2c == nullptr) { |
| 75 | sim::CommunicationMemoryManager::GetInstance().ReleaseCommMem(c2s.data()); | 74 | sim::CommunicationMemoryManager::GetInstance().ReleaseCommMem(c2s.data()); |
| 76 | - RA_SOCK_STUB_ERROR(" alloc sock name {} mem failed", s2c.data()); | 75 | + HCCL_VM_ERROR(" alloc sock name {} mem failed", s2c.data()); |
| 77 | return false; | 76 | return false; |
| 78 | } | 77 | } |
| 79 | return true; | 78 | return true; |
| @@ -84,7 +83,7 @@ static bool AcquireRaSocketBufKeyByPairId(uint64_t pairId) | |||
| 84 | std::string c2s = RA_SOCK_BUF_PREFIX + std::to_string(pairId) + "_c2s"; | 83 | std::string c2s = RA_SOCK_BUF_PREFIX + std::to_string(pairId) + "_c2s"; |
| 85 | void *ptrC2s = sim::CommunicationMemoryManager::GetInstance().AcquireCommMem(c2s.data()); | 84 | void *ptrC2s = sim::CommunicationMemoryManager::GetInstance().AcquireCommMem(c2s.data()); |
| 86 | if (ptrC2s == nullptr) { | 85 | if (ptrC2s == nullptr) { |
| 87 | - RA_SOCK_STUB_ERROR(" acquire sock name {} mem failed", c2s.data()); | 86 | + HCCL_VM_ERROR(" acquire sock name {} mem failed", c2s.data()); |
| 88 | return false; | 87 | return false; |
| 89 | } | 88 | } |
| 90 | 89 | ||
| @@ -92,7 +91,7 @@ static bool AcquireRaSocketBufKeyByPairId(uint64_t pairId) | |||
| 92 | void *ptrS2c = sim::CommunicationMemoryManager::GetInstance().AcquireCommMem(s2c.data()); | 91 | void *ptrS2c = sim::CommunicationMemoryManager::GetInstance().AcquireCommMem(s2c.data()); |
| 93 | if (ptrS2c == nullptr) { | 92 | if (ptrS2c == nullptr) { |
| 94 | sim::CommunicationMemoryManager::GetInstance().ReleaseCommMem(c2s.data()); | 93 | sim::CommunicationMemoryManager::GetInstance().ReleaseCommMem(c2s.data()); |
| 95 | - RA_SOCK_STUB_ERROR(" acquire sock name {} mem failed", s2c.data()); | 94 | + HCCL_VM_ERROR(" acquire sock name {} mem failed", s2c.data()); |
| 96 | return false; | 95 | return false; |
| 97 | } | 96 | } |
| 98 | return true; | 97 | return true; |
| @@ -131,7 +130,7 @@ int RaSocketInit(int mode, struct rdev rdevInfo, void **socketHandle) | |||
| 131 | (void) mode; | 130 | (void) mode; |
| 132 | sim::Device device{}; | 131 | sim::Device device{}; |
| 133 | if (GetDeviceByPhysicalId(rdevInfo.phyId, device) != ACL_SUCCESS) { | 132 | if (GetDeviceByPhysicalId(rdevInfo.phyId, device) != ACL_SUCCESS) { |
| 134 | - RA_SOCK_STUB_ERROR(" get device by phy id {} failed.", rdevInfo.phyId); | 133 | + HCCL_VM_ERROR(" get device by phy id {} failed.", rdevInfo.phyId); |
| 135 | return -1; | 134 | return -1; |
| 136 | } | 135 | } |
| 137 | 136 | ||
| @@ -141,7 +140,7 @@ int RaSocketInit(int mode, struct rdev rdevInfo, void **socketHandle) | |||
| 141 | 140 | ||
| 142 | sim::EndPoint endPoint{}; | 141 | sim::EndPoint endPoint{}; |
| 143 | if (GetEndPointByIpAddr(ipAddr, endPoint) != 0) { | 142 | if (GetEndPointByIpAddr(ipAddr, endPoint) != 0) { |
| 144 | - RA_SOCK_STUB_ERROR(" cannot find remote ip {} ", ipAddr); | 143 | + HCCL_VM_ERROR(" cannot find remote ip {} ", ipAddr); |
| 145 | return -1; | 144 | return -1; |
| 146 | } | 145 | } |
| 147 | 146 | ||
| @@ -152,7 +151,7 @@ int RaSocketInit(int mode, struct rdev rdevInfo, void **socketHandle) | |||
| 152 | [deviceIdx, endpointId](const sim::RaSocket &so) { return (so.device_id == deviceIdx && so.endpoint_id == endpointId); }); | 151 | [deviceIdx, endpointId](const sim::RaSocket &so) { return (so.device_id == deviceIdx && so.endpoint_id == endpointId); }); |
| 153 | if (ret.second) { | 152 | if (ret.second) { |
| 154 | *socketHandle = reinterpret_cast<void *>(static_cast<uintptr_t>(ret.first.id)); | 153 | *socketHandle = reinterpret_cast<void *>(static_cast<uintptr_t>(ret.first.id)); |
| 155 | - RA_SOCK_STUB_INFO(" Found socket socketFd:{:d}", ret.first.id); | 154 | + HCCL_VM_INFO(" Found socket socketFd:{:d}", ret.first.id); |
| 156 | return 0; | 155 | return 0; |
| 157 | } | 156 | } |
| 158 | 157 | ||
| @@ -166,15 +165,15 @@ int RaSocketInit(int mode, struct rdev rdevInfo, void **socketHandle) | |||
| 166 | [deviceIdx, endpointId](const sim::RaSocket &so) { return (so.device_id == deviceIdx && so.endpoint_id == endpointId); }); | 165 | [deviceIdx, endpointId](const sim::RaSocket &so) { return (so.device_id == deviceIdx && so.endpoint_id == endpointId); }); |
| 167 | if (retry.second) { | 166 | if (retry.second) { |
| 168 | *socketHandle = reinterpret_cast<void *>(static_cast<uintptr_t>(retry.first.id)); | 167 | *socketHandle = reinterpret_cast<void *>(static_cast<uintptr_t>(retry.first.id)); |
| 169 | - RA_SOCK_STUB_INFO(" Insert failed, found existing socketFd:{:d}", retry.first.id); | 168 | + HCCL_VM_WARN(" Insert conflict, reused existing socketFd:{:d}", retry.first.id); |
| 170 | return 0; | 169 | return 0; |
| 171 | } | 170 | } |
| 172 | - RA_SOCK_STUB_ERROR(" Insert and re-lookup both failed for device:{:d} endpointId:{:d}", deviceIdx, endpointId); | 171 | + HCCL_VM_ERROR(" Insert and re-lookup both failed for device:{:d} endpointId:{:d}", deviceIdx, endpointId); |
| 173 | return -1; | 172 | return -1; |
| 174 | } | 173 | } |
| 175 | 174 | ||
| 176 | *socketHandle = reinterpret_cast<void *>(static_cast<uintptr_t>(socketfd)); | 175 | *socketHandle = reinterpret_cast<void *>(static_cast<uintptr_t>(socketfd)); |
| 177 | - RA_SOCK_STUB_INFO(" add socketHandle:{:d} device:{:d} ip:{} endpointId:{:d}", socketfd, deviceIdx, ipAddr, endpointId); | 176 | + HCCL_VM_INFO(" add socketHandle:{:d} device:{:d} ip:{} endpointId:{:d}", socketfd, deviceIdx, ipAddr, endpointId); |
| 178 | return 0; | 177 | return 0; |
| 179 | } | 178 | } |
| 180 | 179 | ||
| @@ -191,7 +190,7 @@ int RaSocketDeinit(void *socketHandle) | |||
| 191 | uint64_t localFd = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(socketHandle)); | 190 | uint64_t localFd = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(socketHandle)); |
| 192 | RunnerDB::Delete<sim::RaSocket>(localFd); | 191 | RunnerDB::Delete<sim::RaSocket>(localFd); |
| 193 | 192 | ||
| 194 | - RA_SOCK_STUB_INFO(" delete socketHandle:{:d}", localFd); | 193 | + HCCL_VM_INFO(" delete socketHandle:{:d}", localFd); |
| 195 | return 0; | 194 | return 0; |
| 196 | } | 195 | } |
| 197 | 196 | ||
| @@ -201,13 +200,13 @@ int RaSocketListenStart(struct SocketListenInfoT conn[], uint32_t num) | |||
| 201 | uint64_t socketHandle = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); | 200 | uint64_t socketHandle = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); |
| 202 | auto socketRes = RunnerDB::GetById<sim::RaSocket>(socketHandle); | 201 | auto socketRes = RunnerDB::GetById<sim::RaSocket>(socketHandle); |
| 203 | if (!socketRes.has_value()) { | 202 | if (!socketRes.has_value()) { |
| 204 | - RA_SOCK_STUB_ERROR(" can not get Socket:{:d}", socketHandle); | 203 | + HCCL_VM_ERROR(" can not get Socket:{:d}", socketHandle); |
| 205 | return -1; | 204 | return -1; |
| 206 | } | 205 | } |
| 207 | RunnerDB::Update<sim::RaSocket>(socketHandle, [](sim::RaSocket &s) { | 206 | RunnerDB::Update<sim::RaSocket>(socketHandle, [](sim::RaSocket &s) { |
| 208 | s.state = 1; | 207 | s.state = 1; |
| 209 | }); | 208 | }); |
| 210 | - RA_SOCK_STUB_INFO(" socket {:d} port:{:d}", socketHandle, conn[i].port); | 209 | + HCCL_VM_INFO(" socket {:d} port:{:d}", socketHandle, conn[i].port); |
| 211 | } | 210 | } |
| 212 | return 0; | 211 | return 0; |
| 213 | } | 212 | } |
| @@ -218,13 +217,13 @@ int RaSocketListenStop(struct SocketListenInfoT conn[], unsigned int num) | |||
| 218 | uint64_t socketHandle = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); | 217 | uint64_t socketHandle = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); |
| 219 | auto socketRes = RunnerDB::GetById<sim::RaSocket>(socketHandle); | 218 | auto socketRes = RunnerDB::GetById<sim::RaSocket>(socketHandle); |
| 220 | if (!socketRes.has_value()) { | 219 | if (!socketRes.has_value()) { |
| 221 | - RA_SOCK_STUB_ERROR(" can not get Socket:{:d}", socketHandle); | 220 | + HCCL_VM_ERROR(" can not get Socket:{:d}", socketHandle); |
| 222 | return -1; | 221 | return -1; |
| 223 | } | 222 | } |
| 224 | RunnerDB::Update<sim::RaSocket>(socketHandle, [](sim::RaSocket &s) { | 223 | RunnerDB::Update<sim::RaSocket>(socketHandle, [](sim::RaSocket &s) { |
| 225 | s.state = 0; | 224 | s.state = 0; |
| 226 | }); | 225 | }); |
| 227 | - RA_SOCK_STUB_INFO(" socket {:d} port:{:d}", socketHandle, conn[i].port); | 226 | + HCCL_VM_INFO(" socket {:d} port:{:d}", socketHandle, conn[i].port); |
| 228 | } | 227 | } |
| 229 | return 0; | 228 | return 0; |
| 230 | } | 229 | } |
| @@ -235,7 +234,7 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 235 | uint64_t localFd = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); | 234 | uint64_t localFd = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); |
| 236 | auto localSock = RunnerDB::GetById<sim::RaSocket>(localFd); | 235 | auto localSock = RunnerDB::GetById<sim::RaSocket>(localFd); |
| 237 | if (!localSock.has_value()) { | 236 | if (!localSock.has_value()) { |
| 238 | - RA_SOCK_STUB_ERROR(" can not get Local Ra Socket:{:d}", localFd); | 237 | + HCCL_VM_ERROR(" can not get Local Ra Socket:{:d}", localFd); |
| 239 | return -1; | 238 | return -1; |
| 240 | } | 239 | } |
| 241 | 240 | ||
| @@ -244,7 +243,7 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 244 | auto ipAddr = IpAddress(ba, AF_INET6).GetIpStr().substr(2); | 243 | auto ipAddr = IpAddress(ba, AF_INET6).GetIpStr().substr(2); |
| 245 | sim::EndPoint endPoint{}; | 244 | sim::EndPoint endPoint{}; |
| 246 | if (GetEndPointByIpAddr(ipAddr, endPoint) != 0) { | 245 | if (GetEndPointByIpAddr(ipAddr, endPoint) != 0) { |
| 247 | - RA_SOCK_STUB_ERROR(" cannot find remote ip:{}", ipAddr); | 246 | + HCCL_VM_ERROR(" cannot find remote ip:{}", ipAddr); |
| 248 | return -1; | 247 | return -1; |
| 249 | } | 248 | } |
| 250 | 249 | ||
| @@ -253,11 +252,11 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 253 | uint64_t tagHash = ComputeTagHash(conn[i].tag); | 252 | uint64_t tagHash = ComputeTagHash(conn[i].tag); |
| 254 | uint32_t remotePort = conn[i].port; | 253 | uint32_t remotePort = conn[i].port; |
| 255 | 254 | ||
| 256 | - RA_SOCK_STUB_INFO(" dev:{} sock:{} connect dev:{} ip addr:{} tag:{}", localSock->device_id, localFd, remoteDevId, ipAddr, conn[i].tag); | 255 | + HCCL_VM_INFO(" dev:{} sock:{} connect dev:{} ip addr:{} tag:{}", localSock->device_id, localFd, remoteDevId, ipAddr, conn[i].tag); |
| 257 | uint32_t count = 0; | 256 | uint32_t count = 0; |
| 258 | while (true) { | 257 | while (true) { |
| 259 | if (count++ >= 600) { // 超时60s | 258 | if (count++ >= 600) { // 超时60s |
| 260 | - RA_SOCK_STUB_ERROR(" can not get break dev:{} sock:{} connect dev:{} ip addr:{} tag:{}", localSock->device_id, localFd, remoteDevId, ipAddr, conn[i].tag); | 259 | + HCCL_VM_ERROR(" can not get break dev:{} sock:{} connect dev:{} ip addr:{} tag:{}", localSock->device_id, localFd, remoteDevId, ipAddr, conn[i].tag); |
| 261 | break; | 260 | break; |
| 262 | } | 261 | } |
| 263 | auto remoteSockRes = RunnerDB::GetOneByPred<sim::RaSocket>( | 262 | auto remoteSockRes = RunnerDB::GetOneByPred<sim::RaSocket>( |
| @@ -266,7 +265,7 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 266 | }); | 265 | }); |
| 267 | if (!remoteSockRes.second) { | 266 | if (!remoteSockRes.second) { |
| 268 | if (count % 100 == 0) { | 267 | if (count % 100 == 0) { |
| 269 | - RA_SOCK_STUB_WARN(" can not find remote dev:{}, endpoint:{}", remoteDevId, remoteEndPointId); | 268 | + HCCL_VM_WARN(" can not find remote dev:{}, endpoint:{}", remoteDevId, remoteEndPointId); |
| 270 | } | 269 | } |
| 271 | 270 | ||
| 272 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); | 271 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); |
| @@ -285,7 +284,7 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 285 | uint64_t pairId = 0; | 284 | uint64_t pairId = 0; |
| 286 | if (existPairRes.second) { | 285 | if (existPairRes.second) { |
| 287 | pairId = existPairRes.first.id; | 286 | pairId = existPairRes.first.id; |
| 288 | - RA_SOCK_STUB_INFO(" pair already exists id:{:d}", pairId); | 287 | + HCCL_VM_INFO(" pair already exists id:{:d}", pairId); |
| 289 | } else { | 288 | } else { |
| 290 | sim::RaSocketPair socketpair{}; | 289 | sim::RaSocketPair socketpair{}; |
| 291 | socketpair.client_id = localFd; | 290 | socketpair.client_id = localFd; |
| @@ -296,7 +295,7 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 296 | socketpair.buf_status = 0; | 295 | socketpair.buf_status = 0; |
| 297 | pairId = RunnerDB::Add<sim::RaSocketPair>(socketpair); | 296 | pairId = RunnerDB::Add<sim::RaSocketPair>(socketpair); |
| 298 | if (pairId == 0) { | 297 | if (pairId == 0) { |
| 299 | - RA_SOCK_STUB_WARN(" add failed retry"); | 298 | + HCCL_VM_WARN(" add failed retry"); |
| 300 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); | 299 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); |
| 301 | continue; | 300 | continue; |
| 302 | } | 301 | } |
| @@ -311,7 +310,7 @@ int RaSocketBatchConnect(struct SocketConnectInfoT conn[], unsigned int num) | |||
| 311 | std::string sendBuf, recvBuf; | 310 | std::string sendBuf, recvBuf; |
| 312 | GetRaSendSocketkeyByFd(pairId, sendBuf); | 311 | GetRaSendSocketkeyByFd(pairId, sendBuf); |
| 313 | GetRaRecvSocketkeyByFd(pairId, recvBuf); | 312 | GetRaRecvSocketkeyByFd(pairId, recvBuf); |
| 314 | - RA_SOCK_STUB_INFO(" add pair id:{:d} local:{:d} peer:{:d} port:{:d} sendBuf:{}, recvBuf:{}", | 313 | + HCCL_VM_INFO(" add pair id:{:d} local:{:d} peer:{:d} port:{:d} sendBuf:{}, recvBuf:{}", |
| 315 | pairId, localFd, peerFd, remotePort, sendBuf, recvBuf); | 314 | pairId, localFd, peerFd, remotePort, sendBuf, recvBuf); |
| 316 | } | 315 | } |
| 317 | break; | 316 | break; |
| @@ -327,7 +326,7 @@ int RaGetSockets(unsigned int role, struct SocketInfoT conn[], unsigned int num, | |||
| 327 | uint64_t loadFd = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); | 326 | uint64_t loadFd = static_cast<uint64_t>(reinterpret_cast<uintptr_t>(conn[i].socketHandle)); |
| 328 | auto localSock = RunnerDB::GetById<sim::RaSocket>(loadFd); | 327 | auto localSock = RunnerDB::GetById<sim::RaSocket>(loadFd); |
| 329 | if (!localSock.has_value()) { | 328 | if (!localSock.has_value()) { |
| 330 | - RA_SOCK_STUB_ERROR(" can not get local socket fd:{:d}", loadFd); | 329 | + HCCL_VM_ERROR(" can not get local socket fd:{:d}", loadFd); |
| 331 | return -1; | 330 | return -1; |
| 332 | } | 331 | } |
| 333 | 332 | ||
| @@ -336,7 +335,7 @@ int RaGetSockets(unsigned int role, struct SocketInfoT conn[], unsigned int num, | |||
| 336 | auto ipAddr = IpAddress(ba, AF_INET6).GetIpStr().substr(2); | 335 | auto ipAddr = IpAddress(ba, AF_INET6).GetIpStr().substr(2); |
| 337 | sim::EndPoint endPoint{}; | 336 | sim::EndPoint endPoint{}; |
| 338 | if (GetEndPointByIpAddr(ipAddr, endPoint) != 0) { | 337 | if (GetEndPointByIpAddr(ipAddr, endPoint) != 0) { |
| 339 | - RA_SOCK_STUB_ERROR(" cannot find remote ip:{}", ipAddr); | 338 | + HCCL_VM_ERROR(" cannot find remote ip:{}", ipAddr); |
| 340 | continue; | 339 | continue; |
| 341 | } | 340 | } |
| 342 | 341 | ||
| @@ -344,7 +343,7 @@ int RaGetSockets(unsigned int role, struct SocketInfoT conn[], unsigned int num, | |||
| 344 | auto remoteEndpointId = endPoint.id; | 343 | auto remoteEndpointId = endPoint.id; |
| 345 | uint64_t tagHash = ComputeTagHash(conn[i].tag); | 344 | uint64_t tagHash = ComputeTagHash(conn[i].tag); |
| 346 | 345 | ||
| 347 | - RA_SOCK_STUB_INFO(" dev:{} sock:{} get remote dev:{} addr:{} tag:{}", localSock->device_id, loadFd, remoteDevId, ipAddr, conn[i].tag); | 346 | + HCCL_VM_INFO(" dev:{} sock:{} get remote dev:{} addr:{} tag:{}", localSock->device_id, loadFd, remoteDevId, ipAddr, conn[i].tag); |
| 348 | 347 | ||
| 349 | uint32_t count = 0; | 348 | uint32_t count = 0; |
| 350 | bool found = false; | 349 | bool found = false; |
| @@ -355,7 +354,7 @@ int RaGetSockets(unsigned int role, struct SocketInfoT conn[], unsigned int num, | |||
| 355 | }); | 354 | }); |
| 356 | if (!remoteSockRes.second) { | 355 | if (!remoteSockRes.second) { |
| 357 | if (count % 60 == 0) { | 356 | if (count % 60 == 0) { |
| 358 | - RA_SOCK_STUB_WARN(" waiting for remote socket dev:{}, endpoint:{}, attempt:{}/180", | 357 | + HCCL_VM_WARN(" waiting for remote socket dev:{}, endpoint:{}, attempt:{}/180", |
| 359 | remoteDevId, remoteEndpointId, count); | 358 | remoteDevId, remoteEndpointId, count); |
| 360 | } | 359 | } |
| 361 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); | 360 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); |
| @@ -373,7 +372,7 @@ int RaGetSockets(unsigned int role, struct SocketInfoT conn[], unsigned int num, | |||
| 373 | 372 | ||
| 374 | if (!pairRes.second) { | 373 | if (!pairRes.second) { |
| 375 | if (count % 60 == 0) { | 374 | if (count % 60 == 0) { |
| 376 | - RA_SOCK_STUB_WARN(" waiting for socket pair local:{:d} peer:{:d} tag_hash:{:d}, attempt:{}/180", | 375 | + HCCL_VM_WARN(" waiting for socket pair local:{:d} peer:{:d} tag_hash:{:d}, attempt:{}/180", |
| 377 | loadFd, peerFd, tagHash, count); | 376 | loadFd, peerFd, tagHash, count); |
| 378 | } | 377 | } |
| 379 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); | 378 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); |
| @@ -393,13 +392,13 @@ int RaGetSockets(unsigned int role, struct SocketInfoT conn[], unsigned int num, | |||
| 393 | *connectedNum += 1; | 392 | *connectedNum += 1; |
| 394 | found = true; | 393 | found = true; |
| 395 | 394 | ||
| 396 | - RA_SOCK_STUB_INFO(" get socket local:{:d} peer:{:d} pairId:{:d} role:{:d}", | 395 | + HCCL_VM_INFO(" get socket local:{:d} peer:{:d} pairId:{:d} role:{:d}", |
| 397 | loadFd, peerFd, pairId, role); | 396 | loadFd, peerFd, pairId, role); |
| 398 | break; | 397 | break; |
| 399 | } | 398 | } |
| 400 | 399 | ||
| 401 | if (!found) { | 400 | if (!found) { |
| 402 | - RA_SOCK_STUB_ERROR(" get socket failed local:{:d} peerAddr:{} role:{:d}", loadFd, ipAddr, role); | 401 | + HCCL_VM_ERROR(" get socket failed local:{:d} peerAddr:{} role:{:d}", loadFd, ipAddr, role); |
| 403 | } | 402 | } |
| 404 | } | 403 | } |
| 405 | return 0; | 404 | return 0; |
| @@ -413,7 +412,7 @@ int RaSocketBatchClose(struct SocketCloseInfoT conn[], unsigned int num) | |||
| 413 | 412 | ||
| 414 | auto pairRes = RunnerDB::GetById<sim::RaSocketPair>(pairId); | 413 | auto pairRes = RunnerDB::GetById<sim::RaSocketPair>(pairId); |
| 415 | if (!pairRes.has_value()) { | 414 | if (!pairRes.has_value()) { |
| 416 | - RA_SOCK_STUB_WARN(" pair {:d} not found during close", pairId); | 415 | + HCCL_VM_WARN(" pair {:d} not found during close", pairId); |
| 417 | continue; | 416 | continue; |
| 418 | } | 417 | } |
| 419 | 418 | ||
| @@ -423,10 +422,10 @@ int RaSocketBatchClose(struct SocketCloseInfoT conn[], unsigned int num) | |||
| 423 | 422 | ||
| 424 | if (pairRes->ref_cnt > 1) { | 423 | if (pairRes->ref_cnt > 1) { |
| 425 | RunnerDB::Update<sim::RaSocketPair>(pairId, [](sim::RaSocketPair &pair) { pair.ref_cnt -= 1; }); | 424 | RunnerDB::Update<sim::RaSocketPair>(pairId, [](sim::RaSocketPair &pair) { pair.ref_cnt -= 1; }); |
| 426 | - RA_SOCK_STUB_INFO(" reduce pair:{:d} ref_cnt:{:d} sendBuf:{}, recvBuf:{}", pairId, pairRes->ref_cnt, sendBuf, recvBuf); | 425 | + HCCL_VM_INFO(" reduce pair:{:d} ref_cnt:{:d} sendBuf:{}, recvBuf:{}", pairId, pairRes->ref_cnt, sendBuf, recvBuf); |
| 427 | } else { | 426 | } else { |
| 428 | RunnerDB::Delete<sim::RaSocketPair>(pairId); | 427 | RunnerDB::Delete<sim::RaSocketPair>(pairId); |
| 429 | - RA_SOCK_STUB_INFO(" delete pair:{:d} ref_cnt:{:d} sendBuf:{}, recvBuf:{}", pairId, pairRes->ref_cnt, sendBuf, recvBuf); | 428 | + HCCL_VM_INFO(" delete pair:{:d} ref_cnt:{:d} sendBuf:{}, recvBuf:{}", pairId, pairRes->ref_cnt, sendBuf, recvBuf); |
| 430 | } | 429 | } |
| 431 | 430 | ||
| 432 | DestoryRaSocketBufKeyByPairId(pairId); | 431 | DestoryRaSocketBufKeyByPairId(pairId); |
| @@ -449,13 +448,13 @@ int RaSocketSend(const void *fdHandle, const void *data, unsigned long long size | |||
| 449 | 448 | ||
| 450 | int ret = sim::CommunicationMemoryManager::GetInstance().WriteCommMem(socketKey.data(), data, size); | 449 | int ret = sim::CommunicationMemoryManager::GetInstance().WriteCommMem(socketKey.data(), data, size); |
| 451 | if (ret != 0) { | 450 | if (ret != 0) { |
| 452 | - RA_SOCK_STUB_ERROR(" cannot pair socket:{:d} role:{:d}, key={}", FD_PAIR_ID(socketFd), | 451 | + HCCL_VM_ERROR(" cannot pair socket:{:d} role:{:d}, key={}", FD_PAIR_ID(socketFd), |
| 453 | FD_ROLE(socketFd), socketKey); | 452 | FD_ROLE(socketFd), socketKey); |
| 454 | return -1; | 453 | return -1; |
| 455 | } | 454 | } |
| 456 | 455 | ||
| 457 | *sentSize = size; | 456 | *sentSize = size; |
| 458 | - RA_SOCK_STUB_INFO(" pair socket:{:d} role:{:d} key={} Send:{:d}", FD_PAIR_ID(socketFd), | 457 | + HCCL_VM_INFO(" pair socket:{:d} role:{:d} key={} Send:{:d}", FD_PAIR_ID(socketFd), |
| 459 | FD_ROLE(socketFd), socketKey.data(), size); | 458 | FD_ROLE(socketFd), socketKey.data(), size); |
| 460 | 459 | ||
| 461 | return 0; | 460 | return 0; |
| @@ -469,16 +468,16 @@ int RaSocketRecv(const void *fdHandle, void *data, unsigned long long size, unsi | |||
| 469 | 468 | ||
| 470 | int ret = sim::CommunicationMemoryManager::GetInstance().ReadCommMem(socketKey.data(), data, size); | 469 | int ret = sim::CommunicationMemoryManager::GetInstance().ReadCommMem(socketKey.data(), data, size); |
| 471 | if (ret == 0) { | 470 | if (ret == 0) { |
| 472 | - RA_SOCK_STUB_WARN(" socket pair:{:d} role:{:d}, key={} read try again", | 471 | + HCCL_VM_WARN(" socket pair:{:d} role:{:d}, key={} read try again", |
| 473 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); | 472 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); |
| 474 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); | 473 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); |
| 475 | return 0; | 474 | return 0; |
| 476 | } else if (ret == -1) { | 475 | } else if (ret == -1) { |
| 477 | - RA_SOCK_STUB_ERROR(" socket pair:{:d} role:{:d}, key={} recv failed", | 476 | + HCCL_VM_ERROR(" socket pair:{:d} role:{:d}, key={} recv failed", |
| 478 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); | 477 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); |
| 479 | return -1; | 478 | return -1; |
| 480 | } else if (ret > 0) { | 479 | } else if (ret > 0) { |
| 481 | - RA_SOCK_STUB_INFO(" socket pair:{:d} role:{:d}, key={} recv bytes:{:d} ", | 480 | + HCCL_VM_INFO(" socket pair:{:d} role:{:d}, key={} recv bytes:{:d} ", |
| 482 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey, ret); | 481 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey, ret); |
| 483 | } | 482 | } |
| 484 | *receivedSize = ret; | 483 | *receivedSize = ret; |
| @@ -545,7 +544,7 @@ int RaSocketListenStopAsync(struct SocketListenInfoT conn[], unsigned int num, v | |||
| 545 | int RaSocketBatchCloseAsync(struct SocketCloseInfoT conn[], unsigned int num, void **reqHandle) | 544 | int RaSocketBatchCloseAsync(struct SocketCloseInfoT conn[], unsigned int num, void **reqHandle) |
| 546 | { | 545 | { |
| 547 | (void) reqHandle; | 546 | (void) reqHandle; |
| 548 | - RA_SOCK_STUB_INFO(" close socket async"); | 547 | + HCCL_VM_INFO(" close socket async"); |
| 549 | return RaSocketBatchClose(conn, num); | 548 | return RaSocketBatchClose(conn, num); |
| 550 | } | 549 | } |
| 551 | 550 | ||
| @@ -567,18 +566,18 @@ int RaSocketRecvAsync(const void *fdHandle, void *data, unsigned long long size, | |||
| 567 | 566 | ||
| 568 | int ret = sim::CommunicationMemoryManager::GetInstance().ReadCommMem(socketKey.data(), data, size); | 567 | int ret = sim::CommunicationMemoryManager::GetInstance().ReadCommMem(socketKey.data(), data, size); |
| 569 | if (ret == 0) { | 568 | if (ret == 0) { |
| 570 | - RA_SOCK_STUB_WARN(" socket pair:{:d} role:{:d} key:{} try again", | 569 | + HCCL_VM_WARN(" socket pair:{:d} role:{:d} key:{} try again", |
| 571 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); | 570 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); |
| 572 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); | 571 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); |
| 573 | return 0; | 572 | return 0; |
| 574 | } else if (ret == -1) { | 573 | } else if (ret == -1) { |
| 575 | - RA_SOCK_STUB_ERROR(" socket pair:{:d} role:{:d} key:{} recv failed", | 574 | + HCCL_VM_ERROR(" socket pair:{:d} role:{:d} key:{} recv failed", |
| 576 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); | 575 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey); |
| 577 | return -1; | 576 | return -1; |
| 578 | } | 577 | } |
| 579 | 578 | ||
| 580 | *receivedSize = ret; | 579 | *receivedSize = ret; |
| 581 | - RA_SOCK_STUB_INFO(" socket pair:{:d} role:{:d} key:{} recv bytes:{:d} ", | 580 | + HCCL_VM_INFO(" socket pair:{:d} role:{:d} key:{} recv bytes:{:d} ", |
| 582 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey, ret); | 581 | FD_PAIR_ID(socketFd), FD_ROLE(socketFd), socketKey, ret); |
| 583 | return 0; | 582 | return 0; |
| 584 | } | 583 | } |
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -17,6 +20,7 @@ | |||
| 17 | 20 | ||
| 18 | 21 | ||
| 19 | 22 | ||
| 23 | + | ||
| 20 | constexpr size_t LOG_MSG_BUFFER_SIZE = 1024; | 24 | constexpr size_t LOG_MSG_BUFFER_SIZE = 1024; |
| 21 | 25 | ||
| 22 | std::once_flag log_initialized_flag; | 26 | std::once_flag log_initialized_flag; |
| @@ -44,19 +48,19 @@ void DlogPrintStub(int level, const char *msgBuffer) | |||
| 44 | 48 | ||
| 45 | switch (level) { | 49 | switch (level) { |
| 46 | case 0: | 50 | case 0: |
| 47 | - HCCL_VM_DEBUG("[HCCL_LOG]{}", msgBuffer); | 51 | + HCCL_VM_DEBUG("{}", msgBuffer); |
| 48 | break; | 52 | break; |
| 49 | case 1: | 53 | case 1: |
| 50 | - HCCL_VM_INFO("[HCCL_LOG]{}", msgBuffer); | 54 | + HCCL_VM_INFO("{}", msgBuffer); |
| 51 | break; | 55 | break; |
| 52 | case 2: | 56 | case 2: |
| 53 | - HCCL_VM_WARN("[HCCL_LOG]{}", msgBuffer); | 57 | + HCCL_VM_WARN("{}", msgBuffer); |
| 54 | break; | 58 | break; |
| 55 | case 3: | 59 | case 3: |
| 56 | - HCCL_VM_ERROR("[HCCL_LOG]{}", msgBuffer); | 60 | + HCCL_VM_ERROR("{}", msgBuffer); |
| 57 | break; | 61 | break; |
| 58 | default: | 62 | default: |
| 59 | - HCCL_VM_TRACE("[HCCL_LOG]{}", msgBuffer); | 63 | + HCCL_VM_TRACE("{}", msgBuffer); |
| 60 | } | 64 | } |
| 61 | } | 65 | } |
| 62 | 66 | ||
| @@ -10,23 +10,21 @@ | |||
| 10 | 10 | ||
| 11 | // 为确保Scatter正常执行打的临时桩,需要适时移除 | 11 | // 为确保Scatter正常执行打的临时桩,需要适时移除 |
| 12 | 12 | ||
| 13 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 14 | + | ||
| 15 | + | ||
| 13 | 16 | ||
| 14 | 17 | ||
| 15 | 18 | ||
| 16 | 19 | ||
| 17 | 20 | ||
| 18 | 21 | ||
| 19 | - | ||
| 20 | - | ||
| 21 | - | ||
| 22 | - | ||
| 23 | - | ||
| 24 | 22 | ||
| 25 | namespace ops_hccl { | 23 | namespace ops_hccl { |
| 26 | bool RunIndependentOpExpansion(DevType deviceType) | 24 | bool RunIndependentOpExpansion(DevType deviceType) |
| 27 | { | 25 | { |
| 28 | (void) deviceType; | 26 | (void) deviceType; |
| 29 | - SCATTER_STUB_INFO("return true"); | 27 | + HCCL_VM_INFO("return true"); |
| 30 | return true; | 28 | return true; |
| 31 | } | 29 | } |
| 32 | 30 | ||
| @@ -35,7 +33,7 @@ bool IsAiCpuMode(DevType deviceType, u32 rankSize) | |||
| 35 | (void) deviceType; | 33 | (void) deviceType; |
| 36 | (void) rankSize; | 34 | (void) rankSize; |
| 37 | const char* env = getenv("HCCL_OP_EXPANSION_MODE"); | 35 | const char* env = getenv("HCCL_OP_EXPANSION_MODE"); |
| 38 | - SCATTER_STUB_INFO("env = {}", env); | 36 | + HCCL_VM_INFO("env = {}", env); |
| 39 | // 判断是否是AI_CPU | 37 | // 判断是否是AI_CPU |
| 40 | if (env != nullptr && strcmp(env, "AI_CPU") == 0) { | 38 | if (env != nullptr && strcmp(env, "AI_CPU") == 0) { |
| 41 | return true; | 39 | return true; |
| @@ -8,6 +8,9 @@ | |||
| 8 | * See LICENSE in the root of the software repository for the full text of the License. | 8 | * See LICENSE in the root of the software repository for the full text of the License. |
| 9 | */ | 9 | */ |
| 10 | 10 | ||
| 11 | +// 日志染色: 模块 tag (须在 include sim_log.h 之前) | ||
| 12 | + | ||
| 13 | + | ||
| 11 | 14 | ||
| 12 | 15 | ||
| 13 | 16 | ||
| @@ -16,6 +19,7 @@ | |||
| 16 | 19 | ||
| 17 | 20 | ||
| 18 | 21 | ||
| 22 | + | ||
| 19 | 23 | ||
| 20 | extern "C" { | 24 | extern "C" { |
| 21 | 25 | ||
| @@ -25,7 +29,7 @@ uint32_t TsdCapabilityGet(const uint32_t logicDeviceId, const int32_t type, cons | |||
| 25 | (void) logicDeviceId; | 29 | (void) logicDeviceId; |
| 26 | (void) type; | 30 | (void) type; |
| 27 | (void) ptr; | 31 | (void) ptr; |
| 28 | - HCCL_VM_INFO("[TSD_STUB]stub"); | 32 | + HCCL_VM_INFO("stub"); |
| 29 | return 0; | 33 | return 0; |
| 30 | } | 34 | } |
| 31 | 35 | ||
| @@ -59,3 +59,6 @@ find_package(SQLite3 REQUIRED) | |||
| 59 | 59 | ||
| 60 | target_link_libraries(runnerdb PUBLIC SQLite::SQLite3) | 60 | target_link_libraries(runnerdb PUBLIC SQLite::SQLite3) |
| 61 | 61 | ||
| 62 | + | ||
| 63 | +# 日志染色: 模块 tag | ||
| 64 | +target_compile_definitions(runnerdb PRIVATE HCCL_VM_MODULE="RUNNERDB") | ||
| @@ -36,3 +36,6 @@ find_package(SQLite3 REQUIRED) | |||
| 36 | 36 | ||
| 37 | target_link_libraries(modeldb PUBLIC SQLite::SQLite3) | 37 | target_link_libraries(modeldb PUBLIC SQLite::SQLite3) |
| 38 | 38 | ||
| 39 | + | ||
| 40 | +# 日志染色: 模块 tag | ||
| 41 | +target_compile_definitions(modeldb PRIVATE HCCL_VM_MODULE="RUNNERDB") | ||
| @@ -32,7 +32,7 @@ HcclVmResult HcclDBSqlite::Connect(const sim::DBConfig& config) { | |||
| 32 | int ret = sqlite3_open(config.dbPath.c_str(), &m_db); | 32 | int ret = sqlite3_open(config.dbPath.c_str(), &m_db); |
| 33 | if (ret != SQLITE_OK) { | 33 | if (ret != SQLITE_OK) { |
| 34 | m_lastError = sqlite3_errmsg(m_db); | 34 | m_lastError = sqlite3_errmsg(m_db); |
| 35 | - HCCL_VM_ERROR("[HcclDBSqlite::Connect] Failed: {}, path: {}", m_lastError, config.dbPath); | 35 | + HCCL_VM_ERROR("Failed: {}, path: {}", m_lastError, config.dbPath); |
| 36 | sqlite3_close(m_db); | 36 | sqlite3_close(m_db); |
| 37 | m_db = nullptr; | 37 | m_db = nullptr; |
| 38 | return HCCL_SIM_E_OPEN_FILE_FAILURE; | 38 | return HCCL_SIM_E_OPEN_FILE_FAILURE; |
| @@ -50,7 +50,7 @@ HcclVmResult HcclDBSqlite::Connect(const sim::DBConfig& config) { | |||
| 50 | sqlite3_exec(m_db, "PRAGMA wal_autocheckpoint = 200;", nullptr, nullptr, nullptr); | 50 | sqlite3_exec(m_db, "PRAGMA wal_autocheckpoint = 200;", nullptr, nullptr, nullptr); |
| 51 | sqlite3_exec(m_db, "PRAGMA mmap_size = 268435456;", nullptr, nullptr, nullptr); | 51 | sqlite3_exec(m_db, "PRAGMA mmap_size = 268435456;", nullptr, nullptr, nullptr); |
| 52 | 52 | ||
| 53 | - HCCL_VM_INFO("[HcclDBSqlite::Connect] Connected: {}", config.dbPath); | 53 | + HCCL_VM_INFO("Connected: {}", config.dbPath); |
| 54 | return HCCL_SIM_SUCCESS; | 54 | return HCCL_SIM_SUCCESS; |
| 55 | } | 55 | } |
| 56 | 56 | ||
| @@ -116,7 +116,7 @@ HcclVmResult HcclDBSqlite::Query(const std::string& sql, const std::vector<Value | |||
| 116 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); | 116 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); |
| 117 | if (ret != SQLITE_OK) { | 117 | if (ret != SQLITE_OK) { |
| 118 | m_lastError = sqlite3_errmsg(m_db); | 118 | m_lastError = sqlite3_errmsg(m_db); |
| 119 | - HCCL_VM_ERROR("[HcclDBSqlite::Query] Prepare failed: {}", m_lastError); | 119 | + HCCL_VM_ERROR("Prepare failed: {}", m_lastError); |
| 120 | return HCCL_SIM_E_INTERNAL; | 120 | return HCCL_SIM_E_INTERNAL; |
| 121 | } | 121 | } |
| 122 | 122 | ||
| @@ -148,7 +148,7 @@ HcclVmResult HcclDBSqlite::Query(const std::string& sql, const std::vector<Value | |||
| 148 | 148 | ||
| 149 | if (ret != SQLITE_DONE) { | 149 | if (ret != SQLITE_DONE) { |
| 150 | m_lastError = sqlite3_errmsg(m_db); | 150 | m_lastError = sqlite3_errmsg(m_db); |
| 151 | - HCCL_VM_ERROR("[HcclDBSqlite::Query] Step failed: {}", m_lastError); | 151 | + HCCL_VM_ERROR("Step failed: {}", m_lastError); |
| 152 | return HCCL_SIM_E_INTERNAL; | 152 | return HCCL_SIM_E_INTERNAL; |
| 153 | } | 153 | } |
| 154 | 154 | ||
| @@ -164,7 +164,7 @@ HcclVmResult HcclDBSqlite::QueryBlob(const std::string& sql, const std::vector<V | |||
| 164 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); | 164 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); |
| 165 | if (ret != SQLITE_OK) { | 165 | if (ret != SQLITE_OK) { |
| 166 | m_lastError = sqlite3_errmsg(m_db); | 166 | m_lastError = sqlite3_errmsg(m_db); |
| 167 | - HCCL_VM_ERROR("[HcclDBSqlite::QueryBlob] Prepare failed: {}", m_lastError); | 167 | + HCCL_VM_ERROR("Prepare failed: {}", m_lastError); |
| 168 | return HCCL_SIM_E_INTERNAL; | 168 | return HCCL_SIM_E_INTERNAL; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| @@ -193,7 +193,7 @@ HcclVmResult HcclDBSqlite::QueryBlob(const std::string& sql, const std::vector<V | |||
| 193 | 193 | ||
| 194 | if (ret != SQLITE_ROW && ret != SQLITE_DONE) { | 194 | if (ret != SQLITE_ROW && ret != SQLITE_DONE) { |
| 195 | m_lastError = sqlite3_errmsg(m_db); | 195 | m_lastError = sqlite3_errmsg(m_db); |
| 196 | - HCCL_VM_ERROR("[HcclDBSqlite::QueryBlob] Step failed: {}", m_lastError); | 196 | + HCCL_VM_ERROR("Step failed: {}", m_lastError); |
| 197 | return HCCL_SIM_E_INTERNAL; | 197 | return HCCL_SIM_E_INTERNAL; |
| 198 | } | 198 | } |
| 199 | 199 | ||
| @@ -209,7 +209,7 @@ HcclVmResult HcclDBSqlite::QueryBlobColumns(const std::string& sql, const std::v | |||
| 209 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); | 209 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); |
| 210 | if (ret != SQLITE_OK) { | 210 | if (ret != SQLITE_OK) { |
| 211 | m_lastError = sqlite3_errmsg(m_db); | 211 | m_lastError = sqlite3_errmsg(m_db); |
| 212 | - HCCL_VM_ERROR("[HcclDBSqlite::QueryBlobColumns] Prepare failed: {}", m_lastError); | 212 | + HCCL_VM_ERROR("Prepare failed: {}", m_lastError); |
| 213 | return HCCL_SIM_E_INTERNAL; | 213 | return HCCL_SIM_E_INTERNAL; |
| 214 | } | 214 | } |
| 215 | 215 | ||
| @@ -242,7 +242,7 @@ HcclVmResult HcclDBSqlite::QueryBlobColumns(const std::string& sql, const std::v | |||
| 242 | 242 | ||
| 243 | if (ret != SQLITE_ROW && ret != SQLITE_DONE) { | 243 | if (ret != SQLITE_ROW && ret != SQLITE_DONE) { |
| 244 | m_lastError = sqlite3_errmsg(m_db); | 244 | m_lastError = sqlite3_errmsg(m_db); |
| 245 | - HCCL_VM_ERROR("[HcclDBSqlite::QueryBlobColumns] Step failed: {}", m_lastError); | 245 | + HCCL_VM_ERROR("Step failed: {}", m_lastError); |
| 246 | return HCCL_SIM_E_INTERNAL; | 246 | return HCCL_SIM_E_INTERNAL; |
| 247 | } | 247 | } |
| 248 | 248 | ||
| @@ -258,7 +258,7 @@ HcclVmResult HcclDBSqlite::QueryEx(const std::string& sql, const std::vector<Val | |||
| 258 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); | 258 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); |
| 259 | if (ret != SQLITE_OK) { | 259 | if (ret != SQLITE_OK) { |
| 260 | m_lastError = sqlite3_errmsg(m_db); | 260 | m_lastError = sqlite3_errmsg(m_db); |
| 261 | - HCCL_VM_ERROR("[HcclDBSqlite::QueryEx] Prepare failed: {}", m_lastError); | 261 | + HCCL_VM_ERROR("Prepare failed: {}", m_lastError); |
| 262 | return HCCL_SIM_E_INTERNAL; | 262 | return HCCL_SIM_E_INTERNAL; |
| 263 | } | 263 | } |
| 264 | 264 | ||
| @@ -306,7 +306,7 @@ HcclVmResult HcclDBSqlite::QueryEx(const std::string& sql, const std::vector<Val | |||
| 306 | 306 | ||
| 307 | if (ret != SQLITE_DONE) { | 307 | if (ret != SQLITE_DONE) { |
| 308 | m_lastError = sqlite3_errmsg(m_db); | 308 | m_lastError = sqlite3_errmsg(m_db); |
| 309 | - HCCL_VM_ERROR("[HcclDBSqlite::QueryEx] Step failed: {}", m_lastError); | 309 | + HCCL_VM_ERROR("Step failed: {}", m_lastError); |
| 310 | return HCCL_SIM_E_INTERNAL; | 310 | return HCCL_SIM_E_INTERNAL; |
| 311 | } | 311 | } |
| 312 | 312 | ||
| @@ -323,7 +323,7 @@ HcclVmResult HcclDBSqlite::Execute(const std::string& sql) { | |||
| 323 | 323 | ||
| 324 | if (ret != SQLITE_OK) { | 324 | if (ret != SQLITE_OK) { |
| 325 | m_lastError = errMsg ? errMsg : "Unknown error"; | 325 | m_lastError = errMsg ? errMsg : "Unknown error"; |
| 326 | - HCCL_VM_ERROR("[HcclDBSqlite::Execute] SQL error: {}, sql: {}", m_lastError, sql); | 326 | + HCCL_VM_ERROR("SQL error: {}, sql: {}", m_lastError, sql); |
| 327 | if (errMsg) { | 327 | if (errMsg) { |
| 328 | sqlite3_free(errMsg); | 328 | sqlite3_free(errMsg); |
| 329 | } | 329 | } |
| @@ -342,7 +342,7 @@ HcclVmResult HcclDBSqlite::Query(const std::string& sql, std::vector<std::vector | |||
| 342 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); | 342 | int ret = sqlite3_prepare_v2(m_db, sql.c_str(), -1, &stmt, nullptr); |
| 343 | if (ret != SQLITE_OK) { | 343 | if (ret != SQLITE_OK) { |
| 344 | m_lastError = sqlite3_errmsg(m_db); | 344 | m_lastError = sqlite3_errmsg(m_db); |
| 345 | - HCCL_VM_ERROR("[HcclDBSqlite::Query] Prepare failed: {}", m_lastError); | 345 | + HCCL_VM_ERROR("Prepare failed: {}", m_lastError); |
| 346 | return HCCL_SIM_E_INTERNAL; | 346 | return HCCL_SIM_E_INTERNAL; |
| 347 | } | 347 | } |
| 348 | 348 | ||
| @@ -361,7 +361,7 @@ HcclVmResult HcclDBSqlite::Query(const std::string& sql, std::vector<std::vector | |||
| 361 | 361 | ||
| 362 | if (ret != SQLITE_DONE) { | 362 | if (ret != SQLITE_DONE) { |
| 363 | m_lastError = sqlite3_errmsg(m_db); | 363 | m_lastError = sqlite3_errmsg(m_db); |
| 364 | - HCCL_VM_ERROR("[HcclDBSqlite::Query] Step failed: {}", m_lastError); | 364 | + HCCL_VM_ERROR("Step failed: {}", m_lastError); |
| 365 | return HCCL_SIM_E_INTERNAL; | 365 | return HCCL_SIM_E_INTERNAL; |
| 366 | } | 366 | } |
| 367 | 367 | ||
| @@ -410,7 +410,7 @@ HcclVmResult HcclDBSqlite::Backup(const std::string& destPath) { | |||
| 410 | rc = sqlite3_open(destPath.c_str(), &pDest); | 410 | rc = sqlite3_open(destPath.c_str(), &pDest); |
| 411 | if (rc != SQLITE_OK) { | 411 | if (rc != SQLITE_OK) { |
| 412 | m_lastError = sqlite3_errstr(rc); | 412 | m_lastError = sqlite3_errstr(rc); |
| 413 | - HCCL_VM_ERROR("[HcclDBSqlite::Backup] Failed to open destination DB: {}", m_lastError); | 413 | + HCCL_VM_ERROR("Failed to open destination DB: {}", m_lastError); |
| 414 | return HCCL_SIM_E_OPEN_FILE_FAILURE; | 414 | return HCCL_SIM_E_OPEN_FILE_FAILURE; |
| 415 | } | 415 | } |
| 416 | 416 | ||
| @@ -428,18 +428,18 @@ HcclVmResult HcclDBSqlite::Backup(const std::string& destPath) { | |||
| 428 | } else { | 428 | } else { |
| 429 | rc = sqlite3_errcode(pDest); | 429 | rc = sqlite3_errcode(pDest); |
| 430 | m_lastError = sqlite3_errmsg(pDest); | 430 | m_lastError = sqlite3_errmsg(pDest); |
| 431 | - HCCL_VM_ERROR("[HcclDBSqlite::Backup] Backup init failed: {}", m_lastError); | 431 | + HCCL_VM_ERROR("Backup init failed: {}", m_lastError); |
| 432 | } | 432 | } |
| 433 | 433 | ||
| 434 | // 5. Close destination | 434 | // 5. Close destination |
| 435 | sqlite3_close(pDest); | 435 | sqlite3_close(pDest); |
| 436 | 436 | ||
| 437 | if (rc == SQLITE_OK) { | 437 | if (rc == SQLITE_OK) { |
| 438 | - HCCL_VM_INFO("[HcclDBSqlite::Backup] Backup completed successfully to: {}", destPath); | 438 | + HCCL_VM_INFO("Backup completed successfully to: {}", destPath); |
| 439 | return HCCL_SIM_SUCCESS; | 439 | return HCCL_SIM_SUCCESS; |
| 440 | } else { | 440 | } else { |
| 441 | m_lastError = sqlite3_errstr(rc); | 441 | m_lastError = sqlite3_errstr(rc); |
| 442 | - HCCL_VM_ERROR("[HcclDBSqlite::Backup] Backup failed: {}", m_lastError); | 442 | + HCCL_VM_ERROR("Backup failed: {}", m_lastError); |
| 443 | return HCCL_SIM_E_INTERNAL; | 443 | return HCCL_SIM_E_INTERNAL; |
| 444 | } | 444 | } |
| 445 | } | 445 | } |
| @@ -453,7 +453,7 @@ HcclVmResult HcclDBSqlite::RunInTransaction(std::function<HcclVmResult()> fn) { | |||
| 453 | int rc = sqlite3_exec(m_db, "BEGIN IMMEDIATE", nullptr, nullptr, &errMsg); | 453 | int rc = sqlite3_exec(m_db, "BEGIN IMMEDIATE", nullptr, nullptr, &errMsg); |
| 454 | if (rc != SQLITE_OK) { | 454 | if (rc != SQLITE_OK) { |
| 455 | m_lastError = errMsg ? errMsg : "BEGIN IMMEDIATE failed"; | 455 | m_lastError = errMsg ? errMsg : "BEGIN IMMEDIATE failed"; |
| 456 | - HCCL_VM_ERROR("[HcclDBSqlite::RunInTransaction] BEGIN failed: {}", m_lastError); | 456 | + HCCL_VM_ERROR("BEGIN failed: {}", m_lastError); |
| 457 | if (errMsg) { | 457 | if (errMsg) { |
| 458 | sqlite3_free(errMsg); | 458 | sqlite3_free(errMsg); |
| 459 | } | 459 | } |
| @@ -466,7 +466,7 @@ HcclVmResult HcclDBSqlite::RunInTransaction(std::function<HcclVmResult()> fn) { | |||
| 466 | rc = sqlite3_exec(m_db, "COMMIT", nullptr, nullptr, &errMsg); | 466 | rc = sqlite3_exec(m_db, "COMMIT", nullptr, nullptr, &errMsg); |
| 467 | if (rc != SQLITE_OK) { | 467 | if (rc != SQLITE_OK) { |
| 468 | m_lastError = errMsg ? errMsg : "COMMIT failed"; | 468 | m_lastError = errMsg ? errMsg : "COMMIT failed"; |
| 469 | - HCCL_VM_ERROR("[HcclDBSqlite::RunInTransaction] COMMIT failed: {}, rolling back", m_lastError); | 469 | + HCCL_VM_ERROR("COMMIT failed: {}, rolling back", m_lastError); |
| 470 | if (errMsg) { | 470 | if (errMsg) { |
| 471 | sqlite3_free(errMsg); | 471 | sqlite3_free(errMsg); |
| 472 | } | 472 | } |
| @@ -36,16 +36,16 @@ OpDbOps::OpDbOps() | |||
| 36 | 36 | ||
| 37 | auto db = HcclSim::DB::HcclDBFactory::Instance().CreateDB(config.type); | 37 | auto db = HcclSim::DB::HcclDBFactory::Instance().CreateDB(config.type); |
| 38 | if (!db || db->Connect(config) != HcclSim::HCCL_SIM_SUCCESS) { | 38 | if (!db || db->Connect(config) != HcclSim::HCCL_SIM_SUCCESS) { |
| 39 | - HCCL_VM_ERROR("[dbInit] Connect database failed"); | 39 | + HCCL_VM_ERROR("Connect database failed"); |
| 40 | return; | 40 | return; |
| 41 | } | 41 | } |
| 42 | 42 | ||
| 43 | SetDB(std::move(db)); | 43 | SetDB(std::move(db)); |
| 44 | 44 | ||
| 45 | if (InitSchema() != HcclSim::HCCL_SIM_SUCCESS) { | 45 | if (InitSchema() != HcclSim::HCCL_SIM_SUCCESS) { |
| 46 | - HCCL_VM_ERROR("[dbInit] Init schema failed"); | 46 | + HCCL_VM_ERROR("Init schema failed"); |
| 47 | } else { | 47 | } else { |
| 48 | - HCCL_VM_INFO("[dbInit] OpDbOps tables initialized successfully"); | 48 | + HCCL_VM_INFO("OpDbOps tables initialized successfully"); |
| 49 | } | 49 | } |
| 50 | } | 50 | } |
| 51 | 51 | ||
| @@ -150,7 +150,7 @@ HcclVmResult OpDbOps::InitSchema() { | |||
| 150 | }); | 150 | }); |
| 151 | 151 | ||
| 152 | if (ret == HCCL_SIM_SUCCESS) { | 152 | if (ret == HCCL_SIM_SUCCESS) { |
| 153 | - HCCL_VM_INFO("[OpDbOps::InitSchema] All tables created"); | 153 | + HCCL_VM_INFO("All tables created"); |
| 154 | } | 154 | } |
| 155 | return ret; | 155 | return ret; |
| 156 | } | 156 | } |
| @@ -180,7 +180,7 @@ int UpdateAndInsertByCcuId(uint64_t& ccuId, uint32_t deviceId, uint32_t rankId, | |||
| 180 | ExRows rows; | 180 | ExRows rows; |
| 181 | int ret = OpDbOps::Instance().ExecQueryEx(querySql, queryParams, rows); | 181 | int ret = OpDbOps::Instance().ExecQueryEx(querySql, queryParams, rows); |
| 182 | if (ret != 0) { | 182 | if (ret != 0) { |
| 183 | - HCCL_VM_ERROR("[UpdateAndInsertByCcuId] query ccuInstrRes failed for deviceId: {} dieId: {}", deviceId, dieId); | 183 | + HCCL_VM_ERROR("query ccuInstrRes failed for deviceId: {} dieId: {}", deviceId, dieId); |
| 184 | return -1; | 184 | return -1; |
| 185 | } | 185 | } |
| 186 | 186 | ||
| @@ -303,14 +303,14 @@ int QueryCurrentOpMemInfoByRank(uint32_t rankId, OpMemInfoTab& out) | |||
| 303 | std::vector<Value> params = {(int64_t)g_currOpDetailId, (int64_t)rankId}; | 303 | std::vector<Value> params = {(int64_t)g_currOpDetailId, (int64_t)rankId}; |
| 304 | std::vector<std::vector<std::string>> rows; | 304 | std::vector<std::vector<std::string>> rows; |
| 305 | if (OpDbOps::Instance().ExecQuery(sql, params, rows) != 0 || rows.empty()) { | 305 | if (OpDbOps::Instance().ExecQuery(sql, params, rows) != 0 || rows.empty()) { |
| 306 | - HCCL_VM_ERROR("[QueryCurrentOpMemInfoByRank] query failed, currentOpDetailId={}, rankId={}", | 306 | + HCCL_VM_ERROR("query failed, currentOpDetailId={}, rankId={}", |
| 307 | g_currOpDetailId, rankId); | 307 | g_currOpDetailId, rankId); |
| 308 | return -1; | 308 | return -1; |
| 309 | } | 309 | } |
| 310 | 310 | ||
| 311 | const auto& row = rows[0]; | 311 | const auto& row = rows[0]; |
| 312 | if (row.size() < 8) { | 312 | if (row.size() < 8) { |
| 313 | - HCCL_VM_ERROR("[QueryCurrentOpMemInfoByRank] invalid row size={}, currentOpDetailId={}, rankId={}", | 313 | + HCCL_VM_ERROR("invalid row size={}, currentOpDetailId={}, rankId={}", |
| 314 | row.size(), g_currOpDetailId, rankId); | 314 | row.size(), g_currOpDetailId, rankId); |
| 315 | return -1; | 315 | return -1; |
| 316 | } | 316 | } |
| @@ -332,7 +332,7 @@ int UpdateOpExpansionMode(uint8_t mode) | |||
| 332 | std::vector<Value> params = {(int64_t)mode, (int64_t)g_currOpDetailId}; | 332 | std::vector<Value> params = {(int64_t)mode, (int64_t)g_currOpDetailId}; |
| 333 | int ret = OpDbOps::Instance().ExecUpdate(sql, params); | 333 | int ret = OpDbOps::Instance().ExecUpdate(sql, params); |
| 334 | if (ret != 0) { | 334 | if (ret != 0) { |
| 335 | - HCCL_VM_ERROR("[UpdateOpExpansionMode] Failed to update mode={}, id={}", mode, g_currOpDetailId); | 335 | + HCCL_VM_ERROR("Failed to update mode={}, id={}", mode, g_currOpDetailId); |
| 336 | } | 336 | } |
| 337 | return ret; | 337 | return ret; |
| 338 | } | 338 | } |
| @@ -343,7 +343,7 @@ int QueryLatestOpExpansionMode() | |||
| 343 | std::vector<Value> params = {(int64_t)g_currOpDetailId}; | 343 | std::vector<Value> params = {(int64_t)g_currOpDetailId}; |
| 344 | std::vector<std::vector<std::string>> rows; | 344 | std::vector<std::vector<std::string>> rows; |
| 345 | if (OpDbOps::Instance().ExecQuery(sql, params, rows) != 0 || rows.empty() || rows[0].empty()) { | 345 | if (OpDbOps::Instance().ExecQuery(sql, params, rows) != 0 || rows.empty() || rows[0].empty()) { |
| 346 | - HCCL_VM_ERROR("[QueryLatestOpExpansionMode] Failed to query, id={}", g_currOpDetailId); | 346 | + HCCL_VM_ERROR("Failed to query, id={}", g_currOpDetailId); |
| 347 | return static_cast<int>(SimOpExpansionMode::SIM_OP_EXPANSION_MODE_RESERVED); | 347 | return static_cast<int>(SimOpExpansionMode::SIM_OP_EXPANSION_MODE_RESERVED); |
| 348 | } | 348 | } |
| 349 | return static_cast<int>(std::stoul(rows[0][0])); | 349 | return static_cast<int>(std::stoul(rows[0][0])); |
| @@ -376,7 +376,7 @@ int QueryCompositeOpDetailBySyncIter(uint32_t syncIter, std::map<uint32_t, std:: | |||
| 376 | std::vector<Value> opParams = {(int64_t)syncIter}; | 376 | std::vector<Value> opParams = {(int64_t)syncIter}; |
| 377 | HcclSim::DB::ExRows opRows; | 377 | HcclSim::DB::ExRows opRows; |
| 378 | if (OpDbOps::Instance().ExecQueryEx(opSql, opParams, opRows) != 0) { | 378 | if (OpDbOps::Instance().ExecQueryEx(opSql, opParams, opRows) != 0) { |
| 379 | - HCCL_VM_ERROR("[QueryCompositeOpDetailBySyncIter] Query opDetails failed for syncIter: {}", syncIter); | 379 | + HCCL_VM_ERROR("Query opDetails failed for syncIter: {}", syncIter); |
| 380 | return -1; | 380 | return -1; |
| 381 | } | 381 | } |
| 382 | 382 | ||
| @@ -371,7 +371,7 @@ std::set<uint64_t> GetUsedServerNum() | |||
| 371 | } | 371 | } |
| 372 | return usedServerIds; | 372 | return usedServerIds; |
| 373 | } catch (const std::exception &e) { | 373 | } catch (const std::exception &e) { |
| 374 | - HCCL_VM_ERROR("[GetUsedServerNum] exception: {}", e.what()); | 374 | + HCCL_VM_ERROR("exception: {}", e.what()); |
| 375 | return {}; | 375 | return {}; |
| 376 | } | 376 | } |
| 377 | } | 377 | } |
| @@ -385,13 +385,13 @@ bool GetRankIdByMPI(uint32_t &rankId, uint64_t &serverId) | |||
| 385 | auto usedServerIds = GetUsedServerNum(); | 385 | auto usedServerIds = GetUsedServerNum(); |
| 386 | if (usedServerIds.size() == 1) { | 386 | if (usedServerIds.size() == 1) { |
| 387 | if (*usedServerIds.begin() != 1) { | 387 | if (*usedServerIds.begin() != 1) { |
| 388 | - HCCL_VM_ERROR("[HVM] env OMPI_COMM_WORLD_RANK or PMI_RANK are not found, serverId is not 1."); | 388 | + HCCL_VM_ERROR("env OMPI_COMM_WORLD_RANK or PMI_RANK are not found, serverId is not 1."); |
| 389 | return false; | 389 | return false; |
| 390 | } | 390 | } |
| 391 | serverId = 1; | 391 | serverId = 1; |
| 392 | return true; | 392 | return true; |
| 393 | } | 393 | } |
| 394 | - HCCL_VM_ERROR("[HVM] env OMPI_COMM_WORLD_RANK or PMI_RANK are not found, usedServerIds size is not 1."); | 394 | + HCCL_VM_ERROR("env OMPI_COMM_WORLD_RANK or PMI_RANK are not found, usedServerIds size is not 1."); |
| 395 | return false; | 395 | return false; |
| 396 | } else if (ompiRankStr != nullptr) { | 396 | } else if (ompiRankStr != nullptr) { |
| 397 | rankId = static_cast<uint32_t>(atoi(ompiRankStr)); | 397 | rankId = static_cast<uint32_t>(atoi(ompiRankStr)); |
| @@ -60,12 +60,12 @@ bool InsertRunner(uint64_t serverKey) | |||
| 60 | g_runner.thread_id = pthread_self(); | 60 | g_runner.thread_id = pthread_self(); |
| 61 | 61 | ||
| 62 | g_cur_server_key = serverKey; | 62 | g_cur_server_key = serverKey; |
| 63 | - HCCL_VM_INFO("[InsertRunner] Init host ip= {}, server key= {}.", ret.first.ip_addr, g_cur_server_key); | 63 | + HCCL_VM_INFO("Init host ip= {}, server key= {}.", ret.first.ip_addr, g_cur_server_key); |
| 64 | 64 | ||
| 65 | g_runner.id = RunnerDB::Add<sim::Runner>(runner); | 65 | g_runner.id = RunnerDB::Add<sim::Runner>(runner); |
| 66 | return true; | 66 | return true; |
| 67 | } catch (const std::exception &e) { | 67 | } catch (const std::exception &e) { |
| 68 | - HCCL_VM_ERROR("[InsertRunner] SQLite exception: {}", e.what()); | 68 | + HCCL_VM_ERROR("SQLite exception: {}", e.what()); |
| 69 | return false; | 69 | return false; |
| 70 | } | 70 | } |
| 71 | } | 71 | } |
| @@ -93,7 +93,7 @@ bool SetCurrCtxTls(uint64_t ctx) | |||
| 93 | [currRunnerId, ctx](sim::Runner &runner) { runner.current_ctx_id = ctx; }); | 93 | [currRunnerId, ctx](sim::Runner &runner) { runner.current_ctx_id = ctx; }); |
| 94 | return true; | 94 | return true; |
| 95 | } catch (const std::exception &e) { | 95 | } catch (const std::exception &e) { |
| 96 | - HCCL_VM_ERROR("[SetCurrCtxTls] SQLite exception: {}", e.what()); | 96 | + HCCL_VM_ERROR("SQLite exception: {}", e.what()); |
| 97 | return false; | 97 | return false; |
| 98 | } | 98 | } |
| 99 | } | 99 | } |
| @@ -139,7 +139,7 @@ uint64_t GetCurrRankId() | |||
| 139 | return r.device_id == devKey; | 139 | return r.device_id == devKey; |
| 140 | }); | 140 | }); |
| 141 | if (!rank.second) { | 141 | if (!rank.second) { |
| 142 | - HCCL_VM_ERROR("[GetCurrRankId] can not find any rank"); | 142 | + HCCL_VM_ERROR("can not find any rank"); |
| 143 | return 0; | 143 | return 0; |
| 144 | } | 144 | } |
| 145 | 145 | ||
| @@ -216,7 +216,7 @@ uint64_t GetServerKeyById(uint32_t superPodIdx, uint32_t serverIdx) | |||
| 216 | return r.pod_id == superPodIdx && r.server_id == serverIdx; | 216 | return r.pod_id == superPodIdx && r.server_id == serverIdx; |
| 217 | }); | 217 | }); |
| 218 | if (!serverRet.second) { | 218 | if (!serverRet.second) { |
| 219 | - HCCL_VM_ERROR("[GetServerKeyById] can not find server by id: {:d}, {:d}", superPodIdx, serverIdx); | 219 | + HCCL_VM_ERROR("can not find server by id: {:d}, {:d}", superPodIdx, serverIdx); |
| 220 | return 0; | 220 | return 0; |
| 221 | } | 221 | } |
| 222 | return serverRet.first.id; | 222 | return serverRet.first.id; |
| @@ -46,3 +46,6 @@ target_include_directories(store PRIVATE | |||
| 46 | ) | 46 | ) |
| 47 | 47 | ||
| 48 | target_link_libraries(store PUBLIC log loader) | 48 | target_link_libraries(store PUBLIC log loader) |
| 49 | + | ||
| 50 | +# 日志染色: 模块 tag | ||
| 51 | +target_compile_definitions(store PRIVATE HCCL_VM_MODULE="STORE") | ||
| @@ -53,7 +53,7 @@ std::string GenDataId() | |||
| 53 | int32_t retryCount = 10; | 53 | int32_t retryCount = 10; |
| 54 | while (true) { | 54 | while (true) { |
| 55 | if (retryCount == 0) { | 55 | if (retryCount == 0) { |
| 56 | - HCCL_VM_ERROR("[GenDataId] failed to gen dataId."); | 56 | + HCCL_VM_ERROR("failed to gen dataId."); |
| 57 | break; | 57 | break; |
| 58 | } | 58 | } |
| 59 | retryCount--; | 59 | retryCount--; |
| @@ -100,7 +100,7 @@ HcclVmResult DumpHcclVmFlagData(HcclSim::HcclVmFlagData &flagData) | |||
| 100 | 100 | ||
| 101 | auto ret = HcclVmFlagDataWrite(fp, flagData); | 101 | auto ret = HcclVmFlagDataWrite(fp, flagData); |
| 102 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 102 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 103 | - HCCL_VM_ERROR("[DumpHcclVmFlagData] HcclVmFlagDataWrite failed. "); | 103 | + HCCL_VM_ERROR("HcclVmFlagDataWrite failed. "); |
| 104 | fclose(fp); | 104 | fclose(fp); |
| 105 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 105 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 106 | } | 106 | } |
| @@ -123,7 +123,7 @@ HcclVmResult GetHcclVmFlagData(HcclSim::HcclVmFlagData &waitFlag) | |||
| 123 | 123 | ||
| 124 | auto ret = HcclVmFlagDataRead(fp, waitFlag, HCCLVM_FLAG_FILE_MAGIC); | 124 | auto ret = HcclVmFlagDataRead(fp, waitFlag, HCCLVM_FLAG_FILE_MAGIC); |
| 125 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { | 125 | if (ret != HcclVmResult::HCCL_SIM_SUCCESS) { |
| 126 | - HCCL_VM_ERROR("[GetHcclVmFlagData] HcclVmFlagDataRead failed. "); | 126 | + HCCL_VM_ERROR("HcclVmFlagDataRead failed. "); |
| 127 | fclose(fp); | 127 | fclose(fp); |
| 128 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 128 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 129 | } | 129 | } |
| @@ -158,7 +158,7 @@ HcclVmResult DumpDataToFile(const std::string &dataId) | |||
| 158 | 158 | ||
| 159 | HcclVmResult CreateMemoryInfo(HcclVmSynData &hvmSynData, const sim::OpMemInfoTab &memInfo, uint32_t rankId) | 159 | HcclVmResult CreateMemoryInfo(HcclVmSynData &hvmSynData, const sim::OpMemInfoTab &memInfo, uint32_t rankId) |
| 160 | { | 160 | { |
| 161 | - HCCL_VM_INFO("[STUB] Enter into create memory info..."); | 161 | + HCCL_VM_INFO("Enter into create memory info..."); |
| 162 | auto addBuf = [&](uint8_t bufType, uint64_t addr, uint64_t size) { | 162 | auto addBuf = [&](uint8_t bufType, uint64_t addr, uint64_t size) { |
| 163 | if (addr != 0 && size > 0) { | 163 | if (addr != 0 && size > 0) { |
| 164 | MemLayoutData memLayoutData{}; | 164 | MemLayoutData memLayoutData{}; |
| @@ -175,14 +175,14 @@ HcclVmResult CreateMemoryInfo(HcclVmSynData &hvmSynData, const sim::OpMemInfoTab | |||
| 175 | addBuf(static_cast<uint8_t>(BufferType::CCL), memInfo.cclAddr, memInfo.cclSize); | 175 | addBuf(static_cast<uint8_t>(BufferType::CCL), memInfo.cclAddr, memInfo.cclSize); |
| 176 | 176 | ||
| 177 | hvmSynData.memory_info.count = hvmSynData.memory_info.data.size(); | 177 | hvmSynData.memory_info.count = hvmSynData.memory_info.data.size(); |
| 178 | - HCCL_VM_INFO("{}", hvmSynData.memory_info.count, hvmSynData.memory_info.data.size()); | 178 | + HCCL_VM_INFO("memory_info count={}, dataSize={}", hvmSynData.memory_info.count, hvmSynData.memory_info.data.size()); |
| 179 | 179 | ||
| 180 | return HcclVmResult::HCCL_SIM_SUCCESS; | 180 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 181 | } | 181 | } |
| 182 | 182 | ||
| 183 | HcclVmResult CreateChannelInfo(HcclVmSynData &hvmSynData) | 183 | HcclVmResult CreateChannelInfo(HcclVmSynData &hvmSynData) |
| 184 | { | 184 | { |
| 185 | - HCCL_VM_INFO("[STUB] Enter into create channel info..."); | 185 | + HCCL_VM_INFO("Enter into create channel info..."); |
| 186 | 186 | ||
| 187 | std::vector<sim::CcuChannelTab> channels; | 187 | std::vector<sim::CcuChannelTab> channels; |
| 188 | loader::Loader dataLoader; | 188 | loader::Loader dataLoader; |
| @@ -251,7 +251,7 @@ HcclVmResult CreateChannelInfo(HcclVmSynData &hvmSynData) | |||
| 251 | 251 | ||
| 252 | HcclVmResult CreateJettyInfo(HcclVmSynData &hvmSynData) | 252 | HcclVmResult CreateJettyInfo(HcclVmSynData &hvmSynData) |
| 253 | { | 253 | { |
| 254 | - HCCL_VM_INFO("[STUB] Enter into create jetty info..."); | 254 | + HCCL_VM_INFO("Enter into create jetty info..."); |
| 255 | auto endpointPairs = RunnerDB::GetByPred<sim::EndPointPair>([](auto &&) { return true; }); | 255 | auto endpointPairs = RunnerDB::GetByPred<sim::EndPointPair>([](auto &&) { return true; }); |
| 256 | hvmSynData.channel_info.count = endpointPairs.size(); | 256 | hvmSynData.channel_info.count = endpointPairs.size(); |
| 257 | for (auto &endpoint : endpointPairs) { | 257 | for (auto &endpoint : endpointPairs) { |
| @@ -334,7 +334,7 @@ HcclVmResult CreateJettyInfo(HcclVmSynData &hvmSynData) | |||
| 334 | 334 | ||
| 335 | HcclVmResult CreateSimSynData(HcclVmSynData &hvmSynData) | 335 | HcclVmResult CreateSimSynData(HcclVmSynData &hvmSynData) |
| 336 | { | 336 | { |
| 337 | - HCCL_VM_INFO("[CreateSimSynData] Start get simulator synthesis data..."); | 337 | + HCCL_VM_INFO("Start get simulator synthesis data..."); |
| 338 | // header | 338 | // header |
| 339 | hvmSynData.header.magic = HCCLVM_SYN_FILE_MAGIC; | 339 | hvmSynData.header.magic = HCCLVM_SYN_FILE_MAGIC; |
| 340 | hvmSynData.header.version = 1; | 340 | hvmSynData.header.version = 1; |
| @@ -345,7 +345,7 @@ HcclVmResult CreateSimSynData(HcclVmSynData &hvmSynData) | |||
| 345 | constexpr uint32_t kDumpSyncIter = 0; // 仅在 round 0 时 dump,syncIter 固定为 0 | 345 | constexpr uint32_t kDumpSyncIter = 0; // 仅在 round 0 时 dump,syncIter 固定为 0 |
| 346 | loader::Loader dataLoader; | 346 | loader::Loader dataLoader; |
| 347 | if (dataLoader.LoadCompositeOpDetailBySyncIter(kDumpSyncIter, compositeDataMap) != HcclResult::HCCL_SUCCESS) { | 347 | if (dataLoader.LoadCompositeOpDetailBySyncIter(kDumpSyncIter, compositeDataMap) != HcclResult::HCCL_SUCCESS) { |
| 348 | - HCCL_VM_ERROR("LoadCompositeOpDetailBySyncIter({}, {}) failed.", kDumpSyncIter); | 348 | + HCCL_VM_ERROR("LoadCompositeOpDetailBySyncIter({}) failed.", kDumpSyncIter); |
| 349 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 349 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 350 | } | 350 | } |
| 351 | if (compositeDataMap.count(0) == 0 || compositeDataMap[0].empty()) { | 351 | if (compositeDataMap.count(0) == 0 || compositeDataMap[0].empty()) { |
| @@ -415,7 +415,7 @@ HcclVmResult CreateSimSynData(HcclVmSynData &hvmSynData) | |||
| 415 | 415 | ||
| 416 | HcclVmResult DumpHcclVmSynthesisData(const std::string &dataId) | 416 | HcclVmResult DumpHcclVmSynthesisData(const std::string &dataId) |
| 417 | { | 417 | { |
| 418 | - HCCL_VM_INFO("[DumpHcclVmSynthesisData] Start dumping hccl vm synthesis data..."); | 418 | + HCCL_VM_INFO("Start dumping hccl vm synthesis data..."); |
| 419 | // 1. 构造完整路径 | 419 | // 1. 构造完整路径 |
| 420 | char fileName[256]; | 420 | char fileName[256]; |
| 421 | snprintf(fileName, sizeof(fileName), HCCLVM_SYN_DATA_FILE.c_str(), dataId.c_str()); | 421 | snprintf(fileName, sizeof(fileName), HCCLVM_SYN_DATA_FILE.c_str(), dataId.c_str()); |
| @@ -436,7 +436,7 @@ HcclVmResult DumpHcclVmSynthesisData(const std::string &dataId) | |||
| 436 | 436 | ||
| 437 | FILE *fp = fopen(fullPath.c_str(), "wb"); | 437 | FILE *fp = fopen(fullPath.c_str(), "wb"); |
| 438 | if (!fp) { | 438 | if (!fp) { |
| 439 | - HCCL_VM_ERROR("Open file failed: {}", fullPath, strerror(errno)); | 439 | + HCCL_VM_ERROR("Open file failed: {}, err: {}", fullPath, strerror(errno)); |
| 440 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 440 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 441 | } | 441 | } |
| 442 | 442 | ||
| @@ -511,7 +511,7 @@ HcclVmResult DumpHcclVmInstrData(const std::string &dataId) | |||
| 511 | std::string fullPath = InstallPath::ResolveToInstallRoot("data" + std::string(fileName)); | 511 | std::string fullPath = InstallPath::ResolveToInstallRoot("data" + std::string(fileName)); |
| 512 | FILE *fp = fopen(fullPath.c_str(), "wb"); | 512 | FILE *fp = fopen(fullPath.c_str(), "wb"); |
| 513 | if (!fp) { | 513 | if (!fp) { |
| 514 | - HCCL_VM_ERROR("Open file failed: {}", fullPath, strerror(errno)); | 514 | + HCCL_VM_ERROR("Open file failed: {}, err: {}", fullPath, strerror(errno)); |
| 515 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 515 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 516 | } | 516 | } |
| 517 | 517 | ||
| @@ -601,7 +601,7 @@ HcclVmResult DumpHcclVmTask(const std::string &dataId) | |||
| 601 | 601 | ||
| 602 | FILE *fp = fopen(fullPath.c_str(), "wb"); | 602 | FILE *fp = fopen(fullPath.c_str(), "wb"); |
| 603 | if (!fp) { | 603 | if (!fp) { |
| 604 | - HCCL_VM_ERROR("Open file failed: {}", fullPath, strerror(errno)); | 604 | + HCCL_VM_ERROR("Open file failed: {}, err: {}", fullPath, strerror(errno)); |
| 605 | return HcclVmResult::HCCL_SIM_E_INTERNAL; | 605 | return HcclVmResult::HCCL_SIM_E_INTERNAL; |
| 606 | } | 606 | } |
| 607 | 607 | ||
| @@ -48,7 +48,7 @@ CommunicationMemoryManager::~CommunicationMemoryManager() | |||
| 48 | void* CommunicationMemoryManager::AllocCommMem(const char* name) | 48 | void* CommunicationMemoryManager::AllocCommMem(const char* name) |
| 49 | { | 49 | { |
| 50 | if (!name) { | 50 | if (!name) { |
| 51 | - HCCL_VM_ERROR("[COMM_MEM] acquire name is null"); | 51 | + HCCL_VM_ERROR("acquire name is null"); |
| 52 | return nullptr; | 52 | return nullptr; |
| 53 | } | 53 | } |
| 54 | 54 | ||
| @@ -56,7 +56,7 @@ void* CommunicationMemoryManager::AllocCommMem(const char* name) | |||
| 56 | 56 | ||
| 57 | void* ptr = MemoryManager::GetInstance().AllocMemByName(name, sizeof(CommMemHead)); | 57 | void* ptr = MemoryManager::GetInstance().AllocMemByName(name, sizeof(CommMemHead)); |
| 58 | if (ptr == nullptr) { | 58 | if (ptr == nullptr) { |
| 59 | - HCCL_VM_ERROR("[COMM_MEM] alloc failed, name: {}", name); | 59 | + HCCL_VM_ERROR("alloc failed, name: {}", name); |
| 60 | return nullptr; | 60 | return nullptr; |
| 61 | } | 61 | } |
| 62 | 62 | ||
| @@ -64,7 +64,7 @@ void* CommunicationMemoryManager::AllocCommMem(const char* name) | |||
| 64 | head->bufferSize = 0; | 64 | head->bufferSize = 0; |
| 65 | head->writeTotalBytes = 0; | 65 | head->writeTotalBytes = 0; |
| 66 | head->readTotalBytes = 0; | 66 | head->readTotalBytes = 0; |
| 67 | - HCCL_VM_INFO("[COMM_MEM] acquire name: {}, ptr: {:p}", name, ptr); | 67 | + HCCL_VM_INFO("acquire name: {}, ptr: {:p}", name, ptr); |
| 68 | m_commMemMap[name] = ptr; | 68 | m_commMemMap[name] = ptr; |
| 69 | return head->data; | 69 | return head->data; |
| 70 | } | 70 | } |
| @@ -72,7 +72,7 @@ void* CommunicationMemoryManager::AllocCommMem(const char* name) | |||
| 72 | void* CommunicationMemoryManager::AcquireCommMem(const char* name) | 72 | void* CommunicationMemoryManager::AcquireCommMem(const char* name) |
| 73 | { | 73 | { |
| 74 | if (!name) { | 74 | if (!name) { |
| 75 | - HCCL_VM_ERROR("[COMM_MEM] acquire name is null"); | 75 | + HCCL_VM_ERROR("acquire name is null"); |
| 76 | return nullptr; | 76 | return nullptr; |
| 77 | } | 77 | } |
| 78 | void* ptr = nullptr; | 78 | void* ptr = nullptr; |
| @@ -83,7 +83,7 @@ void* CommunicationMemoryManager::AcquireCommMem(const char* name) | |||
| 83 | } else { | 83 | } else { |
| 84 | ptr = MemoryManager::GetInstance().AcquireMemByName(name); | 84 | ptr = MemoryManager::GetInstance().AcquireMemByName(name); |
| 85 | if (ptr == nullptr) { | 85 | if (ptr == nullptr) { |
| 86 | - HCCL_VM_ERROR("[COMM_MEM] acquire failed, name: {}", name); | 86 | + HCCL_VM_ERROR("acquire failed, name: {}", name); |
| 87 | return nullptr; | 87 | return nullptr; |
| 88 | } | 88 | } |
| 89 | m_commMemMap[name] = ptr; | 89 | m_commMemMap[name] = ptr; |
| @@ -95,17 +95,17 @@ void* CommunicationMemoryManager::AcquireCommMem(const char* name) | |||
| 95 | 95 | ||
| 96 | int CommunicationMemoryManager::ReleaseCommMem(const char* name) { | 96 | int CommunicationMemoryManager::ReleaseCommMem(const char* name) { |
| 97 | if (!name) { | 97 | if (!name) { |
| 98 | - HCCL_VM_ERROR("[COMM_MEM] release name is nullptr"); | 98 | + HCCL_VM_ERROR("release name is nullptr"); |
| 99 | return -1; | 99 | return -1; |
| 100 | } | 100 | } |
| 101 | 101 | ||
| 102 | std::lock_guard<std::mutex> lock(m_mutex); | 102 | std::lock_guard<std::mutex> lock(m_mutex); |
| 103 | if (m_commMemMap.find(name) == m_commMemMap.end()) { | 103 | if (m_commMemMap.find(name) == m_commMemMap.end()) { |
| 104 | - HCCL_VM_ERROR("[COMM_MEM] release comm mem not found, name: {}", name); | 104 | + HCCL_VM_ERROR("release comm mem not found, name: {}", name); |
| 105 | return -1; | 105 | return -1; |
| 106 | } | 106 | } |
| 107 | m_commMemMap.erase(name); | 107 | m_commMemMap.erase(name); |
| 108 | - HCCL_VM_INFO("[COMM_MEM] release name: {}", name); | 108 | + HCCL_VM_INFO("release name: {}", name); |
| 109 | MemoryManager::GetInstance().ReleaseMemByName(name); | 109 | MemoryManager::GetInstance().ReleaseMemByName(name); |
| 110 | return 0; | 110 | return 0; |
| 111 | } | 111 | } |
| @@ -113,13 +113,13 @@ int CommunicationMemoryManager::ReleaseCommMem(const char* name) { | |||
| 113 | int CommunicationMemoryManager::WriteCommMem(const char* name, const void* dataPtr, size_t size) | 113 | int CommunicationMemoryManager::WriteCommMem(const char* name, const void* dataPtr, size_t size) |
| 114 | { | 114 | { |
| 115 | if (!name || !dataPtr || size == 0) { | 115 | if (!name || !dataPtr || size == 0) { |
| 116 | - HCCL_VM_ERROR("[COMM_MEM] write invalid params"); | 116 | + HCCL_VM_ERROR("write invalid params, name: {}, dataPtr: {}, size: {}", name, dataPtr, size); |
| 117 | return -1; | 117 | return -1; |
| 118 | } | 118 | } |
| 119 | 119 | ||
| 120 | std::lock_guard<std::mutex> lock(m_mutex); | 120 | std::lock_guard<std::mutex> lock(m_mutex); |
| 121 | if (m_commMemMap.find(name) == m_commMemMap.end()) { | 121 | if (m_commMemMap.find(name) == m_commMemMap.end()) { |
| 122 | - HCCL_VM_ERROR("[COMM_MEM] write comm mem not found, name: {}", name); | 122 | + HCCL_VM_ERROR("write comm mem not found, name: {}", name); |
| 123 | return -1; | 123 | return -1; |
| 124 | } | 124 | } |
| 125 | void* ptr = m_commMemMap[name]; | 125 | void* ptr = m_commMemMap[name]; |
| @@ -129,7 +129,7 @@ int CommunicationMemoryManager::WriteCommMem(const char* name, const void* dataP | |||
| 129 | 129 | ||
| 130 | if (size + head->bufferSize > RA_SOCKET_BUF_SIZE) { | 130 | if (size + head->bufferSize > RA_SOCKET_BUF_SIZE) { |
| 131 | MemoryManager::GetInstance().UnlockMemByName(name); | 131 | MemoryManager::GetInstance().UnlockMemByName(name); |
| 132 | - HCCL_VM_ERROR("[COMM_MEM] write size too large, size: {}, max: {}", size, RA_SOCKET_BUF_SIZE); | 132 | + HCCL_VM_ERROR("write size too large, size: {}, max: {}", size, RA_SOCKET_BUF_SIZE); |
| 133 | return -1; | 133 | return -1; |
| 134 | } | 134 | } |
| 135 | 135 | ||
| @@ -139,20 +139,20 @@ int CommunicationMemoryManager::WriteCommMem(const char* name, const void* dataP | |||
| 139 | uint64_t totalWrite = head->writeTotalBytes; | 139 | uint64_t totalWrite = head->writeTotalBytes; |
| 140 | 140 | ||
| 141 | MemoryManager::GetInstance().UnlockMemByName(name); | 141 | MemoryManager::GetInstance().UnlockMemByName(name); |
| 142 | - HCCL_VM_INFO("[COMM_MEM] write name: {}, size: {}, totalWriteBytes: {}", name, size, totalWrite); | 142 | + HCCL_VM_INFO("write name: {}, size: {}, totalWriteBytes: {}", name, size, totalWrite); |
| 143 | return 0; | 143 | return 0; |
| 144 | } | 144 | } |
| 145 | 145 | ||
| 146 | int CommunicationMemoryManager::ReadCommMem(const char* name, void* dataPtr, size_t size) | 146 | int CommunicationMemoryManager::ReadCommMem(const char* name, void* dataPtr, size_t size) |
| 147 | { | 147 | { |
| 148 | if (!name || !dataPtr || size == 0) { | 148 | if (!name || !dataPtr || size == 0) { |
| 149 | - HCCL_VM_ERROR("[COMM_MEM] read invalid params"); | 149 | + HCCL_VM_ERROR("read invalid params, name: {}, dataPtr: {}, size: {}", name, dataPtr, size); |
| 150 | return -1; | 150 | return -1; |
| 151 | } | 151 | } |
| 152 | 152 | ||
| 153 | std::lock_guard<std::mutex> lock(m_mutex); | 153 | std::lock_guard<std::mutex> lock(m_mutex); |
| 154 | if (m_commMemMap.find(name) == m_commMemMap.end()) { | 154 | if (m_commMemMap.find(name) == m_commMemMap.end()) { |
| 155 | - HCCL_VM_ERROR("[COMM_MEM] read comm mem not found, name: {}", name); | 155 | + HCCL_VM_ERROR("read comm mem not found, name: {}", name); |
| 156 | return -1; | 156 | return -1; |
| 157 | } | 157 | } |
| 158 | void* ptr = m_commMemMap[name]; | 158 | void* ptr = m_commMemMap[name]; |
| @@ -177,7 +177,7 @@ int CommunicationMemoryManager::ReadCommMem(const char* name, void* dataPtr, siz | |||
| 177 | 177 | ||
| 178 | MemoryManager::GetInstance().UnlockMemByName(name); | 178 | MemoryManager::GetInstance().UnlockMemByName(name); |
| 179 | 179 | ||
| 180 | - HCCL_VM_INFO("[COMM_MEM] read name: {}, size: {}, totalReadBytes: {}", name, recvSize, totalRead); | 180 | + HCCL_VM_INFO("read name: {}, size: {}, totalReadBytes: {}", name, recvSize, totalRead); |
| 181 | return recvSize; | 181 | return recvSize; |
| 182 | } | 182 | } |
| 183 | 183 | ||
| @@ -65,7 +65,7 @@ void* MemoryManager::AllocMemByName(const char* name, size_t size) { | |||
| 65 | memInfo.size = size; | 65 | memInfo.size = size; |
| 66 | memInfo.type = 0; // 0: 共享内存 | 66 | memInfo.type = 0; // 0: 共享内存 |
| 67 | memInfo.refCount = 1; | 67 | memInfo.refCount = 1; |
| 68 | - HCCL_VM_INFO("[MEM_MANAGER] alloc name: {}, size: {:d}, ptr: {:p}", name, size, ptr); | 68 | + HCCL_VM_INFO("alloc name: {}, size: {:d}, ptr: {:p}", name, size, ptr); |
| 69 | m_memMap[nameStr] = memInfo; | 69 | m_memMap[nameStr] = memInfo; |
| 70 | return ptr; | 70 | return ptr; |
| 71 | } | 71 | } |
| @@ -81,7 +81,7 @@ void MemoryManager::FreeMemByName(const char* name) { | |||
| 81 | auto it = m_memMap.find(nameStr); | 81 | auto it = m_memMap.find(nameStr); |
| 82 | if (it != m_memMap.end()) { | 82 | if (it != m_memMap.end()) { |
| 83 | // 关闭共享内存 | 83 | // 关闭共享内存 |
| 84 | - HCCL_VM_INFO("[MEM_MANAGER] free name: {}, size: {:d}, ptr: {:p}", name, it->second.size, it->second.ptr); | 84 | + HCCL_VM_INFO("free name: {}, size: {:d}, ptr: {:p}", name, it->second.size, it->second.ptr); |
| 85 | ShmClose(it->second.ptr); | 85 | ShmClose(it->second.ptr); |
| 86 | // 从映射表中移除 | 86 | // 从映射表中移除 |
| 87 | m_memMap.erase(it); | 87 | m_memMap.erase(it); |
| @@ -145,7 +145,7 @@ void* MemoryManager::AcquireMemByName(const char* name) { | |||
| 145 | memInfo.size = size; // 使用ShmOpen返回的大小 | 145 | memInfo.size = size; // 使用ShmOpen返回的大小 |
| 146 | memInfo.type = 0; // 0: 共享内存 | 146 | memInfo.type = 0; // 0: 共享内存 |
| 147 | memInfo.refCount = 1; | 147 | memInfo.refCount = 1; |
| 148 | - HCCL_VM_INFO("[MEM_MANAGER] acquire name: {}, size: {:d}, ptr: {:p}", name, size, ptr); | 148 | + HCCL_VM_INFO("acquire name: {}, size: {:d}, ptr: {:p}", name, size, ptr); |
| 149 | m_memMap[nameStr] = memInfo; | 149 | m_memMap[nameStr] = memInfo; |
| 150 | return ptr; | 150 | return ptr; |
| 151 | } | 151 | } |
| @@ -164,7 +164,7 @@ void MemoryManager::ReleaseMemByName(const char* name) { | |||
| 164 | int refCount = __sync_fetch_and_sub(&it->second.refCount, 1); | 164 | int refCount = __sync_fetch_and_sub(&it->second.refCount, 1); |
| 165 | if (refCount == 1) { | 165 | if (refCount == 1) { |
| 166 | // 引用计数归0,关闭共享内存并从映射表中移除 | 166 | // 引用计数归0,关闭共享内存并从映射表中移除 |
| 167 | - HCCL_VM_INFO("[MEM_MANAGER] release name: {}, ptr: {:p}", name, it->second.ptr); | 167 | + HCCL_VM_INFO("release name: {}, ptr: {:p}", name, it->second.ptr); |
| 168 | ShmClose(it->second.ptr); | 168 | ShmClose(it->second.ptr); |
| 169 | m_memMap.erase(it); | 169 | m_memMap.erase(it); |
| 170 | } | 170 | } |
| @@ -92,13 +92,13 @@ static uint64_t FromLe64(uint64_t value) { | |||
| 92 | 92 | ||
| 93 | void* ShmCreate(const char* name, size_t size) { | 93 | void* ShmCreate(const char* name, size_t size) { |
| 94 | if (!name || size == 0) { | 94 | if (!name || size == 0) { |
| 95 | - HCCL_VM_ERROR("[SHM_OPS] create: name is nullptr or size is 0"); | 95 | + HCCL_VM_ERROR("create: name is nullptr or size is 0"); |
| 96 | return nullptr; | 96 | return nullptr; |
| 97 | } | 97 | } |
| 98 | 98 | ||
| 99 | int shmFd = shm_open(name, O_CREAT | O_RDWR | O_EXCL, 0666); | 99 | int shmFd = shm_open(name, O_CREAT | O_RDWR | O_EXCL, 0666); |
| 100 | if (shmFd == -1) { | 100 | if (shmFd == -1) { |
| 101 | - HCCL_VM_ERROR("[SHM_OPS] create: shm_open failed, name: {}", name); | 101 | + HCCL_VM_ERROR("create: shm_open failed, name: {}", name); |
| 102 | return nullptr; | 102 | return nullptr; |
| 103 | } | 103 | } |
| 104 | 104 | ||
| @@ -106,7 +106,7 @@ void* ShmCreate(const char* name, size_t size) { | |||
| 106 | if (ftruncate(shmFd, totalSize) == -1) { | 106 | if (ftruncate(shmFd, totalSize) == -1) { |
| 107 | close(shmFd); | 107 | close(shmFd); |
| 108 | shm_unlink(name); | 108 | shm_unlink(name); |
| 109 | - HCCL_VM_ERROR("[SHM_OPS] create: ftruncate failed, name: {}", name); | 109 | + HCCL_VM_ERROR("create: ftruncate failed, name: {}", name); |
| 110 | return nullptr; | 110 | return nullptr; |
| 111 | } | 111 | } |
| 112 | 112 | ||
| @@ -115,7 +115,7 @@ void* ShmCreate(const char* name, size_t size) { | |||
| 115 | 115 | ||
| 116 | if (addr == MAP_FAILED) { | 116 | if (addr == MAP_FAILED) { |
| 117 | shm_unlink(name); | 117 | shm_unlink(name); |
| 118 | - HCCL_VM_ERROR("[SHM_OPS] create: mmap failed, name: {}", name); | 118 | + HCCL_VM_ERROR("create: mmap failed, name: {}", name); |
| 119 | return nullptr; | 119 | return nullptr; |
| 120 | } | 120 | } |
| 121 | 121 | ||
| @@ -127,25 +127,25 @@ void* ShmCreate(const char* name, size_t size) { | |||
| 127 | head->size = ToLe64(size); | 127 | head->size = ToLe64(size); |
| 128 | head->lock = 0; | 128 | head->lock = 0; |
| 129 | head->refCount = 1; | 129 | head->refCount = 1; |
| 130 | - HCCL_VM_INFO("[SHM_OPS] create name: {}, size: {:d}, ptr: {:p} ref:{:d}", head->name, size, (char*)addr + SHM_HEAD_SIZE, (int)head->refCount); | 130 | + HCCL_VM_INFO("create name: {}, size: {:d}, ptr: {:p} ref:{:d}", head->name, size, (char*)addr + SHM_HEAD_SIZE, (int)head->refCount); |
| 131 | return (char*)addr + SHM_HEAD_SIZE; | 131 | return (char*)addr + SHM_HEAD_SIZE; |
| 132 | } | 132 | } |
| 133 | 133 | ||
| 134 | void* ShmOpen(const char* name, size_t* size) { | 134 | void* ShmOpen(const char* name, size_t* size) { |
| 135 | if (!name) { | 135 | if (!name) { |
| 136 | - HCCL_VM_ERROR("[SHM_OPS] open: name is nullptr"); | 136 | + HCCL_VM_ERROR("open: name is nullptr"); |
| 137 | return nullptr; | 137 | return nullptr; |
| 138 | } | 138 | } |
| 139 | 139 | ||
| 140 | int shmFd = shm_open(name, O_RDWR, 0666); | 140 | int shmFd = shm_open(name, O_RDWR, 0666); |
| 141 | if (shmFd == -1) { | 141 | if (shmFd == -1) { |
| 142 | - HCCL_VM_ERROR("[SHM_OPS] open: shm_open failed, name: {}", name); | 142 | + HCCL_VM_ERROR("open: shm_open failed, name: {}", name); |
| 143 | return nullptr; | 143 | return nullptr; |
| 144 | } | 144 | } |
| 145 | 145 | ||
| 146 | struct stat statBuf; | 146 | struct stat statBuf; |
| 147 | if (fstat(shmFd, &statBuf) == -1) { | 147 | if (fstat(shmFd, &statBuf) == -1) { |
| 148 | - HCCL_VM_ERROR("[SHM_OPS] open: fstat failed, name: {}", name); | 148 | + HCCL_VM_ERROR("open: fstat failed, name: {}", name); |
| 149 | close(shmFd); | 149 | close(shmFd); |
| 150 | return nullptr; | 150 | return nullptr; |
| 151 | } | 151 | } |
| @@ -154,13 +154,13 @@ void* ShmOpen(const char* name, size_t* size) { | |||
| 154 | close(shmFd); | 154 | close(shmFd); |
| 155 | 155 | ||
| 156 | if (addr == MAP_FAILED) { | 156 | if (addr == MAP_FAILED) { |
| 157 | - HCCL_VM_ERROR("[SHM_OPS] open: mmap failed, name: {}", name); | 157 | + HCCL_VM_ERROR("open: mmap failed, name: {}", name); |
| 158 | return nullptr; | 158 | return nullptr; |
| 159 | } | 159 | } |
| 160 | 160 | ||
| 161 | ShmHead* head = (ShmHead*)addr; | 161 | ShmHead* head = (ShmHead*)addr; |
| 162 | if (FromLe32(head->magic) != SHM_MAGIC) { | 162 | if (FromLe32(head->magic) != SHM_MAGIC) { |
| 163 | - HCCL_VM_ERROR("[SHM_OPS] open: invalid magic number, name: {}", name); | 163 | + HCCL_VM_ERROR("open: invalid magic number, name: {}", name); |
| 164 | munmap(addr, statBuf.st_size); | 164 | munmap(addr, statBuf.st_size); |
| 165 | return nullptr; | 165 | return nullptr; |
| 166 | } | 166 | } |
| @@ -173,14 +173,14 @@ void* ShmOpen(const char* name, size_t* size) { | |||
| 173 | current_count = FromLe32(head->refCount); | 173 | current_count = FromLe32(head->refCount); |
| 174 | new_count = current_count + 1; | 174 | new_count = current_count + 1; |
| 175 | } | 175 | } |
| 176 | - HCCL_VM_INFO("[SHM_OPS] open name: {}, size: {:d}, ptr: {:p}", head->name, *size, (char*)addr + SHM_HEAD_SIZE); | 176 | + HCCL_VM_INFO("open name: {}, size: {:d}, ptr: {:p}", head->name, *size, (char*)addr + SHM_HEAD_SIZE); |
| 177 | return (void*)((char*)addr + SHM_HEAD_SIZE); | 177 | return (void*)((char*)addr + SHM_HEAD_SIZE); |
| 178 | } | 178 | } |
| 179 | 179 | ||
| 180 | void ShmClose(void* shm) | 180 | void ShmClose(void* shm) |
| 181 | { | 181 | { |
| 182 | if (!shm) { | 182 | if (!shm) { |
| 183 | - HCCL_VM_ERROR("[SHM_OPS] close: shm is nullptr"); | 183 | + HCCL_VM_ERROR("close: shm is nullptr"); |
| 184 | return; | 184 | return; |
| 185 | } | 185 | } |
| 186 | 186 | ||
| @@ -195,10 +195,10 @@ void ShmClose(void* shm) | |||
| 195 | munmap(head, totalSize); | 195 | munmap(head, totalSize); |
| 196 | if (refCount == 1) { | 196 | if (refCount == 1) { |
| 197 | // 最后一个引用,删除共享内存 | 197 | // 最后一个引用,删除共享内存 |
| 198 | - HCCL_VM_INFO("[SHM_OPS] close name: {}, ptr: {:p}", tmp, (char*)head); | 198 | + HCCL_VM_INFO("close name: {}, ptr: {:p}", tmp, (char*)head); |
| 199 | shm_unlink(tmp); | 199 | shm_unlink(tmp); |
| 200 | } else { | 200 | } else { |
| 201 | - HCCL_VM_INFO("[SHM_OPS] close name: {}, ptr: {:p}, refCount: {} {}", tmp, (char*)head, refCount, ref); | 201 | + HCCL_VM_INFO("close name: {}, ptr: {:p}, refCount: {} {}", tmp, (char*)head, refCount, ref); |
| 202 | } | 202 | } |
| 203 | } | 203 | } |
| 204 | 204 | ||
| @@ -32,7 +32,7 @@ HcclVmResult GetAddrByOffset(uint64_t offset, VmUniquePtr& addrPtr) // todo | |||
| 32 | { | 32 | { |
| 33 | // 1. 入参合法性检查(避免空指针和无效输出) | 33 | // 1. 入参合法性检查(避免空指针和无效输出) |
| 34 | if (addrPtr) { | 34 | if (addrPtr) { |
| 35 | - HCCL_VM_ERROR("[GetAddrByOffset] 错误:指针已持有资源,不可重复获取!"); | 35 | + HCCL_VM_ERROR("错误:指针已持有资源,不可重复获取!"); |
| 36 | return HcclVmResult::HCCL_SIM_E_PARA; | 36 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 37 | } | 37 | } |
| 38 | 38 | ||
| @@ -40,12 +40,12 @@ HcclVmResult GetAddrByOffset(uint64_t offset, VmUniquePtr& addrPtr) // todo | |||
| 40 | sim::PhyMemBlock phyMem{}; | 40 | sim::PhyMemBlock phyMem{}; |
| 41 | void *addr = sim::AcquireDevPtrInNoHostProcess(offserPtr, phyMem); | 41 | void *addr = sim::AcquireDevPtrInNoHostProcess(offserPtr, phyMem); |
| 42 | if (addr == nullptr) { | 42 | if (addr == nullptr) { |
| 43 | - HCCL_VM_ERROR("[GetAddrByOffset] 错误:无法获取设备地址(addr= {})!", offserPtr); | 43 | + HCCL_VM_ERROR("错误:无法获取设备地址(addr= {})!", offserPtr); |
| 44 | return HcclVmResult::HCCL_SIM_E_PTR; | 44 | return HcclVmResult::HCCL_SIM_E_PTR; |
| 45 | } | 45 | } |
| 46 | 46 | ||
| 47 | // 5. 日志打印(可选,便于问题排查) | 47 | // 5. 日志打印(可选,便于问题排查) |
| 48 | - HCCL_VM_INFO("[GetAddrByOffset] 计算成功, 虚拟地址: {}, 物理地址: {}", offset, addr); | 48 | + HCCL_VM_INFO("计算成功, 虚拟地址: {}, 物理地址: {}", offset, addr); |
| 49 | 49 | ||
| 50 | addrPtr = VmUniquePtr(addr, VmPtrReleaser{phyMem}); | 50 | addrPtr = VmUniquePtr(addr, VmPtrReleaser{phyMem}); |
| 51 | 51 | ||
| @@ -58,11 +58,11 @@ HcclVmResult InsertTaskToCollection(HcclTaskMetaData *task, uint32_t *index) | |||
| 58 | 58 | ||
| 59 | // 1. 入参合法性检查(避免空指针访问) | 59 | // 1. 入参合法性检查(避免空指针访问) |
| 60 | if (task == nullptr) { | 60 | if (task == nullptr) { |
| 61 | - HCCL_VM_ERROR("[InsertTaskToCollection] 错误:输入任务指针 task 不能为空!"); | 61 | + HCCL_VM_ERROR("错误:输入任务指针 task 不能为空!"); |
| 62 | return HcclVmResult::HCCL_SIM_E_PARA; | 62 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 63 | } | 63 | } |
| 64 | if (index == nullptr) { | 64 | if (index == nullptr) { |
| 65 | - HCCL_VM_ERROR("[InsertTaskToCollection] 错误:输出索引指针 index 不能为空!"); | 65 | + HCCL_VM_ERROR("错误:输出索引指针 index 不能为空!"); |
| 66 | return HcclVmResult::HCCL_SIM_E_PARA; | 66 | return HcclVmResult::HCCL_SIM_E_PARA; |
| 67 | } | 67 | } |
| 68 | 68 | ||
| @@ -81,7 +81,7 @@ HcclVmResult InsertTaskToCollection(HcclTaskMetaData *task, uint32_t *index) | |||
| 81 | 81 | ||
| 82 | auto ret = sim::InsertOpTask(opTaskInfo); | 82 | auto ret = sim::InsertOpTask(opTaskInfo); |
| 83 | if (ret != 0) { | 83 | if (ret != 0) { |
| 84 | - HCCL_VM_ERROR("[InsertTaskToCollection] 错误:插入任务到数据库失败 - {}", ret); | 84 | + HCCL_VM_ERROR("错误:插入任务到数据库失败 - {}", ret); |
| 85 | return HcclVmResult::HCCL_SIM_SHM_FAIL; | 85 | return HcclVmResult::HCCL_SIM_SHM_FAIL; |
| 86 | } | 86 | } |
| 87 | return HcclVmResult::HCCL_SIM_SUCCESS; | 87 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| @@ -40,3 +40,6 @@ target_compile_options(topo PRIVATE | |||
| 40 | ) | 40 | ) |
| 41 | 41 | ||
| 42 | target_link_libraries(topo PRIVATE log) | 42 | target_link_libraries(topo PRIVATE log) |
| 43 | + | ||
| 44 | +# 日志染色: 模块 tag | ||
| 45 | +target_compile_definitions(topo PRIVATE HCCL_VM_MODULE="TOPO") | ||
| @@ -11,6 +11,7 @@ | |||
| 11 | 11 | ||
| 12 | 12 | ||
| 13 | 13 | ||
| 14 | + | ||
| 14 | 15 | ||
| 15 | 16 | ||
| 16 | 17 | ||
| @@ -137,7 +138,7 @@ HcclVmResult AscendClusterTopoParser::InitClusterTopo(const std::string &cluster | |||
| 137 | */ | 138 | */ |
| 138 | HcclVmResult AscendClusterTopoParser::ParseRanktableAndInitCommDomain(const std::string &ranktablePath) | 139 | HcclVmResult AscendClusterTopoParser::ParseRanktableAndInitCommDomain(const std::string &ranktablePath) |
| 139 | { | 140 | { |
| 140 | - HCCL_VM_DEBUG("Enter ParseRanktableAndInitCommDomain, status: {}"); | 141 | + HCCL_VM_DEBUG("Enter ParseRanktableAndInitCommDomain, ranktablePath: {}", ranktablePath); |
| 141 | 142 | ||
| 142 | TopoMeta topoMeta; | 143 | TopoMeta topoMeta; |
| 143 | if (ParseRanktable(ranktablePath, topoMeta) != HcclVmResult::HCCL_SIM_SUCCESS) { | 144 | if (ParseRanktable(ranktablePath, topoMeta) != HcclVmResult::HCCL_SIM_SUCCESS) { |
| @@ -416,13 +417,7 @@ HcclVmResult AscendClusterTopoParser::CreateRankTableFile(const TopoMeta &topoMe | |||
| 416 | } | 417 | } |
| 417 | } | 418 | } |
| 418 | 419 | ||
| 419 | - std::error_code ec; | 420 | + fs::path dataDir = fs::path(InstallPath::ResolveToInstallRoot("data")); |
| 420 | - fs::path exePath = fs::read_symlink("/proc/self/exe", ec); | ||
| 421 | - fs::path binDir = ec ? fs::current_path() : exePath.parent_path(); | ||
| 422 | - if (binDir.filename() == "bin") { | ||
| 423 | - binDir = binDir.parent_path(); | ||
| 424 | - } | ||
| 425 | - fs::path dataDir = binDir / "data"; | ||
| 426 | std::error_code ec2; | 421 | std::error_code ec2; |
| 427 | fs::create_directories(dataDir, ec2); | 422 | fs::create_directories(dataDir, ec2); |
| 428 | std::string outputPath = (dataDir / "ranktable.json").string(); | 423 | std::string outputPath = (dataDir / "ranktable.json").string(); |
| @@ -435,6 +430,8 @@ HcclVmResult AscendClusterTopoParser::CreateRankTableFile(const TopoMeta &topoMe | |||
| 435 | ofs << rankTable.dump(4) << std::endl; | 430 | ofs << rankTable.dump(4) << std::endl; |
| 436 | ofs.close(); | 431 | ofs.close(); |
| 437 | 432 | ||
| 433 | + setenv("RANK_TABLE_FILE", outputPath.c_str(), 1); | ||
| 434 | + | ||
| 438 | HCCL_VM_DEBUG("[{}] ranktable.json generated with {:d} ranks", __func__, rankId); | 435 | HCCL_VM_DEBUG("[{}] ranktable.json generated with {:d} ranks", __func__, rankId); |
| 439 | return HcclVmResult::HCCL_SIM_SUCCESS; | 436 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 440 | } | 437 | } |
| @@ -685,7 +682,7 @@ HcclVmResult AscendClusterTopoParser::ParseRanktable(const std::string &ranktabl | |||
| 685 | !network_.superPods[0].servers[0].devices.empty()) { | 682 | !network_.superPods[0].servers[0].devices.empty()) { |
| 686 | } | 683 | } |
| 687 | 684 | ||
| 688 | - HCCL_VM_DEBUG("[{}] parsed ranktable: podNum={}, serverCount={}, rankNum={}", | 685 | + HCCL_VM_DEBUG("[{}] parsed ranktable: topoMetaSize={}", |
| 689 | __func__, topoMeta.size()); | 686 | __func__, topoMeta.size()); |
| 690 | return HcclVmResult::HCCL_SIM_SUCCESS; | 687 | return HcclVmResult::HCCL_SIM_SUCCESS; |
| 691 | } | 688 | } |
| @@ -12,6 +12,7 @@ | |||
| 12 | 12 | ||
| 13 | 13 | ||
| 14 | 14 | ||
| 15 | + | ||
| 15 | 16 | ||
| 16 | std::string ClusterTopoDumper::Indent(int level) { | 17 | std::string ClusterTopoDumper::Indent(int level) { |
| 17 | return std::string(level * 2, ' '); | 18 | return std::string(level * 2, ' '); |
| @@ -20,12 +21,12 @@ std::string ClusterTopoDumper::Indent(int level) { | |||
| 20 | void ClusterTopoDumper::DumpToFile(const Network& network, const std::string& filePath) { | 21 | void ClusterTopoDumper::DumpToFile(const Network& network, const std::string& filePath) { |
| 21 | std::ofstream ofs(filePath); | 22 | std::ofstream ofs(filePath); |
| 22 | if (!ofs.is_open()) { | 23 | if (!ofs.is_open()) { |
| 23 | - fprintf(stderr, "Error: cannot open file for writing: %s\n", filePath.c_str()); | 24 | + HCCL_VM_ERROR("cannot open file for writing: {}", filePath.c_str()); |
| 24 | return; | 25 | return; |
| 25 | } | 26 | } |
| 26 | DumpToStream(network, ofs); | 27 | DumpToStream(network, ofs); |
| 27 | ofs.close(); | 28 | ofs.close(); |
| 28 | - printf("IR data dumped to: %s\n", filePath.c_str()); | 29 | + HCCL_VM_INFO("IR data dumped to: {}", filePath.c_str()); |
| 29 | } | 30 | } |
| 30 | 31 | ||
| 31 | void ClusterTopoDumper::DumpToStream(const Network& network, std::ostream& os) { | 32 | void ClusterTopoDumper::DumpToStream(const Network& network, std::ostream& os) { |
| @@ -18,6 +18,7 @@ | |||
| 18 | 18 | ||
| 19 | 19 | ||
| 20 | 20 | ||
| 21 | + | ||
| 21 | 22 | ||
| 22 | using json = nlohmann::json; | 23 | using json = nlohmann::json; |
| 23 | 24 | ||
| @@ -36,7 +37,7 @@ ParseStatus ClusterTopoParser::ParseClusterDir(const std::string& clusterDir) { | |||
| 36 | std::string serversInfoPath = clusterDir + "/servers_info.json"; | 37 | std::string serversInfoPath = clusterDir + "/servers_info.json"; |
| 37 | ParseStatus siStatus = ParseServersInfoJson(serversInfoPath); | 38 | ParseStatus siStatus = ParseServersInfoJson(serversInfoPath); |
| 38 | if (siStatus != ParseStatus::OK) { | 39 | if (siStatus != ParseStatus::OK) { |
| 39 | - fprintf(stderr, "Warning: failed to parse servers_info.json\n"); | 40 | + HCCL_VM_WARN("failed to parse servers_info.json"); |
| 40 | } | 41 | } |
| 41 | 42 | ||
| 42 | return network_.BuildIndexes(); | 43 | return network_.BuildIndexes(); |
| @@ -81,7 +82,7 @@ ParseStatus ClusterTopoParser::ScanSuperPodDirs(const std::string& clusterDir) { | |||
| 81 | try { | 82 | try { |
| 82 | superpod.superPodId = std::stoi(numStr); | 83 | superpod.superPodId = std::stoi(numStr); |
| 83 | } catch (...) { | 84 | } catch (...) { |
| 84 | - fprintf(stderr, "Convert numStr[%s] to int error.\n", numStr.c_str()); | 85 | + HCCL_VM_ERROR("Convert numStr[{}] to int error", numStr.c_str()); |
| 85 | } | 86 | } |
| 86 | } | 87 | } |
| 87 | 88 | ||
| @@ -139,7 +140,7 @@ ParseStatus ClusterTopoParser::ScanServerDirs(const std::string& superpodDir, Su | |||
| 139 | try { | 140 | try { |
| 140 | server.serverId = std::stoi(numStr); | 141 | server.serverId = std::stoi(numStr); |
| 141 | } catch (...) { | 142 | } catch (...) { |
| 142 | - fprintf(stderr, "Convert numStr[%s] to int error.\n", numStr.c_str()); | 143 | + HCCL_VM_ERROR("Convert numStr[{}] to int error", numStr.c_str()); |
| 143 | } | 144 | } |
| 144 | } | 145 | } |
| 145 | 146 | ||
| @@ -159,10 +160,10 @@ ParseStatus ClusterTopoParser::ParseServer(const std::string& serverDir, Server& | |||
| 159 | std::string rootinfoPath = FindFileBySuffix(serverDir, "rootinfo"); | 160 | std::string rootinfoPath = FindFileBySuffix(serverDir, "rootinfo"); |
| 160 | 161 | ||
| 161 | if (topoPath.empty()) { | 162 | if (topoPath.empty()) { |
| 162 | - fprintf(stderr, "Warning: no topo.json found in %s\n", serverDir.c_str()); | 163 | + HCCL_VM_WARN("no topo.json found in {}", serverDir.c_str()); |
| 163 | } | 164 | } |
| 164 | if (rootinfoPath.empty()) { | 165 | if (rootinfoPath.empty()) { |
| 165 | - fprintf(stderr, "Warning: no rootinfo.json found in %s\n", serverDir.c_str()); | 166 | + HCCL_VM_WARN("no rootinfo.json found in {}", serverDir.c_str()); |
| 166 | } | 167 | } |
| 167 | 168 | ||
| 168 | if (!topoPath.empty()) { | 169 | if (!topoPath.empty()) { |
| @@ -299,7 +300,7 @@ ParseStatus ClusterTopoParser::ParseTopoJson(const std::string& topoPath, Server | |||
| 299 | } | 300 | } |
| 300 | } | 301 | } |
| 301 | } catch (const json::exception& e) { | 302 | } catch (const json::exception& e) { |
| 302 | - fprintf(stderr, "JSON parse error in %s: %s\n", topoPath.c_str(), e.what()); | 303 | + HCCL_VM_ERROR("JSON parse error in {}: {}", topoPath.c_str(), e.what()); |
| 303 | return ParseStatus::PARSE_ERROR; | 304 | return ParseStatus::PARSE_ERROR; |
| 304 | } | 305 | } |
| 305 | 306 | ||
| @@ -413,7 +414,7 @@ ParseStatus ClusterTopoParser::ParseRootinfoJson(const std::string& rootinfoPath | |||
| 413 | } | 414 | } |
| 414 | } | 415 | } |
| 415 | } catch (const json::exception& e) { | 416 | } catch (const json::exception& e) { |
| 416 | - fprintf(stderr, "JSON parse error in %s: %s\n", rootinfoPath.c_str(), e.what()); | 417 | + HCCL_VM_ERROR("JSON parse error in {}: {}", rootinfoPath.c_str(), e.what()); |
| 417 | return ParseStatus::PARSE_ERROR; | 418 | return ParseStatus::PARSE_ERROR; |
| 418 | } | 419 | } |
| 419 | 420 | ||
| @@ -649,7 +650,7 @@ int ClusterTopoParser::ExtractDieIdFromPortId(const std::string& portId) { | |||
| 649 | try { | 650 | try { |
| 650 | return std::stoi(portId.substr(0, slashPos)); | 651 | return std::stoi(portId.substr(0, slashPos)); |
| 651 | } catch (...) { | 652 | } catch (...) { |
| 652 | - fprintf(stderr, "ExtractDieIdFromPortId error.\n"); | 653 | + HCCL_VM_ERROR("ExtractDieIdFromPortId error"); |
| 653 | return -1; | 654 | return -1; |
| 654 | } | 655 | } |
| 655 | } | 656 | } |
| @@ -659,7 +660,7 @@ int ClusterTopoParser::ExtractDieIdFromPortId(const std::string& portId) { | |||
| 659 | ParseStatus ClusterTopoParser::ParseServersInfoJson(const std::string& serversInfoPath) { | 660 | ParseStatus ClusterTopoParser::ParseServersInfoJson(const std::string& serversInfoPath) { |
| 660 | std::ifstream ifs(serversInfoPath); | 661 | std::ifstream ifs(serversInfoPath); |
| 661 | if (!ifs.is_open()) { | 662 | if (!ifs.is_open()) { |
| 662 | - fprintf(stderr, "Warning: servers_info.json not found: %s\n", serversInfoPath.c_str()); | 663 | + HCCL_VM_WARN("servers_info.json not found: {}", serversInfoPath.c_str()); |
| 663 | return ParseStatus::FILE_NOT_FOUND; | 664 | return ParseStatus::FILE_NOT_FOUND; |
| 664 | } | 665 | } |
| 665 | 666 | ||
| @@ -704,7 +705,7 @@ ParseStatus ClusterTopoParser::ParseServersInfoJson(const std::string& serversIn | |||
| 704 | } | 705 | } |
| 705 | } | 706 | } |
| 706 | } catch (const json::exception& e) { | 707 | } catch (const json::exception& e) { |
| 707 | - fprintf(stderr, "JSON parse error in %s: %s\n", serversInfoPath.c_str(), e.what()); | 708 | + HCCL_VM_ERROR("JSON parse error in {}: {}", serversInfoPath.c_str(), e.what()); |
| 708 | return ParseStatus::PARSE_ERROR; | 709 | return ParseStatus::PARSE_ERROR; |
| 709 | } | 710 | } |
| 710 | 711 | ||
| @@ -82,3 +82,9 @@ target_link_libraries(loader PRIVATE | |||
| 82 | runnerdb | 82 | runnerdb |
| 83 | rt | 83 | rt |
| 84 | ) | 84 | ) |
| 85 | + | ||
| 86 | +# 日志染色: 模块 tag | ||
| 87 | +target_compile_definitions(log PRIVATE HCCL_VM_MODULE="UTILS") | ||
| 88 | + | ||
| 89 | +# 日志染色: 模块 tag | ||
| 90 | +target_compile_definitions(loader PRIVATE HCCL_VM_MODULE="UTILS") | ||
| @@ -31,15 +31,15 @@ HcclResult Loader::LoadOpTaskFile(const std::string dbPath) | |||
| 31 | 31 | ||
| 32 | if (!dbPath.empty()) { | 32 | if (!dbPath.empty()) { |
| 33 | if (!std::filesystem::exists(dbPath)) { | 33 | if (!std::filesystem::exists(dbPath)) { |
| 34 | - HCCL_VM_ERROR("[Loader] Backup database file not found: {}", dbPath); | 34 | + HCCL_VM_ERROR("Backup database file not found: {}", dbPath); |
| 35 | return HcclResult::HCCL_E_PARA; | 35 | return HcclResult::HCCL_E_PARA; |
| 36 | } | 36 | } |
| 37 | targetPath = dbPath; | 37 | targetPath = dbPath; |
| 38 | - HCCL_VM_INFO("[Loader] Loading from specific backup path: {}", targetPath); | 38 | + HCCL_VM_INFO("Loading from specific backup path: {}", targetPath); |
| 39 | } else { | 39 | } else { |
| 40 | targetPath = InstallPath::ResolveToInstallRoot("data/hccl_vm_data.db"); | 40 | targetPath = InstallPath::ResolveToInstallRoot("data/hccl_vm_data.db"); |
| 41 | std::string absPath = std::filesystem::absolute(targetPath).string(); | 41 | std::string absPath = std::filesystem::absolute(targetPath).string(); |
| 42 | - HCCL_VM_INFO("[Loader] Loading using default configuration path: {}", absPath); | 42 | + HCCL_VM_INFO("Loading using default configuration path: {}", absPath); |
| 43 | } | 43 | } |
| 44 | 44 | ||
| 45 | config.dbPath = targetPath; | 45 | config.dbPath = targetPath; |
| @@ -62,67 +62,67 @@ HcclResult Loader::LoadRunnerSingleSync(const uint32_t& outSyncIter, | |||
| 62 | HCCL_VM_ERROR("QueryCompositeOpDetailBySyncIter failed for syncIter: {}", outSyncIter); | 62 | HCCL_VM_ERROR("QueryCompositeOpDetailBySyncIter failed for syncIter: {}", outSyncIter); |
| 63 | return HcclResult::HCCL_E_PARA; | 63 | return HcclResult::HCCL_E_PARA; |
| 64 | } | 64 | } |
| 65 | - HCCL_VM_INFO("[LoadRunnerSingleSync] syncIter={} loaded. Rank Count: {}", outSyncIter, compositeDataMap.size()); | 65 | + HCCL_VM_INFO("syncIter={} loaded. Rank Count: {}", outSyncIter, compositeDataMap.size()); |
| 66 | return HcclResult::HCCL_SUCCESS; | 66 | return HcclResult::HCCL_SUCCESS; |
| 67 | } | 67 | } |
| 68 | 68 | ||
| 69 | HcclResult Loader::GetCcuChannelInfo(std::vector<sim::CcuChannelTab>& channels) | 69 | HcclResult Loader::GetCcuChannelInfo(std::vector<sim::CcuChannelTab>& channels) |
| 70 | { | 70 | { |
| 71 | if (sim::QueryCcuChannelAll(channels) != 0) { | 71 | if (sim::QueryCcuChannelAll(channels) != 0) { |
| 72 | - HCCL_VM_ERROR("[GetCcuChannelInfo] QueryCcuChannelAll failed"); | 72 | + HCCL_VM_ERROR("QueryCcuChannelAll failed"); |
| 73 | return HcclResult::HCCL_E_PARA; | 73 | return HcclResult::HCCL_E_PARA; |
| 74 | } | 74 | } |
| 75 | - HCCL_VM_INFO("[GetCcuChannelInfo] Get {} channels", channels.size()); | 75 | + HCCL_VM_INFO("Get {} channels", channels.size()); |
| 76 | return HcclResult::HCCL_SUCCESS; | 76 | return HcclResult::HCCL_SUCCESS; |
| 77 | } | 77 | } |
| 78 | 78 | ||
| 79 | HcclResult Loader::GetJettyMapInfo(std::vector<sim::JettyMapTab>& jettyMaps) | 79 | HcclResult Loader::GetJettyMapInfo(std::vector<sim::JettyMapTab>& jettyMaps) |
| 80 | { | 80 | { |
| 81 | if (sim::QueryJettyMapAll(jettyMaps) != 0) { | 81 | if (sim::QueryJettyMapAll(jettyMaps) != 0) { |
| 82 | - HCCL_VM_ERROR("[GetJettyMapInfo] QueryJettyMapAll failed"); | 82 | + HCCL_VM_ERROR("QueryJettyMapAll failed"); |
| 83 | return HcclResult::HCCL_E_PARA; | 83 | return HcclResult::HCCL_E_PARA; |
| 84 | } | 84 | } |
| 85 | - HCCL_VM_INFO("[GetJettyMapInfo] Get {} jetty maps", jettyMaps.size()); | 85 | + HCCL_VM_INFO("Get {} jetty maps", jettyMaps.size()); |
| 86 | return HcclResult::HCCL_SUCCESS; | 86 | return HcclResult::HCCL_SUCCESS; |
| 87 | } | 87 | } |
| 88 | 88 | ||
| 89 | HcclResult Loader::GetInstrResInfo(std::vector<sim::CcuInstrResTab>& instrRes) | 89 | HcclResult Loader::GetInstrResInfo(std::vector<sim::CcuInstrResTab>& instrRes) |
| 90 | { | 90 | { |
| 91 | if (sim::QueryCcuInstrResAll(instrRes) != 0) { | 91 | if (sim::QueryCcuInstrResAll(instrRes) != 0) { |
| 92 | - HCCL_VM_ERROR("[GetInstrResInfo] QueryCcuInstrResAll failed"); | 92 | + HCCL_VM_ERROR("QueryCcuInstrResAll failed"); |
| 93 | return HcclResult::HCCL_E_PARA; | 93 | return HcclResult::HCCL_E_PARA; |
| 94 | } | 94 | } |
| 95 | - HCCL_VM_INFO("[GetInstrResInfo] Get {} instr res records", instrRes.size()); | 95 | + HCCL_VM_INFO("Get {} instr res records", instrRes.size()); |
| 96 | return HcclResult::HCCL_SUCCESS; | 96 | return HcclResult::HCCL_SUCCESS; |
| 97 | } | 97 | } |
| 98 | 98 | ||
| 99 | HcclResult Loader::GetSyncInfo(std::vector<sim::SyncRecordTab>& syncRecords) | 99 | HcclResult Loader::GetSyncInfo(std::vector<sim::SyncRecordTab>& syncRecords) |
| 100 | { | 100 | { |
| 101 | if (sim::QuerySyncRecordAll(syncRecords) != 0) { | 101 | if (sim::QuerySyncRecordAll(syncRecords) != 0) { |
| 102 | - HCCL_VM_ERROR("[GetSyncInfo] QuerySyncRecordAll failed"); | 102 | + HCCL_VM_ERROR("QuerySyncRecordAll failed"); |
| 103 | return HcclResult::HCCL_E_PARA; | 103 | return HcclResult::HCCL_E_PARA; |
| 104 | } | 104 | } |
| 105 | - HCCL_VM_INFO("[GetSyncInfo] Get {} sync records", syncRecords.size()); | 105 | + HCCL_VM_INFO("Get {} sync records", syncRecords.size()); |
| 106 | return HcclResult::HCCL_SUCCESS; | 106 | return HcclResult::HCCL_SUCCESS; |
| 107 | } | 107 | } |
| 108 | 108 | ||
| 109 | HcclResult Loader::GetSyncRecordsByStatus(uint8_t status, std::vector<sim::SyncRecordTab>& syncRecords) | 109 | HcclResult Loader::GetSyncRecordsByStatus(uint8_t status, std::vector<sim::SyncRecordTab>& syncRecords) |
| 110 | { | 110 | { |
| 111 | if (sim::QuerySyncRecordByStatus(status, syncRecords) != 0) { | 111 | if (sim::QuerySyncRecordByStatus(status, syncRecords) != 0) { |
| 112 | - HCCL_VM_ERROR("[GetSyncRecordsByStatus] QuerySyncRecordByStatus failed for status: {}", status); | 112 | + HCCL_VM_ERROR("QuerySyncRecordByStatus failed for status: {}", status); |
| 113 | return HcclResult::HCCL_E_PARA; | 113 | return HcclResult::HCCL_E_PARA; |
| 114 | } | 114 | } |
| 115 | - HCCL_VM_INFO("[GetSyncRecordsByStatus] Get {} sync records for status: {}", syncRecords.size(), status); | 115 | + HCCL_VM_INFO("Get {} sync records for status: {}", syncRecords.size(), status); |
| 116 | return HcclResult::HCCL_SUCCESS; | 116 | return HcclResult::HCCL_SUCCESS; |
| 117 | } | 117 | } |
| 118 | 118 | ||
| 119 | HcclResult Loader::LoadCompositeOpDetailBySyncIter(uint32_t syncIter, std::map<uint32_t, std::vector<sim::CompositeOpDetail>>& compositeDataMap) | 119 | HcclResult Loader::LoadCompositeOpDetailBySyncIter(uint32_t syncIter, std::map<uint32_t, std::vector<sim::CompositeOpDetail>>& compositeDataMap) |
| 120 | { | 120 | { |
| 121 | if (sim::QueryCompositeOpDetailBySyncIter(syncIter, compositeDataMap) != 0) { | 121 | if (sim::QueryCompositeOpDetailBySyncIter(syncIter, compositeDataMap) != 0) { |
| 122 | - HCCL_VM_ERROR("[LoadCompositeOpDetailBySyncIter] QueryCompositeOpDetailsBySyncIter failed for syncIter: {}", syncIter); | 122 | + HCCL_VM_ERROR("QueryCompositeOpDetailsBySyncIter failed for syncIter: {}", syncIter); |
| 123 | return HcclResult::HCCL_E_PARA; | 123 | return HcclResult::HCCL_E_PARA; |
| 124 | } | 124 | } |
| 125 | - HCCL_VM_INFO("[LoadCompositeOpDetailBySyncIter] syncIter={}, Rank Count: {}", syncIter, compositeDataMap.size()); | 125 | + HCCL_VM_INFO("syncIter={}, Rank Count: {}", syncIter, compositeDataMap.size()); |
| 126 | return HcclResult::HCCL_SUCCESS; | 126 | return HcclResult::HCCL_SUCCESS; |
| 127 | } | 127 | } |
| 128 | } // namespace loader | 128 | } // namespace loader |
| @@ -38,7 +38,7 @@ bool LoadYamlStringMap(const std::string& yamlPath, | |||
| 38 | } | 38 | } |
| 39 | return true; | 39 | return true; |
| 40 | } catch (const std::exception& e) { | 40 | } catch (const std::exception& e) { |
| 41 | - HCCL_VM_ERROR("[SimYamlConfig] Failed to parse YAML file {}: {}", yamlPath, e.what()); | 41 | + HCCL_VM_ERROR("Failed to parse YAML file {}: {}", yamlPath, e.what()); |
| 42 | return false; | 42 | return false; |
| 43 | } | 43 | } |
| 44 | 44 | ||
| @@ -1468,13 +1468,11 @@ protected: | |||
| 1468 | sim::MemoryManager::GetInstance().FreeMemByName(sim::CommPoolPolicy::kPoolName); | 1468 | sim::MemoryManager::GetInstance().FreeMemByName(sim::CommPoolPolicy::kPoolName); |
| 1469 | shm_unlink(sim::CommPoolPolicy::kPoolName); | 1469 | shm_unlink(sim::CommPoolPolicy::kPoolName); |
| 1470 | unsetenv("HCCL_OP_EXPANSION_MODE"); | 1470 | unsetenv("HCCL_OP_EXPANSION_MODE"); |
| 1471 | - unsetenv("HCCL_VM_INSTALL_DIR"); | ||
| 1472 | } | 1471 | } |
| 1473 | void TearDown() override { | 1472 | void TearDown() override { |
| 1474 | sim::MemoryManager::GetInstance().FreeMemByName("HcclAicpuData"); | 1473 | sim::MemoryManager::GetInstance().FreeMemByName("HcclAicpuData"); |
| 1475 | sim::MemoryManager::GetInstance().FreeMemByName(sim::CommPoolPolicy::kPoolName); | 1474 | sim::MemoryManager::GetInstance().FreeMemByName(sim::CommPoolPolicy::kPoolName); |
| 1476 | unsetenv("HCCL_OP_EXPANSION_MODE"); | 1475 | unsetenv("HCCL_OP_EXPANSION_MODE"); |
| 1477 | - unsetenv("HCCL_VM_INSTALL_DIR"); | ||
| 1478 | } | 1476 | } |
| 1479 | }; | 1477 | }; |
| 1480 | 1478 | ||
| @@ -1490,13 +1488,6 @@ TEST_F(InitHvmEnvTest, InitializesSharedMemoryWithoutAivValidation) { | |||
| 1490 | EXPECT_EQ(sim::MemoryManager::GetInstance().AcquireMemByName(sim::CommPoolPolicy::kPoolName), nullptr); | 1488 | EXPECT_EQ(sim::MemoryManager::GetInstance().AcquireMemByName(sim::CommPoolPolicy::kPoolName), nullptr); |
| 1491 | } | 1489 | } |
| 1492 | 1490 | ||
| 1493 | -TEST_F(InitHvmEnvTest, FailsInAivModeWithoutInstallDir) { | ||
| 1494 | - setenv("HCCL_OP_EXPANSION_MODE", "AIV", 1); | ||
| 1495 | - | ||
| 1496 | - HcclVmResult ret = InitHvmEnv("/nonexistent/path/for/ut", 2, false); | ||
| 1497 | - EXPECT_EQ(ret, HCCL_SIM_HOST_ERROR_CMD); | ||
| 1498 | -} | ||
| 1499 | - | ||
| 1500 | TEST_F(InitHvmEnvTest, FailsWhenCommPoolNameAlreadyExists) { | 1491 | TEST_F(InitHvmEnvTest, FailsWhenCommPoolNameAlreadyExists) { |
| 1501 | // 仅校验模式下池名被预先占用时,InitHvmEnv 建池失败、返回错误,且不创建 HcclAicpuData。 | 1492 | // 仅校验模式下池名被预先占用时,InitHvmEnv 建池失败、返回错误,且不创建 HcclAicpuData。 |
| 1502 | ASSERT_NE(sim::MemoryManager::GetInstance().AllocMemByName( | 1493 | ASSERT_NE(sim::MemoryManager::GetInstance().AllocMemByName( |
| @@ -1553,33 +1544,14 @@ TEST_F(IsAivExpansionModeEnabledTest, ToggleOnOff) { | |||
| 1553 | class ValidateAivTaskJsonByRankTest : public testing::Test { | 1544 | class ValidateAivTaskJsonByRankTest : public testing::Test { |
| 1554 | protected: | 1545 | protected: |
| 1555 | void SetUp() override { | 1546 | void SetUp() override { |
| 1556 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 1547 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 1557 | - setenv("HCCL_VM_INSTALL_DIR", "/tmp/hvm_ut_aiv_dummy", 1); | 1548 | + setenv("HCCL_VM_INSTALL_ROOT", "/tmp/hvm_ut_aiv_dummy", 1); |
| 1558 | } | 1549 | } |
| 1559 | void TearDown() override { | 1550 | void TearDown() override { |
| 1560 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 1551 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 1561 | } | 1552 | } |
| 1562 | }; | 1553 | }; |
| 1563 | 1554 | ||
| 1564 | -TEST_F(ValidateAivTaskJsonByRankTest, FailsWithoutInstallDir) { | ||
| 1565 | - std::vector<sim::Rank> allRank; | ||
| 1566 | - sim::Rank r; | ||
| 1567 | - r.rank_id = 0; | ||
| 1568 | - allRank.push_back(r); | ||
| 1569 | - HcclVmResult ret = ValidateAivTaskJsonByRank(allRank); | ||
| 1570 | - EXPECT_EQ(ret, HcclVmResult::HCCL_SIM_HOST_ERROR_CMD); | ||
| 1571 | -} | ||
| 1572 | - | ||
| 1573 | -TEST_F(ValidateAivTaskJsonByRankTest, FailsWithEmptyInstallDir) { | ||
| 1574 | - setenv("HCCL_VM_INSTALL_DIR", "", 1); | ||
| 1575 | - std::vector<sim::Rank> allRank; | ||
| 1576 | - sim::Rank r; | ||
| 1577 | - r.rank_id = 0; | ||
| 1578 | - allRank.push_back(r); | ||
| 1579 | - HcclVmResult ret = ValidateAivTaskJsonByRank(allRank); | ||
| 1580 | - EXPECT_EQ(ret, HcclVmResult::HCCL_SIM_HOST_ERROR_CMD); | ||
| 1581 | -} | ||
| 1582 | - | ||
| 1583 | TEST_F(ValidateAivTaskJsonByRankTest, SucceedsWithEmptyRankList) { | 1555 | TEST_F(ValidateAivTaskJsonByRankTest, SucceedsWithEmptyRankList) { |
| 1584 | std::vector<sim::Rank> allRank; | 1556 | std::vector<sim::Rank> allRank; |
| 1585 | HcclVmResult ret = ValidateAivTaskJsonByRank(allRank); | 1557 | HcclVmResult ret = ValidateAivTaskJsonByRank(allRank); |
| @@ -1587,7 +1559,7 @@ TEST_F(ValidateAivTaskJsonByRankTest, SucceedsWithEmptyRankList) { | |||
| 1587 | } | 1559 | } |
| 1588 | 1560 | ||
| 1589 | TEST_F(ValidateAivTaskJsonByRankTest, FailsWithNonExistentDataDir) { | 1561 | TEST_F(ValidateAivTaskJsonByRankTest, FailsWithNonExistentDataDir) { |
| 1590 | - setenv("HCCL_VM_INSTALL_DIR", "/tmp/hvm_ut_no_such_dir_validate", 1); | 1562 | + setenv("HCCL_VM_INSTALL_ROOT", "/tmp/hvm_ut_no_such_dir_validate", 1); |
| 1591 | std::vector<sim::Rank> allRank; | 1563 | std::vector<sim::Rank> allRank; |
| 1592 | sim::Rank r; | 1564 | sim::Rank r; |
| 1593 | r.rank_id = 0; | 1565 | r.rank_id = 0; |
| @@ -1602,10 +1574,10 @@ protected: | |||
| 1602 | void SetUp() override { | 1574 | void SetUp() override { |
| 1603 | installDir_ = std::filesystem::temp_directory_path() / "hvm_ut_aiv_validate"; | 1575 | installDir_ = std::filesystem::temp_directory_path() / "hvm_ut_aiv_validate"; |
| 1604 | std::filesystem::create_directories(installDir_ / "data"); | 1576 | std::filesystem::create_directories(installDir_ / "data"); |
| 1605 | - setenv("HCCL_VM_INSTALL_DIR", installDir_.string().c_str(), 1); | 1577 | + setenv("HCCL_VM_INSTALL_ROOT", installDir_.string().c_str(), 1); |
| 1606 | } | 1578 | } |
| 1607 | void TearDown() override { | 1579 | void TearDown() override { |
| 1608 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 1580 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 1609 | std::filesystem::remove_all(installDir_); | 1581 | std::filesystem::remove_all(installDir_); |
| 1610 | } | 1582 | } |
| 1611 | void WriteAivTaskFile(uint32_t rankId, uint32_t launchIdx) { | 1583 | void WriteAivTaskFile(uint32_t rankId, uint32_t launchIdx) { |
| @@ -68,6 +68,7 @@ set_target_properties(test_allgather_v_semantics_checker PROPERTIES | |||
| 68 | add_executable(test_allreduce_semantics_checker semantics_check/allreduce_semantics_checker_test.cc main.cc) | 68 | add_executable(test_allreduce_semantics_checker semantics_check/allreduce_semantics_checker_test.cc main.cc) |
| 69 | target_sources(test_allreduce_semantics_checker PRIVATE | 69 | target_sources(test_allreduce_semantics_checker PRIVATE |
| 70 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/allreduce_semantics_checker.cc | 70 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/allreduce_semantics_checker.cc |
| 71 | + ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/utils/dump/dump_json_utils.cc | ||
| 71 | ${CHECKER_LOG_SRC} | 72 | ${CHECKER_LOG_SRC} |
| 72 | ) | 73 | ) |
| 73 | set_target_properties(test_allreduce_semantics_checker PROPERTIES | 74 | set_target_properties(test_allreduce_semantics_checker PROPERTIES |
| @@ -86,6 +87,7 @@ set_target_properties(test_broadcast_semantics_checker PROPERTIES | |||
| 86 | add_executable(test_reduce_semantics_checker semantics_check/reduce_semantics_checker_test.cc main.cc) | 87 | add_executable(test_reduce_semantics_checker semantics_check/reduce_semantics_checker_test.cc main.cc) |
| 87 | target_sources(test_reduce_semantics_checker PRIVATE | 88 | target_sources(test_reduce_semantics_checker PRIVATE |
| 88 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/reduce_semantics_checker.cc | 89 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/reduce_semantics_checker.cc |
| 90 | + ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/utils/dump/dump_json_utils.cc | ||
| 89 | ${CHECKER_LOG_SRC} | 91 | ${CHECKER_LOG_SRC} |
| 90 | ) | 92 | ) |
| 91 | set_target_properties(test_reduce_semantics_checker PROPERTIES | 93 | set_target_properties(test_reduce_semantics_checker PROPERTIES |
| @@ -122,6 +124,7 @@ set_target_properties(test_all2all_semantics_checker PROPERTIES | |||
| 122 | add_executable(test_reduce_scatter_semantics_checker semantics_check/reduce_scatter_semantics_checker_test.cc main.cc) | 124 | add_executable(test_reduce_scatter_semantics_checker semantics_check/reduce_scatter_semantics_checker_test.cc main.cc) |
| 123 | target_sources(test_reduce_scatter_semantics_checker PRIVATE | 125 | target_sources(test_reduce_scatter_semantics_checker PRIVATE |
| 124 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/reduce_scatter_semantics_checker.cc | 126 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/reduce_scatter_semantics_checker.cc |
| 127 | + ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/utils/dump/dump_json_utils.cc | ||
| 125 | ${CHECKER_LOG_SRC} | 128 | ${CHECKER_LOG_SRC} |
| 126 | ) | 129 | ) |
| 127 | set_target_properties(test_reduce_scatter_semantics_checker PROPERTIES | 130 | set_target_properties(test_reduce_scatter_semantics_checker PROPERTIES |
| @@ -131,6 +134,7 @@ set_target_properties(test_reduce_scatter_semantics_checker PROPERTIES | |||
| 131 | add_executable(test_reduce_scatter_v_semantics_checker semantics_check/reduce_scatter_v_semantics_checker_test.cc main.cc) | 134 | add_executable(test_reduce_scatter_v_semantics_checker semantics_check/reduce_scatter_v_semantics_checker_test.cc main.cc) |
| 132 | target_sources(test_reduce_scatter_v_semantics_checker PRIVATE | 135 | target_sources(test_reduce_scatter_v_semantics_checker PRIVATE |
| 133 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/reduce_scatter_v_semantics_checker.cc | 136 | ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/framework/semantics_check/reduce_scatter_v_semantics_checker.cc |
| 137 | + ${CMAKE_SOURCE_DIR}/src/plugin/checker/src/utils/dump/dump_json_utils.cc | ||
| 134 | ${CHECKER_LOG_SRC} | 138 | ${CHECKER_LOG_SRC} |
| 135 | ) | 139 | ) |
| 136 | set_target_properties(test_reduce_scatter_v_semantics_checker PROPERTIES | 140 | set_target_properties(test_reduce_scatter_v_semantics_checker PROPERTIES |
| @@ -107,22 +107,22 @@ public: | |||
| 107 | dataDir_ = root_ + "/data"; | 107 | dataDir_ = root_ + "/data"; |
| 108 | EnsureDir(dataDir_); | 108 | EnsureDir(dataDir_); |
| 109 | 109 | ||
| 110 | - const char *oldEnv = std::getenv("HCCL_VM_INSTALL_DIR"); | 110 | + const char *oldEnv = std::getenv("HCCL_VM_INSTALL_ROOT"); |
| 111 | if (oldEnv != nullptr) { | 111 | if (oldEnv != nullptr) { |
| 112 | hadOldEnv_ = true; | 112 | hadOldEnv_ = true; |
| 113 | oldEnv_ = oldEnv; | 113 | oldEnv_ = oldEnv; |
| 114 | } | 114 | } |
| 115 | - if (setenv("HCCL_VM_INSTALL_DIR", root_.c_str(), 1) != 0) { | 115 | + if (setenv("HCCL_VM_INSTALL_ROOT", root_.c_str(), 1) != 0) { |
| 116 | - throw std::runtime_error("failed to set HCCL_VM_INSTALL_DIR"); | 116 | + throw std::runtime_error("failed to set HCCL_VM_INSTALL_ROOT"); |
| 117 | } | 117 | } |
| 118 | } | 118 | } |
| 119 | 119 | ||
| 120 | ~TempAivInstallDir() | 120 | ~TempAivInstallDir() |
| 121 | { | 121 | { |
| 122 | if (hadOldEnv_) { | 122 | if (hadOldEnv_) { |
| 123 | - setenv("HCCL_VM_INSTALL_DIR", oldEnv_.c_str(), 1); | 123 | + setenv("HCCL_VM_INSTALL_ROOT", oldEnv_.c_str(), 1); |
| 124 | } else { | 124 | } else { |
| 125 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 125 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 126 | } | 126 | } |
| 127 | if (!root_.empty()) { | 127 | if (!root_.empty()) { |
| 128 | RemoveTree(root_); | 128 | RemoveTree(root_); |
| @@ -30,12 +30,12 @@ protected: | |||
| 30 | void SetUp() override { | 30 | void SetUp() override { |
| 31 | testDir_ = fs::temp_directory_path() / ("aiv_exec_test_" + std::to_string(::getpid())); | 31 | testDir_ = fs::temp_directory_path() / ("aiv_exec_test_" + std::to_string(::getpid())); |
| 32 | fs::create_directories(testDir_ / "data"); | 32 | fs::create_directories(testDir_ / "data"); |
| 33 | - setenv("HCCL_VM_INSTALL_DIR", testDir_.c_str(), 1); | 33 | + setenv("HCCL_VM_INSTALL_ROOT", testDir_.c_str(), 1); |
| 34 | AivResourceManager::GetInstance().Reset(); | 34 | AivResourceManager::GetInstance().Reset(); |
| 35 | } | 35 | } |
| 36 | 36 | ||
| 37 | void TearDown() override { | 37 | void TearDown() override { |
| 38 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 38 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 39 | AivResourceManager::GetInstance().Reset(); | 39 | AivResourceManager::GetInstance().Reset(); |
| 40 | fs::remove_all(testDir_); | 40 | fs::remove_all(testDir_); |
| 41 | } | 41 | } |
| @@ -440,32 +440,14 @@ TEST_F(AivTaskJsonTest, SerializeExecutor_WithTasks) { | |||
| 440 | 440 | ||
| 441 | // ========== ResolveExecutorJsonFilePath ========== | 441 | // ========== ResolveExecutorJsonFilePath ========== |
| 442 | 442 | ||
| 443 | -TEST_F(AivTaskJsonTest, ResolveExecutorJsonFilePath_NoEnvVar) { | ||
| 444 | - unsetenv("HCCL_VM_INSTALL_DIR"); | ||
| 445 | - auto &executor = AivKernelExecutor::GetInstance(); | ||
| 446 | - std::string filePath; | ||
| 447 | - bool result = ResolveExecutorJsonFilePath(executor, 0, filePath); | ||
| 448 | - EXPECT_FALSE(result); | ||
| 449 | - EXPECT_TRUE(filePath.empty()); | ||
| 450 | -} | ||
| 451 | - | ||
| 452 | -TEST_F(AivTaskJsonTest, ResolveExecutorJsonFilePath_EmptyEnvVar) { | ||
| 453 | - setenv("HCCL_VM_INSTALL_DIR", "", 1); | ||
| 454 | - auto &executor = AivKernelExecutor::GetInstance(); | ||
| 455 | - std::string filePath; | ||
| 456 | - bool result = ResolveExecutorJsonFilePath(executor, 0, filePath); | ||
| 457 | - EXPECT_FALSE(result); | ||
| 458 | - unsetenv("HCCL_VM_INSTALL_DIR"); | ||
| 459 | -} | ||
| 460 | - | ||
| 461 | TEST_F(AivTaskJsonTest, ResolveExecutorJsonFilePath_Success) { | 443 | TEST_F(AivTaskJsonTest, ResolveExecutorJsonFilePath_Success) { |
| 462 | - setenv("HCCL_VM_INSTALL_DIR", "/tmp/hccl_test", 1); | 444 | + setenv("HCCL_VM_INSTALL_ROOT", "/tmp/hccl_test", 1); |
| 463 | auto &executor = AivKernelExecutor::GetInstance(); | 445 | auto &executor = AivKernelExecutor::GetInstance(); |
| 464 | std::string filePath; | 446 | std::string filePath; |
| 465 | bool result = ResolveExecutorJsonFilePath(executor, 3, filePath); | 447 | bool result = ResolveExecutorJsonFilePath(executor, 3, filePath); |
| 466 | EXPECT_TRUE(result); | 448 | EXPECT_TRUE(result); |
| 467 | EXPECT_NE(filePath.find("hcclvm_aiv_rank0_launch3_task.json"), std::string::npos); | 449 | EXPECT_NE(filePath.find("hcclvm_aiv_rank0_launch3_task.json"), std::string::npos); |
| 468 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 450 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 469 | } | 451 | } |
| 470 | 452 | ||
| 471 | // ========== WriteJsonAtomically ========== | 453 | // ========== WriteJsonAtomically ========== |
| @@ -524,16 +506,9 @@ TEST_F(AivTaskJsonTest, WriteJsonAtomically_RenameFallback) { | |||
| 524 | 506 | ||
| 525 | // ========== DumpExecutorToJsonFile ========== | 507 | // ========== DumpExecutorToJsonFile ========== |
| 526 | 508 | ||
| 527 | -TEST_F(AivTaskJsonTest, DumpExecutorToJsonFile_NoEnvVar) { | ||
| 528 | - unsetenv("HCCL_VM_INSTALL_DIR"); | ||
| 529 | - auto &executor = AivKernelExecutor::GetInstance(); | ||
| 530 | - bool result = DumpExecutorToJsonFile(executor, 0); | ||
| 531 | - EXPECT_FALSE(result); | ||
| 532 | -} | ||
| 533 | - | ||
| 534 | TEST_F(AivTaskJsonTest, DumpExecutorToJsonFile_Success) { | 509 | TEST_F(AivTaskJsonTest, DumpExecutorToJsonFile_Success) { |
| 535 | std::error_code ec; | 510 | std::error_code ec; |
| 536 | - setenv("HCCL_VM_INSTALL_DIR", "/tmp/hccl_dump_test", 1); | 511 | + setenv("HCCL_VM_INSTALL_ROOT", "/tmp/hccl_dump_test", 1); |
| 537 | fs::remove_all("/tmp/hccl_dump_test", ec); | 512 | fs::remove_all("/tmp/hccl_dump_test", ec); |
| 538 | 513 | ||
| 539 | auto &executor = AivKernelExecutor::GetInstance(); | 514 | auto &executor = AivKernelExecutor::GetInstance(); |
| @@ -554,7 +529,7 @@ TEST_F(AivTaskJsonTest, DumpExecutorToJsonFile_Success) { | |||
| 554 | EXPECT_NE(content.find("\"aiv\""), std::string::npos); | 529 | EXPECT_NE(content.find("\"aiv\""), std::string::npos); |
| 555 | 530 | ||
| 556 | fs::remove_all("/tmp/hccl_dump_test", ec); | 531 | fs::remove_all("/tmp/hccl_dump_test", ec); |
| 557 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 532 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 558 | } | 533 | } |
| 559 | 534 | ||
| 560 | // ========== aiv_task.h coverage: AivTask base ========== | 535 | // ========== aiv_task.h coverage: AivTask base ========== |
| @@ -25,11 +25,11 @@ protected: | |||
| 25 | void SetUp() override { | 25 | void SetUp() override { |
| 26 | testDir_ = fs::temp_directory_path() / ("aiv_test_" + std::to_string(::getpid())); | 26 | testDir_ = fs::temp_directory_path() / ("aiv_test_" + std::to_string(::getpid())); |
| 27 | fs::create_directories(testDir_ / "data"); | 27 | fs::create_directories(testDir_ / "data"); |
| 28 | - setenv("HCCL_VM_INSTALL_DIR", testDir_.c_str(), 1); | 28 | + setenv("HCCL_VM_INSTALL_ROOT", testDir_.c_str(), 1); |
| 29 | } | 29 | } |
| 30 | 30 | ||
| 31 | void TearDown() override { | 31 | void TearDown() override { |
| 32 | - unsetenv("HCCL_VM_INSTALL_DIR"); | 32 | + unsetenv("HCCL_VM_INSTALL_ROOT"); |
| 33 | fs::remove_all(testDir_); | 33 | fs::remove_all(testDir_); |
| 34 | } | 34 | } |
| 35 | 35 | ||
| @@ -300,14 +300,6 @@ TEST_F(AivTaskSnapshotLoaderTest, LoadByLaunchDirect_ErrorNullPointer_Succeeds) | |||
| 300 | 300 | ||
| 301 | // ==================== LoadRuntimeTaskSnapshotByLaunchDirect Error Tests ==================== | 301 | // ==================== LoadRuntimeTaskSnapshotByLaunchDirect Error Tests ==================== |
| 302 | 302 | ||
| 303 | -TEST_F(AivTaskSnapshotLoaderTest, LoadByLaunchDirect_EnvNotSet_ReturnsFalse) { | ||
| 304 | - unsetenv("HCCL_VM_INSTALL_DIR"); | ||
| 305 | - AivRuntimeTaskSnapshot snapshot; | ||
| 306 | - std::string errorMsg; | ||
| 307 | - EXPECT_FALSE(AivTaskSnapshotLoader::LoadRuntimeTaskSnapshotByLaunchDirect(0, 0, snapshot, &errorMsg)); | ||
| 308 | - EXPECT_FALSE(errorMsg.empty()); | ||
| 309 | -} | ||
| 310 | - | ||
| 311 | TEST_F(AivTaskSnapshotLoaderTest, LoadByLaunchDirect_FileNotExist_ReturnsFalse) { | 303 | TEST_F(AivTaskSnapshotLoaderTest, LoadByLaunchDirect_FileNotExist_ReturnsFalse) { |
| 312 | AivRuntimeTaskSnapshot snapshot; | 304 | AivRuntimeTaskSnapshot snapshot; |
| 313 | std::string errorMsg; | 305 | std::string errorMsg; |
| @@ -41,9 +41,9 @@ cd {HCCL_VM路径}/test | |||
| 41 | ./run_ut.sh | 41 | ./run_ut.sh |
| 42 | ``` | 42 | ``` |
| 43 | 43 | ||
| 44 | -- 执行所有 `test` 目录下的测试 | 44 | +- 执行所有 `test` 目录下的测试。 |
| 45 | -- 完整三步流程:编译 → 生成可执行文件 → 执行所有测试 | 45 | +- 完整三步流程:编译 → 生成可执行文件 → 执行所有测试。 |
| 46 | -- 适用于完整回归测试 | 46 | +- 适用于完整回归测试。 |
| 47 | 47 | ||
| 48 | ### 2. 覆盖率报告 | 48 | ### 2. 覆盖率报告 |
| 49 | 49 | ||
| @@ -51,11 +51,11 @@ cd {HCCL_VM路径}/test | |||
| 51 | ./run_ut.sh --cov | 51 | ./run_ut.sh --cov |
| 52 | ``` | 52 | ``` |
| 53 | 53 | ||
| 54 | -- 全量编译 + 执行所有测试 + 生成 gcov/lcov 覆盖率 HTML 报告 | 54 | +- 全量编译 + 执行所有测试 + 生成 gcov/lcov 覆盖率 HTML 报告。 |
| 55 | -- 自动启用 `--coverage` 编译选项,生成 `.gcno`/`.gcda` 文件 | 55 | +- 自动启用 `--coverage` 编译选项,生成 `.gcno`/`.gcda` 文件。 |
| 56 | -- 四步流程:编译(含覆盖率插桩)→ 执行 → 收集覆盖率数据 → 生成 HTML 报告 | 56 | +- 四步流程:编译(含覆盖率插桩)→ 执行 → 收集覆盖率数据 → 生成 HTML 报告。 |
| 57 | -- 报告输出路径:`$CODE_DIR/coverage_report/html/index.html` | 57 | +- 报告输出路径:`$CODE_DIR/coverage_report/html/index.html`。 |
| 58 | -- 会自动过滤系统头文件、第三方库、stub 文件等非业务代码 | 58 | +- 会自动过滤系统头文件、第三方库、stub 文件等非业务代码。 |
| 59 | 59 | ||
| 60 | ### 3. 目录执行 | 60 | ### 3. 目录执行 |
| 61 | 61 | ||
| @@ -65,9 +65,9 @@ cd {HCCL_VM路径}/test | |||
| 65 | ./run_ut.sh store | 65 | ./run_ut.sh store |
| 66 | ``` | 66 | ``` |
| 67 | 67 | ||
| 68 | -- 递归查找指定目录下所有 `*_test.cc` 文件 | 68 | +- 递归查找指定目录下所有 `*_test.cc` 文件。 |
| 69 | -- 编译并执行该目录下所有测试 | 69 | +- 编译并执行该目录下所有测试。 |
| 70 | -- 适用于模块级测试 | 70 | +- 适用于模块级测试。 |
| 71 | 71 | ||
| 72 | ### 4. 二进制执行 | 72 | ### 4. 二进制执行 |
| 73 | 73 | ||
| @@ -76,8 +76,8 @@ cd {HCCL_VM路径}/test | |||
| 76 | ./run_ut.sh test_allgather_semantics_checker | 76 | ./run_ut.sh test_allgather_semantics_checker |
| 77 | ``` | 77 | ``` |
| 78 | 78 | ||
| 79 | -- 编译并执行指定的测试二进制 | 79 | +- 编译并执行指定的测试二进制。 |
| 80 | -- 二进制名以 `test_` 开头 | 80 | +- 二进制名以 `test_` 开头。 |
| 81 | 81 | ||
| 82 | ### 5. 文件执行 | 82 | ### 5. 文件执行 |
| 83 | 83 | ||
| @@ -86,8 +86,8 @@ cd {HCCL_VM路径}/test | |||
| 86 | ./run_ut.sh allgather_semantics_checker_test.cc | 86 | ./run_ut.sh allgather_semantics_checker_test.cc |
| 87 | ``` | 87 | ``` |
| 88 | 88 | ||
| 89 | -- 根据测试文件名自动匹配对应二进制 | 89 | +- 根据测试文件名自动匹配对应二进制。 |
| 90 | -- 编译并执行 | 90 | +- 编译并执行。 |
| 91 | 91 | ||
| 92 | ### 6. 单个用例执行 | 92 | ### 6. 单个用例执行 |
| 93 | 93 | ||
| @@ -96,8 +96,8 @@ cd {HCCL_VM路径}/test | |||
| 96 | ./run_ut.sh allgather_semantics_checker_test.cc AllgatherSemanticsCheckerTest.CheckBasic | 96 | ./run_ut.sh allgather_semantics_checker_test.cc AllgatherSemanticsCheckerTest.CheckBasic |
| 97 | ``` | 97 | ``` |
| 98 | 98 | ||
| 99 | -- 执行指定测试文件中的单个测试用例 | 99 | +- 执行指定测试文件中的单个测试用例。 |
| 100 | -- 用例名格式:`TestSuiteName.TestCaseName` | 100 | +- 用例名格式:`TestSuiteName.TestCaseName`。 |
| 101 | 101 | ||
| 102 | ### 7. 列出测试 | 102 | ### 7. 列出测试 |
| 103 | 103 | ||
| @@ -106,8 +106,8 @@ cd {HCCL_VM路径}/test | |||
| 106 | ./run_ut.sh --list | 106 | ./run_ut.sh --list |
| 107 | ``` | 107 | ``` |
| 108 | 108 | ||
| 109 | -- 列出所有可用测试文件及其状态 | 109 | +- 列出所有可用测试文件及其状态。 |
| 110 | -- 显示用例数量和对应二进制名 | 110 | +- 显示用例数量和对应二进制名。 |
| 111 | 111 | ||
| 112 | ## 示例 | 112 | ## 示例 |
| 113 | 113 | ||
| @@ -244,10 +244,10 @@ source /home/workspace/Ascend/cann/set_env.sh | |||
| 244 | 244 | ||
| 245 | ## 注意事项 | 245 | ## 注意事项 |
| 246 | 246 | ||
| 247 | -1. **首次执行**:首次执行会进行完整的 CMake 配置,耗时较长 | 247 | +1. **首次执行**:首次执行会进行完整的 CMake 配置,耗时较长。 |
| 248 | -2. **编译失败**:如果编译失败,请检查 `build.log` 查看详细错误信息 | 248 | +2. **编译失败**:如果编译失败,请检查 `build.log` 查看详细错误信息。 |
| 249 | -3. **测试失败**:如果测试失败,请检查 `run.log` 查看具体失败的用例 | 249 | +3. **测试失败**:如果测试失败,请检查 `run.log` 查看具体失败的用例。 |
| 250 | -4. **日志清理**:日志目录按时间戳命名,定期清理旧日志以节省空间 | 250 | +4. **日志清理**:日志目录按时间戳命名,定期清理旧日志以节省空间。 |
| 251 | 251 | ||
| 252 | ## 常见问题 | 252 | ## 常见问题 |
| 253 | 253 | ||
| @@ -281,8 +281,8 @@ cd <coverage_report/html所在路径> | |||
| 281 | python3 -m http.server 8080 | 281 | python3 -m http.server 8080 |
| 282 | ``` | 282 | ``` |
| 283 | 283 | ||
| 284 | -- 本机浏览器访问:`http://localhost:8080` | 284 | +- 本机浏览器访问:`http://localhost:8080`。 |
| 285 | -- 远程服务器:通过 SSH 端口转发后本地访问 | 285 | +- 远程服务器:通过 SSH 端口转发后本地访问。 |
| 286 | 286 | ||
| 287 | ```bash | 287 | ```bash |
| 288 | ssh -L 8080:localhost:8080 <user>@<server_ip> | 288 | ssh -L 8080:localhost:8080 <user>@<server_ip> |
| @@ -293,6 +293,6 @@ ssh -L 8080:localhost:8080 <user>@<server_ip> | |||
| 293 | 293 | ||
| 294 | ## 版本历史 | 294 | ## 版本历史 |
| 295 | 295 | ||
| 296 | -- v1.0 - 初始版本,支持全量/目录/文件/用例级测试执行 | 296 | +- v1.0 - 初始版本,支持全量/目录/文件/用例级测试执行。 |
| 297 | -- v2.0 - 优化命令行参数,支持自动路径推导,三步流程日志记录 | 297 | +- v2.0 - 优化命令行参数,支持自动路径推导,三步流程日志记录。 |
| 298 | -- v2.1 - 新增 `--cov` 参数,支持 gcov/lcov 覆盖率 HTML 报告生成 | 298 | +- v2.1 - 新增 `--cov` 参数,支持 gcov/lcov 覆盖率 HTML 报告生成。 |