Closed
[Bug-Report|缺陷反馈]:[工具检测]:mssanitizer 检测到 matmul_callback 样例 DMA 从 GM 0x0 非法越界读取 #1672
h-glueAdd on  Sep 16Closed at  Sep 28
h-glue
Sep 16 Create

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

使用 mssanitizer(memcheck 模式)对 examples/01_simd_cpp_api/04_advanced_api/00_matmul/matmul_callback 样例进行内存检测,报告 129 处 illegal read of size 256:

  • 均指向同一条指令:pc 0x1644 (serialNo:47),即 DMA_MOV_ND2NZ_D,src:(type:GM;addr:0x0;loop1Stride:1024),D:128/N:128,一次搬运跨度覆盖 0x0~0x1fc00;
  • 调用栈均为 matmul_callback.asc:92(CustomDataCopyInA 中 DataCopy)← :174(IterateAll)← :212(Process);
  • 出错核:aic(0,2-5,7-8,10-12,14-18),非法读取地址(相对 GM 偏移)从 0x0 到 0x1fc00;
  • 日志中所有 MALLOC 记录的 GM 地址均为设备真实分配地址(0x1200.../0x1201...),偏移 0x0 处从未被分配,即 DMA 从 GM 0x0 读取,确为越界访问。

Environment / 环境信息 (Mandatory / 必填)

Ascend 950PR
cann-9.1.0
检测工具:mssanitizer(memcheck)
基于asc-devlit: 9.1.0分支代码检测

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

  1. 进入样例目录:
    cd examples/01_simd_cpp_api/04_advanced_api/00_matmul/matmul_callback
    source ${install_path}/cann/set_env.sh
    
  2. 编译并生成数据(dav-3510):
    mkdir -p build && cd build
    cmake -DSCENARIO_NUM=1 -DCMAKE_ASC_ARCHITECTURES=dav-3510 -DCMAKE_ASC_LINK_FLAGS=--cce-enable-sanitizer .. && make -j
    python3 ../scripts/gen_data.py
    mssanitizer -t memcheck -t racecheck -t initcheck -t synccheck --log-level=error -- ./demo
    

Describe the expected behavior / 预期结果 (Mandatory / 必填)

算子在 mssanitizer memcheck 下运行不应出现非法内存访问:回调 DataCopy 搬运 A 矩阵的源地址应为各核实际分配到的 A 分块地址,而非从未分配的 GM 偏移 0x0。

mssanitizer memcheck 报告(地址均为相对 GM 偏移):

====== ERROR: illegal read of size 256
======    at 0x13800 on GM in "matmul_callback_custom(...)"
======    in block aic(0,2-5,7-8,10-12,14-18) on device 3
======    code in pc current 0x1644 (serialNo:47)
======    #0 matmul_callback.asc:92:5
======    #1 matmul_callback.asc:174:15
======    #2 matmul_callback.asc:212:18

其余 128 处错误调用栈与上完全相同(同一 serialNo:47 指令),仅报错地址不同,完整地址列表如下:

0x0, 0x400, 0x800, 0xc00, 0x1000, 0x1400, 0x1800, 0x1c00,
0x2000, 0x2400, 0x2800, 0x2c00, 0x3000, 0x3400, 0x3800, 0x3c00,
0x4000, 0x4400, 0x4800, 0x4c00, 0x5000, 0x5400, 0x5800, 0x5c00,
0x6000, 0x6400, 0x6800, 0x6c00, 0x7000, 0x7400, 0x7800, 0x7c00,
0x8000, 0x8400, 0x8800, 0x8c00, 0x9000, 0x9400, 0x9800, 0x9c00,
0xa000, 0xa400, 0xa800, 0xac00, 0xb000, 0xb400, 0xb800, 0xbc00,
0xc000, 0xc400, 0xc800, 0xcc00, 0xd000, 0xd400, 0xd800, 0xdc00,
0xe000, 0xe400, 0xe800, 0xec00, 0xf000, 0xf400, 0xf800, 0xfc00,
0x10000, 0x10400, 0x10800, 0x10c00, 0x11000, 0x11400, 0x11800, 0x11c00,
0x12000, 0x12400, 0x12800, 0x12c00, 0x13000, 0x13400, 0x13800, 0x13c00,
0x14000, 0x14400, 0x14800, 0x14c00, 0x15000, 0x15400, 0x15800, 0x15c00,
0x16000, 0x16400, 0x16800, 0x16c00, 0x17000, 0x17400, 0x17800, 0x17c00,
0x18000, 0x18400, 0x18800, 0x18c00, 0x19000, 0x19400, 0x19800, 0x19c00,
0x1a000, 0x1a400, 0x1a800, 0x1ac00, 0x1b000, 0x1b400, 0x1b800, 0x1bc00,
0x1c000, 0x1c400, 0x1c800, 0x1cc00, 0x1d000, 0x1d400, 0x1d800, 0x1dc00,
0x1e000, 0x1e400, 0x1e800, 0x1ec00, 0x1f000, 0x1f400, 0x1f800, 0x1fc00

Special notes for this issue/备注 (Optional / 选填)

初步根因分析:

  • offset list(50×uint64)由 scripts/gen_data.py 初始化为全 0(仅 index 0=256)。
  • kernel 在 Process() 中通过 offsetListGlobal.SetValue(curBlockIdx, srcBaseAddr1)(标量 GM 写)写入各核自己的 A 分块地址,随后 IterateAll() 触发回调 CustomDataCopyInA,经 dataGM.GetValue(curBlockIdx) 读取该地址作为 DataCopy 源地址(第 92 行)。
  • 标量写 GM 与后续 DMA 读同一 GM 之间缺少内存屏障(如 AscendC::Mb()),导致部分核的 DMA 读到 offset list 初始值 0,从而从 GM 偏移 0x0 非法读取。
  • 成功核(aic 1/6/9/13/19)与失败核(0,2-5,7-8,10-12,14-18)呈非确定性散布,符合写-读时序竞争特征,而非确定性逻辑错误。
likedislike
Hh-glue
Sep 16 Modify description of issue
li-yuanjie-daCollaborators
Sep 16 Comments:

感谢反馈。我们确认所报GM 0x0访问与该代码路径吻合。当前不能将原因直接定为缺少Mb(),还需排查跨核DCache写回覆盖。我们将在 950 环境下以mssanitizer进行验证。

likedislike
Lli-yuanjie-daCollaborators
Sep 16 Assign li-yuanjie-da as responsible person
HhyoleeCollaborators
Sep 28 Assign ca1kw as responsible person
ca1kw
ca1kwCollaborators
Sep 28 Comments:

您好,对于该问题,已经修改样例,上述提到的128出报错经验证已经解决,具体修改可以查看关联pr~

likedislike
Hh-glue
Sep 28 changed custom state from 待办的 to 已解决
Hh-glue
Sep 28 Close issue
CANN-robotCANN-robotCollaborators
Sep 28 Added label: resolved