Closed
[Bug-Report|缺陷反馈][工具检测]: 检测到mhc_sinkhorn_backward算子有内存越界异常 #4282
h-glueAdd on  Aug 12Closed at  Aug 24
h-glue
h-glue
Aug 12 Create

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

[mssanitizer] Start memcheck, racecheck, initcheck and synccheck sanitizer on kernel MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
MhcSinkhornBackward compute success!
====== ERROR: illegal write of size 32
====== at 0x820 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-42) on device 0
====== code in pc current 0x16d8 (serialNo:505)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:329:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:430:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:563:13
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #5 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5

====== ERROR: uninitialized read of size 160
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-41) on device 0
====== code in pc current 0x245c (serialNo:5525)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5

====== ERROR: uninitialized read of size 96
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(42) on device 0
====== code in pc current 0x245c (serialNo:237597)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5

Environment / 环境信息 (Mandatory / 必填)

A5
cann 730商用9.1.0
ops-transformer 9.1.0分支

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

bash build.sh --pkg --ops=mhc_sinkhorn_backward --soc=ascend950 --op_debug_config "sanitizer" -j32
cd build_out && bash cann-ops-transformer-custom_linux-x86_64.run --force && cd -
bash build.sh --run_example mhc_sinkhorn_backward eager --soc=ascend950
timeout 1800 mssanitizer --tool=memcheck --tool=racecheck --tool=synccheck --tool=initcheck
--kernel-name=MhcSinkhornBackward --log-level=error -- build/test_aclnn_mhc_sinkhorn_backward

Describe the expected behavior / 预期结果 (Mandatory / 必填)

无异常

[mssanitizer] Start memcheck, racecheck, initcheck and synccheck sanitizer on kernel MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
MhcSinkhornBackward compute success!
====== ERROR: illegal write of size 32
====== at 0x820 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-42) on device 0
====== code in pc current 0x16d8 (serialNo:505)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:329:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:430:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:563:13
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #5 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5

====== ERROR: uninitialized read of size 160
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-41) on device 0
====== code in pc current 0x245c (serialNo:5525)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5

====== ERROR: uninitialized read of size 96
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(42) on device 0
====== code in pc current 0x245c (serialNo:237597)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5

Special notes for this issue/备注 (Optional / 选填)

结论总览

报错 位置 判定
illegal write of size 32 @ 0x820 (所有 43 核) CopyInNormAndSumPad 里 sum 的 DataCopyPad 真 bug(根因)
uninitialized read of size 160 @ 0x880 (核 0-41) CopyOut 读输出队列 同一根因的次生问题
uninitialized read of size 96 @ 0x880 (核 42) CopyOut 读输出队列 同一根因的次生问题

根因:inputSumQue_ 缓冲区尺寸算小了 32 字节

我直接在插桩日志里找到了报错指令(serialNo:505 / pc 0x16d8):

MSTX_STUB, 505, ... DataCopyPad
  dst:(addr:0x820, size:24, UB), lenBurst:12, nBurst:4, rightPad:5

这就是 CopyInNormAndSumPad(simd.h:329)里 sum 的 DataCopyPad。它按 nSize_(4) 块 × INDEX_BLOCK_LEN(8) 个 float = 32 个 float(128 字节) 往 sum 本地缓冲里写,步长 32 字节,落点 0x820 / 0x840 / 0x860 / 0x880。

但 kernel 在 Init(simd.h:142)里给 inputSumQue_ 分配的是

normCorePerLoopTSize(=3) × nAlignSize_(=8) = 24 float = 96 字节/缓冲

写 128 字节、分配 96 字节 → 第 4 块(0x880 起)越界 32 字节。

  • 报错 1 = 这次越界写,illegal write of size 32 与 128−96=32 完全吻合。
  • 报错 2/3 = 越界写刚好落到相邻的输出队列缓冲 0x880(CopyOut 的 MOV_ALIGN 日志 dst src UB 0x880 可证实),把输出缓冲前 32 字节“写脏”。于是 CopyOut 拷贝整段输出时:
    • 普通核读 192 字节,仅 32 字节被(越界的 sum 数据)初始化 → 未初始化 192−32 = 160 字节 ✓
    • 尾核(perloopSize=2)只读 128 字节 → 未初始化 128−32 = 96 字节 ✓

两个数字都与日志精确吻合,说明三条报错是同一条越界写在不同核上的连锁反应。

触发条件(边界场景,但真实存在)

normCorePerLoopTSize(=tUbFactor)=3 < nSize_(=4),且 perloopSize % 8 != 0(本示例 perloopSize=3/2 都走 pad 分支 CopyInNormAndSumPad)。即 小 T、每核分到的 token 数小于 N 时必触发;T 大、perloopSize≥8 且为 8 的倍数时走非 pad 分支则不会。

建议修复

  • 首选:把 inputSumQue_ 每缓冲尺寸改为 max(normCorePerLoopTSize * nAlignSize_, nSize_ * INDEX_BLOCK_LEN)(= max(24, 32)),或直接按 pad 布局 nSize_ * ((perloopSize+7)/8*8) 分配。
  • 同时核对 CalcRowBackward 读 sum 用的 sumAddr + k * perloopSizePad + j * repeatSize([n][padded_t] 布局)与非 pad 路径 CopyInNormAndSum 的扁平 [t*align_n] 布局是否一致——两者对 sum 的排布语义不同,需确认不会读错位。
  • 示例 test_aclnn_mhc_sinkhorn_backward.cpp 目前不做结果断言,建议补一个 CPU 参考对比,否则这类输出被污染的问题会被静默放过。

判定已记入仓库记忆。

一句话总结:mhc_sinkhorn_backward 的 3 个报错都是真阳性,且是同一个 bug——CopyInNormAndSumPad 的 sum DataCopyPad 往只有 96 字节的 inputSumQue_ 里写了 128 字节,越界 32 字节落到相邻的输出缓冲 0x880。illegal write@0x820 就是这次越界本身;两个 uninitialized read(160/96 字节)是它把输出缓冲前 32 字节写脏后,CopyOut 拷贝整段输出时的连锁结果,字节数与越界量精确吻合。

建议修法:inputSumQue_ 的每缓冲尺寸从 tUbFactor*nAlignSize(24 float)放宽到 max(tUbFactor*nAlignSize, nSize_*INDEX_BLOCK_LEN)(32 float);并顺手给示例补输出断言,避免这类输出污染被静默放过。

likedislike
weihao18Collaborators
Aug 12 Comments:

/assign @xuejinghui

likedislike
CANN-robotCANN-robotCollaborators
Aug 12 Assign xuejinghui as responsible person
小王!
小王!
Aug 18 Comments:

根因分析在 :CopyInNormAndSumPad路径中,sum 数据通过 DataCopyPad以下布局写入inputSumQue_:nSize_ × INDEX_BLOCK_LEN,实际写入 128 Bytes,但仅分配 96 Bytes,导致尾部越界写 32 Bytes。该越界区域正好覆盖相邻的输出队列缓冲区,因此后续 CopyOut 读取输出缓冲区时又产生两条次生报错:
普通核读取 192 Bytes,其中仅前 32 Bytes 被越界写入,剩余 160 Bytes 未初始化,因此报告 uninitialized read of size 160;
尾核读取 128 Bytes,其中仅前 32 Bytes 被越界写入,剩余 96 Bytes 未初始化,因此报告 uninitialized read of size 96。
修复时将根据 Pad 路径的真实目标布局和最大写入跨度,调整 inputSumQue_ 每个 buffer 的分配容量,使其同时覆盖普通路径和 Pad 路径。

likedislike
CANN-robotCANN-robotCollaborators
Aug 24 Close issue
CANN-robotCANN-robotCollaborators
Aug 24 Added label: resolved
Hh-glue
Sep 30 Change title to "[Bug-Report|缺陷反馈][工具检测]: 检测到mhc_sinkhorn_backward算子有内存越界异常", originally titled "[Bug-Report|缺陷反馈]: 检测到mhc_sinkhorn_backward算子有内存越界异常".