Closed
[Bug-Report|缺陷反馈][工具检测]: 检测到mhc_sinkhorn_backward算子有内存越界异常 #4282
h-glueAdd on Aug 12Closed at Aug 24
weihao18
Aug 12 Comments:
Aug 12 Comments:
/assign @xuejinghui


Aug 12 Assign xuejinghui as responsible person
小王!
Aug 18 Comments:
Aug 18 Comments:
根因分析在 :CopyInNormAndSumPad路径中,sum 数据通过 DataCopyPad以下布局写入inputSumQue_:nSize_ × INDEX_BLOCK_LEN,实际写入 128 Bytes,但仅分配 96 Bytes,导致尾部越界写 32 Bytes。该越界区域正好覆盖相邻的输出队列缓冲区,因此后续 CopyOut 读取输出缓冲区时又产生两条次生报错:
普通核读取 192 Bytes,其中仅前 32 Bytes 被越界写入,剩余 160 Bytes 未初始化,因此报告 uninitialized read of size 160;
尾核读取 128 Bytes,其中仅前 32 Bytes 被越界写入,剩余 96 Bytes 未初始化,因此报告 uninitialized read of size 96。
修复时将根据 Pad 路径的真实目标布局和最大写入跨度,调整 inputSumQue_ 每个 buffer 的分配容量,使其同时覆盖普通路径和 Pad 路径。


Aug 24 Close issue
Aug 24 Added label: resolved
Sep 30 Change title to "[Bug-Report|缺陷反馈][工具检测]: 检测到mhc_sinkhorn_backward算子有内存越界异常", originally titled "[Bug-Report|缺陷反馈]: 检测到mhc_sinkhorn_backward算子有内存越界异常".
Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.
Describe the current behavior / 问题描述 (Mandatory / 必填)
[mssanitizer] Start memcheck, racecheck, initcheck and synccheck sanitizer on kernel MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
MhcSinkhornBackward compute success!
====== ERROR: illegal write of size 32
====== at 0x820 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-42) on device 0
====== code in pc current 0x16d8 (serialNo:505)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:329:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:430:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:563:13
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #5 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5
====== ERROR: uninitialized read of size 160
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-41) on device 0
====== code in pc current 0x245c (serialNo:5525)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5
====== ERROR: uninitialized read of size 96
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(42) on device 0
====== code in pc current 0x245c (serialNo:237597)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5
Environment / 环境信息 (Mandatory / 必填)
A5
cann 730商用9.1.0
ops-transformer 9.1.0分支
Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
bash build.sh --pkg --ops=mhc_sinkhorn_backward --soc=ascend950 --op_debug_config "sanitizer" -j32
cd build_out && bash cann-ops-transformer-custom_linux-x86_64.run --force && cd -
bash build.sh --run_example mhc_sinkhorn_backward eager --soc=ascend950
timeout 1800 mssanitizer --tool=memcheck --tool=racecheck --tool=synccheck --tool=initcheck
--kernel-name=MhcSinkhornBackward --log-level=error -- build/test_aclnn_mhc_sinkhorn_backward
Describe the expected behavior / 预期结果 (Mandatory / 必填)
无异常
Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
[mssanitizer] Start memcheck, racecheck, initcheck and synccheck sanitizer on kernel MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
MhcSinkhornBackward compute success!
====== ERROR: illegal write of size 32
====== at 0x820 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-42) on device 0
====== code in pc current 0x16d8 (serialNo:505)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:329:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:430:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:563:13
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #5 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5
====== ERROR: uninitialized read of size 160
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(0-41) on device 0
====== code in pc current 0x245c (serialNo:5525)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5
====== ERROR: uninitialized read of size 96
====== at 0x880 on UB in MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_0
====== in block aiv(42) on device 0
====== code in pc current 0x245c (serialNo:237597)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:378:5
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/arch35/mhc_sinkhorn_backward_simd.h:575:9
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/mhc_sinkhorn_backward_apt/op_kernel/mhc_sinkhorn_backward_apt.cpp:34:8
====== #3 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:32:5
====== #4 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9/kernel_meta/MhcSinkhornBackward_6b91c6bfa95195531f5a512c65ae05d9_kernel.cpp:43:5
Special notes for this issue/备注 (Optional / 选填)
结论总览
illegal write of size 32 @ 0x820(所有 43 核)CopyInNormAndSumPad里 sum 的DataCopyPaduninitialized read of size 160 @ 0x880(核 0-41)CopyOut读输出队列uninitialized read of size 96 @ 0x880(核 42)CopyOut读输出队列根因:
inputSumQue_缓冲区尺寸算小了 32 字节我直接在插桩日志里找到了报错指令(serialNo:505 / pc 0x16d8):
这就是
CopyInNormAndSumPad(simd.h:329)里 sum 的DataCopyPad。它按nSize_(4) 块 ×INDEX_BLOCK_LEN(8) 个 float = 32 个 float(128 字节) 往 sum 本地缓冲里写,步长 32 字节,落点0x820 / 0x840 / 0x860 / 0x880。但 kernel 在
Init(simd.h:142)里给inputSumQue_分配的是写 128 字节、分配 96 字节 → 第 4 块(0x880 起)越界 32 字节。
illegal write of size 32与 128−96=32 完全吻合。0x880(CopyOut的 MOV_ALIGN 日志 dst srcUB 0x880可证实),把输出缓冲前 32 字节“写脏”。于是CopyOut拷贝整段输出时:两个数字都与日志精确吻合,说明三条报错是同一条越界写在不同核上的连锁反应。
触发条件(边界场景,但真实存在)
normCorePerLoopTSize(=tUbFactor)=3 <nSize_(=4),且perloopSize % 8 != 0(本示例 perloopSize=3/2 都走 pad 分支CopyInNormAndSumPad)。即 小 T、每核分到的 token 数小于 N 时必触发;T 大、perloopSize≥8 且为 8 的倍数时走非 pad 分支则不会。建议修复
inputSumQue_每缓冲尺寸改为max(normCorePerLoopTSize * nAlignSize_, nSize_ * INDEX_BLOCK_LEN)(= max(24, 32)),或直接按 pad 布局nSize_ * ((perloopSize+7)/8*8)分配。CalcRowBackward读 sum 用的sumAddr + k * perloopSizePad + j * repeatSize([n][padded_t]布局)与非 pad 路径CopyInNormAndSum的扁平[t*align_n]布局是否一致——两者对 sum 的排布语义不同,需确认不会读错位。判定已记入仓库记忆。
一句话总结:
mhc_sinkhorn_backward的 3 个报错都是真阳性,且是同一个 bug——CopyInNormAndSumPad的 sumDataCopyPad往只有 96 字节的inputSumQue_里写了 128 字节,越界 32 字节落到相邻的输出缓冲0x880。illegal write@0x820就是这次越界本身;两个uninitialized read(160/96 字节)是它把输出缓冲前 32 字节写脏后,CopyOut拷贝整段输出时的连锁结果,字节数与越界量精确吻合。建议修法:
inputSumQue_的每缓冲尺寸从tUbFactor*nAlignSize(24 float)放宽到max(tUbFactor*nAlignSize, nSize_*INDEX_BLOCK_LEN)(32 float);并顺手给示例补输出断言,避免这类输出污染被静默放过。