已关闭
[Bug-Report|缺陷反馈][工具检测]: 检测moe\moe_token_unpermute_with_routing_map有内存异常 #4278
h-glue创建于 8月12日关闭于 28 天前
weihao18
8月12日 评论:
8月12日 评论:
/assign @bartonfang


8月12日 将 fazhenyao123 设为负责人
CANN-robot
8月27日 评论:
8月27日 评论:
8月29日 关联了pull request:fix: moe_token_unpermute_with_routing_map pad path 0-size UB buffer OOB
8月29日 关联了pull request:fix: moe_token_unpermute_with_routing_map pad path 0-size UB buffer OOB
9月24日 关联了pull request:fix: moe_token_unpermute_with_routing_map pad path 0-size UB buffer OOB
9月24日 关联了pull request:fix: moe_token_unpermute_with_routing_map pad path 0-size UB buffer OOB
28 天前 关闭了 issue
28 天前 添加了label:resolved
26 天前 修改标题为 “[Bug-Report|缺陷反馈][工具检测]: 检测moe\moe_token_unpermute_with_routing_map有内存异常”,原标题为“[Bug-Report|缺陷反馈]: 检测moe\moe_token_unpermute_with_routing_map有内存异常”
Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.
Describe the current behavior / 问题描述 (Mandatory / 必填)
[mssanitizer] Start memcheck, racecheck, initcheck and synccheck sanitizer on kernel MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_1000
====== ERROR: illegal read of size 4
====== at 0x0 on UB in MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_1000
====== in block aiv(0-1) on device 0
====== code in pc current 0xcfc (serialNo:217)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/moe_token_unpermute_with_routing_map/op_kernel/moe_token_unpermute_with_routing_map.cpp:67:8
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed/kernel_meta/MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_kernel.cpp:27:5
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed/kernel_meta/MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_kernel.cpp:32:5
Environment / 环境信息 (Mandatory / 必填)
A5
基于ops-transformer 9.1.0分支
cann包730商法
Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
bash build.sh --pkg --ops=moe_token_unpermute_with_routing_map --soc=ascend950 --op_debug_config "sanitizer" -j32
cd build_out && bash cann-ops-transformer-custom_linux-x86_64.run --force && cd -
bash build.sh --run_example moe_token_unpermute_with_routing_map eager --soc=ascend950
timeout 1800 mssanitizer --tool=memcheck --tool=racecheck --tool=synccheck --tool=initcheck
--kernel-name=MoeTokenUnpermuteWithRoutingMap --log-level=error -- build/test_aclnn_moe_token_unpermute_with_routing_map
Describe the expected behavior / 预期结果 (Mandatory / 必填)
无内存异常
Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
[mssanitizer] Start memcheck, racecheck, initcheck and synccheck sanitizer on kernel MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_1000
====== ERROR: illegal read of size 4
====== at 0x0 on UB in MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_1000
====== in block aiv(0-1) on device 0
====== code in pc current 0xcfc (serialNo:217)
====== #0 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/src/moe_token_unpermute_with_routing_map/op_kernel/moe_token_unpermute_with_routing_map.cpp:67:8
====== #1 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed/kernel_meta/MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_kernel.cpp:27:5
====== #2 /home/npu_user3/huangguang/workspace/ops-transformer/build/binary/ascend950/gen/kernel_meta_MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed/kernel_meta/MoeTokenUnpermuteWithRoutingMap_5a04a8d5a5f8b1d188f09777225257ed_kernel.cpp:32:5
Special notes for this issue/备注 (Optional / 选填)
结论总览
illegal read of size 4 @ 0x0_1000(paddedMode=true)真 bug —— pad 路径的 0 字节 UB 输出队列
栈顶
moe_token_unpermute_with_routing_map.cpp:67=op.Process(),即 TILING_KEY 1000 分支的KernelMoeTokenUnpermuteWithRoutingMapPad。根因在 host tiling 与 kernel 的 buffer 尺寸联动上:_tiling.cpp:166): 本例num_experts=2, capacity=1 → rowVal=2,num_tokens_each_front_core = ceil(2/coreNum) = 1,而maxNPerLoopForUb ≈ UB/4 ≈ 49k,所以GetAlign(1, 49152) = (1/49152)*49152 = **0**。_pad.h:83) 用这个值分配输出队列:Compute()(_pad.h:117) 再往这个 0 字节 buffer 里SetValue+DataCopyPad→ 访问地址 0x0 → sanitizer 抓到的illegal read @ 0x0。旁证:示例输出的
original unpermutedTokensData = [0,0,2,2]是垃圾值——probs 全为 1,正确 gather 结果应为[1,1,0,0]。因为示例对 pad 场景没有做断言,所以功能错误被静默放过了。这不是 sanitizer 误报,而是一个真实的小 shape 越界 bug(生产大 shape 下
num_tokens_each_front_core ≥ maxNPerLoopForUb时不会触发,所以平时没暴露)。建议
必改(真 bug):修复 pad 路径的 buffer 尺寸——按
max(num_tokens_each_loop_current_core, num_tokens_last_loop_current_core)分配输出队列,或在 host 侧当num_tokens_each_front_core < maxNPerLoopForUb时让each_loop兜底为实际值(而非 0)。