Closed
[Bug-Report|缺陷反馈]: aclnnBatchMatMul 在 self batch=0、mat2 batch=1 时返回成功但输出未写入 #7976
liuhanwen8Add on 19 days agoClosed at 10 days ago
19 days ago Added label: bug-report
19 days ago Modify description of issue
19 days ago Modify description of issue
19 days ago Assign fanqirui as responsible person
yuning_chen
19 days ago Comments:
19 days ago Comments:
yuning_chen
19 days ago Comments:
19 days ago Comments:
您好,该问题已收到,当前问题 @fanqirui 正在处理中


19 days ago Assign linzehui2058 as responsible person
19 days ago Removed responsible person fanqirui
16 days ago Associated pull request
16 days ago Deleted associated pull request: fix_issue_11_5
10 days ago Close issue
10 days ago changed custom state from 进行中 to 已完成
9 days ago Added label: resolved
chenjiayi_
5 days ago Comments:
5 days ago Comments:
补充说明本问题的后续处理:
已修复并合入 master。
处理结果:
CheckBmmResIsEmpty改为仅在self[0] == 0且mat2[0] == 0时走空 tensor 早退;单侧 batch 为 0、另一侧为 1 时进入正常计算路径。ProcessEmptyTensor的输出 batch 改为max(self[0], mat2[0]),非空输出通过Fill写成全 0。- PR:!13786(同时覆盖 #7997)。
验证:
- master 源码
matmul/batch_mat_mul_v3/op_host/op_api/aclnn_batch_matmul.cpp:150-156当前为空 tensor 判定;matmul/common/op_host/op_api/batch_matmul_util.cpp:81-96为按 max batch 生成并填 0 的逻辑。 - PR !13786 的改动与 issue 给出的
self=[0,4,8]、mat2=[1,8,16]场景对应。
说明:该修复随 PR 合入关闭;本环境基于源码核对,未上板复跑。
如果还有后续需要,请重新提 issue。


3 days ago Assign chenjiayi_ as responsible person
23 hours ago Removed responsible person chenjiayi_
问题描述
aclnnBatchMatMul在self的 batch 维为 0、mat2的 batch 维为 1 时:aclnnBatchMatMulGetWorkspaceSize返回 0,且workspaceSize=0;aclnnBatchMatMul返回 0;out的全部元素保持调用前的内存内容,没有被写入任何值。该组合能通过接口自身的形状校验:batch 维允许一侧为 1 时广播,
firstDim=max(0,1)=1,out=[1,M,N]合法。因此调用方无法从返回码发现问题,会读到未初始化的显存内容。相关代码(
matmul/batch_mat_mul_v3/op_host/op_api/aclnn_batch_matmul.cpp):150行CheckBmmResIsEmpty只判断self[0]==0 || self[1]==0 || mat2[2]==0,不检查mat2的 batch 维;334-338行命中后仅执行*workspaceSize = 0; return ACLNN_SUCCESS;,没有追加任何写out的节点。当
self[0]==0、mat2[0]==1时,out非空,却走了空 tensor 早退分支。对比:
selfbatch=1、mat2batch=0 时走正常路径并写出全 0,两种方向行为不一致。环境信息
aclnnBatchMatMul重现步骤
bmm_repro.cpp;./bmm_repro,观察输出。// g++ -std=c++17 -I$ASCEND_HOME/include -L$ASCEND_HOME/lib64 -Wl,-rpath,$ASCEND_HOME/lib64 \ // -o bmm_repro bmm_repro.cpp -lopapi -lnnopbase -lascendcl #include <acl/acl.h> #include <aclnnop/aclnn_batch_matmul.h> #include <cstdint> #include <cstdio> #include <vector> static aclTensor* Make(const std::vector<int64_t>& s, void* d) { std::vector<int64_t> st(s.size(), 1); for (int i = static_cast<int>(s.size()) - 2; i >= 0; --i) { st[i] = st[i + 1] * s[i + 1]; } return aclCreateTensor(s.data(), s.size(), ACL_FLOAT, st.data(), 0, ACL_FORMAT_ND, s.data(), s.size(), d); } int main() { aclInit(nullptr); aclrtSetDevice(0); aclrtContext ctx = nullptr; aclrtCreateContext(&ctx, 0); aclrtSetCurrentContext(ctx); aclrtStream stream = nullptr; aclrtCreateStream(&stream); const int64_t outElems = 64; void* sD = nullptr; void* mD = nullptr; void* oD = nullptr; aclrtMalloc(&sD, 4, ACL_MEM_MALLOC_HUGE_FIRST); aclrtMalloc(&mD, 128 * 4, ACL_MEM_MALLOC_HUGE_FIRST); aclrtMalloc(&oD, outElems * 4, ACL_MEM_MALLOC_HUGE_FIRST); std::vector<float> one(128, 1.0f); std::vector<uint32_t> nan(outElems, 0x7FC00000u); aclrtMemcpy(mD, 128 * 4, one.data(), 128 * 4, ACL_MEMCPY_HOST_TO_DEVICE); aclrtMemcpy(oD, outElems * 4, nan.data(), outElems * 4, ACL_MEMCPY_HOST_TO_DEVICE); aclTensor* s = Make({0, 4, 8}, sD); aclTensor* m = Make({1, 8, 16}, mD); aclTensor* o = Make({1, 4, 16}, oD); uint64_t ws = 0; aclOpExecutor* ex = nullptr; aclnnStatus r1 = aclnnBatchMatMulGetWorkspaceSize(s, m, o, 1, &ws, &ex); aclnnStatus r2 = aclnnBatchMatMul(nullptr, ws, ex, stream); aclrtSynchronizeStream(stream); std::vector<uint32_t> back(outElems, 0); aclrtMemcpy(back.data(), outElems * 4, oD, outElems * 4, ACL_MEMCPY_DEVICE_TO_HOST); int nanLeft = 0; for (uint32_t bits : back) { if ((bits & 0x7F800000u) == 0x7F800000u && (bits & 0x007FFFFFu) != 0) { ++nanLeft; } } printf("gws=%d exec=%d workspace=%llu nan=%d/%lld\n", static_cast<int>(r1), static_cast<int>(r2), static_cast<unsigned long long>(ws), nanLeft, static_cast<long long>(outElems)); aclrtDestroyStream(stream); aclrtDestroyContext(ctx); aclrtResetDevice(0); aclFinalize(); return 0; }预期结果
二选一:
out被正确写出:self的 batch 为 0、mat2只有 1 份数据,广播后每个 batch 的空和为 0;当前行为是第三种:放行、返回 0、但不写
out。日志 / 截图
nan=64/64表示out的 64 个元素全部仍是预填的 NaN 哨兵。备注
self=[0,1,1]、mat2=[1,1,1]、out=[1,1,1]同样未写;self=[1,4,8]同样输入时输出正常写入,且 cubeMathType=0 与 fp64 参考值最大误差 2.8e-8,说明检测手段有效;out->IsEmpty()判断,仅当输出为空时才直接返回;否则补一个零填充节点,或对self[0]==0 && mat2[0]!=0的组合返回 161002。同族实现对照(修复建议参考)
同目录的
aclnnBaddbmm(matmul/batch_mat_mul_v3/op_host/op_api/aclnn_baddbmm.cpp,92-110 行)对同类场景有显式防线:bool selfIsEmpty = selfTensor->IsEmpty(); bool matIsEmpty = (batch1[FIRST_DIM] == 0 || batch1[SHAPE_LIMIT - PENULTIMATE_DIM] == 0 || batch2[SHAPE_LIMIT - LAST_DIM] == 0) ? true : false; if ((selfIsEmpty && !matIsEmpty) || (!selfIsEmpty && matIsEmpty)) { OP_LOGE(ACLNN_ERR_PARAM_INVALID, "self and batch1@batch2 should be both empty tensors, or not, ..."); return false; }即:一方为空、另一方非空时直接返回 161002。
aclnnBatchMatMul只需要在 334 行的早退前补同样的判定(或补out->IsEmpty()条件),即可消除本问题。实测同一环境(Ascend 910B / CANN 9.2.0-beta.2)下
aclnnBaddbmm的行为:即同族接口对「一方空、一方非空」明确拒绝,与上面的源码防线一致;
aclnnBatchMatMul缺这一层。