Closed
[Bug-Report|缺陷反馈]: aclnnBinaryCrossEntropyWithLogitsBackward 空 self 跳过 shape 校验,非法入参返回 0 且不写输出 #7133
liuhanwen8Add on  Oct 21Closed at  19 days ago
liuhanwen8
liuhanwen8
Oct 21 Create

Describe the current behavior / 问题描述 (Mandatory / 必填)

self 为空张量(numel==0)时,第一段接口跳过全部 shape 校验,第二段直接返回成功且一个字节都不写 out:

# self target gradOutput out 实测 资料要求
A [0,4] [0,4] [0,4] [2,4] 两段 rc=0,out 未被写(NaN 金丝雀 7fc00000 原样) 161002(self 与 out shape 不一致)
B [0,4] [2,4] [0,4] [0,4] 两段 rc=0 161002(self 与 target shape 不一致)
C [0,4] [0,4] [5] [0,4] 两段 rc=0 161002(gradOutput 不能 broadcast 到 self)

对照:同一批非法形状在 self 非空时均正确返回 161002(资料错误码子项 E161002-3 / E161002-5 实测 MATCH)。

Environment Information / 环境信息 (Mandatory / 必填)

  • 昇腾硬件型号:Atlas A2 训练系列产品(Ascend910B2,DAV_2201)
  • CANN 版本:9.2.0(aclnn 交付包)
  • 复现源码基线:cann/ops-nn master 8288926

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

  1. 保存下方 repro.cpp,按注释里的 g++ 命令编译(用 CANN 自带 aclnn 头文件与库)。
  2. ./repro empty_out_nonempty —— self=[0,4]、target=[0,4]、gradOutput=[0,4]、out=[2,4](out 预填 NaN)。
  3. 记录两段式返回码,回读 out 前 4 字节。
  4. 另两个子用例:./repro empty_bypass(target=[2,4])、./repro empty_go_not_bcast(gradOutput=[5])。
  5. 对照:./repro normal(self=[2,3] 全部一致)返回 0 且 out 被写。

Describe the expected behavior / 预期结果 (Mandatory / 必填)

三种都应返回 161002。即使允许空输入,也不应返回成功且不写 out。

实际输出(device 0,CANN 9.2.0):

mode=normal GetWorkspaceSize rc=0 ws=1024
mode=normal Execute rc=0
out shape=[2,3] first bytes: 3d9fd59a 3da89e10 3da99b84 3da2c3d2

mode=empty_out_nonempty GetWorkspaceSize rc=0 ws=0
mode=empty_out_nonempty Execute rc=0
out shape=[2,4] first bytes: 7fc00000 7fc00000 7fc00000 7fc00000
out_untouched=YES (canary NaN preserved -> output was NOT written)

mode=empty_bypass GetWorkspaceSize rc=0
mode=empty_bypass Execute rc=0
mode=empty_go_not_bcast GetWorkspaceSize rc=0
mode=empty_go_not_bcast Execute rc=0

根因位置(ops-nn master 8288926,loss/sigmoid_cross_entropy_with_logits_grad_v2/op_api/aclnn_binary_cross_entropy_with_logits_backward.cpp):

  • CheckShape() L178-180:if (self->IsEmpty()) { return true; } —— 空 self 直接放行 target/out 同 shape 以及 gradOutput / weight / posWeight 三条 broadcast 校验;
  • GetWorkspaceSize() L323-329:空 self 提前 return ACLNN_SUCCESS,不建执行器,第二段为空操作,因此 out 保持调用方预分配的内容;
  • 另外 CheckPromoteType() 位于 BinaryCrossEntropyWithLogitsBackwardStub()(L248),也在空张量早退之后,dtype 提升校验同样被跳过。

建议:删掉 CheckShape() 里的提前返回(0 尺寸维度的 shape 比较与 broadcast 推导本身是合法的),保留计算侧的空张量短路;并把 dtype 提升校验移到早退之前。

Special notes for this issue/备注 (Optional / 选填)

动态 shape 训练里空 batch 是常态;若上层按非空 shape 预分配或复用 out buffer,会静默读到未初始化数据(NaN / 上一轮残留),且没有任何错误码。

同类缺陷在兄弟接口上已单独提单:#7097(aclnnBinaryCrossEntropyWithLogitsTargetBackward,不同文件、需分别修)。

repro.cpp(编译:g++ -std=c++17 repro_bcewb.cpp -o repro -I$ASCEND_HOME_PATH/include -I$ASCEND_HOME_PATH/include/aclnnop -L$ASCEND_HOME_PATH/lib64 -lopapi_nn -lopapi -lnnopbase -lascendcl -Wl,-rpath,$ASCEND_HOME_PATH/lib64 -Wl,--allow-shlib-undefined)
// Minimal reproducer for aclnnBinaryCrossEntropyWithLogitsBackward (A2 / Ascend910B, CANN 9.2.0)
//
// Build:
//   g++ -std=c++17 repro_bcewb.cpp -o repro \
//     -I$ASCEND_HOME_PATH/include -I$ASCEND_HOME_PATH/include/aclnnop \
//     -L$ASCEND_HOME_PATH/lib64 -lopapi_nn -lopapi -lnnopbase -lascendcl \
//     -Wl,-rpath,$ASCEND_HOME_PATH/lib64 -Wl,--allow-shlib-undefined
//
// Usage: ./repro <mode> [device]
//   normal               self=[2,3] all shapes consistent  -> rc=0, out written
//   empty_out_nonempty   self=[0,4] out=[2,4] (doc: shape must match) -> rc=0, out NOT written
//   empty_bypass         self=[0,4] target=[2,4]  (doc: shape must match) -> rc=0
//   empty_go_not_bcast   self=[0,4] gradOutput=[5] (doc: must be broadcastable) -> rc=0
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <cmath>
#include <string>
#include <vector>
#include <cstdint>

#include "acl/acl.h"
#include "aclnnop/aclnn_binary_cross_entropy_with_logits_backward.h"

#define CHK(cond, msg) do { if (!(cond)) { printf("FAIL %s (line %d)\n", msg, __LINE__); std::exit(2);} } while (0)

static int g_dev = 0;
static aclrtStream g_stream = nullptr;

struct Ten {
    void* dev = nullptr;
    aclTensor* t = nullptr;
    int64_t numel = 1;
    void build(const std::vector<int64_t>& shp) {
        numel = 1;
        for (auto d : shp) numel *= d;
        size_t bytes = (size_t)numel * 4;
        if (bytes == 0) bytes = 4;
        CHK(aclrtMalloc(&dev, bytes, ACL_MEM_MALLOC_HUGE_FIRST) == ACL_SUCCESS, "aclrtMalloc");
        std::vector<int64_t> st(shp.size(), 1);
        for (int64_t i = (int64_t)shp.size() - 2; i >= 0; --i) st[i] = shp[i + 1] * st[i + 1];
        t = aclCreateTensor(shp.data(), shp.size(), ACL_FLOAT, st.data(), 0, ACL_FORMAT_ND,
                            shp.data(), shp.size(), dev);
        CHK(t != nullptr, "aclCreateTensor");
    }
    void fill_ramp() {
        std::vector<float> h((size_t)numel);
        for (int64_t i = 0; i < numel; ++i) h[i] = 0.25f + 0.05f * (float)(i % 5);
        aclrtMemcpy(dev, (size_t)numel * 4, h.data(), (size_t)numel * 4, ACL_MEMCPY_HOST_TO_DEVICE);
    }
    void fill_nan() {  // canary: 0x7fc00000 (float NaN)
        std::vector<uint32_t> h((size_t)numel, 0x7fc00000u);
        aclrtMemcpy(dev, (size_t)numel * 4, h.data(), (size_t)numel * 4, ACL_MEMCPY_HOST_TO_DEVICE);
    }
    std::vector<uint32_t> bits() const {
        std::vector<uint32_t> h((size_t)numel, 0u);
        aclrtMemcpy(h.data(), (size_t)numel * 4, dev, (size_t)numel * 4, ACL_MEMCPY_DEVICE_TO_HOST);
        return h;
    }
    void destroy() { if (t) aclDestroyTensor(t); if (dev) aclrtFree(dev); }
};

int main(int argc, char** argv) {
    std::string mode = argc > 1 ? argv[1] : "empty_out_nonempty";
    if (argc > 2) g_dev = std::atoi(argv[2]);

    std::vector<int64_t> selfShape{0, 4}, goShape{0, 4}, tgtShape{0, 4}, outShape{0, 4};
    bool canary = false;
    if (mode == "normal") { selfShape = {2, 3}; goShape = {2, 3}; tgtShape = {2, 3}; outShape = {2, 3}; }
    else if (mode == "empty_out_nonempty") { canary = true; outShape = {2, 4}; }
    else if (mode == "empty_bypass") { tgtShape = {2, 4}; }
    else if (mode == "empty_go_not_bcast") { goShape = {5}; }

    CHK(aclInit(nullptr) == ACL_SUCCESS, "aclInit");
    CHK(aclrtSetDevice(g_dev) == ACL_SUCCESS, "aclrtSetDevice");
    CHK(aclrtCreateStream(&g_stream) == ACL_SUCCESS, "aclrtCreateStream");

    Ten go, self, tgt, out;
    go.build(goShape); go.fill_ramp();
    self.build(selfShape); self.fill_ramp();
    tgt.build(tgtShape); tgt.fill_ramp();
    out.build(outShape);
    if (canary) out.fill_nan(); else out.fill_ramp();

    uint64_t ws = 0; aclOpExecutor* ex = nullptr;
    aclnnStatus rc1 = aclnnBinaryCrossEntropyWithLogitsBackwardGetWorkspaceSize(
        go.t, self.t, tgt.t, nullptr, nullptr, 0, out.t, &ws, &ex);
    printf("mode=%s GetWorkspaceSize rc=%d ws=%llu\n", mode.c_str(), (int)rc1, (unsigned long long)ws);

    int rc2 = -1;
    if (rc1 == 0) {
        void* wsp = nullptr;
        if (ws > 0) aclrtMalloc(&wsp, ws, ACL_MEM_MALLOC_HUGE_FIRST);
        rc2 = (int)aclnnBinaryCrossEntropyWithLogitsBackward(wsp, ws, ex, g_stream);
        aclrtSynchronizeStream(g_stream);
        printf("mode=%s Execute rc=%d\n", mode.c_str(), rc2);
        if (wsp) aclrtFree(wsp);
    }

    std::vector<uint32_t> b = out.bits();
    printf("out shape=[");
    for (size_t i = 0; i < outShape.size(); ++i) printf("%lld%s", (long long)outShape[i], i + 1 < outShape.size() ? "," : "");
    printf("] first bytes:");
    for (size_t i = 0; i < b.size() && i < 4; ++i) printf(" %08x", b[i]);
    printf("\n");
    if (canary) {
        bool untouched = !b.empty();
        for (auto v : b) if (v != 0x7fc00000u) { untouched = false; break; }
        printf("out_untouched=%s\n", untouched ? "YES (canary NaN preserved -> output was NOT written)" : "NO (written)");
    }

    out.destroy(); tgt.destroy(); self.destroy(); go.destroy();
    aclrtDestroyStream(g_stream); aclrtResetDevice(g_dev); aclFinalize();
    return 0;
}

likedislike
yuning_chenyuning_chenCollaborators
Oct 21 Assign ji-songyuan as responsible person
yuning_chen
yuning_chenCollaborators
Oct 22 Comments:

初步分析

感谢反馈。

现象理解:aclnnBinaryCrossEntropyWithLogitsBackward 在 self 为空张量时跳过 shape 校验,第二段返回成功但不写 out。self 非空时,同样的非法形状会返回 161002。

当前判断:初步判断,待复现确认。

涉及范围:binary_cross_entropy_with_logits 反向的空张量路径。

下一步:用空 self 配不一致的 target、gradOutput 或 out,看返回码和 out 是否被写。确认后,空张量也先做 shape 校验。

likedislike
yuning_chen
yuning_chenCollaborators
Oct 22 Comments:

您好,该问题已收到,当前问题 @ji-songyuan 处理中

likedislike
sunhao_hwsunhao_hwCollaborators
22 days ago Associated pull request
sunhao_hw
sunhao_hwCollaborators
22 days ago Comments:

感谢您的反馈。我们已经对 Issue 中描述的问题进行了核查,并在 CANN 9.2.0、Atlas A2(910B3)上复现,确认问题属实。

aclnnBinaryCrossEntropyWithLogitsBackward 在 self 为空张量时会跳过 shape 校验。CheckShape 在 self->IsEmpty() 时直接返回成功,self 与 target、out 的 shape 一致性,以及 gradOutput、weight、posWeight 向 self 的 broadcast 约束都未执行;GetWorkspaceSize 随后对空 self 提前返回 ACLNN_SUCCESS,不建立计算任务,因此 out 保持调用方原有内容。对照结果:self 非空且 out shape 不一致时正确返回 161002;self、target、gradOutput、out 均为 [0,4] 的合法空输入返回 0。Issue 中的三种非法形状(out=[2,4]、target=[2,4]、gradOutput=[5])两段返回码均为 0,其中 out 非空的用例预填 NaN 未被改写。

修复已提交:https://gitcode.com/cann/ops-nn/pull/13156

该修改去掉 CheckShape 中的空张量提前返回,使 0 尺寸维度仍走既有 shape 与 broadcast 校验。shape 合法的空 self 仍在计算侧短路返回。用修改后的 op_api 复测,上述三种非法入参的 GetWorkspaceSize 均返回 161002,合法空输入与非空一致输入仍返回 0。再次感谢您的反馈。

likedislike
CANN-robotCANN-robotCollaborators
19 days ago Close issue
sunhao_hwsunhao_hwCollaborators
19 days ago Assign sunhao_hw as responsible person
sunhao_hwsunhao_hwCollaborators
19 days ago changed custom state from 进行中 to 已完成
CANN-robotCANN-robotCollaborators
19 days ago Added label: resolved