Closed
[Bug-Report|缺陷反馈]: [aclnnMseLossOut] 空 tensor 输入返回成功但 out 未被写出(整块残留未初始化显存) #7542
liuhanwen8Add on  24 days agoClosed at  11 days ago
liuhanwen8
liuhanwen8
24 days ago Create

现象

aclnnMseLossOut 在 self/target 含 0 元素、reduction=1(mean) 或 2(sum) 时,两段接口都返回 0,但 out 一个字节都没被改写,用户拿到的是 device 上未初始化的旧内容。

实测(Ascend 910B2 / CANN 9.2.0 / ops-nn 711a4f5d,out 在 device 上预填 -777.0 当金丝雀):

[empty_1d_mean] ret1=0 ret2=0 workspace=0 out_n=1 canary_left=1/1
[empty_1d_sum]  ret1=0 ret2=0 workspace=0 out_n=1 canary_left=1/1
[empty_2d_mean] ret1=0 ret2=0 workspace=0 out_n=4 canary_left=4/4
[empty_2d_sum]  ret1=0 ret2=0 workspace=0 out_n=4 canary_left=4/4
[nonempty_mean_control] ret1=0 ret2=0 workspace=1536 out_n=4 canary_left=0/4   ← 非空对照,正常写出

canary_left = n/n 表示 out 的 n 个元素全部仍是预填值。同仓 aclnnMseLoss 在同一组入参下 out 得到 NaN(reduction=1) / 0(reduction=2)。

成因:loss/mse_loss/op_api/aclnn_mse_loss_out.cpp 的早退分支 if (self->IsEmpty() || target->IsEmpty()) 里只有一行注释 // 根据实际支持情况补充,没有填充动作;而 aclnn_mse_loss.cpp 同一位置调用 MseLossEmptyTensorCompute,用 loss/common/level2_base_loss.h 的 CheckFillScalarLoss 按 reduction 填 NaN/0(aclnnL1Loss、aclnnSoftMarginLoss 同)。另一个独立入口(valid-fuzz 冒烟,out 预填 0xA5)也判出 输出未写出(仍是预填金丝雀):out(len=4, a5a5a5a5)。

期望

reduction=1 → out 全 NaN;reduction=2 → out 全 0;reduction=0 → out 本身是空张量,不写是正确行为。与同仓 aclnnMseLoss 及 PyTorch mse_loss 的空输入语义一致(资料未声明空 tensor 语义,另见 #7466)。

环境

  • 硬件:Atlas A2 训练服务器,Ascend 910B2,单 die
  • CANN:9.2.0(toolkit cann-9.2.0);算子仓 cann/ops-nn 711a4f5d(loss/mse_loss)
  • 形态:aclnn 两段式 C 直调,g++ 11.3 / aarch64

复现步骤

  1. self=[0]、target=[0]、fp32、ND;out 取 0 维 fp32,device 内存预填 -777.0
  2. reduction=1 调 aclnnMseLossOutGetWorkspaceSize,记录返回码与 workspaceSize
  3. 调 aclnnMseLossOut + aclrtSynchronizeStream,记录返回码
  4. out 拷回 host,统计仍等于 -777.0 的元素个数
  5. reduction 换 2、形状换 self=[0,4] / out=[4] 重复 2-4;再换 self=[2,4] 做非空对照

复现脚本

实测输出(本脚本跑出的就是下面五行,与「现象」一致):

[empty_1d_mean] ret1=0 ret2=0 workspace=0 out_n=1 canary_left=1/1
[empty_1d_sum]  ret1=0 ret2=0 workspace=0 out_n=1 canary_left=1/1
[empty_2d_mean] ret1=0 ret2=0 workspace=0 out_n=4 canary_left=4/4
[empty_2d_sum]  ret1=0 ret2=0 workspace=0 out_n=4 canary_left=4/4
[nonempty_mean_control] ret1=0 ret2=0 workspace=1536 out_n=4 canary_left=0/4
// g++ -std=c++17 -O1 -o repro repro.cpp -I$ASCEND_TOOLKIT_HOME/include -L$ASCEND_TOOLKIT_HOME/lib64 \
//     -lascendcl -lnnopbase -lopapi -Wl,--allow-shlib-undefined   &&   ./repro 0
#include <cstdio>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_mse_loss_out.h"

static aclTensor* Make(const std::vector<int64_t>& shape, void** dev, float fill) {
    int64_t n = 1; for (auto v : shape) n *= v;
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = (int64_t)shape.size() - 2; i >= 0; --i) strides[i] = shape[i + 1] * strides[i + 1];
    size_t bytes = (size_t)(n > 0 ? n : 1) * 4;
    aclrtMalloc(dev, bytes, ACL_MEM_MALLOC_HUGE_FIRST);
    std::vector<float> h((size_t)(n > 0 ? n : 1), fill);
    aclrtMemcpy(*dev, bytes, h.data(), bytes, ACL_MEMCPY_HOST_TO_DEVICE);
    std::vector<int64_t> storage{n > 0 ? n : 1};
    return aclCreateTensor(shape.data(), shape.size(), ACL_FLOAT, strides.data(), 0, ACL_FORMAT_ND,
                           storage.data(), storage.size(), *dev);
}

static void Run(const char* name, std::vector<int64_t> ssh, std::vector<int64_t> tsh,
                std::vector<int64_t> osh, int64_t red) {
    void *sp = nullptr, *tp = nullptr, *op = nullptr;
    aclTensor* self = Make(ssh, &sp, 1.0f);
    aclTensor* target = Make(tsh, &tp, 0.0f);
    aclTensor* out = Make(osh, &op, -777.0f);
    uint64_t ws = 0; aclOpExecutor* ex = nullptr;
    aclnnStatus r1 = aclnnMseLossOutGetWorkspaceSize(self, target, red, out, &ws, &ex);
    aclnnStatus r2 = -1;
    if (r1 == 0) {
        void* wsp = nullptr;
        if (ws > 0) aclrtMalloc(&wsp, ws, ACL_MEM_MALLOC_HUGE_FIRST);
        aclrtStream st = nullptr; aclrtCreateStream(&st);
        r2 = aclnnMseLossOut(wsp, ws, ex, st);
        aclrtSynchronizeStream(st);
        if (wsp) aclrtFree(wsp);
        aclrtDestroyStream(st);
    }
    int64_t on = 1; for (auto v : osh) on *= v;
    std::vector<float> h((size_t)(on > 0 ? on : 1), 0.0f);
    aclrtMemcpy(h.data(), (size_t)on * 4, op, (size_t)on * 4, ACL_MEMCPY_DEVICE_TO_HOST);
    int canary = 0;
    for (int64_t i = 0; i < on; ++i) if (h[(size_t)i] == -777.0f) canary++;
    printf("[%s] ret1=%d ret2=%d workspace=%llu out_n=%lld canary_left=%d/%lld\n",
           name, (int)r1, (int)r2, (unsigned long long)ws, (long long)on, canary, (long long)on);
    aclDestroyTensor(self); aclDestroyTensor(target); aclDestroyTensor(out);
    aclrtFree(sp); aclrtFree(tp); aclrtFree(op);
}

int main(int argc, char** argv) {
    setvbuf(stdout, nullptr, _IONBF, 0);
    int dev = (argc > 1) ? atoi(argv[1]) : 0;
    if (aclInit(nullptr) != ACL_SUCCESS || aclrtSetDevice(dev) != ACL_SUCCESS) { printf("init failed\n"); return 1; }
    Run("empty_1d_mean", {0}, {0}, {}, 1);
    Run("empty_1d_sum", {0}, {0}, {}, 2);
    Run("empty_2d_mean", {0, 4}, {0, 4}, {4}, 1);
    Run("empty_2d_sum", {0, 4}, {0, 4}, {4}, 2);
    Run("nonempty_mean_control", {2, 4}, {2, 4}, {4}, 1);
    aclrtResetDevice(dev); aclFinalize(); _Exit(0);
}

影响

  • batch=0、seqlen=0、切片得到空张量时,loss 静默返回未初始化显存:无报错、无日志,数值随机(NaN、极大值,或看起来像正常值的脏数据),上层无法感知,训练被无声污染。
  • 显存复用场景下 out 直接暴露上一轮其它算子的残留数据;同族语义分裂(aclnnMseLoss 给 NaN/0,本接口给垃圾)。

建议

早退分支复用 loss/common/level2_base_loss.h 的 CheckFillScalarLoss:reduction=1 填 NAN、reduction=2 填 0.0f,reduction=0 保持不写;并补 UT(fp32/fp16/bf16 × reduction 1/2 × rank 1/2)断言 out 全 NaN / 全 0。

likedislike
liuhanwen8liuhanwen8
24 days ago Added label: bug-report
yuning_chen
yuning_chenCollaborators
24 days ago Comments:

初步分析

感谢反馈。

现象理解:
aclnnMseLossOut 空 tensor + mean/sum 时两段返回 0 但 out 未写,残留未初始化显存;同仓 aclnnMseLoss 会写出 NaN/0。

当前判断:初步判断,待复现确认。
偏空输入早退未填充出参,属静默脏数据。

涉及范围:
aclnn_mse_loss_out 空 Tensor 早退与 MseLossEmptyTensorCompute。

下一步:
对齐 aclnnMseLoss 空输入填充语义;补金丝雀用例。

likedislike
yuning_chenyuning_chenCollaborators
24 days ago Assign ji-songyuan as responsible person
yuning_chen
yuning_chenCollaborators
24 days ago Comments:

您好,问题已收到,当前 @ji-songyuan 正在处理中。

likedislike
佐助大王佐助大王Collaborators
11 days ago Associated pull request
佐助大王
佐助大王Collaborators
11 days ago Comments:

该接口没有竞品实现,只有aclnnMseLoss有对应的实现,所以不存在污染现象

likedislike
佐助大王佐助大王Collaborators
11 days ago changed custom state from 进行中 to 已完成
佐助大王佐助大王Collaborators
11 days ago Close issue
CANN-robotCANN-robotCollaborators
6 days ago Added label: resolved
chenjiayi_chenjiayi_Collaborators
3 days ago Assign chenjiayi_ as responsible person
chenjiayi_
chenjiayi_Collaborators
3 days ago Comments:

补充说明本问题的后续处理:

已修复并随关联 PR 合入。

处理结果:

  • !14080:fix(mse_loss): MseLossOut空tensor输入按reduction语义填充输出

验证:

  • 关联 PR(2026-11-16)已合入,变更可从 PR 记录追溯。
  • 本次以 PR 记录与 issue 讨论核对为主,未重新执行复现或真机验证。

说明:合入分支与版本以 PR 记录为准;如仍有问题,请重新提 issue 并附版本、shape 与日志。

如果还有后续需要,请重新提 issue。

likedislike
chenjiayi_chenjiayi_Collaborators
18 hours ago Removed responsible person chenjiayi_