Closed
[Bug-Report|缺陷反馈]: aclnnForeachCopy 在 x/out 为同一存储的前向重叠视图时静默算错:两段返回 0,结果偏离 memmove 语义且逐次不定 #8261
majinglanAdd on  16 days agoClosed at  8 days ago
majinglan
majinglan
16 days ago Create

现象

aclnnForeachCopy 的 x 与 out 指向同一块 device 内存的重叠区间(out 起址 = x 起址 + 5 个 fp32),两段式调用均返回 0(成功),但输出与 memmove 参考语义不符,且同一构造多次执行结果不同。

构造(单元素 TensorList,37 元素 fp32/ND buffer,x/out 各自均为连续 [32] 张量,仅数据区间重叠):

  • x[0] = 视图 buf[0:32](dataPtr = buf)
  • out[0] = 视图 buf[5:37](dataPtr = buf + 5×4 字节)
  • 参考语义(等价 out_view[:] = x_view[:] 的 memmove 结果):buf[5:37] = base[0:32],其余元素保持原值

真机重复 10 次(每次重新灌入随机数据):

iter 0: rc1=0 rc2=0 与 memmove 参考位级一致
iter 1: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 5 个, 首个位置 13(重叠区=5..36)
iter 2: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 10 个, 首个位置 13(重叠区=5..36)
iter 3: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 10 个, 首个位置 13(重叠区=5..36)
iter 4: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 5 个, 首个位置 13(重叠区=5..36)
iter 5: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 5 个, 首个位置 13(重叠区=5..36)
iter 6: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 10 个, 首个位置 13(重叠区=5..36)
iter 7: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 5 个, 首个位置 13(重叠区=5..36)
iter 8: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 5 个, 首个位置 13(重叠区=5..36)
iter 9: rc1=0 rc2=0 与 memmove 参考不一致, 差异元素 5 个, 首个位置 13(重叠区=5..36)
VERDICT: BUG——9/10 次结果与 memmove 参考不一致(前向重叠未按 memmove 语义处理)
  • 本轮 10 次中 9 次不一致(此前一轮为 10/10 不一致,两轮合计 19/20);
  • 差异元素数在 5 与 10 之间波动,另一批运行首个差异位置在 13 或 29 之间波动——同一构造逐次结果不同;
  • 差异位置聚类于 buf[13:18)、buf[21:26)、buf[29:34),恰为 8 元素(32B)搬运块边界与 5 元素前向重叠的交叠窗口,与 kernel 升序分块拷贝的自覆盖特征一致(见根因)。

根因(源码定位,master 46ff6fdb)

  1. host 层无重叠校验:foreach/foreach_copy/op_api/aclnn_foreach_copy.cpp 的 CheckParams()(:128-151)仅校验空指针、逐成员非空、dtype 映射(含列表内 dtype 一致性)、rank 上限与 format,未对 x[i]/out[i] 的 device 地址区间做重叠检测;foreach/foreach_copy/docs/aclnnForeachCopy.md(全文 385 行)亦未声明「x 与 out 不得别名/地址重叠」的约束。
  2. kernel 层升序块拷贝、无重叠处理:foreach/foreach_copy/op_kernel/foreach_copy.h 的 ForeachCopyND::SingleTensorProcess() 按 CopyIn(i) → ComputeAndCopyOut(i) 升序逐块搬运(BUFFER_NUM = 1,块大小 maxDataCount = inputsTensorUbSize / sizeof(T))。前向重叠(out 起址 > x 起址)时,第 i+1 块的源数据区间已被第 i 块的写入覆盖,读到污染数据;块间 MTE3→MTE2 事件同步并未保证 GM 写对后续 GM 读的可见性,导致污染窗口逐次不同(非确定性错误)。

框架层影响(torch_npu)

torch._foreach_copy_ 的 NPU 实现直接路由到本接口(libtorch_npu.so 内含 aclnnForeachCopy / aclnnForeachCopyGetWorkspaceSize 符号)。同一重叠构造(buf[5:37] ← buf[0:32] 视图拷贝,torch 2.14.0+cpu / torch_npu 2.14.0):

  • CPU:直接抛 RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.(显式拒绝重叠输入);
  • NPU(910B3):5/5 次静默算错,无任何异常抛出:
no_overlap_ok: True
run 0 diff_pos [13, 14, 15, 16, 17, 29, 30, 31, 32, 33] count 10
run 1 diff_pos [13, 14, 15, 16, 17, 21, 22, 23, 24, 25] count 10
run 2 diff_pos [13, 14, 15, 16, 17, 29, 30, 31, 32, 33] count 10
run 3 diff_pos [13, 14, 15, 16, 17] count 5
run 4 diff_pos [13, 14, 15, 16, 17, 21, 22, 23, 24, 25, 29, 30] count 15
npu_overlap: corrupt=5 rejected=0 ok=0 of 5
cpu_overlap_rejected: unsupported operation: some elements of the input tensor ...

非重叠路径 no_overlap_ok: True,排除 kernel 基本功能问题——地址重叠是唯一触发条件。torch 层最小验证片段:

import torch, torch_npu
buf = torch.arange(64, dtype=torch.float32, device="npu")
torch._foreach_copy_([buf[5:37]], [buf[0:32]])   # CPU 上同构造直接抛 RuntimeError,NPU 上静默算错

期望

「返回成功但静默算错」不可接受,两种处置任选其一:

  • 方案 A(与 torch CPU 语义一致,推荐):aclnnForeachCopyGetWorkspaceSize 增加对 x[i]/out[i] device 地址区间(dataPtr + offset,长度 = numel × sizeof(dtype))的重叠检测,检测到重叠时返回 161002,并在《aclnnForeachCopy》资料错误码表补充该子项与约束声明;
  • 方案 B(memmove 语义):kernel 侧按重叠安全顺序搬运(降序分块拷贝,或整段 UB 暂存后再写出)。

环境

  • 硬件:Atlas A2 训练系列(910B3),单卡,Health=OK
  • CANN:9.2.0(安装包二进制实测),npu-smi 26.1.1
  • 仓库:cann/ops-nn master 46ff6fdb(2026-11-06 fetch;三处源码定位均为当前 master 状态)
  • 复现脚本依赖:仅 ctypes + numpy(不依赖 torch 与任何第三方仓)

复现步骤

  1. 将下方脚本保存为 repro_fc_alias.py;
  2. 执行 python3 repro_fc_alias.py <npu设备号>(缺省 3,任意空闲设备均可);
  3. 观察输出:按 memmove 语义应 10 次全部「与 memmove 参考位级一致」,实际多数次数不一致、差异元素数 5/10 波动、首个差异位置 13 或 29。

复现脚本

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# aclnnForeachCopy 别名前向重叠探针(同一 device buffer 上的重叠视图)
# 构造: x = buf[0:32], out = buf[5:37] (fp32, ND), 重复 10 次, 每次灌入随机数据
# 参考语义 (memmove): expected[5:37] = base[0:32], 其余保持 base 原值
# 判定: PASS = 10 次全部位级一致; BUG = 任一次不一致
import ctypes
import sys

import numpy as np

DEV = int(sys.argv[1]) if len(sys.argv) > 1 else 3
ACL_FLOAT = 0
ACL_FORMAT_ND = 2
H2D, D2H = 1, 2
HUGE_FIRST = 0
N = 32          # 视图元素数
OVERLAP = 5     # 前向重叠元素数
TOTAL = N + OVERLAP
REPEAT = 10


def mk_tensor(nn, data_ptr, dims, dtype=ACL_FLOAT):
    n = len(dims)
    arr_dims = (ctypes.c_int64 * max(n, 1))(*dims)
    strides, acc = [], 1
    for d in reversed(dims):
        strides.insert(0, acc)
        acc *= d
    arr_strides = (ctypes.c_int64 * max(n, 1))(*strides)
    arr_storage = (ctypes.c_int64 * max(n, 1))(*dims)
    fn = nn.aclCreateTensor
    fn.argtypes = [ctypes.POINTER(ctypes.c_int64), ctypes.c_uint64, ctypes.c_int32,
                   ctypes.POINTER(ctypes.c_int64), ctypes.c_int64, ctypes.c_int32,
                   ctypes.POINTER(ctypes.c_int64), ctypes.c_uint64, ctypes.c_void_p]
    fn.restype = ctypes.c_void_p
    h = fn(arr_dims, ctypes.c_uint64(n), ctypes.c_int32(dtype), arr_strides,
           ctypes.c_int64(0), ctypes.c_int32(ACL_FORMAT_ND), arr_storage,
           ctypes.c_uint64(n), ctypes.c_void_p(data_ptr))
    if not h:
        sys.exit("aclCreateTensor 返回空")
    return int(h or 0)


def mk_list(nn, handles):
    fn = nn.aclCreateTensorList
    fn.argtypes = [ctypes.POINTER(ctypes.c_void_p), ctypes.c_uint64]
    fn.restype = ctypes.c_void_p
    arr = (ctypes.c_void_p * max(len(handles), 1))(*[ctypes.c_void_p(h) for h in handles])
    h = fn(arr, ctypes.c_uint64(len(handles)))
    if not h:
        sys.exit("aclCreateTensorList 返回空")
    return int(h or 0)


def main():
    la = ctypes.CDLL("libascendcl.so")
    nn = ctypes.CDLL("libnnopbase.so")
    op = ctypes.CDLL("libopapi.so")
    la.aclInit.argtypes = [ctypes.c_char_p]
    la.aclInit.restype = ctypes.c_int32
    rc = la.aclInit(None)
    if rc != 0:
        print("aclInit rc=%d(已初始化场景返回非 0, 继续观察)" % rc)
    la.aclrtSetDevice.argtypes = [ctypes.c_int32]
    la.aclrtSetDevice.restype = ctypes.c_int32
    rc = la.aclrtSetDevice(DEV)
    if rc != 0:
        sys.exit("aclrtSetDevice(%d) rc=%d" % (DEV, rc))
    stream = ctypes.c_void_p()
    la.aclrtCreateStream.argtypes = [ctypes.POINTER(ctypes.c_void_p)]
    la.aclrtCreateStream.restype = ctypes.c_int32
    rc = la.aclrtCreateStream(ctypes.byref(stream))
    if rc != 0:
        sys.exit("aclrtCreateStream rc=%d" % rc)

    p = ctypes.c_void_p()
    rc = la.aclrtMalloc(ctypes.byref(p), ctypes.c_size_t(TOTAL * 4),
                        ctypes.c_int32(HUGE_FIRST))
    if rc != 0:
        sys.exit("aclrtMalloc rc=%d" % rc)
    buf = int(p.value or 0)

    la.aclrtMemcpy.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_void_p,
                               ctypes.c_size_t, ctypes.c_int32]
    la.aclrtMemcpy.restype = ctypes.c_int32
    la.aclrtSynchronizeStream.argtypes = [ctypes.c_void_p]
    la.aclrtSynchronizeStream.restype = ctypes.c_int32

    xt = mk_tensor(nn, buf, [N])                       # x   = buf[0:32]
    ot = mk_tensor(nn, buf + OVERLAP * 4, [N])         # out = buf[5:37]
    xl = mk_list(nn, [xt])
    ol = mk_list(nn, [ot])

    gws = op.aclnnForeachCopyGetWorkspaceSize
    gws.argtypes = [ctypes.c_void_p, ctypes.c_void_p,
                    ctypes.POINTER(ctypes.c_uint64), ctypes.POINTER(ctypes.c_void_p)]
    gws.restype = ctypes.c_int32
    run = op.aclnnForeachCopy
    run.argtypes = [ctypes.c_void_p, ctypes.c_uint64, ctypes.c_void_p, ctypes.c_void_p]
    run.restype = ctypes.c_int32

    n_bad = 0
    for it in range(REPEAT):
        rng = np.random.default_rng(1000 + it)
        base = (rng.uniform(-2, 2, TOTAL)).astype(np.float32)
        # memmove 参考:out 视图写到 buf[5:37],取 base[0:32]
        expected = base.copy()
        expected[OVERLAP:TOTAL] = base[0:N].copy()
        la.aclrtMemcpy(buf, TOTAL * 4, base.ctypes.data, TOTAL * 4, H2D)

        ws_size = ctypes.c_uint64(0)
        executor = ctypes.c_void_p()
        rc1 = gws(ctypes.c_void_p(xl), ctypes.c_void_p(ol),
                  ctypes.byref(ws_size), ctypes.byref(executor))
        if rc1 != 0:
            print("iter %d: GetWorkspaceSize rc=%d" % (it, rc1))
            n_bad += 1
            continue
        ws = ctypes.c_void_p()
        if ws_size.value:
            rc = la.aclrtMalloc(ctypes.byref(ws), ctypes.c_size_t(ws_size.value),
                                ctypes.c_int32(HUGE_FIRST))
            if rc != 0:
                sys.exit("workspace malloc rc=%d" % rc)
        rc2 = run(ws, ctypes.c_uint64(ws_size.value), executor, stream)
        la.aclrtSynchronizeStream(stream)
        hres = np.zeros(TOTAL, dtype=np.float32)
        la.aclrtMemcpy(hres.ctypes.data, TOTAL * 4, buf, TOTAL * 4, D2H)
        ok = bool(np.array_equal(hres.view(np.uint32), expected.view(np.uint32)))
        if not ok:
            n_bad += 1
            diff = np.nonzero(hres.view(np.uint32) != expected.view(np.uint32))[0]
            print("iter %d: rc1=%d rc2=%d 与 memmove 参考不一致, 差异元素 %d 个, "
                  "首个位置 %d(重叠区=%d..%d)"
                  % (it, rc1, rc2, len(diff), int(diff[0]), OVERLAP, TOTAL - 1))
        else:
            print("iter %d: rc1=%d rc2=%d 与 memmove 参考位级一致" % (it, rc1, rc2))
    if n_bad == 0:
        print("VERDICT: PASS(%d 次别名前向重叠均与 memmove 语义一致)" % REPEAT)
    else:
        print("VERDICT: BUG——%d/%d 次结果与 memmove 参考不一致(前向重叠未按 "
              "memmove 语义处理)" % (n_bad, REPEAT))


if __name__ == "__main__":
    main()
likedislike
majinglanmajinglan
16 days ago Change title to "[Bug-Report|缺陷反馈]: aclnnForeachCopy 在 x/out 为同一存储的前向重叠视图时静默算错:两段返回 0,结果偏离 memmove 语义且逐次不定", originally titled "[Bug-Report|缂洪櫡鍙嶉]: aclnnForeachCopy 鍦?x/out 涓哄悓涓€瀛樺偍鐨勫墠鍚戦噸鍙犺鍥炬椂闈欓粯绠楅敊锛氫袱娈佃繑鍥?0锛岀粨鏋滃亸绂?memmove 璇箟涓旈€愭涓嶅畾".
majinglan
majinglan
16 days ago Comments:

补充实测:x/out 维度数超过 8 时返回 161002,与文档错误码表(561002)不一致

现象

x(或 out)维度数为 9(如 fp32[1x1x1x1x1x1x1x2x1024],out 同步为同一 shape)时,aclnnForeachCopyGetWorkspaceSize 第一段即返回 161002(ACLNN_ERR_PARAM_INVALID),而《aclnnForeachCopy》错误码表把「x/out 的维度数超过 8」列在 561002(ACLNN_ERR_INNER_TILING_ERROR)。普通直调与确定性复跑两种方式均复现,第一段即返回。

根因(master 46ff6fdb)

foreach/foreach_copy/op_api/aclnn_foreach_copy.cpp:148 的 CHECK_RET(CheckShape(x, out), ACLNN_ERR_PARAM_INVALID):CheckShape()(:114-124)对列表内每个成员以 OP_CHECK_MAX_DIM(:122-123,上限 MAX_SUPPORT_DIMS_NUMS=8,x 与 out 均查)在参数校验阶段直接判失败 → 161002,不会进入文档所述的 tiling 错误路径(561002)。

框架层影响

调用方按文档错误码表写错误处理分支时,「维度超过 8」场景实际返回 161002,会落到 default 分支(漏判参数错误)。

复现

在 issue 正文 repro_fc_alias.py 框架(ctypes + numpy,不依赖 torch)下,把 x/out 换成 9 维同构 fp32 张量即可:

dims = [1, 1, 1, 1, 1, 1, 1, 2, 1024]        # rank = 9(超过文档上界 8)
xt = mk_tensor(nn, buf,  dims)               # x   = fp32 9 维
ot = mk_tensor(nn, buf2, dims)               # out = fp32 9 维(独立 device 内存)
xl, ol = mk_list(nn, [xt]), mk_list(nn, [ot])
rc1 = gws(xl, ol, ctypes.byref(ws_size), ctypes.byref(executor))
print("GetWorkspaceSize rc =", rc1)           # 实测 161002(文档口径 561002)

实测输出:GetWorkspaceSize rc = 161002。

同轮其余阴性结论(供参考)

  • 内存线:mssanitizer memcheck 24 个用例全部通过,无越界读写 / 未初始化访问报告;
  • 错误场景线中另有 2 条「已接受未复核」变异经复核为合法行为(rank 1→2 首维插 1 保持元素数不变、out fp32→fp16 属合法转换对),两段返回 0 为正确行为,不构成缺陷。

期望:修正文档错误码表(「x/out 的维度数超过 8」的返回码应为 161002),或将该校验移入 tiling 路径以与文档 561002 对齐。

likedislike
yuning_chenyuning_chenCollaborators
16 days ago Assign tramp-ll as responsible person
yuning_chen
yuning_chenCollaborators
16 days ago Comments:

初步分析

感谢反馈,我们已开始核对这个问题。

现象理解:[Bug-Report|缺陷反馈]: aclnnForeachCopy 在 x/out 为同一存储的前向重叠视图时静默算错:两段返回 0,结果偏离 memmove 语义且逐次不定。

当前判断:初步判断,待复现确认,还不能当成已确认根因。

涉及范围:根据标题与描述,先对照对应算子的实现/资料/校验路径排查。

下一步:我们会按描述构造用例复现,并对照代码与资料给出修复或文档修正方向。

likedislike
yuning_chen
yuning_chenCollaborators
16 days ago Comments:

您好,该问题已收到,当前问题 @tramp-ll 正在处理中

likedislike
yuning_chenyuning_chenCollaborators
11 days ago Assign east_yang as responsible person, remove original responsible person: tramp-ll
east_yangeast_yangCollaborators
11 days ago Assign tramp-ll as responsible person
east_yangeast_yangCollaborators
11 days ago Removed responsible person east_yang
tramp-ll
tramp-llCollaborators
10 days ago Comments:

aclnnForeachCopy 不支持在 x/out 为同一存储的前向重叠视图场景,精度无法保证正确,已添加文档说明

likedislike
CANN-robotCANN-robotCollaborators
8 days ago Close issue
CANN-robotCANN-robotCollaborators
8 days ago changed custom state from 进行中 to 已完成
CANN-robotCANN-robotCollaborators
8 days ago Added label: resolved