Closed
[Bug] lightning_indexer sparse_count=0/-1024 被错误守卫静默接受 #4095
Agent-Bug-HunterAdd on  Aug 3Closed at  Aug 25
Agent-Bug-Hunter
Aug 3 Create

摘要

lightning_indexer 的 host tiling 对 sparse_count 的守卫写成
(!((>0 && <=2048) && %1024!=0) || >8192),等价于接受 0 以及 1024 的负倍数(如 -1024)。
错误文案与同仓 quant_lightning_indexer 均要求 sparse_count > 0。非法值在
aclnnLightningIndexerGetWorkspaceSize 被静默接受。已在 Ascend910B3 + CANN 9.1.0-beta.3
上通过源码 --pkg vendor 内核复现(exit 42)。



环境

项 值
芯片 Ascend910B3
CANN 9.1.0-beta.3
ops-transformer HEAD 4b3d39658e1d
编译命令 见下文「完整复现过程」
安装方式 cann-ops-transformer-cann_scan_repro_linux-*.run → $ASCEND_HOME_PATH/vendors/cann_scan_repro_transformer
Track / Pattern A / novel-sparse-count-guard
Finding ATT-LI-SPARSECNT-001
运行时证明 见下文「为何确认跑的是源码编译内核」

为什么认为这是问题:期望 vs 实际

输入(与复现用例一致)

query/key: [1,2,1,128] float16;weights: [1,2,1] float16
sparse_indices/values: [1,2,1,sparseCount];layout BSND;sparseMode=3

期望行为

用例 期望
legal sparseCount=1024 成功
sparseCount=0 拒绝(须 > 0)
sparseCount=-1024 拒绝

实际行为(aclnn / 源码编译安装路径)

用例 GetWorkspace Launch Sync 说明
legal 1024 SUCCESS SUCCESS SUCCESS 对照
zero 0 SUCCESS SUCCESS SUCCESS 静默接受
neg -1024 SUCCESS SUCCESS 507015 tiling 静默接受(后续 device 异常)

对比结论

守卫布尔式错误,与 quant_lightning_indexer 的 !(>0 && <=LIMIT) 正确形式不一致。



根因

  • attention/lightning_indexer/op_host/lightning_indexer_tiling.cpp L286–292:错误的 sparse_count 复合条件。
  • 对照:attention/quant_lightning_indexer/op_host/quant_lightning_indexer_tiling.cpp L318–322 使用正确形式。


为何确认跑的是源码编译内核(非 built-in)

  1. 环境:安装自定义包后执行 source $ASCEND_HOME_PATH/vendors/cann_scan_repro_transformer/bin/set_env.bash,ASCEND_CUSTOM_OPP_PATH 指向该 vendor。
  2. 二进制路径:复现链接/加载 vendor 下 op_api / op_impl,而非仅依赖 CANN built-in 同名算子(见 repro_manifest.json 中 built_from_source=true)。
  3. (推荐)排除 built-in:临时移走 built-in 对应算子内核目录后仍能 exit 42。

完整复现过程

占位符:$ASCEND_HOME_PATH / $OPS_TRANSFORMER / $REPRO_ROOT。

步骤 1:环境使能

source <CANN安装目录>/ascend-toolkit/set_env.sh

步骤 2:源码编译单算子包

cd "$OPS_TRANSFORMER"
bash build.sh --pkg --soc=ascend910b --ops=lightning_indexer \
  --vendor_name=cann_scan_repro -j8

成功标志:生成 $OPS_TRANSFORMER/build_out/cann-ops-transformer-cann_scan_repro_linux-*.run

步骤 3:安装并切换到自定义包

bash $OPS_TRANSFORMER/build_out/cann-ops-transformer-cann_scan_repro_linux-*.run \
  --quiet --install-path="$ASCEND_HOME_PATH"
source "$ASCEND_HOME_PATH/vendors/cann_scan_repro_transformer/bin/set_env.bash"
echo "ASCEND_CUSTOM_OPP_PATH=$ASCEND_CUSTOM_OPP_PATH"

步骤 4:放置复现源码与执行脚本

$REPRO_ROOT/
  test_aclnn_lightning_indexer_sparsecnt.cpp
  build_and_run_repro.sh

(本 draft 目录内已含上述文件,可直接作为 $REPRO_ROOT。)

步骤 5:编译并运行

cd "$REPRO_ROOT"
chmod +x build_and_run_repro.sh
bash build_and_run_repro.sh --vendor cann_scan_repro --src test_aclnn_lightning_indexer_sparsecnt.cpp
# 退出码 42 = BUG_REPRODUCED

成功复现标志:进程退出码 42。

实际运行摘录

[env] sourced $ASCEND_HOME_PATH/vendors/cann_scan_repro_transformer/bin/set_env.bash
[env] ASCEND_CUSTOM_OPP_PATH=$ASCEND_HOME_PATH/vendors/cann_scan_repro_transformer:
[build] g++ $CANN_SCAN/workspace/runs/20260803_math_transformer_cv/repro/lightning_indexer/test_aclnn_lightning_indexer_sparsecnt.cpp
[run] $CANN_SCAN/workspace/runs/20260803_math_transformer_cv/repro/lightning_indexer/test_aclnn_lightning_indexer_sparsecnt

===== legal sparseCount=1024 =====
GetWorkspaceSize ret=0 workspace=64004096
Launch ret=0
Sync ret=0
accepted=1

===== illegal_zero sparseCount=0 =====
GetWorkspaceSize ret=0 workspace=0
Launch ret=0
Sync ret=0
accepted=1

===== illegal_neg1024 sparseCount=-1024 =====
GetWorkspaceSize ret=0 workspace=64004096
Launch ret=0
Sync ret=507015
accepted=1

===== SUMMARY =====
legal accepted=1 (expect 1)
zero  accepted=1 (expect 0 if correct; 1 = BUG)
neg   accepted=1 (expect 0 if correct; 1 = BUG)
VERDICT: BUG_REPRODUCED (illegal sparse_count silently accepted at GetWorkspace)
[exit] 42  (42=BUG_REPRODUCED, 0=NO_REPRO, 1=harness error)

复现用例源码

文件名:test_aclnn_lightning_indexer_sparsecnt.cpp

/**
 * Track A — lightning_indexer sparse_count guard hole (ATT-LI-SPARSECNT-001).
 *
 * tiling.cpp accepts sparse_count when:
 *   !((>0 && <=2048) && %1024!=0) || >8192
 * rewritten: rejects only if (out-of-(0,2048] AND not 1024-aligned) OR >8192.
 * So sparse_count=0 and negative multiples of 1024 (e.g. -1024) pass the guard.
 * Error text / sibling quant_lightning_indexer require sparse_count > 0.
 *
 * Exit 42 = BUG_REPRODUCED; 0 = NO_REPRO; 1 = harness error.
 */
#include <cstdio>
#include <cstdint>
#include <climits>
#include <cstring>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_lightning_indexer.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

static int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t n = 1;
    for (auto i : shape) {
        if (i <= 0)
            return 0;
        n *= i;
    }
    return n;
}

template <typename T>
static int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                           aclDataType dataType, aclTensor** tensor)
{
    auto nelem = GetShapeSize(shape);
    auto size = (nelem > 0 ? nelem : 1) * sizeof(T); // allocate at least 1 elem for zero-dim last
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    if (!hostData.empty()) {
        ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), hostData.size() * sizeof(T), ACL_MEMCPY_HOST_TO_DEVICE);
        CHECK_RET(ret == ACL_SUCCESS, return ret);
    }
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = static_cast<int64_t>(shape.size()) - 2; i >= 0; i--) {
        int64_t next = shape[i + 1] > 0 ? shape[i + 1] : 1;
        strides[i] = next * strides[i + 1];
    }
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

struct CaseResult {
    int wsRet = -1;
    int launchRet = -1;
    int syncRet = -1;
    bool accepted = false;
};

// Minimal BSND shapes matching official example; last dim of outs = sparseCount.
static CaseResult RunOne(const char* name, int64_t sparseCount, aclrtStream stream)
{
    CaseResult cr;
    printf("\n===== %s sparseCount=%ld =====\n", name, (long)sparseCount);

    // last dim must equal sparseCount for shape check (tiling L905)
    const int64_t sc = sparseCount;
    std::vector<int64_t> queryShape = {1, 2, 1, 128};
    std::vector<int64_t> keyShape = {1, 2, 1, 128};
    std::vector<int64_t> weightsShape = {1, 2, 1};
    std::vector<int64_t> sparseIndicesShape = {1, 2, 1, sc};
    std::vector<int64_t> sparseValuesShape = {1, 2, 1, sc};

    int64_t qn = GetShapeSize(queryShape);
    int64_t kn = GetShapeSize(keyShape);
    int64_t wn = GetShapeSize(weightsShape);
    int64_t sn = GetShapeSize(sparseIndicesShape);

    std::vector<float> queryHost(qn > 0 ? qn : 1, 1.f);
    std::vector<float> keyHost(kn > 0 ? kn : 1, 1.f);
    std::vector<float> weightsHost(wn > 0 ? wn : 1, 1.f);
    std::vector<int32_t> sparseIndicesHost(sn > 0 ? sn : 1, 0);
    std::vector<float> sparseValuesHost(sn > 0 ? sn : 1, 0.f);

    void *qDev = nullptr, *kDev = nullptr, *wDev = nullptr, *siDev = nullptr, *svDev = nullptr;
    aclTensor *q = nullptr, *k = nullptr, *w = nullptr, *si = nullptr, *sv = nullptr;

    if (CreateAclTensor(queryHost, queryShape, &qDev, ACL_FLOAT16, &q) != 0)
        return cr;
    if (CreateAclTensor(keyHost, keyShape, &kDev, ACL_FLOAT16, &k) != 0)
        return cr;
    if (CreateAclTensor(weightsHost, weightsShape, &wDev, ACL_FLOAT16, &w) != 0)
        return cr;
    if (CreateAclTensor(sparseIndicesHost, sparseIndicesShape, &siDev, ACL_INT32, &si) != 0)
        return cr;
    if (CreateAclTensor(sparseValuesHost, sparseValuesShape, &svDev, ACL_FLOAT16, &sv) != 0)
        return cr;

    char layoutQuery[] = "BSND";
    char layoutKey[] = "BSND";
    int64_t sparseMode = 3;
    int64_t preTokens = INT64_MAX;
    int64_t nextTokens = INT64_MAX;
    bool returnValue = true;

    uint64_t workspaceSize = 0;
    aclOpExecutor* executor = nullptr;
    cr.wsRet = aclnnLightningIndexerGetWorkspaceSize(q, k, w, nullptr, nullptr, nullptr, layoutQuery, layoutKey,
                                                     sparseCount, sparseMode, preTokens, nextTokens, returnValue, si,
                                                     sv, &workspaceSize, &executor);
    printf("GetWorkspaceSize ret=%d workspace=%llu\n", cr.wsRet, (unsigned long long)workspaceSize);

    void* workspaceAddr = nullptr;
    if (cr.wsRet == ACL_SUCCESS) {
        if (workspaceSize > 0) {
            if (aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST) != ACL_SUCCESS) {
                printf("workspace malloc failed\n");
                return cr;
            }
        }
        cr.launchRet = aclnnLightningIndexer(workspaceAddr, workspaceSize, executor, stream);
        printf("Launch ret=%d\n", cr.launchRet);
        if (cr.launchRet == ACL_SUCCESS) {
            cr.syncRet = aclrtSynchronizeStream(stream);
            printf("Sync ret=%d\n", cr.syncRet);
        }
    }

    // Accepted = GetWorkspace succeeded (tiling did not reject illegal sparse_count)
    cr.accepted = (cr.wsRet == ACL_SUCCESS);
    printf("accepted=%d\n", (int)cr.accepted);

    if (workspaceAddr)
        aclrtFree(workspaceAddr);
    aclDestroyTensor(q);
    aclDestroyTensor(k);
    aclDestroyTensor(w);
    aclDestroyTensor(si);
    aclDestroyTensor(sv);
    aclrtFree(qDev);
    aclrtFree(kDev);
    aclrtFree(wDev);
    aclrtFree(siDev);
    aclrtFree(svDev);
    return cr;
}

int main()
{
    aclrtStream stream = nullptr;
    CHECK_RET(aclInit(nullptr) == ACL_SUCCESS, return 1);
    CHECK_RET(aclrtSetDevice(0) == ACL_SUCCESS, return 1);
    CHECK_RET(aclrtCreateStream(&stream) == ACL_SUCCESS, return 1);

    // Control: legal sparseCount=1024 should be accepted
    CaseResult legal = RunOne("legal", 1024, stream);
    // Bug cases: 0 and -1024 must be rejected per docs / quant sibling, but buggy guard accepts
    CaseResult zero = RunOne("illegal_zero", 0, stream);
    CaseResult neg = RunOne("illegal_neg1024", -1024, stream);

    bool legalOk = legal.accepted;
    bool zeroBug = zero.accepted; // should be false if fixed
    bool negBug = neg.accepted;

    printf("\n===== SUMMARY =====\n");
    printf("legal accepted=%d (expect 1)\n", (int)legalOk);
    printf("zero  accepted=%d (expect 0 if correct; 1 = BUG)\n", (int)zeroBug);
    printf("neg   accepted=%d (expect 0 if correct; 1 = BUG)\n", (int)negBug);

    int rc = 1;
    if (!legalOk) {
        printf("VERDICT: harness/legal path failed\n");
        rc = 1;
    } else if (zeroBug || negBug) {
        printf("VERDICT: BUG_REPRODUCED (illegal sparse_count silently accepted at GetWorkspace)\n");
        rc = 42;
    } else {
        printf("VERDICT: NO_REPRO (illegal sparse_count rejected)\n");
        rc = 0;
    }

    aclrtDestroyStream(stream);
    aclrtResetDevice(0);
    aclFinalize();
    return rc;
}

执行脚本

文件名:build_and_run_repro.sh

#!/usr/bin/env bash
set -euo pipefail
VENDOR=""
SRC_NAME=""
while [[ $# -gt 0 ]]; do
  case "$1" in
    --vendor) VENDOR="$2"; shift 2 ;;
    --src) SRC_NAME="$2"; shift 2 ;;
    *) echo "unknown arg: $1" >&2; exit 2 ;;
  esac
done
: "${ASCEND_HOME_PATH:?请先 source CANN set_env.sh}"
ROOT="$(cd "$(dirname "$0")" && pwd)"
if [[ -z "$SRC_NAME" ]]; then
  mapfile -t _srcs < <(ls "$ROOT"/test_aclnn_*.cpp 2>/dev/null || true)
  if [[ ${#_srcs[@]} -ne 1 ]]; then
    echo "请用 --src 指定测试文件(当前匹配到 ${#_srcs[@]} 个)" >&2
    exit 2
  fi
  SRC="${_srcs[0]}"
else
  SRC="$ROOT/$SRC_NAME"
fi
BIN="${SRC%.cpp}"
VENDOR_DIR=""
if [[ -n "$VENDOR" ]]; then
  for cand in \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}_transformer" \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}" \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}_nn"
  do
    if [[ -f "$cand/bin/set_env.bash" ]]; then
      set +u; source "$cand/bin/set_env.bash"; set -u
      VENDOR_DIR="$cand"
      echo "[env] sourced $cand/bin/set_env.bash"
      echo "[env] ASCEND_CUSTOM_OPP_PATH=${ASCEND_CUSTOM_OPP_PATH:-}"
      break
    fi
  done
fi
INC="${ASCEND_HOME_PATH}/aarch64-linux/include"
LIB="${ASCEND_HOME_PATH}/aarch64-linux/lib64"
[[ -d "$INC" ]] || INC="${ASCEND_HOME_PATH}/include"
[[ -d "$LIB" ]] || LIB="${ASCEND_HOME_PATH}/lib64"
VENDOR_INC=""; VENDOR_LIB=""
if [[ -n "$VENDOR_DIR" ]]; then
  VENDOR_INC="$VENDOR_DIR/op_api/include"
  VENDOR_LIB="$VENDOR_DIR/op_api/lib"
fi
echo "[build] g++ $SRC"
g++ -std=c++17 -O2 "$SRC" \
  ${VENDOR_INC:+-I"$VENDOR_INC"} -I"$INC" -I"$INC/aclnnop" \
  -L"$LIB" ${VENDOR_LIB:+-L"$VENDOR_LIB"} \
  -Wl,-rpath,"$LIB"${VENDOR_LIB:+ -Wl,-rpath,"$VENDOR_LIB"} \
  -lascendcl -lnnopbase ${VENDOR_LIB:+-lcust_opapi} -lopapi \
  -o "$BIN"
echo "[run] $BIN"
set +e; "$BIN"; RC=$?; set -e
echo "[exit] $RC  (42=BUG_REPRODUCED, 0=NO_REPRO, 1=harness error)"
exit "$RC"

修复建议

在 host/tiling/kernel 入口拒绝非法索引或控制量,返回明确参数错误;勿静默 OOB。


人工复核说明

Agent 复现结果供参考;语义边界(是否允许 wrap、文档是否声明调用方契约)以人工结论与产品规范为准。

likedislike
weihao18Collaborators
Aug 3 Comments:

/assign @east_yang

likedislike
CANN-robotCANN-robotCollaborators
Aug 3 Assign songjionghui as responsible person
CANN-robotCANN-robotCollaborators
Aug 19 Assign east_yang as responsible person, remove original responsible person: songjionghui
guijianwei
guijianweiCollaborators
Aug 19 Comments:

校验可以被负数绕过问题,近期修复上库,感谢反馈。

likedislike
CANN-robotCANN-robotCollaborators
Aug 25 Close issue
CANN-robotCANN-robotCollaborators
Aug 25 Added label: resolved