[Bug] apply_top_k_top_p_with_sorted / aclnnApplyTopKTopP 未钳制 k∈[1,vocab]-ops-nn-AtomGit

已关闭

[Bug] apply_top_k_top_p_with_sorted / aclnnApplyTopKTopP 未钳制 k∈[1,vocab] #4759

Agent-Bug-Hunter创建于 8月13日关闭于 8月17日

8月13日创建

摘要

aclnnApplyTopKTopP（apply_top_k_top_p_with_sorted）文档要求 k 值域为 [1, logits.size(1)]，但 aclnn Check*/tiling/kernel 均未钳制。k=V+1 等越界控制标量仍 GetWorkspaceSize SUCCESS 并完成计算；kernel 用 sortedValue[vocabSize - k] 取阈值，越界时行为未定义。Ascend910B3 真机 exit 42。本地 draft，未远程提交。

说明：历史预期为 Sync 507015；本环境更常表现为静默接受并产生异常 finite 计数（如 V=64、k=65 时 finite_out=65）。

环境

项	值
芯片	Ascend910B3
CANN	9.1.0-beta.3
ops-nn HEAD	`e16ef626bb3a`
编译命令	见下文「完整复现过程」
安装方式	`cann-ops-nn-cann_scan_repro_linux-*.run` → `$ASCEND_HOME_PATH/vendors/cann_scan_repro_nn`
Track / Pattern	A / novel-k-unclamped
Finding	IDX-ATKTP-001
运行时证明	见下文「为何确认跑的是源码编译内核」

为什么认为这是问题：期望 vs 实际

输入（与复现用例一致）

`aclnnApplyTopKTopP`（`apply_top_k_top_p_with_sorted`）文档要求 `k` 值域为 `[1, logits.size(1)]`，但 aclnn Check*/tiling/kernel **均未钳制**。`k=V+1` 等越界控制标量仍 GetWorkspaceSize SUCCESS 并完成计算；kernel 用 `sortedValue[vocabSize - k]` 取阈值，越界时行为未定义。Ascend910B3 真机 exit 42。本地 draft，未远程提交。

期望行为

用例	期望
k∈[1,V]	SUCCESS
k=V+1 / V*2 / 1e6	拒绝（PARAM_INVALID）

实际行为（aclnn / 源码编译安装路径）

用例	API 返回 / 实际	关键输出	说明
k∈[1,V]	SUCCESS（对照 OK）	见运行摘录	exit 42
k=V+1 / V*2 / 1e6	ws SUCCESS，可 launch/sync	见运行摘录	exit 42

对比结论

用例	期望	实际
k∈[1,V]	SUCCESS	SUCCESS（对照 OK）
k=V+1 / V*2 / 1e6	拒绝（PARAM_INVALID）	ws SUCCESS，可 launch/sync

根因

op_kernel/apply_top_k_top_p_with_sorted.h：kthValue = mGmSortedValue_[baseGmIdx_ + vocabSize_ - kValue]，无 clamp。文档明确值域 [1, logits.size(1)]。

Anti-pattern check (AP-001): 路径始终可达。非数据索引 P-011，属 control-scalar 缺界。

为何确认跑的是源码编译内核（非 built-in）

环境：安装自定义包后执行 source $ASCEND_HOME_PATH/vendors/cann_scan_repro_nn/bin/set_env.bash，ASCEND_CUSTOM_OPP_PATH 指向该 vendor。
二进制路径：复现链接/加载 vendor 下 op_api / op_impl，而非仅依赖 CANN built-in 同名算子（见 repro_manifest.json 中 built_from_source=true）。
（推荐）排除 built-in：临时移走 built-in 对应算子内核目录后仍能 exit 42。

完整复现过程

占位符：$ASCEND_HOME_PATH / $OPS_NN / $REPRO_ROOT。

步骤 1：环境使能

source <CANN安装目录>/ascend-toolkit/set_env.sh

步骤 2：源码编译单算子包

cd "$OPS_NN"
bash build.sh --pkg --soc=ascend910b --ops=apply_top_k_top_p_with_sorted \
  --vendor_name=cann_scan_repro --no_force -j8

成功标志：生成 $OPS_NN/build_out/cann-ops-nn-cann_scan_repro_linux-*.run

步骤 3：安装并切换到自定义包

bash $OPS_NN/build_out/cann-ops-nn-cann_scan_repro_linux-*.run \
  --quiet --install-path="$ASCEND_HOME_PATH"
source "$ASCEND_HOME_PATH/vendors/cann_scan_repro_nn/bin/set_env.bash"
echo "ASCEND_CUSTOM_OPP_PATH=$ASCEND_CUSTOM_OPP_PATH"

步骤 4：放置复现源码与执行脚本

$REPRO_ROOT/
  test_aclnn_apply_top_k_top_p_k_oob.cpp
  build_and_run_repro.sh

（本 draft 目录内已含上述文件，可直接作为 $REPRO_ROOT。）

步骤 5：编译并运行

cd "$REPRO_ROOT"
chmod +x build_and_run_repro.sh
bash build_and_run_repro.sh --vendor cann_scan_repro --src test_aclnn_apply_top_k_top_p_k_oob.cpp
# 退出码 42 = BUG_REPRODUCED

成功复现标志：进程退出码 42。

实际运行摘录

[build] g++ $WORKSPACE/ops/cann_scan/workspace/runs/20260731_ops_nn_all/repro/apply_top_k_top_p_with_sorted/test_aclnn_apply_top_k_top_p_k_oob.cpp
[run] $WORKSPACE/ops/cann_scan/workspace/runs/20260731_ops_nn_all/repro/apply_top_k_top_p_with_sorted/test_aclnn_apply_top_k_top_p_k_oob

===== CONTROL k=8 k=8 =====
  wsRet=0 ws=16780800
  launch=0 sync=0
  finite_out=16 / 128 first8: -inf -inf -inf -inf -inf -inf -inf -inf

===== BOUNDARY k=V k=64 =====
  wsRet=0 ws=16780800
  launch=0 sync=0
  finite_out=128 / 128 first8: 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000

===== TRIGGER k=V+1 k=65 =====
  wsRet=0 ws=16780800
  launch=0 sync=0
  finite_out=65 / 128 first8: 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000

===== TRIGGER k=V*2 k=128 =====
  wsRet=0 ws=16780800
  launch=0 sync=0
  finite_out=128 / 128 first8: 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000

===== TRIGGER k=1000000 k=1000000 =====
  wsRet=0 ws=16780800
  launch=0 sync=0
  finite_out=128 / 128 first8: 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000

VERDICT: BUG_REPRODUCED (exit 42) — k>vocab accepted (ws SUCCESS) without reject
  bad1(V+1) ws=0 ok=1 sync=0; bad2 ws=0; bad3 ws=0
[exit] 42  (42=BUG_REPRODUCED, 0=NO_REPRO, 1=harness error)

复现用例源码

文件名：test_aclnn_apply_top_k_top_p_k_oob.cpp

/**
 * IDX-ATKTP-001: unclamped k. Doc requires 1<=k<=vocab.
 * CONTROL k in range; TRIGGER k=V+1 and k=huge.
 * Exit 42 if: device fault OR host accepts OOB k (ws SUCCESS) while producing
 * wrong-vs-control filter / silent SUCCESS without reject.
 */
#include <cstdio>
#include <cstdint>
#include <vector>
#include <cmath>
#include "acl/acl.h"
#include "aclnnop/aclnn_apply_top_k_top_p.h"

#define LOG_PRINT(message, ...) do { printf(message, ##__VA_ARGS__); } while (0)

static int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t n = 1;
    for (auto i : shape) n *= i;
    return n;
}

template <typename T>
static int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                           aclDataType dataType, aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    if (ret) return ret;
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    if (ret) return ret;
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = (int64_t)shape.size() - 2; i >= 0; i--) strides[i] = shape[i + 1] * strides[i + 1];
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, ACL_FORMAT_ND, shape.data(),
                              shape.size(), *deviceAddr);
    return 0;
}

struct CaseResult {
    int wsRet = -1, launchRet = -1, syncRet = -1;
    std::vector<float> out;
    bool ok = false;
};

static CaseResult Run(const char* label, int32_t kVal, int64_t B, int64_t V, const std::vector<float>& logitsHost,
                      aclrtStream stream)
{
    CaseResult cr;
    LOG_PRINT("\n===== %s k=%d =====\n", label, kVal);
    std::vector<int64_t> logitsShape = {B, V};
    std::vector<int64_t> pShape = {B};
    std::vector<int64_t> kShape = {B};
    std::vector<float> pHost(B, 1.0f); // p=1 => top-p keeps all after top-k
    std::vector<int32_t> kHost(B, kVal);
    std::vector<float> outHost(B * V, -123.f);

    void *logitsDev = nullptr, *pDev = nullptr, *kDev = nullptr, *outDev = nullptr;
    aclTensor *logits = nullptr, *p = nullptr, *k = nullptr, *out = nullptr;
    if (CreateAclTensor(logitsHost, logitsShape, &logitsDev, ACL_FLOAT, &logits)) return cr;
    if (CreateAclTensor(pHost, pShape, &pDev, ACL_FLOAT, &p)) return cr;
    if (CreateAclTensor(kHost, kShape, &kDev, ACL_INT32, &k)) return cr;
    if (CreateAclTensor(outHost, logitsShape, &outDev, ACL_FLOAT, &out)) return cr;

    uint64_t ws = 0;
    aclOpExecutor* ex = nullptr;
    cr.wsRet = aclnnApplyTopKTopPGetWorkspaceSize(logits, p, k, out, &ws, &ex);
    LOG_PRINT("  wsRet=%d ws=%lu\n", cr.wsRet, (unsigned long)ws);
    if (cr.wsRet == 0) {
        void* waddr = nullptr;
        if (ws) aclrtMalloc(&waddr, ws, ACL_MEM_MALLOC_HUGE_FIRST);
        cr.launchRet = aclnnApplyTopKTopP(waddr, ws, ex, stream);
        cr.syncRet = aclrtSynchronizeStream(stream);
        LOG_PRINT("  launch=%d sync=%d\n", cr.launchRet, cr.syncRet);
        if (cr.launchRet == 0 && cr.syncRet == 0) {
            cr.out.assign(B * V, 0.f);
            aclrtMemcpy(cr.out.data(), cr.out.size() * sizeof(float), outDev, cr.out.size() * sizeof(float),
                        ACL_MEMCPY_DEVICE_TO_HOST);
            cr.ok = true;
            // count non -inf
            int kept = 0;
            for (float v : cr.out) {
                if (std::isfinite(v)) kept++;
            }
            LOG_PRINT("  finite_out=%d / %ld first8:", kept, (long)(B * V));
            for (int i = 0; i < 8 && i < (int)cr.out.size(); i++) LOG_PRINT(" %.3f", cr.out[i]);
            LOG_PRINT("\n");
        }
        if (waddr) aclrtFree(waddr);
    }
    aclDestroyTensor(logits); aclDestroyTensor(p); aclDestroyTensor(k); aclDestroyTensor(out);
    aclrtFree(logitsDev); aclrtFree(pDev); aclrtFree(kDev); aclrtFree(outDev);
    return cr;
}

int main()
{
    if (aclInit(nullptr)) return 1;
    if (aclrtSetDevice(0)) return 1;
    aclrtStream stream;
    if (aclrtCreateStream(&stream)) return 1;

    const int64_t B = 2, V = 64;
    std::vector<float> logitsHost(B * V);
    for (int64_t b = 0; b < B; b++)
        for (int64_t v = 0; v < V; v++)
            logitsHost[b * V + v] = (float)(v); // sorted ascending → after Sort descending top is V-1

    auto ctrl = Run("CONTROL k=8", 8, B, V, logitsHost, stream);
    auto eqV = Run("BOUNDARY k=V", (int32_t)V, B, V, logitsHost, stream);
    auto bad1 = Run("TRIGGER k=V+1", (int32_t)(V + 1), B, V, logitsHost, stream);
    auto bad2 = Run("TRIGGER k=V*2", (int32_t)(V * 2), B, V, logitsHost, stream);
    auto bad3 = Run("TRIGGER k=1000000", 1000000, B, V, logitsHost, stream);

    aclrtDestroyStream(stream);
    aclrtResetDevice(0);
    aclFinalize();

    if (!ctrl.ok) {
        LOG_PRINT("CONTROL failed\n");
        return 1;
    }

    auto accepted_oob = [](const CaseResult& c) { return c.wsRet == 0; };
    auto faulted = [](const CaseResult& c) {
        return c.wsRet == 0 && (c.syncRet != 0 || c.launchRet != 0);
    };

    bool anyFault = faulted(bad1) || faulted(bad2) || faulted(bad3);
    bool anyAccepted = accepted_oob(bad1) || accepted_oob(bad2) || accepted_oob(bad3);
    bool anyRejected = (bad1.wsRet != 0) && (bad2.wsRet != 0) && (bad3.wsRet != 0);

    // Compare: for k>V, correct would reject. If accepted silently, that's the bug even without 507015.
    // Also if output differs from k=V in a "expanded" way incorrectly — still bug via accept.
    if (anyFault) {
        LOG_PRINT("\nVERDICT: BUG_REPRODUCED (exit 42) — OOB k caused device fault\n");
        return 42;
    }
    if (anyAccepted) {
        // Host/tiling did not reject k>vocab — confirms missing clamp (primary claim)
        LOG_PRINT("\nVERDICT: BUG_REPRODUCED (exit 42) — k>vocab accepted (ws SUCCESS) without reject\n");
        LOG_PRINT("  bad1(V+1) ws=%d ok=%d sync=%d; bad2 ws=%d; bad3 ws=%d\n", bad1.wsRet, bad1.ok, bad1.syncRet,
                  bad2.wsRet, bad3.wsRet);
        return 42;
    }
    if (anyRejected) {
        LOG_PRINT("\nVERDICT: NO_REPRO — OOB k rejected at host\n");
        return 0;
    }
    return 1;
}

执行脚本

文件名：build_and_run_repro.sh

#!/usr/bin/env bash
# 模板：一键编译并运行同目录下的 test_aclnn_*.cpp
# 用法（已 source CANN set_env.sh）：
#   bash build_and_run_repro.sh [--vendor cann_scan_repro] [--src test_aclnn_xxx.cpp]
set -euo pipefail

VENDOR=""
SRC_NAME=""
while [[ $# -gt 0 ]]; do
  case "$1" in
    --vendor) VENDOR="$2"; shift 2 ;;
    --src) SRC_NAME="$2"; shift 2 ;;
    *) echo "unknown arg: $1" >&2; exit 2 ;;
  esac
done

: "${ASCEND_HOME_PATH:?请先 source CANN set_env.sh}"
ROOT="$(cd "$(dirname "$0")" && pwd)"

if [[ -z "$SRC_NAME" ]]; then
  # 默认取目录内唯一的 test_aclnn_*.cpp
  mapfile -t _srcs < <(ls "$ROOT"/test_aclnn_*.cpp 2>/dev/null || true)
  if [[ ${#_srcs[@]} -ne 1 ]]; then
    echo "请用 --src 指定测试文件（当前匹配到 ${#_srcs[@]} 个 test_aclnn_*.cpp）" >&2
    exit 2
  fi
  SRC="${_srcs[0]}"
else
  SRC="$ROOT/$SRC_NAME"
fi
BIN="${SRC%.cpp}"

if [[ -n "$VENDOR" ]]; then
  for cand in \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}_nn/bin/set_env.bash" \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}/bin/set_env.bash" \
    "${ASCEND_HOME_PATH}/opp/vendors/${VENDOR}/bin/set_env.bash"
  do
    if [[ -f "$cand" ]]; then
      # shellcheck disable=SC1090
      source "$cand"
      echo "[env] sourced $cand"
      echo "[env] ASCEND_CUSTOM_OPP_PATH=${ASCEND_CUSTOM_OPP_PATH:-}"
      break
    fi
  done
fi

INC="${ASCEND_HOME_PATH}/aarch64-linux/include"
LIB="${ASCEND_HOME_PATH}/aarch64-linux/lib64"
[[ -d "$INC" ]] || INC="${ASCEND_HOME_PATH}/include"
[[ -d "$LIB" ]] || LIB="${ASCEND_HOME_PATH}/lib64"

echo "[build] g++ $SRC"
g++ -std=c++17 -O2 "$SRC" \
  -I"$INC" -I"$INC/aclnnop" \
  -L"$LIB" \
  -Wl,-rpath,"$LIB" \
  -lascendcl -lnnopbase -lopapi \
  -o "$BIN"

echo "[run] $BIN"
set +e
"$BIN"
RC=$?
set -e
echo "[exit] $RC  (42=BUG_REPRODUCED, 0=NO_REPRO, 1=harness error)"
exit "$RC"