[Bug] top_k_top_p_sample 对 k==0 缺少下界守卫导致设备异常-ops-nn-AtomGit

已关闭

[Bug] top_k_top_p_sample 对 k==0 缺少下界守卫导致设备异常 #4521

Agent-Bug-Hunter创建于 8月3日关闭于 4 天前

8月3日创建

摘要

aclnnTopKTopPSample（top_k_top_p_sample）文档写明 topK 有效范围 1≤k≤min(voc,1024)，超出应跳过 topK。kernel 守卫仅判断 temp > topKMax（上界），k==0 误入 classic topK，随后 (kCount-1)*2 无符号下溢，在 rowLen>8192 时设备异常（本机 Sync 507035）。Ascend910B3 真机 exit 42。本地 draft，未远程提交。

环境

项	值
芯片	Ascend910B3
CANN	9.1.0-beta.3
ops-nn HEAD	`e16ef626bb3a`
编译命令	见下文「完整复现过程」
安装方式	`cann-ops-nn-cann_scan_repro_linux-*.run` → `$ASCEND_HOME_PATH/vendors/cann_scan_repro_nn`
Track / Pattern	A / novel-k-zero
Finding	IDX-TKPS-001
运行时证明	见下文「为何确认跑的是源码编译内核」

为什么认为这是问题：期望 vs 实际

输入（与复现用例一致）

k=128, V=16384: 期望=SUCCESS，idx=argmax; 实际线索=SUCCESS，idx=1234

期望行为

用例	期望
k=128, V=16384	SUCCESS，idx=argmax
k=0, V=16384 / {48,131072}	跳过 topK → argmax / 合法

实际行为（aclnn / 源码编译安装路径）

用例	API 返回 / 实际	关键输出	说明
k=128, V=16384	SUCCESS，idx=1234	见运行摘录	exit 42
k=0, V=16384 / {48,131072}	sync=507035	见运行摘录	exit 42

对比结论

用例	期望	实际
k=128, V=16384	SUCCESS，idx=argmax	SUCCESS，idx=1234
k=0, V=16384 / {48,131072}	跳过 topK → argmax / 合法	sync=507035

根因

op_kernel/top_k_top_p_sample.h：上界守卫遗漏 k>0；sibling top_k_top_p_sample_v2 已正确使用 k>0 && k<=min(...)。

Anti-pattern check (AP-001): 910B 配置；无 host k 下界。control-scalar，非 P-011。

为何确认跑的是源码编译内核（非 built-in）

环境：安装自定义包后执行 source $ASCEND_HOME_PATH/vendors/cann_scan_repro_nn/bin/set_env.bash，ASCEND_CUSTOM_OPP_PATH 指向该 vendor。
二进制路径：复现链接/加载 vendor 下 op_api / op_impl，而非仅依赖 CANN built-in 同名算子（见 repro_manifest.json 中 built_from_source=true）。
（推荐）排除 built-in：临时移走 built-in 对应算子内核目录后仍能 exit 42。

完整复现过程

占位符：$ASCEND_HOME_PATH / $OPS_NN / $REPRO_ROOT。

步骤 1：环境使能

source <CANN安装目录>/ascend-toolkit/set_env.sh

步骤 2：源码编译单算子包

cd "$OPS_NN"
bash build.sh --pkg --soc=ascend910b --ops=top_k_top_p_sample \
  --vendor_name=cann_scan_repro --no_force -j8

成功标志：生成 $OPS_NN/build_out/cann-ops-nn-cann_scan_repro_linux-*.run

步骤 3：安装并切换到自定义包

bash $OPS_NN/build_out/cann-ops-nn-cann_scan_repro_linux-*.run \
  --quiet --install-path="$ASCEND_HOME_PATH"
source "$ASCEND_HOME_PATH/vendors/cann_scan_repro_nn/bin/set_env.bash"
echo "ASCEND_CUSTOM_OPP_PATH=$ASCEND_CUSTOM_OPP_PATH"

步骤 4：放置复现源码与执行脚本

$REPRO_ROOT/
  test_aclnn_top_k_top_p_sample_k0.cpp
  build_and_run_repro.sh

（本 draft 目录内已含上述文件，可直接作为 $REPRO_ROOT。）

步骤 5：编译并运行

cd "$REPRO_ROOT"
chmod +x build_and_run_repro.sh
bash build_and_run_repro.sh --vendor cann_scan_repro --src test_aclnn_top_k_top_p_sample_k0.cpp
# 退出码 42 = BUG_REPRODUCED

成功复现标志：进程退出码 42。

实际运行摘录

[build] g++ $WORKSPACE/ops/cann_scan/workspace/runs/20260731_ops_nn_all/repro/top_k_top_p_sample/test_aclnn_top_k_top_p_sample_k0.cpp
[run] $WORKSPACE/ops/cann_scan/workspace/runs/20260731_ops_nn_all/repro/top_k_top_p_sample/test_aclnn_top_k_top_p_sample_k0

===== EXAMPLE-shape k=128 k=128 B=48 V=131072 =====
  ws=0 size=167772160
  launch=0 sync=0
  idx=1234

===== SMALL k=128 k=128 B=1 V=16384 =====
  ws=0 size=17170432
  launch=0 sync=0
  idx=1234

===== TRIGGER k=0 V=16384 k=0 B=1 V=16384 =====
  ws=0 size=17170432
  launch=0 sync=507035

===== TRIGGER k=0 example-shape k=0 B=48 V=131072 =====
  ws=0 size=167772160
  launch=0 sync=507035
VERDICT: BUG_REPRODUCED (exit 42) k=0 device fault
[exit] 42  (42=BUG_REPRODUCED, 0=NO_REPRO, 1=harness error)

复现用例源码

文件名：test_aclnn_top_k_top_p_sample_k0.cpp

/**
 * IDX-TKPS-001: mirror official example dtypes (bf16 logits/topP), trigger k=0.
 */
#include <cstdio>
#include <cstdint>
#include <vector>
#include <cstring>
#include "acl/acl.h"
#include "aclnnop/aclnn_top_k_top_p_sample.h"

#define LOG_PRINT(message, ...) do { printf(message, ##__VA_ARGS__); } while (0)

static int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t n = 1;
    for (auto i : shape) n *= i;
    return n;
}

template <typename T>
static int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                           aclDataType dataType, aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    if (ret) return (int)ret;
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    if (ret) return (int)ret;
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = (int64_t)shape.size() - 2; i >= 0; i--) strides[i] = shape[i + 1] * strides[i + 1];
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, ACL_FORMAT_ND, shape.data(),
                              shape.size(), *deviceAddr);
    return 0;
}

// pack fp32 -> bf16 (truncate mantissa)
static uint16_t f32_to_bf16(float f)
{
    uint32_t u;
    memcpy(&u, &f, 4);
    return (uint16_t)(u >> 16);
}

struct CaseR {
    int ws = -1, launch = -1, sync = -1;
    int64_t idx = -999;
};

static CaseR Run(const char* label, int32_t kVal, int64_t B, int64_t V, aclrtStream stream)
{
    CaseR cr;
    LOG_PRINT("\n===== %s k=%d B=%ld V=%ld =====\n", label, kVal, (long)B, (long)V);
    std::vector<int64_t> logitsShape = {B, V};
    std::vector<int64_t> topKPShape = {B};

    std::vector<uint16_t> logitsHost(B * V);
    for (int64_t i = 0; i < B * V; i++) logitsHost[i] = f32_to_bf16(0.01f);
    // max at 1234 for batch 0
    logitsHost[1234] = f32_to_bf16(50.0f);

    std::vector<int32_t> topKHost(B, kVal);
    std::vector<uint16_t> topPHost(B, f32_to_bf16(1.0f));
    std::vector<float> qHost(B * V, 1.0f);
    std::vector<int64_t> idxHost(B, -1);
    std::vector<float> selHost(B * V, 0.f);

    void *logitsDev = nullptr, *topKDev = nullptr, *topPDev = nullptr, *qDev = nullptr;
    void *idxDev = nullptr, *selDev = nullptr;
    aclTensor *logits = nullptr, *topK = nullptr, *topP = nullptr, *q = nullptr;
    aclTensor *idxOut = nullptr, *selOut = nullptr;

    if (CreateAclTensor(logitsHost, logitsShape, &logitsDev, ACL_BF16, &logits)) return cr;
    if (CreateAclTensor(topKHost, topKPShape, &topKDev, ACL_INT32, &topK)) return cr;
    if (CreateAclTensor(topPHost, topKPShape, &topPDev, ACL_BF16, &topP)) return cr;
    if (CreateAclTensor(qHost, logitsShape, &qDev, ACL_FLOAT, &q)) return cr;
    if (CreateAclTensor(idxHost, topKPShape, &idxDev, ACL_INT64, &idxOut)) return cr;
    if (CreateAclTensor(selHost, logitsShape, &selDev, ACL_FLOAT, &selOut)) return cr;

    float eps = 1e-8f;
    // header: bool isNeedLogits; example used int64 — try false
    bool isNeedLogits = false;
    int64_t topKGuess = 32;
    uint64_t ws = 0;
    aclOpExecutor* ex = nullptr;
    cr.ws = aclnnTopKTopPSampleGetWorkspaceSize(logits, topK, topP, q, eps, isNeedLogits, topKGuess, idxOut, selOut,
                                                &ws, &ex);
    LOG_PRINT("  ws=%d size=%lu\n", cr.ws, (unsigned long)ws);
    if (cr.ws == 0) {
        void* waddr = nullptr;
        if (ws) aclrtMalloc(&waddr, ws, ACL_MEM_MALLOC_HUGE_FIRST);
        cr.launch = aclnnTopKTopPSample(waddr, ws, ex, stream);
        cr.sync = aclrtSynchronizeStream(stream);
        LOG_PRINT("  launch=%d sync=%d\n", cr.launch, cr.sync);
        if (cr.launch == 0 && cr.sync == 0) {
            aclrtMemcpy(&cr.idx, sizeof(cr.idx), idxDev, sizeof(cr.idx), ACL_MEMCPY_DEVICE_TO_HOST);
            LOG_PRINT("  idx=%ld\n", (long)cr.idx);
        }
        if (waddr) aclrtFree(waddr);
    }
    aclDestroyTensor(logits); aclDestroyTensor(topK); aclDestroyTensor(topP); aclDestroyTensor(q);
    aclDestroyTensor(idxOut); aclDestroyTensor(selOut);
    aclrtFree(logitsDev); aclrtFree(topKDev); aclrtFree(topPDev); aclrtFree(qDev);
    aclrtFree(idxDev); aclrtFree(selDev);
    return cr;
}

int main()
{
    if (aclInit(nullptr)) return 1;
    if (aclrtSetDevice(0)) return 1;
    aclrtStream stream;
    if (aclrtCreateStream(&stream)) return 1;

    // official example shape first
    auto ex = Run("EXAMPLE-shape k=128", 128, 48, 131072, stream);
    auto small = Run("SMALL k=128", 128, 1, 16384, stream);
    auto k0 = Run("TRIGGER k=0 V=16384", 0, 1, 16384, stream);
    auto k0big = Run("TRIGGER k=0 example-shape", 0, 48, 131072, stream);

    aclrtDestroyStream(stream);
    aclrtResetDevice(0);
    aclFinalize();

    if (ex.ws != 0 && small.ws != 0) {
        LOG_PRINT("All controls failed — cannot reach kernel (ws ex=%d small=%d)\n", ex.ws, small.ws);
        return 1;
    }

    auto check = [](const CaseR& c) -> int {
        if (c.ws != 0) return 0; // rejected
        if (c.sync != 0 || c.launch != 0) return 42;
        if (c.idx != 1234) return 42; // for B=1 case only meaningful
        return 0;
    };

    if (k0.ws == 0 && (k0.sync != 0 || k0.launch != 0)) {
        LOG_PRINT("VERDICT: BUG_REPRODUCED (exit 42) k=0 device fault\n");
        return 42;
    }
    if (k0big.ws == 0 && (k0big.sync != 0 || k0big.launch != 0)) {
        LOG_PRINT("VERDICT: BUG_REPRODUCED (exit 42) k=0 big device fault\n");
        return 42;
    }
    if (k0.ws == 0 && k0.idx != 1234) {
        LOG_PRINT("VERDICT: BUG_REPRODUCED (exit 42) k=0 wrong idx=%ld\n", (long)k0.idx);
        return 42;
    }
    if (k0.ws != 0 && k0big.ws != 0) {
        LOG_PRINT("VERDICT: NO_REPRO — k=0 host-rejected (and/or schema)\n");
        return 0;
    }
    LOG_PRINT("VERDICT: NO_REPRO — k=0 behaved OK\n");
    return 0;
}

执行脚本

文件名：build_and_run_repro.sh

#!/usr/bin/env bash
# 模板：一键编译并运行同目录下的 test_aclnn_*.cpp
# 用法（已 source CANN set_env.sh）：
#   bash build_and_run_repro.sh [--vendor cann_scan_repro] [--src test_aclnn_xxx.cpp]
set -euo pipefail

VENDOR=""
SRC_NAME=""
while [[ $# -gt 0 ]]; do
  case "$1" in
    --vendor) VENDOR="$2"; shift 2 ;;
    --src) SRC_NAME="$2"; shift 2 ;;
    *) echo "unknown arg: $1" >&2; exit 2 ;;
  esac
done

: "${ASCEND_HOME_PATH:?请先 source CANN set_env.sh}"
ROOT="$(cd "$(dirname "$0")" && pwd)"

if [[ -z "$SRC_NAME" ]]; then
  # 默认取目录内唯一的 test_aclnn_*.cpp
  mapfile -t _srcs < <(ls "$ROOT"/test_aclnn_*.cpp 2>/dev/null || true)
  if [[ ${#_srcs[@]} -ne 1 ]]; then
    echo "请用 --src 指定测试文件（当前匹配到 ${#_srcs[@]} 个 test_aclnn_*.cpp）" >&2
    exit 2
  fi
  SRC="${_srcs[0]}"
else
  SRC="$ROOT/$SRC_NAME"
fi
BIN="${SRC%.cpp}"

if [[ -n "$VENDOR" ]]; then
  for cand in \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}_nn/bin/set_env.bash" \
    "${ASCEND_HOME_PATH}/vendors/${VENDOR}/bin/set_env.bash" \
    "${ASCEND_HOME_PATH}/opp/vendors/${VENDOR}/bin/set_env.bash"
  do
    if [[ -f "$cand" ]]; then
      # shellcheck disable=SC1090
      source "$cand"
      echo "[env] sourced $cand"
      echo "[env] ASCEND_CUSTOM_OPP_PATH=${ASCEND_CUSTOM_OPP_PATH:-}"
      break
    fi
  done
fi

INC="${ASCEND_HOME_PATH}/aarch64-linux/include"
LIB="${ASCEND_HOME_PATH}/aarch64-linux/lib64"
[[ -d "$INC" ]] || INC="${ASCEND_HOME_PATH}/include"
[[ -d "$LIB" ]] || LIB="${ASCEND_HOME_PATH}/lib64"

echo "[build] g++ $SRC"
g++ -std=c++17 -O2 "$SRC" \
  -I"$INC" -I"$INC/aclnnop" \
  -L"$LIB" \
  -Wl,-rpath,"$LIB" \
  -lascendcl -lnnopbase -lopapi \
  -o "$BIN"

echo "[run] $BIN"
set +e
"$BIN"
RC=$?
set -e
echo "[exit] $RC  (42=BUG_REPRODUCED, 0=NO_REPRO, 1=harness error)"
exit "$RC"