| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(anti_cheat): 补全内置算子禁用清单，修复 TopK 等可绕过评测的漏洞 Co-authored-by: Deng Pan<pan.deng@huawei.com> # message auto-generated for no-merge-commit merge: !146 merge fix/anti-cheat-missing-kernels into master fix(anti_cheat): 补全内置算子禁用清单，修复 TopK 等可绕过评测的漏洞 Created-by: Deng_Pan Commit-by: Deng Pan Merged-by: cann-robot Description: ## 背景 反作弊 kernel-disable 清单（ scripts/anti_cheat/）由 tasks/**/cases.csv 的 baseline_kernels 标注推导。TopK 等算子从未标注，导致其内置 AiCore kernel 二进制始终留在 OPP 中——提交可**完全不实现 AscendC kernel**，仅在 host 侧把计算 委托给内置算子（如 C++ at::topk → aclnnTopk）即可通过全部评测刷分（已实证）。 ## 审计结论 对照真实 OPP 树（ascend910_93 / 910C）、aclnn 头文件与 torch_npu wrapper 逐一审计 原报告列出的 8 个缺标注 task，区分「缺标注」与「可被绕过」： | Task | 可达内置路径 | 有可移除 AiCore kernel? | 处理 | |------|--------------|--------------------------|------| | TopK | at::topk/aclnnTopk（已实证） | ✅ | 补全 | | WeightQuantBatchMatmul | torch_npu.npu_weight_quant_batchmatmul | ✅ | 补全 | | NMS | torch_npu.npu_nms_* | ✅ | 补全 | | Dilation2D | 仅 raw GE | ✅ | 补全（防御性） | | StridedSlice | 原生 x[b:e:s] + DMA | ✅（部分） | 补全 + 记录残留 | | Unique | at::unique | ❌ AICPU | 记录残留 | | GRU | torch_npu.npu_gru | ❌ 分解为 matmul | 记录残留 | | EngramGateFusion | 无 | ❌ 全新融合算子 | 缺标注属正确，不在残留清单 | ## 改动 - **补全 5 个有可移除 AiCore kernel 的算子**：cases.csv 追加 | baseline_kernels: ... 标注；kernel_map.json 新增 5 个算子 → 10 个 kernel 目录映射；benchmarked_kernels.txt 新增 10 个目录（均已确认存在于 910C kernel 树并实跑 disable_builtin_kernels.sh 验证移除）。 - **README 记录残留风险**：Unique（AICPU）、GRU（分解为受保护的 matmul）、StridedSlice （原生切片走 view+DMA，不经算子 kernel）这三类 kernel-disable **结构性无法覆盖**， 需 host 侧 TorchOpGuard 或 C++/dispatch 层防护。 ## 已知局限 已实证的 TopK 作弊走 **C++ at::topk**，而 TorchOpGuard 是 Python TorchFunctionMode， 无法拦截 C++ PrivateUse1 plugin 内部的 aten 调用——故本 PR 用 kernel-disable 兜底。 Unique/GRU 的根治需在 dispatch 层补防护，留作后续。 ## 测试 - disable_builtin_kernels.sh --dry-run 与实跑均确认 10 个目录正确移出 OPP（含 top_k_d/top_k_v2），restore 可一键还原。 - cases.csv 列完整性校验通过（7 列，无错行）；kernel_map.json 为合法 JSON。 🤖 Generated with [Claude Code](https://claude.com/claude-code) See merge request: cann/cann-bench!146 | 28 天前 | |
refactor: 迁移 StanfordBench 到 bench_lab/stanford_bench，移除 thirdparty 目录 Co-authored-by: Developer user<suyueming@huawei.com> # message auto-generated for no-merge-commit merge: !124 merge master into master refactor: 迁移 StanfordBench 到 bench_lab/stanford_bench，移除 thirdparty 目录 Created-by: su-yueming Commit-by: Developer user Merged-by: cann-robot Description: ## 变更描述 / Description 1. baseline 数据迁移 - 将 tasks/ 和 bench_lab/ 下所有 cases.yaml 中的 baseline_perf_us/t_hw_us 字段剥离，数据集中存储到各评测集 metadata/910b2.json - 删除 data/stanford_baseline.json，StanfordBench baseline 改由 bench_lab/stanford_bench/metadata/910b2.json 加载 - 新增 scripts/migrate_baseline_to_data.py 迁移工具脚本 2. src 代码变更 - __init__.py: 版本号改为动态读取（_version.py → VERSION 文件） - perf_strategy.py: KernelDetailsStrategy 以 kernel_details.csv 为唯一权威源， 不再 fallback 到 trace_view；TraceViewStrategy 标注待收编 - cann_loader.py: 集成 BaselineStore，baseline 优先从 JSON 查询 - stanford_loader.py: baseline 改由 BaselineStore 加载，移除 _load_baseline - report 模块: 微调 HTML/MD 报告和 setup_info 输出 3. 新增文件 - tasks/metadata/910b2.json + VERSION - bench_lab/*/metadata/910b2.json (cv_agent_bench, kernel_bench, pypto_cann_bench, stanford_bench) - pyproject.toml, VERSION (根目录), _version.py, test_version_consistency.py - docs/design/micro_benchmark_selection.md, module_panorama.md - docs/guide/version_policy.md 4. tests 更新 - test_benchmark_pipeline: 适配 baseline 迁移 - test_cases_yaml_csv_consistency: 新增 metadata JSON 与 cases.yaml 一致性校验 - test_version_consistency: 版本号一致性测试 1. 新增 PLATFORM_ALIAS 和 resolve_hardware() - 在 baseline_resolver.py 中添加产品型号→逻辑名映射表 - 支持精确匹配（Ascend910_9362 → 910b2）和前缀匹配（Ascend310P* → 310p） - 最长前缀优先，避免短 key 误匹配 - BaselineStore/BaselineResolver/__init__ 统一调用 resolve_hardware - has_baseline_for/resolve_baseline_us 内部自动解析硬件名 - 导出 resolve_hardware 和 PLATFORM_ALIAS 至 utils/__init__.py 2. 将 examples/eval_task → examples/tasks - 按 tasks/ 结构重构：add → level2/add, sqrt → level1/sqrt - 新增 metadata/910b2.json（fixture baseline，值均为 0.0 placeholder） - 新增 metadata/VERSION - 新增 cases.csv（与生产 tasks 格式一致） - 重写 README.md（目录结构、路径引用） - 更新 docs/guide/quick_start.md 和根 README.md 的路径引用 - 删除旧 examples/eval_task/ 目录 - 移动 scripts/download_benchmarks.sh → bench_lab/stanford_bench/download.sh， 下载目标从 thirdparty/KernelBench 改为 bench_lab/stanford_bench/KernelBench 将 direct_launch_example 从仅支持 910B 扩展为支持多种 SOC， 与 aclnn_launch_example 的多 SOC 能力对齐。 <!-- 本 PR 做了什么，为什么需要 / What does this PR do and why --> [#31](https://gitcode.com/cann/cann-bench/issues/31) ## 改动类型 / Change Type - [ ] Bug 修复 / Bug Fix - [ ] 新功能 / New Feature - [ ] 性能优化 / Performance - [x] 代码重构 / Refactoring - [ ] 文档更新 / Documentation - [ ] 测试相关 / Test - [ ] 其它 / Other ## 关联 Issue / Related Issues <!-- Closes #000 可自动关闭 / Closes #000 to auto-close --> - Closes # - References # ## 测试信息 / Testing <!-- 简要测试说明或关键结果 / Brief test description or key results --> - [ ] 单元测试通过 / UT passed - [ ] 集成测试通过 / ST passed - [ ] 人工验证通过 / Manual verified ## 检查清单 / Checklist - [ ] 代码符合规范 / Code follows style guide - [ ] 测试添加并通过 / Tests added and passed - [ ] 文档已更新 / Docs updated if needed - [ ] 无硬编码敏感信息 / No secrets hardcoded - [ ] 提交信息符合规范 / Commit message follows convention See merge request: cann/cann-bench!124 | 1 个月前 | |
refactor(tasks): mla_prolog baselines, add_rms_norm int8-only, weight_quant_batch_matmul decode redesign, dynamic_quant scale semantics, L1 support range, mhc_sinkhorn DSV4 alignment Co-authored-by: gxj1123<gao.xuejian1@huawei.com> # message auto-generated for no-merge-commit merge: !67 merge refactor/tasks-data-followup into master refactor(tasks): mla_prolog baselines, add_rms_norm int8-only, weight_quant_batch_matmul decode redesign, dynamic_quant scale semantics, L1 support range, mhc_sinkhorn DSV4 alignment Created-by: gxj1123 Commit-by: gxj1123 Merged-by: cann-robot Description: **118 files, +1878 / -1870.** Touches cases.yaml / cases.csv / desc.md / proto.yaml / golden.py across many operators in tasks/level1-4. ### Highlights - **tasks realign (6 ops)**: P1/P2 priority hierarchy + L1 spec-consistency framework + wrapper Python adapter for the validate-tasks pipeline. - **level3/add_rms_norm_dynamic_quant**: delete the unused int4 output path, redesign case 20, re-bench 3 cases against the int8-only API surface; semantic alignment of scale output with level2/dynamic_quant. - **level3/weight_quant_batch_matmul**: redesign decode cases + switch to the native torch_npu ref + drop the int4 baseline branch. - **level3/mhc_sinkhorn**: align cases + golden to the DeepSeek V4 kernel reality (the previous spec drifted from what aclnnMHCSinkhornV4 actually computes). - **level3/dilation_2d**: SAME ceil_mode off-by-one fix; cases shape adjusted accordingly. - **level3/transpose**: case 18 upper bound fix for x.shape[i]. - **level4/mla_prolog**: fill prefill/MTP baselines (10 cases) + decode baselines (8 cases); ref function corrected for the V3 input layout; supported-range tightened to match aclnnMlaPrologV3 docs. - **level2/dynamic_quant**: golden + proto drop the dead dst_type parameter and add the scale output; div-by-zero guard + int8 clamp; cases updated accordingly. - **level3/engram_gate_fusion**: golden refinement. - **level4/gru, lstm, grouped_matmul_swiglu_quant**: cases shape trims for stability; LSTM precision threshold relax for fp16 LSTM gates. - **level2/rms_norm + level3/add_rms_norm**: fp16 overflow guards (input range clamp before scale). - **level4 attention ops**: full mask NaN guard (attn_mask all -inf no longer produces NaN at row level). - **level2/apply_adam_w**: bias-correction expression matches the reference Adam-W formula in the spec. - **level1 support range** (1M = 1024, not 1000) propagated through desc + cases dtype/shape ranges for the 8 L1 ops. - **level3** (transpose, mhc_sinkhorn, weight_quant_batch_matmul, nms, grouped_matmul, conv_3d_backprop_filter, dilation_2d, moe_*, unique, strided_slice, quant_matmul): cases / desc / proto refresh after the t_hw clip + perf-score fallback + audit follow-up rounds. ### Status This PR is now **rebased off the current master** (after #63 / #64 / #66 merged). Mergeable, no conflicts. See merge request: cann/cann-bench!67 | 2 个月前 | |
refactor: rename kernel_bench/ → tasks/; level1-4 yaml parsing fixes; baseline_perf_us fills; kernel_eval evolution; docs/scripts sync Co-authored-by: gxj1123<gao.xuejian1@huawei.com> # message auto-generated for no-merge-commit merge: !62 merge refactor/kernel-bench-to-tasks into master refactor: rename kernel_bench/ → tasks/; level1-4 yaml parsing fixes; baseline_perf_us fills; kernel_eval evolution; docs/scripts sync Created-by: gxj1123 Commit-by: gxj1123 Merged-by: cann-robot Description: **308 files, +2679 / -1562.** The big move is renaming ** kernel_bench/ → tasks/** to reflect that the tree holds evaluation tasks (operator tasks for now, with room for model tasks alongside later). git tracks the move as R100 renames for 204 unchanged files; the remaining 64 files inside the renamed tree pick up content updates as listed below. ### Highlights - **level1 YAML parsing** — bare .inf / .nan tokens, none / null strings, and scientific notation now parse correctly across exp / foreach_addcdiv_scalar / foreach_norm / gelu / masked_scale / mish / sigmoid / swi_glu. Older PyYAML versions silently treated some of these as strings, poisoning downstream dtype handling. - **level2** — extend the YAML-parsing fix to remaining cases; group_norm / maximum / gcd case_16 — proto/cases consistency + desc broadcast rules + input_fail repair. - **level3** — large content sweep: - moe_finalize_routing — baseline_perf_us applied, ref + shape/range fixes after the code-review pass - transpose — cases refreshed to Attention BSHD ↔ BHSD patterns; 20/20 baselines benched - depthwise_conv_2d — spec corrected to the depthwise academic definition (depthwise = conv2d with groups=Cin, weight [Cin,1,kH,kW]) - dequant_swiglu_quant / dynamic_quant / moe_gating_top_k_softmax / grid_sampler_3d — smaller golden/case adjustments - **level4** — attention bench tightening + grouped_matmul_swiglu_quant spec follow-ups. - **src/kernel_eval** — evolution pass: process_pool flow refinements, precision spec touch-ups, cli/config polish, accuracy_eval bug fixes. tests/unit grows with new pytest cases (test_path_resolver, test_summarize_results); run_simple.py refreshed. - **docs** — new design notes (kernel_eval_architecture, cases_yaml_spec); evaluator_design / multi_card_parallel_analysis / perf_collection refreshed; guide/contributing + quick_start reworded; README scoring formula wording. - **scripts** — run_test.sh / run_evaluation.sh / run_ut.sh + README brought in line with the multi-card pool flow and the renamed task tree. - **examples** — aclnn_launch_example + direct_launch_example content refreshed. - **.gitignore** — covers new artifact paths produced by the harness. ### Note on the rename The new tasks/ name keeps the same levelN/<op>/ layout under it, so any code or CI that references kernel_bench/... needs a path update. The evaluator CLI / configs in src/kernel_eval have been updated accordingly. See merge request: cann/cann-bench!62 | 2 个月前 | |
fix(level3/top_k): 索引输出改用 tie 顺序无关校验，修复 golden 索引非确定无法验证 (#40) Co-authored-by: gxj1123<gao.xuejian1@huawei.com> # message auto-generated for no-merge-commit merge: !172 merge fix/topk-index-validation into master fix(level3/top_k): 索引输出改用 tie 顺序无关校验，修复 golden 索引非确定无法验证 (#40) Created-by: gxj1123 Commit-by: gxj1123 Merged-by: cann-robot Description: ## 关联 issue Closes #40 ## 问题 TopK 的 golden 用 torch.topk，相等元素的索引顺序非确定；NPU 候选对并列元素可能选不同下标，导致索引输出逐元素比对必然不过。此前把 idx 标 compare: false 直接跳过，等于放弃验证索引，留下“返回正确值却乱填索引也能过”的防作弊缺口。 ## 方案 索引改用 **tie 顺序无关的“指向值”校验**：候选返回的索引必须指向其值输出对应的元素，即 x.gather(dim, idx) == values。配合框架已有的“值输出 vs golden”比对： - 值算错 → 值比对（关系 golden）判失败； - 索引乱填/越界 → gather 自洽校验判失败（堵住防作弊缺口）； - 仅并列顺序不同 → 两者都通过（正确接受，正是本 issue 的症结）。 > 不采用 get_input 去并列的思路：TopK 用例（如 int8[-127,127] 含 2048+ 元素、fp16 含 1M 元素）受鸽笼原理必然存在并列，无法消除。 ## 改动（声明式，可复用于 ArgSort/ArgMax 等同类算子） - 新增 src/kernel_eval/eval/index_check.py：validate_index_output / validate_index_gather_outputs； - OutputSpec 增加可选字段 index_gather，cann_loader 从 proto 透传； - evaluator 在精度比对后调用校验，**仅当某输出声明了 index_gather 时才生效，对其他算子零影响**； - tasks/level3/top_k/proto.yaml 的 idx 输出声明 index_gather: {input: x, dim_attr: dim, value_output: y}（保留 compare: false，不再逐元素比对索引）。 ## 测试 新增 tests/ut/test_index_check.py（9 条）：异序正确索引 / 并列等价 → 通过；乱填 / 越界 / 形状不符 → 失败；声明路径与无声明零影响；dim=0 方向。本地相关 UT 全绿（既有的 grouped_mat_mul_allto_allv yaml/csv 不一致与本 PR 无关）。 See merge request: cann/cann-bench!172 | 25 天前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 28 天前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 25 天前 |