TTK（Ops Test Tool Kit）是CANN算子库提供的全链路、自动化、批量化算子测试框架，帮助开发者快速完成算子批量功能验证、性能评估以及Golden值比对，提升算子开发质量和效率。

TTK —— Operator Test Framework

TTK (ops Test Tool Kit) is a full-pipeline, automated, batch operator testing framework provided by the CANN operator library. It helps developers quickly perform batch operator functional verification, performance evaluation, and Golden value comparison, improving operator development quality and efficiency.

Hardware vendor names appearing in this document are for illustration only; TTK is configuration-driven and supports any hardware accelerator that conforms to the interface.

Full-stack test paths: Kernel (AscendC) / GEIR (GE graph) / ACLNN (aclnn* C API) / E2E (torch/torch_npu, tensorflow, and onnx domains) — covering operator to framework layers
Multi-device + simulation: Real devices support NPU / MLU / CPU; simulation mode mimics NPU behavior on CPU, enabling development and debugging without real hardware
Multi-card parallel: Multi-NPU parallel testing
Multiple comparison methods: Mixed tolerance (ecosystem open-source standard, default), statistical relative error, cosine similarity, binary exact, requantization, cross-check
Extensible plugins: Custom Golden / input generation, separate namespaces for Kernel and ACLNN/E2E

中文文档

Architecture Layers & Test Coverage

┌──────────────────────────────────────────────────┐
│  Framework Layer  ✅ Torch · TensorFlow · ...    │  ← E2E (end-to-end, full pipeline)
├──────────────────────────────────────────────────┤
│  Engine Layer     ✅ GEIR · ✅ ACLNN · ✅ ONNX    │  ← GEIR (GE graph compile + execute) / ACLNN (engine API) / ONNX (atc→om, via the e2e onnx domain)
├──────────────────────────────────────────────────┤
│  Operator Layer   ✅ AiCore · AiCpu              │  ← Kernel (operator kernel, compile + execute)
└──────────────────────────────────────────────────┘

Installation

Install CANN from the Ascend community: Official Link
Python 3.8+ recommended
The GEIR path requires the system toolchain g++ (for GE graph compilation; verify with g++ --version)
The ACLNN path additionally requires torch (input generation/golden computation); the E2E path additionally requires torch/torch_npu (neither in requirements.txt; install the pair matching your local CANN version, see the torch_npu installation guide)
The E2E onnx domain (api_name = onnx.<OpType>) needs no torch/tensorflow: it requires the onnx package (≥1.13 with onnx.defs.onnx_opset_version() ≤ 18; onnx 1.13.x pairs with CANN 9.2.x) plus the CANN atc compiler and libascendcl runtime — see the ONNX Operator Test Guide (Chinese)

git clone https://gitcode.com/cann/ops-test-kit.git
cd ops-test-kit
pip install -r requirements.txt

Quick Start

Define test cases via CSV files (see Test Case Generation) and run them in batch:

# Show device info
python3 -m ttk info

# Kernel: compile + execute + precision comparison (--plugin provides golden for some cases)
python3 -m ttk kernel -i examples/case_store/kernel/cases.csv --plugin examples/case_store/kernel/golden.py

# GEIR: GE graph compile + execute
python3 -m ttk geir -i examples/case_store/kernel/cases.csv --plugin examples/case_store/kernel/golden.py

# ACLNN: aclnn* C API
python3 -m ttk aclnn -i examples/case_store/aclnn/cases.csv --plugin examples/case_store/aclnn/golden.py

# E2E: framework end-to-end (--backend cpu forces CPU backend)
python3 -m ttk e2e -i examples/case_store/e2e/torch_cases.csv

# E2E onnx domain: single ONNX op (atc-compiled .om + aclmdl execution, no torch/tensorflow needed)
python3 -m ttk e2e -i examples/case_store/e2e/onnx_cases.csv

Per-case precision status and overall pass rate are printed to the terminal. Add -o to output a result CSV. Run python3 -m ttk kernel --help for parameters.

Documentation

Common

Test Case Generation (CSV Format) — CSV format, field reference, data types
Task Execution — CLI parameters, case filtering, device parallelism, advanced scenarios
Precision Comparison (Chinese) — comparison formulas, tolerance config, method selection

Test Paths

Four test paths map to different layers of the Ascend stack (E2E hosts three domains — torch/tensorflow/onnx; the onnx domain runs at the engine layer via atc→om and has its own row).

Path	Layer	Guide
Kernel	Operator (AiCore/AiCpu), compile + execute	Kernel Operator Test Guide
GEIR	Engine, GE graph compile + execute	GEIR Operator Test Guide (Chinese)
ACLNN	Engine, aclnn* C API	ACLNN Operator Test Guide
E2E	Framework (torch/torch_npu, tensorflow), end-to-end	E2E Operator Test Guide
E2E-onnx	Engine, single ONNX op via atc-compiled .om + aclmdl execution	ONNX Operator Test Guide (Chinese)

Advanced Scenarios

Offline data prepare, XPU cross-check, deterministic compute, dump debug, NPUSim simulation — see Task Execution - Advanced Scenarios.

Tool Adaptation

mssanitizer Adaptation (Chinese) — memory/race/sync detection
msopprof Adaptation (Chinese) — operator performance profiling
msdebug Adaptation (Chinese) — operator source-level debugging

FAQ

FAQ — common issues and self-diagnosis

AI Assistant

TTK ships with Agent Skills that provide TTK usage guidance for CLI-based AI coding assistants (Claude Code, OpenCode, etc.). See AGENTS.md for details.