Closed
[Bug-Report|缺陷反馈] 省略可选bandwidth_gbs的合法benchmark在会话报告阶段崩溃 #9
tianjunAdd on  29 days agoClosed at  27 days ago
tianjun
29 days ago Create

场景与影响

benchmark_record签名默认bandwidth_gbs=None,JSONL schema标注该字段可选,记录入口也只在传入时写入;但matched基线分支的终端报告使用rec[bandwidth_gbs],合法省略字段的成功测试会话最终KeyError退出。

版本与位置

master af459ebfa36cf7f15532cc63385e8d4289704adb;生产位置 tests/pytest_benchmark_plugin.py。

复现与验证

在精确基线 af459ebfa36cf7f15532cc63385e8d4289704adb 与同一最终回归文件上,真实 pytest 原版 1 failed、3 passed;最终 4 passed,0 skip。完整原始插件 record/terminal-summary 路径和实际嵌套pytest会话,matched/missing基线与省略/传入带宽四种组合;记录可选字段及基线字节核对。

公开 CPU 复现(基线或修复仓库根目录):

python3 -m pip install pytest "torch==2.7.1"
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 python3 - <<'PY'
import ast
import importlib.util
from pathlib import Path
import sys
import types

import pytest

root = Path.cwd()
import json
import tempfile
directory_owner = tempfile.TemporaryDirectory(prefix="tilelang-benchmark-host-")
directory = Path(directory_owner.name)
(directory / "pytest.ini").write_text("[pytest]\n")
record = dict(kernel="host_fixture", operation="record", params={}, param_id=None,
              cluster="a5", time_us=1.0, bandwidth_gbs=1.0)
baseline = directory / "benchmark/test_cpu_refresh.jsonl"
baseline.parent.mkdir()
original = json.dumps(record) + "\n"
baseline.write_text(original)
(directory / "test_cpu_refresh.py").write_text(
    'def test_record(benchmark_record):\n'
    '    benchmark_record(kernel="host_fixture", operation="record", params={}, time_us=1.0)\n')

sys.path[:0] = [str(root), str(root / "src")]
# Isolate unrelated package kernel imports; every exercised plugin method remains real.
package = types.ModuleType("cann_ops_tilelang")
package.__path__ = [str(root / "src/cann_ops_tilelang")]
package.__spec__ = importlib.util.spec_from_loader(package.__name__, loader=None, is_package=True)
sys.modules[package.__name__] = package
path = root / "tests/pytest_benchmark_plugin.py"
tree = ast.parse(path.read_text())
excluded = [node for node in tree.body if isinstance(node, ast.ImportFrom)
            and node.module == "tilelang.profiler.bench"]
assert len(excluded) == 1
# The timer backend is unavailable without TileLang/NPU and is never called here.
tree.body = [node for node in tree.body if node not in excluded]
module = types.ModuleType("host_benchmark_plugin")
module.__file__ = str(path)
sys.modules[module.__name__] = module
exec(compile(tree, str(path), "exec"), vars(module))
setattr(module, "_TESTS_DIR", str(directory))
setattr(module, "_BASELINES_LAYOUT", str(directory / "benchmark"))
args = [str(directory / "test_cpu_refresh.py"), "-q", "-c", str(directory / "pytest.ini"),
        "--confcutdir=" + str(directory), "--run-benchmark",
        "--benchmark-output=" + str(directory / "results.jsonl")]
try:
    status = pytest.main(args, plugins=[module])
except KeyError as exc:
    print("KeyError:", str(exc))
    status = 1
print("session_exit=", int(status))
print("baseline_time_us=", json.loads(baseline.read_text())["time_us"])
directory_owner.cleanup()

PY

公开复现原版成功测试尾部 KeyError(bandwidth_gbs),session_exit=1;修复后 session_exit=0,基线字节不变。

关联 PR 拟新增 tests/host/test_benchmark_optional_bandwidth.py;将同一文件复制到精确基线或检出修复分支,执行 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 python3 -m pytest -q tests/host/test_benchmark_optional_bandwidth.py --confcutdir=tests/host。

只排除一个未调用的 tilelang.profiler.bench 后端导入,并隔离无关 kernel 包初始化;执行的插件源码/pytest hooks/fixture/JSONL文件操作均为真实实现。受控时间字段用于Host文件/报告行为验证,不是实测NPU性能;没有模拟NPU计时,没有调用benchmark_timer。最终实际仓库 Ruff 检查/格式化、diff --check通过。

预期与修复建议

终端报告使用.get读取可选带宽,并沿用已有_fmt_bw(None)占位;增加实际pytest会话覆盖matched/missing基线与传入/省略带宽四种组合,不影响基线或记录字段。

本地无 NPU;仅核查实际源码及 Host 行为,未运行 CANN/NPU 编译、设备精度或性能测试。

likedislike
tianjun
29 days ago Comments:

/assign @Ethan_Zou

likedislike
CANN-robotCANN-robotCollaborators
29 days ago Assign Ethan_Zou as responsible person
Ttianjun
29 days ago Associated pull request
Ttianjun
27 days ago Close issue
CANN-robotCANN-robotCollaborators
27 days ago Added label: resolved