Closed
[Bug-Report|缺陷反馈] 省略可选bandwidth_gbs的合法benchmark在会话报告阶段崩溃 #9
tianjunAdd on 29 days agoClosed at 27 days ago
tianjun
29 days ago Comments:
29 days ago Comments:
/assign @Ethan_Zou


29 days ago Assign Ethan_Zou as responsible person
27 days ago Added label: resolved
场景与影响
benchmark_record签名默认bandwidth_gbs=None,JSONL schema标注该字段可选,记录入口也只在传入时写入;但matched基线分支的终端报告使用rec[bandwidth_gbs],合法省略字段的成功测试会话最终KeyError退出。
版本与位置
master
af459ebfa36cf7f15532cc63385e8d4289704adb;生产位置tests/pytest_benchmark_plugin.py。复现与验证
在精确基线
af459ebfa36cf7f15532cc63385e8d4289704adb与同一最终回归文件上,真实 pytest 原版 1 failed、3 passed;最终 4 passed,0 skip。完整原始插件 record/terminal-summary 路径和实际嵌套pytest会话,matched/missing基线与省略/传入带宽四种组合;记录可选字段及基线字节核对。公开 CPU 复现(基线或修复仓库根目录):
python3 -m pip install pytest "torch==2.7.1" PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 python3 - <<'PY' import ast import importlib.util from pathlib import Path import sys import types import pytest root = Path.cwd() import json import tempfile directory_owner = tempfile.TemporaryDirectory(prefix="tilelang-benchmark-host-") directory = Path(directory_owner.name) (directory / "pytest.ini").write_text("[pytest]\n") record = dict(kernel="host_fixture", operation="record", params={}, param_id=None, cluster="a5", time_us=1.0, bandwidth_gbs=1.0) baseline = directory / "benchmark/test_cpu_refresh.jsonl" baseline.parent.mkdir() original = json.dumps(record) + "\n" baseline.write_text(original) (directory / "test_cpu_refresh.py").write_text( 'def test_record(benchmark_record):\n' ' benchmark_record(kernel="host_fixture", operation="record", params={}, time_us=1.0)\n') sys.path[:0] = [str(root), str(root / "src")] # Isolate unrelated package kernel imports; every exercised plugin method remains real. package = types.ModuleType("cann_ops_tilelang") package.__path__ = [str(root / "src/cann_ops_tilelang")] package.__spec__ = importlib.util.spec_from_loader(package.__name__, loader=None, is_package=True) sys.modules[package.__name__] = package path = root / "tests/pytest_benchmark_plugin.py" tree = ast.parse(path.read_text()) excluded = [node for node in tree.body if isinstance(node, ast.ImportFrom) and node.module == "tilelang.profiler.bench"] assert len(excluded) == 1 # The timer backend is unavailable without TileLang/NPU and is never called here. tree.body = [node for node in tree.body if node not in excluded] module = types.ModuleType("host_benchmark_plugin") module.__file__ = str(path) sys.modules[module.__name__] = module exec(compile(tree, str(path), "exec"), vars(module)) setattr(module, "_TESTS_DIR", str(directory)) setattr(module, "_BASELINES_LAYOUT", str(directory / "benchmark")) args = [str(directory / "test_cpu_refresh.py"), "-q", "-c", str(directory / "pytest.ini"), "--confcutdir=" + str(directory), "--run-benchmark", "--benchmark-output=" + str(directory / "results.jsonl")] try: status = pytest.main(args, plugins=[module]) except KeyError as exc: print("KeyError:", str(exc)) status = 1 print("session_exit=", int(status)) print("baseline_time_us=", json.loads(baseline.read_text())["time_us"]) directory_owner.cleanup() PY公开复现原版成功测试尾部 KeyError(bandwidth_gbs),session_exit=1;修复后 session_exit=0,基线字节不变。
关联 PR 拟新增
tests/host/test_benchmark_optional_bandwidth.py;将同一文件复制到精确基线或检出修复分支,执行PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 python3 -m pytest -q tests/host/test_benchmark_optional_bandwidth.py --confcutdir=tests/host。只排除一个未调用的 tilelang.profiler.bench 后端导入,并隔离无关 kernel 包初始化;执行的插件源码/pytest hooks/fixture/JSONL文件操作均为真实实现。受控时间字段用于Host文件/报告行为验证,不是实测NPU性能;没有模拟NPU计时,没有调用benchmark_timer。最终实际仓库 Ruff 检查/格式化、diff --check通过。
预期与修复建议
终端报告使用.get读取可选带宽,并沿用已有_fmt_bw(None)占位;增加实际pytest会话覆盖matched/missing基线与传入/省略带宽四种组合,不影响基线或记录字段。
本地无 NPU;仅核查实际源码及 Host 行为,未运行 CANN/NPU 编译、设备精度或性能测试。