Merged
fix: 省略可选bandwidth_gbs的合法benchmark在会话报告阶段崩溃 #37
fix: 省略可选bandwidth_gbs的合法benchmark在会话报告阶段崩溃 #37
Merged
tianjunAdd on 29 days ago
tianjun
29 days ago

当前 PR 是否有 AI 参与

  1. AI Agent 平台:OpenAI Codex(桌面运行环境)
  2. AI 模型:GPT-6 系列
  3. Prompt 上下文:核查真实 pytest 插件和 JSONL 文件路径,按真人评审移除新增 Host 测试文件,并保留真实修复前后复现证据。

PR 功能描述 / 为什么需要这个合入

benchmark_record签名默认bandwidth_gbs=None,JSONL schema标注该字段可选,记录入口也只在传入时写入;但matched基线分支的终端报告使用rec[bandwidth_gbs],合法省略字段的成功测试会话最终KeyError退出。

终端报告使用.get读取可选带宽,并沿用已有_fmt_bw(None)占位;实际自测覆盖matched/missing基线与传入/省略带宽四种组合,不影响基线或记录字段。

该 PR 关联的 Issue

Fixes #9

https://gitcode.com/cann/ops-tilelang/issues/9

希望检视人员了解

在精确基线 af459ebfa36cf7f15532cc63385e8d4289704adb 与同一最终回归文件上,真实 pytest 原版 1 failed、3 passed;最终 4 passed,0 skip。完整原始插件 record/terminal-summary 路径和实际嵌套pytest会话,matched/missing基线与省略/传入带宽四种组合;记录可选字段及基线字节核对。

公开 CPU 复现(基线或修复仓库根目录):

python3 -m pip install pytest "torch==2.7.1"
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 python3 - <<'PY'
import ast
import importlib.util
from pathlib import Path
import sys
import types

import pytest

root = Path.cwd()
import json
import tempfile
directory_owner = tempfile.TemporaryDirectory(prefix="tilelang-benchmark-host-")
directory = Path(directory_owner.name)
(directory / "pytest.ini").write_text("[pytest]\n")
record = dict(kernel="host_fixture", operation="record", params={}, param_id=None,
              cluster="a5", time_us=1.0, bandwidth_gbs=1.0)
baseline = directory / "benchmark/test_cpu_refresh.jsonl"
baseline.parent.mkdir()
original = json.dumps(record) + "\n"
baseline.write_text(original)
(directory / "test_cpu_refresh.py").write_text(
    'def test_record(benchmark_record):\n'
    '    benchmark_record(kernel="host_fixture", operation="record", params={}, time_us=1.0)\n')

sys.path[:0] = [str(root), str(root / "src")]
# Isolate unrelated package kernel imports; every exercised plugin method remains real.
package = types.ModuleType("cann_ops_tilelang")
package.__path__ = [str(root / "src/cann_ops_tilelang")]
package.__spec__ = importlib.util.spec_from_loader(package.__name__, loader=None, is_package=True)
sys.modules[package.__name__] = package
path = root / "tests/pytest_benchmark_plugin.py"
tree = ast.parse(path.read_text())
excluded = [node for node in tree.body if isinstance(node, ast.ImportFrom)
            and node.module == "tilelang.profiler.bench"]
assert len(excluded) == 1
# The timer backend is unavailable without TileLang/NPU and is never called here.
tree.body = [node for node in tree.body if node not in excluded]
module = types.ModuleType("host_benchmark_plugin")
module.__file__ = str(path)
sys.modules[module.__name__] = module
exec(compile(tree, str(path), "exec"), vars(module))
setattr(module, "_TESTS_DIR", str(directory))
setattr(module, "_BASELINES_LAYOUT", str(directory / "benchmark"))
args = [str(directory / "test_cpu_refresh.py"), "-q", "-c", str(directory / "pytest.ini"),
        "--confcutdir=" + str(directory), "--run-benchmark",
        "--benchmark-output=" + str(directory / "results.jsonl")]
try:
    status = pytest.main(args, plugins=[module])
except KeyError as exc:
    print("KeyError:", str(exc))
    status = 1
print("session_exit=", int(status))
print("baseline_time_us=", json.loads(baseline.read_text())["time_us"])
directory_owner.cleanup()

PY

公开复现原版成功测试尾部 KeyError(bandwidth_gbs),session_exit=1;修复后 session_exit=0,基线字节不变。

已按 ChaoyangJi 的评审意见移除新增 Host 测试文件,最终源码只保留原生产修复。原真实回归的完整文件、命令和结果已归档到提交之外;上方内联 CPU 命令仍可在精确基线与当前修复源码直接复现,公开复现不依赖归档文件。

只排除一个未调用的 tilelang.profiler.bench 后端导入,并隔离无关 kernel 包初始化;执行的插件源码/pytest hooks/fixture/JSONL文件操作均为真实实现。受控时间字段用于Host文件/报告行为验证,不是实测NPU性能;没有模拟NPU计时,没有调用benchmark_timer。最终实际仓库 Ruff 检查/格式化、diff --check通过。

本地无 NPU;仅核查实际源码及 Host 行为,未运行 CANN/NPU 编译、设备精度或性能测试。

真实自测:aarch64 Host、Python 3.11.4、PyTorch 2.7.1+cpu、NumPy 1.26.4、pytest 8.3.2。执行原始真实回归时设置 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1、TORCH_DEVICE_BACKEND_AUTOLOAD=0,并采用上方说明的 kernel 初始化/未调用 timer 导入隔离。当前保留的生产源码与通过回归的源码逐字相同。原项目完整 pre-commit 及 Ruff 0.14.14 检查/格式化、Git diff/object/LFS 检查通过。独立补充 OAT 实际扫描插件 1 文件,基线与最终均报告该既有文件缺少许可证头、版权头两项;未声称 OAT 通过,原许可、头部和检查规则保持。

改动类型 / Change Type

likedislike
Pull Request has been successfully merged by CANN-robot.
(Thank you for tianjun's contribution.)
Ttianjun
29 days ago Created pull request, commit 58cf9479
Ttianjun
29 days ago Associated issue: [Bug-Report|缺陷反馈] 省略可选bandwidth_gbs的合法benchmark在会话报告阶段崩溃
CANN-robotCANN-robotCollaborators
29 days ago Added label: cann-cla/yes
tianjun
29 days ago Comments:

/compile

likedislike
Ttianjun
29 days ago Pre merge succeeded (commit_id: 5c72a692143e3561b53ee523ba59ff9f9d1d8b84)
CANN-robot
CANN-robotCollaborators
29 days ago Comments:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here.
You can get sig-info at here.
You can self-configure the PR merge rules for this repository. For more details, please refer to Here.
For more, you also can visit HICANN.


PR Approval Progress

✅ Congratulations! All modules have met the lgtm and approve requirements.

Module Approval Details

module lgtm status approve status
repo-cann/ops-tilelang ✅ ChaoyangJi, kevin365 (2/2) ✅ kevin365 (1/1)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

Ethan_Zou, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
CANN-robot
CANN-robotCollaborators
29 days ago Comments:

CLA Signature Pass

Ethan_Zou, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
CANN-robotCANN-robotCollaborators
29 days ago Set ChaoyangJi,kevin365,monologue815,jiang-lirui,Allan_Yu,jiawei_zhuang,liudan12,mabing1118 as approver
CANN-robotCANN-robotCollaborators
29 days ago Set ChaoyangJi,kevin365,monologue815,jiang-lirui,Allan_Yu,jiawei_zhuang,liudan12,mabing1118 as approver
CANN-robot
CANN-robotCollaborators
29 days ago Comments:
🚀 CI 流水线已启动
📋 执行详情: 点击查看流水线
likedislike
CANN-robotCANN-robotCollaborators
29 days ago Added label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
29 days ago Removed label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
29 days ago Added label: ci-pipeline-passed
ChaoyangJi
ChaoyangJiCollaborators29 days agoStart code review2
tests/host/test_benchmark_optional_bandwidth.py
Outdated
@@ -0,0 +1,0 @@
1+# MIT License
ChaoyangJi
ChaoyangJi29 days agoComments:

修改比较简单,请删除该test文件。

代码中的test文件主要用于算子相关的功能,性能测试。

likedislike
System
System MessageSystem
29 days ago Comments:

changed this line on 6ac9fc69 view diff detail

Ttianjun
29 days ago Pre merge succeeded (commit_id: 1e3569c7dd8b20e9b0d51945978faa2a14d06e19)
Ttianjun
29 days ago Force-pushed 1 commits:6ac9fc69-fix: 省略可选bandwidth_gbs的合法benchmark在会话报告阶段崩溃
Ttianjun
29 days ago Pre merge succeeded (commit_id: 6958e2977c687f1c34be04d939c38d43a2866e9f)
CANN-robotCANN-robotCollaborators
29 days ago Removed label: cann-cla/yes
CANN-robotCANN-robotCollaborators
29 days ago Removed label: ci-pipeline-passed
CANN-robot
CANN-robotCollaborators
29 days ago Comments:

Notification

This pull request has been changed(code update) or closed, so removes the following label(s): ci-pipeline-passed.

likedislike
CANN-robotCANN-robotCollaborators
29 days ago Added label: cann-cla/yes
Ttianjun
29 days ago Modify description of pull request
tianjun
29 days ago Comments:

已按您的意见移除新增 Host 测试文件;原生产修复保持原样。相同的真实归档回归在原基线 1 failed、3 passed,修复后 4 passed、0 skip。PR 正文保留可直接复现的内联 CPU 命令,并注明真实 Host 自测与隔离范围。原项目 pre-commit/Ruff、Git/LFS 检查通过;独立补充 OAT 的既有插件头部两项诊断保持,未修改许可或检查规则。当前单源提交:6ac9fc6906681a8a672e0961e9b736db55c0ad96。未运行 NPU 成功配置、精度或性能测试。

likedislike
tianjun
29 days ago Comments:

/compile

likedislike
Ttianjun
29 days ago Pre merge succeeded (commit_id: 6021d102d63eb6032d2d1f10f0f28a77cbc55a72)
CANN-robotCANN-robotCollaborators
29 days ago Added label: ci-pipeline-failed
CANN-robot
CANN-robotCollaborators
29 days ago Comments:

流水线分析报告 — ops-tilelang run#94

流水线: https://gitcode.com/CANN/ops-tilelang/actions/runs/e772612a365547dd8e280897efd585f2 | PR#37 | 2026-10-14 14:17:37 ~ 2026-10-14 14:22:24 | 失败 Job 2 个

定界结论: 环境问题 / Runner机器异常 — 平台弹性资源 Pod 供给失败, PR 代码未参与构建
  • 失败类型: 环境问题(非 PR 代码问题)
  • 根因: CI 平台为 Job 申请弹性资源(Pod)成功后, 平台侧持续无法连接自接入 k8s 集群(CP.COMP1076), 导致弹性资源状态查询异常(CP.COMP1042), Job 启动失败; 两个失败 Job 的业务步骤日志均为空, 代码从未被编译/检查
  • 处置建议: ① 直接重新触发流水线(平台弹性资源瞬时故障, 平台错误码自身建议"稍后重试", 大概率恢复) ② 重试仍失败 → 联系 CI 平台/流水线维护者检查弹性资源池(自接入 k8s 集群 url/kubeconfig/namespace 配置)
  • 排除项: 已排除镜像问题(资源申请成功, image: swr.cn-north-4.myhuaweicloud.com/ci_cann/ubuntu20.04.05_x86:k8s_v1.1006, successCount=1, failedCount=0); 已排除代码问题(失败发生在容器初始化阶段, 业务步骤未执行)

失败根因分析

根因 #1: 环境问题 / Runner机器异常 — 平台弹性资源 Pod 供给失败(CP.COMP1076 → CP.COMP1042)
  • 判据: 平台弹性资源状态异常(CP错误码, Runner机器异常)
  • 定位步骤: 0_初始化步骤.log(Test_image / Test_pre_pipeline 两个 Job 初始化日志同型)
  • 影响 Job: Test_image, Test_pre_pipeline
  • 时间线: 14:17:39 资源申请成功(Pod 处于 Pending) → 14:18:39 起平台持续报 CP.COMP1076 自接入 k8s 集群连接失败(每分钟重试, 历时约 4 分钟) → 14:22:04 Unhealthy container status: UNKNOWN → CP.COMP1102 任务申请资源错误 → CP.COMP1042 CloudEnv弹性资源状态查询异常 → Job 失败
  • 根因证据(日志原文):
[2026-10-14 14:18:39.596+08:00] [ERROR] [server] - Query resource status with failure, reason: [CP.COMP1076][Description] 用户自接入k8s集群连接失败。[RootCause] 无法连接到用户自接入的k8s集群,可能是k8s集群的url地址、kubeconfig、namespace等配置不正确。
[2026-10-14 14:22:04.384+08:00] [ERROR] [server] - Unhealthy container status: UNKNOWN
[2026-10-14 14:22:04.448+08:00] [ERROR] [server] - [CP.COMP1042][Description] CloudEnv弹性资源状态查询异常。...[RootCause] 查询CloudEnv容器状态失败,返回状态不在有效状态范围内,无法识别。[Solution] 检查弹性资源服务状态是否正常,或者稍后重试。
  • 级联说明: 其余 Job(Antipoison / CodeCheck_* / SCA / ST_test)均因前置失败处于 INIT 未启动, 无独立失败; Test_get-pr / Test_get-shell / Test_finish 正常完成
结构化定界摘要
[{"repo": "ops-tilelang", "run_number": 94, "Failed_Type": "环境问题",
  "Failed_Sub_Type": "Runner机器异常",
  "Failed_Reason": "CI平台申请弹性资源Pod成功后, 平台侧持续无法连接自接入k8s集群(CP.COMP1076), 弹性资源状态查询异常(CP.COMP1042)导致Job启动失败, 业务步骤日志为空, PR代码未参与构建",
  "evidence": "[2026-10-14 14:18:39.596+08:00] [ERROR] [server] - Query resource status with failure, reason: [CP.COMP1076][Description] 用户自接入k8s集群连接失败。",
  "suggestion": "直接重新触发流水线(平台弹性资源瞬时故障, 可重试); 重试仍失败则联系CI平台维护者检查自接入k8s集群(url/kubeconfig/namespace)配置",
  "jobs": "Test_image, Test_pre_pipeline"}]

由门禁失败定界分析自动生成 | repo: ops-tilelang | run: e772612a365547dd8e280897efd585f2 | PR#37

likedislike
tianjun
29 days ago Comments:

Current reviewed source 6ac9fc6906681a8a672e0961e9b736db55c0ad96: Native #94 failed before the image and pre-pipeline tasks started. The original source-bound startup failure has been retained for maintenance review; it did not establish a code or test assertion failure. I will retry the existing pipeline once while retaining this source, the reviewer-requested test removal and all original checks.

likedislike
tianjun
29 days ago Comments:

/compile

likedislike
CANN-robot
CANN-robotCollaborators
29 days ago Comments:
🚀 CI 流水线已启动
📋 执行详情: 点击查看流水线
likedislike
CANN-robotCANN-robotCollaborators
29 days ago Removed label: ci-pipeline-failed
CANN-robotCANN-robotCollaborators
29 days ago Added label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
29 days ago Removed label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
29 days ago Added label: ci-pipeline-passed
ChaoyangJi
ChaoyangJiCollaborators
29 days ago Comments:

/lgtm

likedislike
zhangwei503Collaborators
29 days ago Comments:

compile

likedislike
Ttianjun
29 days ago Pre merge succeeded (commit_id: f280da34b14d63457d79bbb43ae6ded9b0c3121b)
CANN-robot
CANN-robotCollaborators
29 days ago Comments:
🚀 CI 流水线已启动
📋 执行详情: 点击查看流水线
likedislike
CANN-robotCANN-robotCollaborators
29 days ago Removed label: ci-pipeline-passed
CANN-robotCANN-robotCollaborators
29 days ago Added label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
28 days ago Removed label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
28 days ago Added label: ci-pipeline-passed
wangkaiCollaborators
28 days ago Comments:

/approve
/lgtm

likedislike
CANN-robotCANN-robotCollaborators
28 days ago Added label: lgtmapproved
CANN-robot
CANN-robotCollaborators
28 days ago Comments:

The MR can not be merged, because of CodeReview discussion not resolved

If you want to solve this problem, you can click here to do it in the FAQs.

likedislike
ChaoyangJiChaoyangJiCollaborators
28 days ago Resolved final issue
CANN-robotCANN-robotCollaborators
28 days ago Merged pull request