Open
feat(multimodal): add torch.compile support for five image ops #68
feat(multimodal): add torch.compile support for five image ops #68
Open
贝比诺Add on 1 day ago
贝比诺
贝比诺Collaborators
1 day ago

当前PR是否有AI参与:

  1. AI Agent 平台:Codex。
  2. AI 模型:GPT-6。
  3. Prompt 上下文:按仓库 .agents/skills/ops-tilelang-torch-compile/SKILL.md,为五个 Multimodal API 增加固定形状 torch.compile / npugraph_ex 支持,并进行 Ascend NPU AUTO、PTO 实机验证。

PR功能描述 / 为什么需要这个合入:

为 attn_rel_h_rel_w、bmp_to_patches、prepare_image_attn_pos_id、replace_input_embeds、shuffle_image_tokens 增加 torch.library.custom_op 与 fake 实现,使它们可在固定形状下进入 torch.compile(..., backend='npugraph_ex', fullgraph=True, dynamic=False)。

  • 将元信息验证与 NPU 环境检查、TileLang JIT/launch 分离。fake 只推导元数据,不读取设备 Tensor 数据。
  • 前三个 API 返回新 Tensor;后两个保持写入 input_embeds、返回 None 的契约,并准确声明 mutates_args。
  • attn_rel_h_rel_w 改为由 PyTorch 显式分配输出并传入 TileLang 内核(out_idx=None)。原先由 TileLang 自动分配输出时,该环境在 ACL Graph 捕获后读取输出元数据发生段错误;数值计算与内核调度保持一致。
  • 新增独立参考结果、同存储更新、同规格新存储、输出及副作用、schema/fake、ACL Graph 捕获/重放测试;覆盖位置 ID 的 single/owner/flat 路径、窗口 1/2、BMP 翻转与归一化分支,以及替换后重排的同图组合。

该PR关联的issue

无。

希望检视人员了解:

  • 保留五个公开 API 的签名和返回语义;未注册自动 autograd,也未声明动态 shape 支持。
  • fake 使用显式的 total_seqlens_vit、total_patches 等参数确定输出形状;调用方须保证这些参数与设备上的前缀和数据一致。
  • shuffle_image_tokens 的每张图 token 数需满足现有交错排列契约;本 PR 沿用已有内核数学逻辑。
  • 验证环境:Python 3.12.3、PyTorch 2.10.0、torch_npu 2.10.0、TileLang 0.1.15+cpu.git194c1b89、CANN 9.3.0、Ascend NPU;PTO 使用已安装的 PTOAS 0.66 配套环境。

改动类型 / Change Type

测试信息 / Testing

在 /home/l00947271/ops-tilelang-multimodal-compile 中,加载 /usr/local/Ascend/ascend-toolkit/set_env.sh,分别设置 TILELANG_DEFAULT_TARGET=auto 与 pto,执行:

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OPS_TILELANG_TEST_LEVEL=0 \
python -m pytest tests/multimodal/test_dsocr.py tests/multimodal/test_image_embed_ops.py \
  -m 'not benchmark' -x -q --tb=short
检查 结果
AUTO 全量非 benchmark 38 passed,6 deselected
PTO 全量非 benchmark 38 passed,6 deselected
五个 API 的 compile 测试 各自验证捕获、后续重放,未观察到 fx_run_eagerly
torch.library.opcheck schema 与 FakeTensor 检查通过
Ruff check / format 通过
git diff --check 通过

日志保存于 /home/l00947271/multimodal_compile_validation_20261110/auto-final.log、pto-final.log。

检查清单 / Checklist

likedislike
Merge blocked
贝比诺贝比诺Collaborators
1 day ago Created pull request, commit fa4aa42c
CANN-robotCANN-robotCollaborators
1 day ago Added label: ai-co-authored
CANN-robotCANN-robotCollaborators
1 day ago Added label: cann-cla/yes
CANN-robot
CANN-robotCollaborators
1 day ago Comments:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here.
You can get sig-info at here.
You can self-configure the PR merge rules for this repository. For more details, please refer to Here.
For more, you also can visit HICANN.


PR Approval Progress

⚠️ This PR does not yet meet the following requirements:lgtm (requires ≥ 2 person(s) per module)、approve (requires ≥ 1 person(s) per module)

Module Approval Details

module lgtm status approve status
repo-cann/ops-tilelang ❌ (0/2)(You can also ask: 马兵, ChaoyangJi, kevin365, jiawei_zhuang, 俞郑中) ❌ (0/1)(You can also ask: ChaoyangJi, jiawei_zhuang, 马兵, 俞郑中, 黄俊健)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

babynull, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
CANN-robotCANN-robotCollaborators
1 day ago Set ChaoyangJi,kevin365,monologue815,jiang-lirui,Allan_Yu,jiawei_zhuang,liudan12,mabing1118 as approver
CANN-robotCANN-robotCollaborators
1 day ago Set ChaoyangJi,kevin365,monologue815,jiang-lirui,Allan_Yu,jiawei_zhuang,liudan12,mabing1118 as approver
贝比诺
贝比诺Collaborators
12 hours ago Comments:

/compile

likedislike
贝比诺贝比诺Collaborators
12 hours ago Pre merge succeeded (commit_id: 14325d3ebb5ca55535b57e0d836052903626824d)
CANN-robot
CANN-robotCollaborators
11 hours ago Comments:
🚀 CI 流水线已启动
📋 执行详情: 点击查看流水线
likedislike
CANN-robotCANN-robotCollaborators
11 hours ago Added label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
11 hours ago Removed label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
11 hours ago Added label: ci-pipeline-failed
CANN-robot
CANN-robotCollaborators
11 hours ago Comments:

流水线分析报告 — ops-tilelang run#154

流水线: https://gitcode.com/CANN/ops-tilelang/actions/runs/7eeab6ca3dde42bba00ba284e6e06bde | PR#68 | 2026-11-11 18:59:09 ~ 2026-11-11 19:01:17 | 失败 Job 1 个

定界结论: 代码问题 / 代码检查错误
  • 失败类型: 代码问题
  • 根因: CodeCheck 静态检查服务返回 no pass(代码检查未通过,具体违规项需查看 codecheck 详情页)
  • 处置建议: 通知 PR 提交者按详情页给出的违规项修复代码后重新触发流水线
  • 排除项: 非环境/网络问题(创建任务与轮询均返回 200,检查服务正常完成并明确给出 no pass);非编译/UT 问题(本 run 仅 CodeCheck 一个 Job 失败,无编译与测试 Job)

失败根因分析

根因 #1: 代码问题 / 代码检查错误 — CodeCheck 服务返回 no pass
  • 判据: codecheck 检查服务返回 no pass,Job 以 exit code 255 结束
  • 定位步骤: 3_codecheck.log(任务创建成功 → 轮询 running → 返回 no pass)
  • 影响 Job: CodeCheck
  • 根因证据(日志原文):
2026-11-11 19:00:54,608 - INFO 响应: {"code":"200","msg":"success","data":"https://www.openlibing.com/apps/entryCheckDashCode/MR_753d35985a1f465e8559bd42a101682b/33631196e339f9cd38780ddf8c9e1600?projectId=300033&codeHostingPlatformFlag=gitcode","state":"no pass",...}
2026-11-11 19:00:54,633+08:00 ::error::Process exited with code 255
结构化定界摘要
[{"repo": "ops-tilelang", "run_number": 154, "Failed_Type": "代码问题", "Failed_Sub_Type": "代码检查错误", "Failed_Reason": "CodeCheck 静态检查服务返回 no pass,代码检查未通过;创建任务与轮询均正常(HTTP 200),检查完成并明确给出 no pass,具体违规项需查看 codecheck 详情页", "evidence": "2026-11-11 19:00:54,608 - INFO 响应: {\"code\":\"200\",\"msg\":\"success\",\"data\":\"https://www.openlibing.com/apps/entryCheckDashCode/MR_753d35985a1f465e8559bd42a101682b/...\",\"state\":\"no pass\"}", "suggestion": "PR 提交者按 codecheck 详情页给出的违规项修复代码后重新触发流水线", "jobs": "CodeCheck"}]

由门禁失败定界分析自动生成 | repo: ops-tilelang | run: 7eeab6ca3dde42bba00ba284e6e06bde | PR#68

likedislike