已开启
[Bug]: [triton_experimental] rank mismatch + fmod #4314
AACAES创建于  17 天前
AACAES成员
17 天前 创建

在提交新问题之前,请确保您已经在社区中搜索过相关问题,并使用了社区中提供的资源/工具后,仍未找到满意的解决方式。

⚠️ 安全信息提醒:请仔细检查提供的文本内容,确保其不包含敏感数据信息,包括但不限于:

  • API 令牌或密钥
  • 密码或身份验证凭证
  • 私有网址或接口地址
  • 个人或机密数据
  • ...

在分享配置信息或代码示例时,请将敏感信息脱敏处理,或使用 <TOKEN> 等占位符替代原有内容。

环境信息

例如:
- 操作系统
- 昇腾硬件信息
- CANN软件版本
- 安装的对应软件版本

torch_npu 2.13.0

🐛 问题描述

self = <torch._inductor.codecache.LambdaFuture object at 0xfffd441ef4c0>
timeout = None

def result(self, timeout: float | None = None) -> Callable[..., Any]:
    if timeout is not None and self.future is not None:
        # Wait on the underlying cross-process future with the caller's
        # timeout; raises concurrent.futures.Time

... [调用栈已省略] ...
E torch._inductor.exc.InductorError: SubprocException: An exception occurred in a subprocess:
E
E Name=triton_unk_fused_add_bmm_permute_sum_view_2
E Traceback (most recent call last):
E File "/home/l00971669/miniconda3/envs/lyt310_ci/lib/python3.10/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 475, in do_job
E result = job()
E File "/home/l00971669/miniconda3/envs/lyt310_ci/lib/python3.10/site-packages/torch/_inductor/runtime/compile_tasks.py", line 204, in _worker_compile_triton
E kernel.precompile(warm_cache_only=True)
E File "/home/l00971669/miniconda3/envs/lyt310_ci/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 648, in precompile
E self._precompile_worker()
E File "/home/l00971669/miniconda3/envs/lyt310_ci/lib/python3.10/site-packages/torch_npu/_inductor/triton_experimental/npu_triton_heuristics.py", line 828, in _precompile_worker
E raise NoTritonConfigsError(
E torch._inductor.runtime.triton_heuristics.NoTritonConfigsError: No valid triton configs. All configs failed compilation (including small-block fallback). Last error: CompilationError: at 30:8:
E r0_1mask = r0_1 < 2
E r0_0 = r0_0inner + tl.arange(0, r0_0_blk)[None, None, :, None]
E r0_0mask = r0_0 < 3
E tmp0 = tl.load(in_ptr0 + (r0_1 + 2r0_0 + 6r0_2), (r0_2mask & r0_1mask & r0_0mask), eviction_policy='evict_last', other=0.0)
E tmp1 = tl.load(in_ptr1 + (r0_0 + 3r0_1), (r0_2mask & r0_1mask & r0_0mask), eviction_policy='evict_last', other=0.0)
E tmp2 = tmp0 + tmp1
E tmp3 = tl.broadcast_to(tmp2, [r0_2_blk, r0_1_blk, r0_0_blk, 1])
E tmp5 = _tmp4 + tmp3
E _tmp4 = tmp5
E _tmp4_rc = tl.reshape(_tmp4, [r0_2_blk
r0_1_blk*r0_0_blk, 1])
E tmp4 = tl.sum(_tmp4_rc, 0)[None, None, None, :]
E tl.store(out_ptr0 + (tl.full([1, 1], 0, tl.int32).broadcast_to(XBLOCK, 1)), tmp4, None)
E ^
E ValueError('Cannot broadcast, rank mismatch: [1, 1, 1, 1], [1, 1]')
E
E
E Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
E
E
E To execute this test, run the following from the base repo dir:
E python test_torchinductor.py NPUTests.test_buffer_use_after_remove_npu
E
E This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

欢迎加入社区,感谢您对社区的贡献 🎉!

likedislike
AAACAES成员
17 天前 添加了label:bug
AAACAES成员
17 天前 关联了看板:FrameworkPTAdapter 版本issue看板
TorchNPU-BotTorchNPU-Bot成员
17 天前 添加了label:triage-review
TorchNPU-Bot
TorchNPU-Bot成员
17 天前 评论:

issue待分派,添加triage-review标签

likedislike
AAACAES成员
17 天前 关联了里程碑:v26.2.0
AAACAES成员
5 天前 修改标题为 “[Bug]: [triton_experimental] rank mismatch + fmod”,原标题为“[Bug]: [triton_experimental] rank mismatch + fmod + remainder + _local_scalar_dense ”