开启TORCHDYNAMO_VERBOSE=1 TORCH_LOGS="+dynamo" 收集的完整报错信息见附件22d50ed75fce4474892227b56cb46d69.log

关联issue https://gitcode.com/Ascend/pytorch/issues/3066

最小复现编译过慢现象测试脚本

ec48fe2002d3413b860f4d9b45c1fa9e.zip

（gitcode居然不支持.py哈哈哈哈哈，只能打包成一个压缩包了）

测试脚本说明

用于复现 Transformer Block 在 NPU 上首次执行 torch.compile 耗时过长的问题。

脚本包含Transformer Block 中与编译相关的 BF16、FP32 权重转换、RMSNorm、Rotary Embedding、Value Embedding gate、SDPA、残差连接和 ReLU² MLP。
将首次 compiled forward、首次 backward 和缓存命中后的迭代分别计时，以区分慢编译阶段和运行性能。

使用方法

安装版本匹配的 PyTorch 和 torch_npu，在项目根目录运行 depth=4 对应的最小结构：

TORCHINDUCTOR_CACHE_DIR=/tmp/inductor-block-standalone-cold \
python tests/npu_test_example/block_compile_repro.py \
  --part block \
  --batch 2 \
  --seq 128 \
  --width 256 \
  --heads 2 \
  2>&1 | tee standalone-cold.log

测试结果

=== environment ===
python: 3.12.3
torch: 2.10.0+cpu
torch_npu: 2.10.0.post1+git3e81424
npu: Ascend950PR_9579
backend: inductor
part=block, batch=2, seq=128, width=256, heads=2, dtype=torch.bfloat16
[W720 23:38:43.872786636 NPUCachingAllocator.cpp:204] Warning: The current CANN and HDK(driver) versions require processing for 32 padding size, with memory allocation. (function operator())
/home/ubuntu/workspace/nanochat/.venv/lib/python3.12/site-packages/torch_npu/op_plugin/meta/_meta_registrations.py:2227: UserWarning: CAUTION: On Ascend950, when npu_fusion_attention is compiled by torch.compile, the seed, offset, and numels may be optimized, which can cause results to differ from eager mode when the random seed is set. This issue will be fixed in later version.
  _warn_npu_fusion_attention_compile_once()
W0720 23:39:38.181000 358407 torch/_inductor/debug.py:518] [0/0] model__0_forward_1 debug trace: /home/ubuntu/workspace/nanochat-ascend/torch_compile_debug/run_2026_07_20_23_38_44_709005-pid_358407/torchinductor/model__0_forward_1.0
compiled first forward: 57.36s
2026-07-20 23:39:41,428 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-20 23:39:41,472 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-20 23:39:41,521 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-20 23:39:41,699 - WARNING - [12*x1 + y0, x1 + 2*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-20 23:39:41,742 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-20 23:39:41,789 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-20 23:39:41,814 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
W0720 23:41:07.665000 358407 torch/_inductor/debug.py:518] [0/0] model__0_backward_3 debug trace: /home/ubuntu/workspace/nanochat-ascend/torch_compile_debug/run_2026_07_20_23_38_44_709005-pid_358407/torchinductor/model__0_backward_3.1
compiled first backward: 89.99s
compiled total: 147.34s
compiled warm iteration: 0.0023s

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

• API 令牌或密钥
• 密码或身份验证凭证
• 私有网址或接口地址
• 个人或机密数据
• ...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

环境问题如下

- 操作系统：ubuntu22.04
- 昇腾硬件信息：Ascend950PR_9579
- CANN软件版本：CANN 9.0.0
- 安装的对应软件版本 torch 2.10.0，torch_npu（v2.10.0源码编译），triton-ascend 3.2.1

其中torch_npu源码版本如下

commit 3e81424f0b0f131f64032bdb4173292303257bea (grafted, HEAD -> v2.10.0, origin/v2.10.0)
Author: Dring <17737727613@163.com>
Date:   Mon Jul 20 14:46:52 2026 +0800

🐛 问题描述

在编译较大参数量模型时，torch.compile会出现硬件超时问题，关键报错信息如下

[rank0]:   File "/home/ubuntu/workspace/nanochat/.venv/lib/python3.12/site-packages/torch_npu/npu/__init__.py", line 528, in get_device_properties
[rank0]:     return torch_npu._C._npu_getDeviceProperties(device_id)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: initDeviceProperty:torch_npu/csrc/npu/Module.cpp:237 NPU function error: aclrtGetMemInfo(ACL_HBM_MEM, &device_free, &device_total), error code is 107002
[rank0]: [ERROR] 2026-07-20-21:18:27 (PID:3342993, Device:0, RankID:0) ERR00100 PTA call acl api failed
[rank0]: [Error]: The context is empty.
[rank0]:         Check whether acl.rt.set_context or acl.rt.set_device is called.
[rank0]: [PID: 3342993] 2026-07-20-21:17:39.695.735 Inner_Error_Device_Subprocess_Startup_Timeout(E39007): Starting a subprocess on the device timed out.
[rank0]:         Possible Cause: 1.Failed to start the subprocess. 2. The HDC link is faulty.
[rank0]:         Solution: Obtain the device log and contact technical support at https://www.hiascend.com/support.
[rank0]:         TraceBack (most recent call last):
[rank0]:         rtsDeviceGetInfo execution failed, reason=driver error:internal error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
[rank0]:         call rtsDeviceGetInfo failed, runtime result = 507899.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[rank0]:         tsd client wait response fail, hostpid:3342993, receive device response data failed, check hdc service.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:340]
[rank0]:         Starting a subprocess on the device timed out.
[rank0]:         TsdOpen failed. devId=0, tdt error=31[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2638]
[rank0]:         Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3558]
[rank0]:         Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3175]
[rank0]:         Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3206]
[rank0]:         Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3319]
[rank0]:         Check param failed, curCtx can not be null.[FUNC:MemGetInfoEx][FILE:api_impl.cc][LINE:3162]
[rank0]:         The argument is invalid.Reason: rtMemGetInfoEx execution failed, the context is a null pointer.
[rank0]:         get memory information failed, runtime result = 107002[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]

且在编译期间使用npu-smi info信息会出现卡死现象

欢迎加入社区，感谢您对社区的贡献 🎉!

经过排查，将nproc_per_node设置为1，使用单卡测试可以跑通：

TORCHDYNAMO_VERBOSE=1 TORCH_LOGS="+dynamo" torchrun --standalone --nproc_per_node=1 -m scripts.base_train -- --depth=4  --target-param-data-ratio=8 --device-batch-size=4 2>&1 | tee run.log

正常跑完：

step 00348/00352 (98.86%) | loss: 4.745445 | lrm: 0.07 | dt: 1557.17ms | tok/sec: 168,346 | bf16_mfu: 2.78 | epoch: 1 pq: 2 rg: 4 | total time: 8.78m | eta: 0.1m
step 00349/00352 (99.15%) | loss: 4.718752 | lrm: 0.06 | dt: 1560.49ms | tok/sec: 167,988 | bf16_mfu: 2.77 | epoch: 1 pq: 2 rg: 5 | total time: 8.81m | eta: 0.1m
step 00350/00352 (99.43%) | loss: 4.707261 | lrm: 0.06 | dt: 1555.96ms | tok/sec: 168,477 | bf16_mfu: 2.78 | epoch: 1 pq: 2 rg: 5 | total time: 8.84m | eta: 0.1m
step 00351/00352 (99.72%) | loss: 4.698171 | lrm: 0.05 | dt: 1559.99ms | tok/sec: 168,042 | bf16_mfu: 2.77 | epoch: 1 pq: 2 rg: 6 | total time: 8.86m | eta: 0.0m
Step 00352 | Validation bpb: 1.436316

进一步聚焦是多卡场景下会存在torch.compile硬件超时的问题。

新增了最小编译复现脚本（test_gpt_compile.py）：

"""可调参数量的最小 GPT 多 NPU torch.compile 测试。"""

import argparse
import os
import time

import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F
# 导入后会为 PyTorch 注册 NPU backend。
import torch_npu  # noqa: F401


class Block(nn.Module):
    def __init__(self, width, heads):
        super().__init__()
        self.heads = heads
        self.norm1 = nn.RMSNorm(width)
        self.qkv = nn.Linear(width, 3 * width, bias=False)
        self.proj = nn.Linear(width, width, bias=False)
        self.norm2 = nn.RMSNorm(width)
        self.mlp = nn.Sequential(
            nn.Linear(width, 4 * width, bias=False),
            nn.GELU(),
            nn.Linear(4 * width, width, bias=False),
        )

    def forward(self, x):
        batch, seq, width = x.shape
        q, k, v = self.qkv(self.norm1(x)).chunk(3, dim=-1)
        shape = (batch, seq, self.heads, width // self.heads)
        q, k, v = (tensor.view(shape).transpose(1, 2) for tensor in (q, k, v))
        attention = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.proj(attention.transpose(1, 2).contiguous().view(batch, seq, width))
        return x + self.mlp(self.norm2(x))


class GPT(nn.Module):
    def __init__(self, layers, width, heads, vocab_size, seq_len):
        super().__init__()
        self.token = nn.Embedding(vocab_size, width)
        self.position = nn.Embedding(seq_len, width)
        self.blocks = nn.ModuleList([Block(width, heads) for _ in range(layers)])
        self.norm = nn.RMSNorm(width)
        self.head = nn.Linear(width, vocab_size, bias=False)

    def forward(self, tokens, targets):
        positions = torch.arange(tokens.size(1), device=tokens.device)
        x = self.token(tokens) + self.position(positions)
        for block in self.blocks:
            x = block(x)
        logits = self.head(self.norm(x))
        return F.cross_entropy(logits.flatten(0, 1).float(), targets.flatten())


parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--layers", type=int, default=4)
parser.add_argument("--width", type=int, default=256)
parser.add_argument("--heads", type=int, default=4)
parser.add_argument("--vocab-size", type=int, default=32768)
parser.add_argument("--seq-len", type=int, default=128)
parser.add_argument("--batch-size", type=int, default=2)
args = parser.parse_args()
assert args.width % args.heads == 0, "width 必须能被 heads 整除"

rank = int(os.environ["RANK"])
local_rank = int(os.environ["LOCAL_RANK"])
device = torch.device("npu", local_rank)
torch.npu.set_device(device)
dist.init_process_group(backend="hccl", device_id=device)
torch.manual_seed(0)

model = GPT(args.layers, args.width, args.heads, args.vocab_size, args.seq_len)
model = model.to(device=device, dtype=torch.bfloat16).train()
tokens = torch.randint(args.vocab_size, (args.batch_size, args.seq_len), device=device)
targets = torch.randint_like(tokens, args.vocab_size)

if rank == 0:
    parameters = sum(parameter.numel() for parameter in model.parameters())
    print(f"GPT parameters: {parameters:,} ({parameters / 1e6:.1f}M)", flush=True)
    print(f"config: {vars(args)}", flush=True)

# 所有 rank 同时进入首次 forward/backward 编译。
dist.barrier()
torch.npu.synchronize()
started = time.monotonic()

model = torch.compile(model)
loss = model(tokens, targets)
loss.backward()

torch.npu.synchronize()
print(
    f"rank {rank}: forward+backward compile={time.monotonic() - started:.2f}s, "
    f"loss={loss.item():.4f}",
    flush=True,
)
dist.destroy_process_group()

在2卡NPU上进行测试，结果如下：

(nanochat) ubuntu@ascend950:~/workspace/nanochat-ascend$ ASCEND_RT_VISIBLE_DEVICES=6,7 timeout 300s torchrun   --standalone   --nproc_per_node=2   test_gpt_compile.py   --layers 4   --width 256   --heads 4   --seq-len 128   --batch-size 2   2>&1 | tee gpt-compile-small.log
W0721 14:40:58.569000 3203689 torch/distributed/run.py:852] 
W0721 14:40:58.569000 3203689 torch/distributed/run.py:852] *****************************************
W0721 14:40:58.569000 3203689 torch/distributed/run.py:852] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0721 14:40:58.569000 3203689 torch/distributed/run.py:852] *****************************************
[rank0]:[W721 14:41:05.320600730 NPUCachingAllocator.cpp:204] Warning: The current CANN and HDK(driver) versions require processing for 32 padding size, with memory allocation. (function operator())
GPT parameters: 19,958,016 (20.0M)
config: {'layers': 4, 'width': 256, 'heads': 4, 'vocab_size': 32768, 'seq_len': 128, 'batch_size': 2}
/home/ubuntu/workspace/nanochat/.venv/lib/python3.12/site-packages/torch_npu/op_plugin/meta/_meta_registrations.py:2227: UserWarning: CAUTION: On Ascend950, when npu_fusion_attention is compiled by torch.compile, the seed, offset, and numels may be optimized, which can cause results to differ from eager mode when the random seed is set. This issue will be fixed in later version.
  _warn_npu_fusion_attention_compile_once()
[rank0]:W0721 14:41:59.823000 3204242 torch/_inductor/debug.py:518] [0/0] model__0_forward_1 debug trace: /home/ubuntu/workspace/nanochat-ascend/torch_compile_debug/run_2026_07_21_14_41_09_079998-pid_3204242/torchinductor/model__0_forward_1.0
2026-07-21 14:42:03,388 - WARNING - [256*x1 + y0, x1 + 32768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,440 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,470 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,517 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,567 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,606 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,627 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,662 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,699 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,735 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,754 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,791 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,827 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,862 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,883 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,919 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:03,955 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
[rank1]:W0721 14:42:21.497000 3204244 torch/_inductor/debug.py:518] [0/0] model__0_forward_1 debug trace: /home/ubuntu/workspace/nanochat-ascend/torch_compile_debug/run_2026_07_21_14_41_09_011532-pid_3204244/torchinductor/model__0_forward_1.0
2026-07-21 14:42:24,867 - WARNING - [256*x1 + y0, x1 + 32768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:24,917 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:24,948 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:24,994 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,044 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,083 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,103 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,139 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,175 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,211 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,230 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,266 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,302 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,337 - WARNING - [1024*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,358 - WARNING - [256*x1 + y0, x1 + 1024*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,393 - WARNING - [256*x1 + y0, x1 + 256*y0] low_dims is null, {0, 1}, {0, 1}
2026-07-21 14:42:25,430 - WARNING - [256*x1 + y0, x1 + 768*y0] low_dims is null, {0, 1}, {0, 1}
[rank0]:W0721 14:42:54.653000 3204242 torch/_inductor/debug.py:518] [0/0] model__0_backward_3 debug trace: /home/ubuntu/workspace/nanochat-ascend/torch_compile_debug/run_2026_07_21_14_41_09_079998-pid_3204242/torchinductor/model__0_backward_3.1
rank 0: forward+backward compile=110.46s, loss=10.5788
^[[A^[[A^[[ATerminated

目前看来一个20.0M的参数量的GPT模型，在rank0进行了110s的编译时间。

不过没遇到npu-smi info的命令卡死问题，使用命令后发现rank1的进程消失了😂。推测原因可能时超时或者别的原因导致了挂掉。