已关闭
[Bug]: reduce-overhead 模式下,跑qwen2.5-vl-7B 报错: Not allow to synchronize captured-stream, stream_id=7 #236
pyf创建于  2025年12月23日关闭于  2月11日
pyf
2025年12月23日 创建

在提交问题之前,请通过搜索现有和历史问题确保该问题尚未被提出并解决。

您的环境信息

-- CANN 8.1.RC1:  
-- Pytorch/Torch_npu v2.7.1:
-- Python 版本 Python 3.11.:
-- 操作系统版本 (e.g., Ubuntu 18.04):

🐛 请描述bug

报错信息如下,请帮忙看下是什么原因:

  File "/opt/mnt2/pyf/MtiI2v/mimogpt/qwen2_vl_patch.py", line 497, in Qwen2VLForConditionalGenerationForward
    image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw, rotary_pos_emb=rotary_pos_emb,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 564, in __call__
    return self._compiled_model(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 305, in compiled_method
    return compiled_fn(model, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mnt2/pyf/MtiI2v/mimogpt/qwen2_vl_patch.py", line 236, in qwen2vision_transformer_pretrained_modelforward
    def qwen2vision_transformer_pretrained_modelforward(self, hidden_states: torch.Tensor, grid_thw: torch.Tensor,
  File "/usr/local/python3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 286, in compiled_fn
    return ge_kernel(*full_args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2066, in kernel
  File "/usr/local/python3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/_acl_concrete_graph/acl_graph.py", line 821, in compile
    self.compile_for_graph_key(graph_key, *args, **kwargs)
  File "/usr/local/python3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/_acl_concrete_graph/acl_graph.py", line 895, in compile_for_graph_key
    captured_outputs = self.capture(graph_key, *args, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/_acl_concrete_graph/acl_graph.py", line 936, in capture
    captured_outputs = self.fx_forward(*args, node_info=self._updated_node_infos, is_capturing=True,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1631, in forward
  File "/usr/local/python3/lib/python3.11/site-packages/torch/_ops.py", line 756, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: copy_between_host_and_device_opapi:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:54 NPU function error: aclrtMemcpy, error code is 107030
[ERROR] 2025-12-23-09:29:42 (PID:3817636, Device:6, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999: [PID: 3817636] 2025-12-23-09:29:42.082.253 Not allow to synchronize captured-stream, stream_id=7.[FUNC:StreamSynchronize][FILE:api_error.cc][LINE:884]
        TraceBack (most recent call last):
       rtStreamSynchronize execute failed, reason=[stream is captured][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 107027[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
       rtMemcpy execute failed, reason=[the current capture mode does not support this operation][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronized memcpy failed, kind = 2, runtime result = 107030[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
likedislike
Ppyf
2025年12月23日 修改标题为 “[Bug]: reduce-overhead 模式下,跑qwen2.5-vl-7B 报错: Not allow to synchronize captured-stream, stream_id=7”,原标题为“[Bug]: reduce-overhead 模型下,跑qwen2.5-vl-7B 报错: Not allow to synchronize captured-stream, stream_id=7”
yurongkun
yurongkun成员
2025年12月23日 评论:

从报错信息来看,capture存在流同步操作(capture期间不允许存在流同步操作),请提供运行脚本进一步分析

likedislike
SunYapingSunYaping成员
2025年12月29日 将 yurongkun 设为负责人
犀利小辣椒
犀利小辣椒
1月8日 评论:

我遇到了同样的报错,编译模式为config.mode = "reduce-overhead"

[W108 06:23:36.660816180 compiler_depend.ts:250] Warning: CAUTION: The operator 'aten::gelu.out' is not currently supported on the NPU backend and will fall back to run on the CPU. This may have performance implications. (function npu_cpu_fallback)
Error with batch_size=1: copy_between_host_and_device_opapi:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:54 NPU function error: aclrtMemcpy, error code is 107030
[ERROR] 2026-01-08-06:23:36 (PID:3072093, Device:7, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999[PID: 3072093] 2026-01-08-06:23:36.451.921 (EE9999):  Not allow to synchronize captured-stream, stream_id=46.[FUNC:StreamSynchronize][FILE:api_error.cc][LINE:962]
        TraceBack (most recent call last):
       rtStreamSynchronize execute failed, reason=[stream is captured][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 107027[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
       rtMemcpy execute failed, reason=[the current capture mode does not support this operation][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronized memcpy failed, kind = 2, runtime result = 107030[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]

详细报错日志,0a7ce077d5b14518a325b39f1336bc34.txt 对应的开源代码,https://github.com/KlingTeam/LivePortrait,设计的相关代码src/modules/convnextv2.py,Block类

likedislike
犀利小辣椒
犀利小辣椒
1月8日 评论:

通过实践分析,代码中,torch.arange(w).type(ref.dtype).to(ref.device) 是在计算图构建过程中创建 CPU 张量并试图传输到 NPU,这在捕获模式下是不允许的。改为 在 NPU 上直接创建张量:可以解决该问题

# 原来的代码:
# x = torch.arange(w).type(ref.dtype).to(ref.device)

# 改为:
x = torch.arange(w, dtype=ref.dtype, device=ref.device)
y = torch.arange(h, dtype=ref.dtype, device=ref.device)
z = torch.arange(d, dtype=ref.dtype, device=ref.device)

但是又遇到了同样的报错,

Error with batch_size=1: copy_between_host_and_device_opapi:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:54 NPU function error: aclrtMemcpy, error code is 107030
[ERROR] 2026-01-08-07:51:27 (PID:3125800, Device:7, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999[PID: 3125800] 2026-01-08-07:51:27.707.740 (EE9999):  Not allow to synchronize captured-stream, stream_id=60.[FUNC:StreamSynchronize][FILE:api_error.cc][LINE:962]
        TraceBack (most recent call last):
       rtStreamSynchronize execute failed, reason=[stream is captured][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 107027[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
       rtMemcpy execute failed, reason=[the current capture mode does not support this operation][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronized memcpy failed, kind = 2, runtime result = 107030[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]  Original traceback:
  File ,

具体的代码为,

class SPADE(nn.Module):
    def __init__(self, norm_nc, label_nc):
        super().__init__()

        self.param_free_norm = nn.InstanceNorm2d(norm_nc, affine=False)
        nhidden = 128

        self.mlp_shared = nn.Sequential(
            nn.Conv2d(label_nc, nhidden, kernel_size=3, padding=1),
            nn.ReLU())
        self.mlp_gamma = nn.Conv2d(nhidden, norm_nc, kernel_size=3, padding=1)
        self.mlp_beta = nn.Conv2d(nhidden, norm_nc, kernel_size=3, padding=1)

    def forward(self, x, segmap):
        normalized = self.param_free_norm(x)
        segmap = F.interpolate(segmap, size=x.size()[2:], mode='nearest')
        actv = self.mlp_shared(segmap)
        gamma = self.mlp_gamma(actv)
        beta = self.mlp_beta(actv)
        out = normalized * (1 + gamma) + beta
        return out

详细报错日志,55f5d15b587640499ee5684b09ffeb55.txt

likedislike
yurongkun
yurongkun成员
1月20日 评论:

经分析脚本中存在copy_between_host_and_device算子操作,导致capture失败。(资料:https://www.hiascend.com/document/detail/zh/canncommercial/850/appdevg/acldevg/aclcppdevg_000519.html ---在捕获过程中,在ACL_MODEL_RI_CAPTURE_MODE_GLOBAL模式(全局禁止,所有线程都不可以调用非安全函数)下,调用内存同步操作类函数(例如aclrtMemset、aclrtMemcpy、aclrtMemcpy2d)是非法的,会校验报错导致捕获失败。)
建议尝试使用其他算子替代该功能

分析过程:
开启TORCH_COMPILE_DEBUG后,文件路径为torch_compile_debug/run_******/torchair/model__0/forward,文件列表如下:
├── 000_aot_forward_graph.txt---查看代码和算子对应关系
├── 001_aot_forward_graph_after_optimize_noop_ops.txt
├── 002_aot_forward_graph_after_recover_view_inplace_pattern.txt
├── 003_aot_forward_graph_after_apply_pattern_passes.txt
├── 004_aot_forward_graph_after_view_to_reshape.txt
├── 005_aot_forward_graph_after_apply_event_closure_with_multi_stream.txt
├── 006_aot_forward_graph_after_apply_event_record.txt
├── 007_aot_forward_graph_after_eliminate_dead_code.txt
├── 008_aot_forward_graph_after_reinplace_inplaceable_ops_pass.txt
├── 009_aot_forward_graph_after_reinplace_input_mutated_ops.txt
├── 010_aot_forward_graph_after_decompose_auto_functionalized.txt
├── 011_aot_forward_graph_after_replace_dynamic_workspace_ops.txt
├── 012_aot_forward_graph_after_replace_core_limit_nodes.txt
└── output_code.py ## 输出代码, 在22行存在该操作
报错日志中显示异常操作在output_code.py中22行
File "/torchair/_acl_concrete_graph/acl_graph.py", line 1127, in capture
captured_outputs = self.fx_forward(*args_list, node_info=self._updated_node_infos, is_capturing=True,
File "", line 22, in forward

output_code.py文件内容如下:
def forward(*args, node_info=[], is_capturing: bool = False):
primals_1, primals_2, primals_3, primals_4, primals_5, primals_6, primals_7, primals_8 = args

view = torch.ops.aten.reshape.default(primals_1, [1, 128, 256, 256]);  primals_1 = None
_native_batch_norm_legit = torch.ops.aten._native_batch_norm_legit.no_stats(view, None, None, True, 0.1, 1e-05);  view = None ### 该操作触发
getitem = _native_batch_norm_legit[0];  _native_batch_norm_legit = None
view_1 = torch.ops.aten.reshape.default(getitem, [2, 64, 256, 256]);  getitem = None
upsample_nearest2d = torch.ops.aten.upsample_nearest2d.default(primals_2, [256, 256]);  primals_2 = None
convolution = torch.ops.aten.convolution.default(upsample_nearest2d, primals_3, primals_4, [1, 1], [1, 1], [1, 1], False, [0, 0], 1);  primals_4 = None
relu = torch.ops.aten.relu_.default(convolution);  relu = None
convolution_1 = torch.ops.aten.convolution.default(convolution, primals_5, primals_6, [1, 1], [1, 1], [1, 1], False, [0, 0], 1);  primals_6 = None
convolution_2 = torch.ops.aten.convolution.default(convolution, primals_7, primals_8, [1, 1], [1, 1], [1, 1], False, [0, 0], 1);  primals_8 = None
add = torch.ops.aten.add_.Tensor(convolution_1, 1);  add = None
mul = torch.ops.aten.mul.Tensor(view_1, convolution_1);  convolution_1 = None
add_1 = torch.ops.aten.add_.Tensor(mul, convolution_2);  convolution_2 = add_1 = None
return (mul, primals_3, primals_5, primals_7, view_1, upsample_nearest2d, convolution)

aot_forward_graph.txt文件内容: 代码对应算子位置:
class GraphModule(torch.nn.Module):
def forward(self, primals_1: "f32[2, 64, 256, 256]", primals_2: "f32[2, 12, 64, 64]", primals_3: "f32[128, 12, 3, 3]", primals_4: "f32[128]", primals_5: "f32[64, 128, 3, 3]", primals_6: "f32[64]", primals_7: "f32[64, 128, 3, 3]", primals_8: "f32[64]"):
# test.py:31 in forward, code: normalized = self.param_free_norm(x) ##该操作中存在host和device直接的内存拷贝导致capture失败。
view: "f32[1, 128, 256, 256]" = torch.ops.aten.view.default(primals_1, [1, 128, 256, 256]); primals_1 = None
_native_batch_norm_legit = torch.ops.aten._native_batch_norm_legit.no_stats(view, None, None, True, 0.1, 1e-05); view = None
getitem: "f32[1, 128, 256, 256]" = _native_batch_norm_legit[0]; _native_batch_norm_legit = None
view_1: "f32[2, 64, 256, 256]" = torch.ops.aten.view.default(getitem, [2, 64, 256, 256]); getitem = None

likedislike
yurongkunyurongkun成员
1月21日 添加了label:aclgraph
SunYapingSunYaping成员
1月21日 添加了label:bug
SunYapingSunYaping成员
2月11日 issue状态由 TODO 改变为 DONE
SunYapingSunYaping成员
2月11日 关闭了 issue