vllm 0.20.2 full decode only图模式， msprobe 26.1.0
启动参数
vllm serve /models/GLM-5.1-w8a8
--host 0.0.0.0
--port 8902
--data-parallel-size 2
--data-parallel-size-local 1
--data-parallel-start-rank 0
--data-parallel-address 30.37.2.2
--data-parallel-rpc-port 13389
--tensor-parallel-size 8
--quantization ascend
--seed 1024
--served-model-name glm-5.1
--enable-expert-parallel
--max-num-seqs 16
--max-model-len 65536
--max-num-batched-tokens 4096
--trust-remote-code
--gpu-memory-utilization 0.95
--no-enable-prefix-caching
--additional-config '{"dump_config_path": "/workspace/config.json"}'
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

/workspace/config.json的配置：
{
"task": "statistics",
"level": "L1",
"dump_path": "/workspace/msprobe_dump",
"statistics": {
"list": []
}
}

启动服务提示错误：
(Worker_DP0_TP3_EP3 pid=52876) INFO 06-29 08:18:31 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP0_EP0 pid=52770) INFO 06-29 08:18:32 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP5_EP5 pid=52964) INFO 06-29 08:18:32 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP4_EP4 pid=52920) INFO 06-29 08:18:32 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP6_EP6 pid=53008) INFO 06-29 08:18:33 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP7_EP7 pid=53052) INFO 06-29 08:18:33 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP2_EP2 pid=52832) INFO 06-29 08:18:33 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP1_EP1 pid=52790) INFO 06-29 08:18:33 [compiler_interface.py:207] enable_npugraph_ex is enabled, which will bring graph compilation optimization.
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/_utils/error_code.py", line 43, in wapper
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return func(*args, **kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/npu_fx_compiler.py", line 574, in call
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return self._get_compiled_gm(gm, example_inputs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/npu_fx_compiler.py", line 598, in _get_compiled_gm
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return _CompiledFxGraph(self._gen_compiled_gm(gm, example_inputs, observer), self.config)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/npu_fx_compiler.py", line 635, in _gen_compiled_gm
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] optimized_gm = _optimize_fx(mutable_gm, self.config, observer)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/npu_fx_compiler.py", line 320, in _optimize_fx
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] observer.apply_gm_pass(_apply_pattern_passes, "apply_pattern_passes")
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/_utils/graph_transform_observer.py", line 41, in apply_gm_pass
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] res = pass_fn(self.gm, self.example_inputs, self.config)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/_utils/graph_utils.py", line 230, in wrapper
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return func(*args, **kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/patterns/pattern_util.py", line 29, in _apply_pattern_passes
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] _register_addrmsnormquant_patterns(pattern_pass_manager)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/patterns/add_rms_norm_quant.py", line 229, in _register_addrmsnormquant_patterns
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] _register_addrmsnormquant_pattern_default(pattern_pass_manager, div_mode=True)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/patterns/add_rms_norm_quant.py", line 142, in _register_addrmsnormquant_pattern_default
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] example_inputs=_get_inputs(),
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/dynamo/npugraph_ex/patterns/add_rms_norm_quant.py", line 79, in _get_inputs
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] zero_points = torch.tensor(gammashape, dtype=torch.int32) # INT32、BFLOAT16
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/msprobe/pytorch/aclgraph_dumper.py", line 116, in torch_dispatch
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] collected = self._dumper._dc(
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/msprobe/pytorch/aclgraph_dumper.py", line 304, in _dc
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/msprobe/pytorch/aclgraph_dumper.py", line 477, in _collect
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] acl_stat(stat_tensor, tag)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/msprobe/pytorch/aclgraph_dump/init.py", line 53, in acl_stat
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return torch.ops.my_ns.acl_stat(x, tag)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_ops.py", line 1209, in call
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/utils/_stats.py", line 29, in wrapper
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 1397, in torch_dispatch
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return self.dispatch(func, types, args, kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 2155, in dispatch
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return self._cached_dispatch_impl(func, types, args, kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 1510, in _cached_dispatch_impl
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] return self._dispatch_impl(func, types, args, kwargs)
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 2451, in _dispatch_impl
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] (flat_args, flat_arg_fake_tensors) = self.validate_and_convert_non_fake_tensors(
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 2909, in validate_and_convert_non_fake_tensors
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] validated_args = [validate(a) for a in flat_args]
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 2909, in
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] validated_args = [validate(a) for a in flat_args]
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] ^^^^^^^^^^^
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] File "/usr/local/python3.11.15/lib/python3.11/site-packages/torch/_subclasses/fake_tensor.py", line 2897, in validate
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] raise AssertionError(
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] AssertionError: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in my_ns.acl_stat.default(tensor([...], size=(1,), dtype=torch.int32), 'Module.root.GlmMoeDsaForCausalLM.Aten.lift_fresh.input.msprobe_fwd_start.0')
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962]
(Worker_DP0_TP3_EP3 pid=52876) ERROR 06-29 08:18:44 [multiproc_executor.py:962] During handling of the above exception, another exception occurred: