Star183
193
Code
Code
Issues5
Pull Requests9
Actions
Discussion
Wiki
Analysis
Star183
193
Closed
quant_qwen_moe_w8a8.py 量化出的qwen3 30b a3b vllm 无法使用 #288
Z字君Add on  Sep 7, 2025Closed at  Oct 27, 2025
Z字君
Z字君
Sep 7, 2025 Create

容器：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.1.RC1-800I-A2-py311-openeuler24.03-lts
执行量化：

python3 quant_qwen_moe_w8a8.py --model_path /root/model/Qwen3-Coder-30B-A3B-Instruct --save_path /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8 --trust_remote_code True

使用vllm-ascend（直接拉取的官方镜像：quay.io/ascend/vllm-ascend:v0.9.2rc1）直接运行原始模型是可以的，

vllm serve /root/model/Qwen3-Coder-30B-A3B-Instruct --host 0.0.0.0 --port 8000  --max-model-len 65536 --served-model-name qwen3-30b-a3b --tensor-parallel-size 8 --gpu-memory-utilization 0.8

但是运行量化后的就不行

vllm serve /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8 --host 0.0.0.0 --port 8000  --max-model-len 65536 --served-model-name qwen3-30b-a3b --tensor-parallel-size 8 --gpu-memory-utilization 0.8

报错信息：

(VllmWorker rank=0 pid=12366) INFO 09-07 09:05:28 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_9b22b065'), local_subscribe_addr='ipc:///tmp/c2f56ebd-632f-43fe-90c5-2f098a970e0e', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=12366) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker rank=1 pid=12367) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
(VllmWorker rank=2 pid=12368) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
(VllmWorker rank=4 pid=12370) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
(VllmWorker rank=3 pid=12369) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 3 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
(VllmWorker rank=5 pid=12371) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
(VllmWorker rank=6 pid=12372) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
(VllmWorker rank=7 pid=12373) INFO 09-07 09:05:28 [parallel_state.py:1076] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
(VllmWorker rank=7 pid=12373) INFO 09-07 09:05:31 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=4 pid=12370) INFO 09-07 09:05:31 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=0 pid=12366) INFO 09-07 09:05:31 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=5 pid=12371) INFO 09-07 09:05:31 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=3 pid=12369) INFO 09-07 09:05:31 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=2 pid=12368) INFO 09-07 09:05:31 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=1 pid=12367) INFO 09-07 09:05:32 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
(VllmWorker rank=6 pid=12372) INFO 09-07 09:05:32 [model_runner_v1.py:1745] Starting to load model /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8...
Loading safetensors checkpoint shards:   0% Completed | 0/8 [00:00<?, ?it/s]
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487] WorkerProc failed to start.
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487] Traceback (most recent call last):
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 461, in worker_main
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 358, in __init__
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.worker.load_model()
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 240, in load_model
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.model_runner.load_model()
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1748, in load_model
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 59, in get_model
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     return loader.load_model(vllm_config=vllm_config,
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 41, in load_model
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.load_weights(model, model_config)
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/default_loader.py", line 269, in load_weights
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     loaded_weights = model.load_weights(
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 541, in load_weights
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     return loader.load_weights(weights)
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     yield from self._load_module(prefix,
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     loaded_params = module_load_weights(weights)
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 446, in load_weights
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     param = params_dict[name]
(VllmWorker rank=7 pid=12373) ERROR 09-07 09:05:33 [multiproc_executor.py:487] KeyError: 'layers.26.mlp.experts.w2_weight_offset'
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487] WorkerProc failed to start.
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487] Traceback (most recent call last):
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 461, in worker_main
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 358, in __init__
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.worker.load_model()
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 240, in load_model
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.model_runner.load_model()
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1748, in load_model
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 59, in get_model
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     return loader.load_model(vllm_config=vllm_config,
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 41, in load_model
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     self.load_weights(model, model_config)
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/default_loader.py", line 269, in load_weights
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     loaded_weights = model.load_weights(
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 541, in load_weights
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     return loader.load_weights(weights)
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     yield from self._load_module(prefix,
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     loaded_params = module_load_weights(weights)
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 446, in load_weights
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487]     param = params_dict[name]
(VllmWorker rank=5 pid=12371) ERROR 09-07 09:05:33 [multiproc_executor.py:487] KeyError: 'layers.26.mlp.experts.w2_weight_offset'
Loading safetensors checkpoint shards:   0% Completed | 0/8 [00:02<?, ?it/s]
(VllmWorker rank=0 pid=12366)
ERROR 09-07 09:05:37 [core.py:586] EngineCore failed to start.
ERROR 09-07 09:05:37 [core.py:586] Traceback (most recent call last):
ERROR 09-07 09:05:37 [core.py:586]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core
ERROR 09-07 09:05:37 [core.py:586]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 09-07 09:05:37 [core.py:586]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__
ERROR 09-07 09:05:37 [core.py:586]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 09-07 09:05:37 [core.py:586]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 75, in __init__
ERROR 09-07 09:05:37 [core.py:586]     self.model_executor = executor_class(vllm_config)
ERROR 09-07 09:05:37 [core.py:586]   File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 53, in __init__
ERROR 09-07 09:05:37 [core.py:586]     self._init_executor()
ERROR 09-07 09:05:37 [core.py:586]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 93, in _init_executor
ERROR 09-07 09:05:37 [core.py:586]     self.workers = WorkerProc.wait_for_ready(unready_workers)
ERROR 09-07 09:05:37 [core.py:586]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 422, in wait_for_ready
ERROR 09-07 09:05:37 [core.py:586]     raise e from None
ERROR 09-07 09:05:37 [core.py:586] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 590, in run_engine_core
    raise e
  File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
  File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 75, in __init__
    self.model_executor = executor_class(vllm_config)
  File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 93, in _init_executor
    self.workers = WorkerProc.wait_for_ready(unready_workers)
  File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 422, in wait_for_ready
    raise e from None
Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Traceback (most recent call last):
  File "/usr/local/python3.10.17/bin/vllm", line 8, in <module>
    sys.exit(main())
  File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 65, in main
    args.dispatch_function(args)
  File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 55, in cmd
    uvloop.run(run_server(args))
  File "/usr/local/python3.10.17/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/python3.10.17/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1431, in run_server
    await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1451, in run_server_worker
    async with build_async_engine_client(args, client_config) as engine_client:
  File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
  File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 162, in from_vllm_config
    return cls(
  File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 124, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 96, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 666, in __init__
    super().__init__(
  File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 403, in __init__
    with launch_core_engines(vllm_config, executor_class,
  File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 434, in launch_core_engines
    wait_for_engine_startup(
  File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
[ERROR] 2025-09-07-09:05:39 (PID:12086, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
root@afaf11cfd9de:/workspace# /usr/local/python3.10.17/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
likedislike
A
anreywmhCollaborators
Sep 23, 2025 Comments：

您好 请尝试vllm 拉量化模型时加上--quantization ascend
image.png
https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_npu_quantization.html

likedislike
A
anreywmhCollaborators
Sep 30, 2025 Comments：

尊敬的用户，您好！由于工单长时间未收到您的响应，现暂为关闭。感谢您对 msModelSlim 开源生态的贡献！后续如有疑问，欢迎随时重新提问。

likedislike
Z字君
Z字君
Oct 10, 2025 Comments：

@anreywmh
仍然报错：

File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/acl_graph.py", line 110, in __call__
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self.runnable(*args, **kwargs)
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/vllm-workspace/vllm/vllm/compilation/cuda_piecewise_backend.py", line 90, in __call__
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self.compiled_graph_for_general_shape(*args)
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self._wrapped_call(self, *args, **kwargs)
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 406, in __call__
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     raise e
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 393, in __call__
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self._call_impl(*args, **kwargs)
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return forward_call(*args, **kwargs)
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "<eval_with_key>.3", line 21, in forward
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     moe_forward = torch.ops.vllm.moe_forward(view_2, linear, 'model.layers.0.mlp.experts');  view_2 = linear = None
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in __call__
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self._op(*args, **(kwargs or {}))
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/vllm-workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1738, in moe_forward
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self.forward_impl(hidden_states, router_logits)
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/common_fused_moe.py", line 496, in forward_impl
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     final_hidden_states = self.quant_method.apply(
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]                           ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/vllm-workspace/vllm-ascend/vllm_ascend/quantization/quant_config.py", line 333, in apply
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self.quant_method.apply(
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/vllm-workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 390, in apply
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return unified_fused_experts(
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/common_fused_moe.py", line 99, in fused_experts
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     gate_up_output = torch_npu.npu_grouped_matmul(
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in __call__
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]     return self._op(*args, **(kwargs or {}))
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596] RuntimeError: npu_grouped_matmul:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:380 NPU function error: call aclnnGroupedMatmulV5 failed, error code is 161002
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596] [ERROR] 2025-10-10-02:45:07 (PID:3939, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596] EZ1001: [PID: 3939] 2025-10-10-02:45:07.052.538 Tensor tensor expected dtype is DT_BFLOAT16 but found DT_INT32.
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]         TraceBack (most recent call last):
(VllmWorker TP0 pid=3939) ERROR 10-10 02:45:07 [multiproc_executor.py:596]         GMM: y dtype does not match with required dtype[DT_BFLOAT16].

命令：

 vllm serve /root/model/Qwen3-Coder-30B-A3B-Instruct-Int8 --host 0.0.0.0 --port 8000  --max-model-len 65536 --served-model-name qwen3-30b-a3b --tensor-parallel-size 8 --gpu-memory-utilization 0.8  --quantization ascend
likedislike
AAiBoGeCollaborators
Oct 20, 2025 changed custom state from TODO to DONE
AAiBoGeCollaborators
Oct 20, 2025 Close issue
Z字君Z字君
Oct 24, 2025 changed custom state from DONE to TODO
Z字君Z字君
Oct 24, 2025 Reopen issue
A
anreywmhCollaborators
Oct 24, 2025 Comments：

您好 量化成功没问题的话，需要按照vllm中部署文档部署量化模型
如果依旧报错的话，请通过昇腾社区向vllm服务提工单咨询
https://www.hiascend.com/zh
image.png

likedislike
A
anreywmhCollaborators
Oct 27, 2025 Comments：

尊敬的用户，您好！由于工单长时间未收到您的响应，现暂为关闭。感谢您对 msModelSlim 开源生态的贡献！后续如有疑问，欢迎随时重新提问

likedislike
AanreywmhCollaborators
Oct 27, 2025 changed custom state from TODO to DONE
AanreywmhCollaborators
Oct 27, 2025 Close issue