在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

• API 令牌或密钥
• 密码或身份验证凭证
• 私有网址或接口地址
• 个人或机密数据
• ...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

- 操作系统 openEuler 22.03 LTS
- 昇腾硬件信息 910B3
- CANN软件版本 8.1.RC1
- 安装的对应软件版本 vllm 0.11.0

🐛 问题描述

在vllm-ascend上加入msprobe数据采集代码后，在采集过程中报错：

(APIServer pid=1115863) INFO 01-13 06:25:46 [chat_utils.py:560] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1115863) INFO 01-13 06:25:56 [loggers.py:127] Engine 000: Avg prompt throughput: 1.4 tokens/s, Avg generation throughput: 3.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:25:58 (1116606) [rank 0] [INFO] msprobe: debugger.start() is set successfully
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:25:58 (1116606) [rank 0] [INFO] Dump switch is turned on at step 38. 
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:25:58 (1116606) [rank 0] [INFO] Dump data will be saved in /AIdata/JW/tmp20260112/dump_data/step38.
.(APIServer pid=1115863) INFO 01-13 06:26:06 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=1115863) INFO 01-13 06:26:16 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(EngineCore_DP0 pid=1116249) INFO 01-13 06:26:58 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:27:22 (1116606) [INFO] dump.json is at /AIdata/JW/tmp20260112/dump_data/step38. 
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:27:23 (1116606) [INFO] dump.json is at /AIdata/JW/tmp20260112/dump_data/step38. 
(EngineCore_DP0 pid=1116249) INFO 01-13 06:27:58 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:28:43 (1116606) [INFO] dump.json is at /AIdata/JW/tmp20260112/dump_data/step38. 
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:28:44 (1116606) [INFO] dump.json is at /AIdata/JW/tmp20260112/dump_data/step38. 
(EngineCore_DP0 pid=1116249) INFO 01-13 06:28:58 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:29:44 (1116606) [rank 0] [INFO] msprobe: debugger.stop() is set successfully. Please set debugger.start() to turn on the dump switch again. 
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:29:44 (1116606) [INFO] dump.json is at /AIdata/JW/tmp20260112/dump_data/step38. 
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:29:44 (1116606) [INFO] ****************************************************************************
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:29:44 (1116606) [INFO] *                        msprobe ends successfully.                        *
(Worker_TP0_EP0 pid=1116606) 2026-01-13 06:29:44 (1116606) [INFO] ****************************************************************************
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:45 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) INFO 01-13 06:29:46 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:46 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:47 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:48 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) ERROR 01-13 06:29:49 [detokenizer.py:239] Encountered invalid token id: -1082130432
(APIServer pid=1115863) ERROR 01-13 06:29:49 [detokenizer.py:239] Traceback (most recent call last):
(APIServer pid=1115863) ERROR 01-13 06:29:49 [detokenizer.py:239]   File "/vllm-workspace/vllm/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=1115863) ERROR 01-13 06:29:49 [detokenizer.py:239]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=1115863) ERROR 01-13 06:29:49 [detokenizer.py:239]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1115863) ERROR 01-13 06:29:49 [detokenizer.py:239] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=1115863) INFO 01-13 06:29:56 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=1115863) INFO 01-13 06:30:06 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(EngineCore_DP0 pid=1116249) INFO 01-13 06:30:49 [shm_broadcast.py:397] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation)
^C[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

欢迎加入社区，感谢您对社区的贡献 🎉!