现象补充：A2场景 cann8.5.0 vllm0.18 性能劣化；cann9.0.0，vllm0.21.rc1性能正常；
先尝试vllm0.18镜像内手动装cann9.0.0测试一下性能
同样的cann，vllm0.21性能正常和0.18性能不一样，怀疑框架侧有影响store传输接口的操作，尝试打profiling
打profiling方法，在vllm-ascend调用store接口的地方，包上with torch_npu.profiler.profile：
import os, torch_npu
rank = os.environ.get("RANK", "0") # 多 worker 分目录，防互相覆盖
with torch_npu.profiler.profile(
activities=[
torch_npu.profiler.ProfilerActivity.CPU,
torch_npu.profiler.ProfilerActivity.NPU,
],
record_shapes=False,
experimental_config=torch_npu.profiler._ExperimentalConfig(
aic_metrics=torch_npu.profiler.AiCMetrics.PipeUtilization,
profiler_level=torch_npu.profiler.ProfilerLevel.Level1,
l2_cache=False,
data_simplification=True,
export_type=[
torch_npu.profiler.ExportType.Text
],
),
on_trace_ready=torch_npu.profiler.tensorboard_trace_handler(
f"./profiling_data/rank{rank}"),
) as prof:
store.batch_get_into_multi_buffers(keys, addrs, sizes)
vllm-ascend 0.18+cann9.0.0的profiling图
16e07966c695444fb87cb64adc6d280a.png
vllm-ascend 0.21+cann9.0.0的profiling图
c90fc958d81b4ccdb0b0b409412e7ec6.png
明显可以看到，0.18对应的profiling上多了stream39 40 43 46,有长时间的notify wait和EVENT_WAIT_SQE在做流间的等待，因此怀疑是vllm-ascend执行时其他流影响了传输接口的执行
因此在store.batch_get_into_multi_buffers前加上torch.npu.synchronize()的流同步操作，等其他流执行完再执行传输，再打profiling（这时候打profiling要把添加的流同步操作放在外面，别放在with profiler的范围内），传输接口耗时明显缩短，接口执行内部硬件上不再有长时间的等待操作
b88573ce8b724451856debb96ffd393b.png
到这一步，能排除传输接口本身的问题，是框架层面其他流的影响导致，需要框架层面进行分析。
vllm 0.18.0 开启mooncake池化功能后，TTFT性能劣化严重
PD混合部署：moocake 配置
{
"local_hostname": "10.44.190.117",
"metadata_server": "P2PHANDSHAKE",
"protocol": "ascend",
"device_name": "",
"master_server_address": "10.44.190.116:50088",
"global_segment_size": "10 GB"
}
主节点启动脚本：
#baseline
source /usr/local/Ascend/cann-9.0.0/set_env.sh
#source /usr/local/Ascend/nnal/atb/set_env.sh
nic_name="eth8"
local_ip="10.44.190.116"
node0_ip="10.44.190.116"
export HCCL_IF_IP=localipexportGLOOSOCKETIFNAME=nic_name
export TP_SOCKET_IFNAME=nicnameexportHCCLSOCKETIFNAME=nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export ASCEND_PROCESS_LOG_PATH=/workspace/logs
#export VLLM_LOGGING_LEVEL=DEBUG
set Ascend store
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export MOONCAKE_CONFIG_PATH="/data_test/zrz/mooncake.json"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1
export VLLM_ENGINE_READY_TIMEOUT_S=1800
export HCCL_RDMA_TIMEOUT=17
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export ASCEND_BUFFER_POOL=4:8
#export HCCL_INTRA_ROCE_ENABLE=1
#export ASCEND_GLOBAL_LOG_LEVEL=1
vllm serve /data_test/Qwen3-235B-A22B-W8A8
--host 0.0.0.0
--port 8000
--data-parallel-size 2
--api-server-count 2
--data-parallel-size-local 1
--data-parallel-address $local_ip
--data-parallel-rpc-port 13389
--seed 1024
--served-model-name qwen3
--quantization ascend
--tensor-parallel-size 8
--enable-expert-parallel
--max-num-seqs 16
--max-model-len 40960
--max-num-batched-tokens 4096
--trust-remote-code
--async-scheduling
--gpu-memory-utilization 0.9
--kv-transfer-config '{"kv_connector":"AscendStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"lookup_rpc_port":"0","backend":"mooncake"}}'
测试脚本：
vllm bench serve
--backend openai-chat
--dataset-name prefix_repetition
--num_prompts 50
--tokenizer "/data_test/Qwen3-235B-A22B-W8A8"
--ignore-eos
--model qwen3
--host 0.0.0.0
--port 8000
--endpoint /v1/chat/completions
--max-concurrency 16
--request-rate inf
--prefix-repetition-prefix-len 22610
--prefix-repetition-suffix-len 10158
--prefix-repetition-num-prefixes 8
--prefix-repetition-output-len 1
--percentile-metrics "ttft,tpot,itl,e2el"
--metric-percentiles "25,50,75,99"
性能劣化：
vllm原生TTFT基线为：38513.98 ms
VLLM+mooncake为：60939.35
打印日志发现
mooncake后端命中后单次batch_get_into_multi_buffers耗时为 s级别，搬运耗时异常，造成ttft劣化严重