非量化版本运行正常。命令差异：
1、模型为非量化模型，https://huggingface.co/Qwen/Qwen3-Reranker-8B
2、启动命令中去掉 --quantization ascend
您好 可以提供一下您量化后权重目录中的config.json文件嘛
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.3",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151669
}
我这边帮您简单询问了一下vllm-ascend的同事，qwen3-8b-reranker模型量化权重，vllm-ascend还没适配，需要适配
具体您可以为vllm提issue询问
vllm-ascend:https://github.com/vllm-project/vllm-ascend/issues
NPU：910b4
镜像：http://cr.rnd.huawei.com/images/vllm-ascend/v0.12.0rc1.tar.gz
量化：msmodelslim quant --model_type Qwen3-8B --model_path qwen3_reranker_8b_open --save_path qwen3_reranker_8b_open_w8a8 --device npu --quant_type w8a8 --trust_remote_code True
启动脚本：
export LOG_LEVEL=DEBUG
export HCCL_OP_EXPANSION_MODE=AIV
#export VLLM_PLUGINS=ascend
#export USE_ACLGRAPH=1
#export CPU_AFFINITY_CONF=0
#export LD_PRELOAD=/usr/local/lib/libjemalloc.so.2:${LD_PRELOAD}
export ASCEND_RT_VISIBLE_DEVICES=1
export VLLM_USE_V1=1
export MODEL_PATH=qwen3_reranker_8b_open_w8a8
export MODEL_NAME=Qwen3-Reranker-8B
export TORCH_DEVICE_BACKEND_AUTOLOAD=0
python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8006
--model ${MODEL_PATH}
--task score
--hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
--max-num-seqs=256
--max-model-len=32768
--max-num-batched-tokens=32768
--tensor-parallel-size=1
--block-size=128
--gpu-memory-utilization=0.9
--trust-remote-code
--quantization ascend
> tmp2.log &
报错信息1：quant_model_description.json文件中加 "score.weight" = "FLOAT" 解决
afcf7fa2ce734f028f71884898b6f60a.png
报错信息2：解决不了
27200e1200e34704af15946ec21f79a3.png