Closed
[Bug-Report|缺陷反馈]: vllm ascend框架自定义算子GroupedMatmulSwigluQuantWeightNzV2整网dsv4推理时遇到coredump问题 #305
shen_lan1234Add on  Aug 10Closed at  Aug 10
shen_lan1234
shen_lan1234
Aug 10 Create

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

vllm ascend框架自定义算子GroupedMatmulSwigluQuantWeightNzV2整网dsv4推理时遇到coredump问题;

堆栈:
#0 0x00007f68f9bcc800 in ?? () from /usr/lib64/libc.so.6
#1 0x00007f68aef8392c in op::internal::operator==(op::internal::OpCacheKey const&, op::internal::OpCacheKey const&) () from /usr/local/Ascend/cann-9.0.0/lib64/libnnopbase.so
#2 0x00007f68aef88357 in op::internal::OpExecCacheManager::AddOpExecCache(op::internal::OpExecCache*) () from /usr/local/Ascend/cann-9.0.0/lib64/libnnopbase.so
#3 0x00007f68aefd66a0 in op::OpExecutorImpl::AddCache() () from /usr/local/Ascend/cann-9.0.0/lib64/libnnopbase.so
#4 0x00007f67ac0f6841 in gmm_dsq::GroupedMatmulSwigluQuantHandler::Process() ()
from /usr/local/python3.11.15/lib/python3.11/site-packages/vllm_ascend/_cann_ops_custom/vendors/custom_transformer/op_api/lib/libcust_opapi.so
#5 0x00007f67ac0f3d86 in ?? () from /usr/local/python3.11.15/lib/python3.11/site-packages/vllm_ascend/_cann_ops_custom/vendors/custom_transformer/op_api/lib/libcust_opapi.so
#6 0x00007f67ac0f4e3e in aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize ()
from /usr/local/python3.11.15/lib/python3.11/site-packages/vllm_ascend/_cann_ops_custom/vendors/custom_transformer/op_api/lib/libcust_opapi.so
#7 0x00007f67ee30038c in ?? () from /usr/local/python3.11.15/lib/python3.11/site-packages/vllm_ascend/vllm_ascend_C.cpython-311-x86_64-linux-gnu.so
#8 0x00007f67ee369684 in ?? () from /usr/local/python3.11.15/lib/python3.11/site-packages/vllm_ascend/vllm_ascend_C.cpython-311-x86_64-linux-gnu.so
#9 0x00007f67ee3416f5 in ?? () from /usr/local/python3.11.15/lib/python3.11/site-packages/vllm_ascend/vllm_ascend_C.cpython-311-x86_64-linux-gnu.so
#10 0x00007f68e948e98c in c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocatorc10::IValue >) const [clone .isra.0] ()
from /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007f68f8825b34 in torch::jit::invokeOperatorFromPython(c10::ArrayRef<std::shared_ptrtorch::jit::Operator >, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) ()
from /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libtorch_python.so
#12 0x00007f68f8826038 in torch::jit::_get_operation_for_overload_or_packet(c10::ArrayRef<std::shared_ptrtorch::jit::Operator >, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) () from /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libtorch_python.so
#13 0x00007f68f8826348 in torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) () from /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libtorch_python.so
#14 0x00007f68f870b9b2 in pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object
)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char

Environment / 环境信息 (Mandatory / 必填)

环境:A2 A+X
版本:cann 9.0.0
场景:vllm ascend自定义算子框架接入aclnn算子,推理部署dsv4 pro模型,偶现coredump

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

vllm ascend自定义算子框架接入aclnn算子,推理部署dsv4 pro模型

Describe the expected behavior / 预期结果 (Mandatory / 必填)

推理服务正常运行,不会崩溃;

无

Special notes for this issue/备注 (Optional / 选填)

likedislike
ren-botaoCollaborators
Aug 10 Comments:

/assign @ren-botao

likedislike
CANN-robotCANN-robotCollaborators
Aug 10 Assign ren-botao as responsible person
ren-botaoCollaborators
Aug 10 Comments:

场景分析:
aclnnGroupedMatmulSwigluQuantWeightNzV2算子,pytorch适配层是自定义的,公共的适配宏为 EXEC_NPU_CMD, 位于 vllm-ascend仓 csrc/aclnn_torch_adapter/op_api_common.h 文件中,aclnn算子一二阶段在两个线程中运行。
第一阶段在调用算子的 Worker 主线程同步执行:
auto workspace_status = call(getWorkspaceSizeFunc, converted_params);
第二阶段被封装为 lambda:
auto acl_call = \[\] () -> int { auto api_ret = opApiFunc(workspace_addr, workspace_size, executor, acl_stream); }; OpCommand cmd; cmd.Name(...); cmd.SetCustomHandler(acl_call); cmd.Run();
torch_npu ACL 消费线程稍后调用该 handler。

问题根因分析:
aclnn算子一阶段正在执行查找key匹配key操作:
bool operator==(const OpCacheKey& lhs, const OpCacheKey& rhs) { *** return lhs.len == key.len && (memcmp(PtrCastTo<char>(lhs.buf), PtrCastTo<char>(key.buf), lhs.len) == 0); }
二阶段线程并发执行 MarkOpCacheInvalid 操作:
if (key_.buf) { delete[] key_.buf; key_.buf = nullptr; } key_.len = 0;
key.buf被值nullptr,但是key.len还没有被置0,一阶段线程memcpy操作,传入了nullptr指针,导致coredump

likedislike
CANN-robotCANN-robotCollaborators
Aug 10 Close issue
CANN-robotCANN-robotCollaborators
Aug 10 Added label: resolved
Rren-botaoCollaborators
Aug 10 Associated pull request
Rren-botaoCollaborators
Aug 10 Associated pull request
kknankknan
Aug 25 Associated pull request
eieieieieiei
16 days ago Associated pull request
eieieieieiei
14 days ago Associated pull request