已关闭
[Bug-Report|缺陷反馈]: A3单边通信CQ overflow #822
hu-yiliang11创建于  28 天前关闭于  27 天前
hu-yiliang11
28 天前 创建

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

使用CANN9.1.0版本,单边通信的时候CQ overflow,报错流同步超时

Environment / 环境信息 (Mandatory / 必填)

A3
海思内部版本:CANN 9.1.0

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

deeps v4 flash+vllm-ascend 0.26.0版本 + pd分离+hixl

Describe the expected behavior / 预期结果 (Mandatory / 必填)

正常运行,CQ不会被打爆

[ERROR] RUNTIME(188164,):2026-08-25-13:13:47.019.026 [stream.cc:1878]195997 SynchronizeExecutedTask:Stream synchronize timeout, device_id=2, stream_id=216, timeout=100000ms.
[ERROR] RUNTIME(188164,):2026-08-25-13:13:47.019.062 [stream.cc:1944]195997 SynchronizeImpl:failed, stream_id=216, error=0x7030010
[ERROR] RUNTIME(188164,):2026-08-25-13:13:47.023.593 [api_impl.cc:1664]195997 StreamSynchronize:Stream synchronize timeout, the current task is type_name=NOTIFY_WAIT, device_id=2, stream_id=216, task_id=679, task_type=14
[ERROR] RUNTIME(188164,):2026-08-25-13:13:47.023.628 [api_error.cc:1101]195997 StreamSynchronize:Stream synchronize failed, stream_id=216, timeout=100000ms.
[ERROR] RUNTIME(188164,):2026-08-25-13:13:47.023.641 [api_c_stream.cc:180]195997 rtStreamSynchronizeWithTimeout:ErrCode=507046, desc=[stream sync timeout], InnerCode=0x7030010
[ERROR] RUNTIME(188164,):2026-08-25-13:13:47.023.643 [error_message_manage.cc:66]195997 FuncErrorReason:rtStreamSynchronizeWithTimeout execution failed.
[ERROR] GE(188164,):2026-08-25-13:13:47.036.220 [comm_channel.cc:516][HIXL] 195997 operator(): ErrorNo: 103901 Call aclrt api failed, ret: 0x7BCA6
[ERROR] GE(188164,):2026-08-25-13:13:47.053.759 [comm_channel.cc:533][HIXL] 195997 TransferSync: ErrorNo: 103901 TransferSync failed.
[ERROR] GE(188164,):2026-08-25-13:13:47.053.764 [adxl_inner_engine.cc:506][HIXL] 195997 TransferSync: ErrorNo: 103901 Failed to transfer sync,

黑匣子日志:
exceptionId 0xa4302003
dev 0x0
arg 0x0
core 0x18
eType 0x3a
module ABNM_CQ_CQE_Err
desc Get Abnormal CQ Error.

[ERROR] KERNEL(13095,sklogd):2026-08-25-21:12:07.010.139 [klogd.c:247][3981956.323356] hns3 0002:79:00.0 hns_0: local work queue 0x172 catast error, sub_event type is: 244 SUBSYSTEM=pci DEVICE=+pci:0002:79:00.0

network info: RoCE Get Abnormal CQ Error

Special notes for this issue/备注 (Optional / 选填)

likedislike
LLeewis成员
28 天前 修改了issue 的描述
Leewis成员
28 天前 评论:

A3单边通信CQ overflow 问题跟踪;

likedislike
Leewis成员
28 天前 评论:

/assign @gcw_NcEfY7mt

likedislike
CANN-robotCANN-robot成员
28 天前 将 gcw_NcEfY7mt 设为负责人
LLeewis成员
28 天前 issue类型由 任务 改变为 缺陷
Leewis成员
27 天前 评论:

针对上述问题,对应PR已经合入,相关链接:https://gitcode.com/cann/hcomm/pull/4953

likedislike
LLeewis成员
27 天前 issue状态由 待办的 改变为 已完成
LLeewis成员
27 天前 关闭了 issue
CANN-robotCANN-robot成员
27 天前 添加了label:resolved
LLeewis成员
27 天前 关联了看板:HCCL
LLeewis成员
27 天前 移除了看板:HCCL