已关闭
[Bug-Report|缺陷反馈]: A2A3 alltoall执行超时 #834
hu-yiliang11创建于  7 天前关闭于  2 天前
hu-yiliang11
7 天前 创建

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

alltoallv_continuous_pipeline算法报错

Environment / 环境信息 (Mandatory / 必填)

A2/A3 CANN9.1.0

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

双机执行alltoall,CCLBUFFER大于80M,跨机跑RDMA

Describe the expected behavior / 预期结果 (Mandatory / 必填)

通信算子执行无报错

[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.009 [alltoallv_continuous_pipeline.cc:896] [17440][AlltoallvContinuousPipeline][WaitValueOfRank] Waiting for the value of rank[5] timed out.
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.049 [alltoallv_continuous_pipeline.cc:918] [17440][WaitAndCalReceiveInfo]call trace: hcclRet -> 9
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.060 [alltoallv_continuous_pipeline.cc:1012] [17440][RunAsync]call trace: hcclRet -> 9
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.067 [coll_all_to_all_v_pipeline_for_910_93_executor.cc:247] [17440][KernelRun]call trace: hcclRet -> 9
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.096 [coll_all_to_all_v_pipeline_for_910_93_executor.cc:188] [17440][CollAlltoAllVPipelineFor91093][Orchestrate]errNo[0x0000000005000009]executor run failed
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.110 [aicpu_communicator.cc:3539] [17440][HcclCommAicpu][Orchestrate]executor process failed algName[RunAlltoAllVPipelineFor91093], ret = 9
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.119 [aicpu_communicator.cc:2570] [17440]OrchestrateHcclOp failed, ret:9
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.139 [aicpu_communicator.cc:2061] [17440][HcclCommAicpu][ExecOp] executor op fail, tag[ALLTOALL_172.16.19.110%eth0_64000_0_1786672840926636RunAlltoAllVPipelineFor91093_device], algName[RunAlltoAllVPipelineFor91093], identifier[172.16.19.110%eth0_64000_0_1786672840926636]
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.162 [aicpu_hccl_process.cc:583] [17440][AicpuHcclProcess][AicpuRunRpcServerV2] newTag[ALLTOALL_172.16.19.110%eth0_64000_0_1786672840926636RunAlltoAllVPipelineFor91093_device] algName[RunAlltoAllVPipelineFor91093]
[ERROR] HCCL(17424,aicpu_scheduler):2026-08-14-10:39:49.604.180 [hccl_aicpu_interface.cc:114] [17440]run AicpuRunRpcServerV2 failed. ret[9]
[ERROR] CCECPU(17424,aicpu_scheduler):2026-08-14-10:39:49.604.192 [ae_kernel_lib_aicpu_kfc.cpp:166][TransformKernelErrorCode][tid:17440][AICPU_PROCESSER] KFC RunAicpuRpcSrvLaunchV2 run failed ret[9]

Special notes for this issue/备注 (Optional / 选填)

likedislike
Leewis成员
7 天前 评论:

/assign @gcw_NcEfY7mt

likedislike
CANN-robotCANN-robot成员
7 天前 将 gcw_NcEfY7mt 设为负责人
LLeewis成员
6 天前 关联了看板:HCOMM
Leewis成员
2 天前 评论:

alltoallv_continuous_pipeline算法报错是由于超时时间配置不妥导致,相关PR已修改合入;

likedislike
LLeewis成员
2 天前 issue类型由 任务 改变为 缺陷
LLeewis成员
2 天前 issue状态由 待办的 改变为 已完成
LLeewis成员
2 天前 关闭了 issue
CANN-robotCANN-robot成员
2 天前 添加了label:resolved