Merged
fix op retry bug #4970
黄建波Add on Sep 1
fix op retry bug #4970
Merged
黄建波Add on Sep 1
黄建波
黄建波Collaborators
Sep 1

描述

batch_send_recv算子注入故障之后没有约束打印,aicpu侧等待stop命令等待超时报错。
修改方案: batch_send_recv 故障算子index不一致,无法重执行,增加约束打印

变更类型

请选择本次引入的变更类型:

关联的Issue

https://gitcode.com/cann/hcomm/issues/814

测试

已完成的测试用例和场景:

  1. 压测2机场景下batch_send_recv算子注入故障,batch_send_recv步长不一致时,看是否有约束打印

补充的UT用例:

文档更新

合入检查

likedislike
Pull Request has been successfully merged by CANN-robot.
(Thank you for 黄建波's contribution.)
黄建波黄建波Collaborators
Sep 1 Created pull request, commit f72a63e7
atomgit-bot
atomgit-bot
Sep 1 Comments:

变更摘要

本 PR 旨在修复算子重试(Op Retry)场景下的问题,改动集中在 src/legacy/ascend910/framework/cluster_maintenance/recovery/operator_retry/opretry_server.cc 中 OpRetryServerHandleError::ProcessEvent 方法的错误日志。原日志笼统地提示"超时"(OpRetryServerHandleError timeout),现改为更明确地说明"aicpu 无法重试,opname 不一致",使该分支的失败原因(操作名不一致导致无法重试)在日志中清晰可辨,便于问题定位。

主要改动

  • 错误日志信息修正:在 OpRetryServerHandleError::ProcessEvent 的超时分支中,将日志文案由 OpRetryServerHandleError timeout 替换为 aicpu can not retry, opname is inconsistent,准确反映该分支的真实失败原因(opname 不一致导致 aicpu 无法重试)。
  • 日志可观测性提升:新日志使运维人员无需排查上下文即可直接从错误信息判断为算子名不一致导致的重试失败,降低定位成本。
likedislike
Incorrect?
atomgit-bot
atomgit-bot
Sep 1 Comments:

代码审查

✅ 未发现问题

likedislike
Incorrect?
CANN-robotCANN-robotCollaborators
Sep 1 Added label: cann-cla/yes
CANN-robot
CANN-robotCollaborators
Sep 1 Comments:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here.
You can get sig-info at here.
You can self-configure the PR merge rules for this repository. For more details, please refer to Here.
For more, you also can visit HICANN.


PR Approval Progress

✅ Congratulations! All modules have met the lgtm and approve requirements.

Module Approval Details

module lgtm status approve status
src/legacy/ascend910/framework ✅ 陈浩, 杨邵华 (2/2) ✅ 陈浩 (1/1)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

muzhiyu6, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
CANN-robotCANN-robotCollaborators
Sep 1 Set leabclove,yanglianxiang2,linyf950,yanzhenghang,chenliang267,yanyefeng,chenhao_,dingweiqin_57,songmingyang,wenxuemin,yangxiaoyuan,laodazhao1,yangshaohua,chenke2026,zhongqin2020,bz_lee,Innoecho,chenyia,temper7620,zhangxp1030,lilin_137,hulk1997,ouyangxizi,one_random as approver
CANN-robotCANN-robotCollaborators
Sep 1 Set leabclove,yanyefeng,chenhao_,dingweiqin_57,wenxuemin as approver
黄建波
黄建波Collaborators
Sep 1 Comments:

compile

likedislike
黄建波黄建波Collaborators
Sep 1 Pre merge succeeded (commit_id: 797c9ea48e418116994a2c188faeff02026ee6ef)
CANN-robotCANN-robotCollaborators
Sep 1 Added label: ci-pipeline-running
黄建波黄建波Collaborators
Sep 1 Modify description of pull request
黄建波黄建波Collaborators
Sep 1 Modify description of pull request
黄建波黄建波Collaborators
Sep 1 Modify description of pull request
CANN-robotCANN-robotCollaborators
Sep 1 Removed label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
Sep 1 Added label: ci-pipeline-passed
yangshaohuaCollaborators
Sep 1 Comments:

/lgtm

likedislike
陈浩
陈浩Collaborators
Sep 1 Comments:

/approve

likedislike
CANN-robotCANN-robotCollaborators
Sep 1 Added label: lgtmapproved
CANN-robotCANN-robotCollaborators
Sep 1 Merged pull request