Merged
fix(presmoke): 自动探测的卡号折成可见设备下标,修复 A5 冒烟批量 SIGSEGV #7366
fix(presmoke): 自动探测的卡号折成可见设备下标,修复 A5 冒烟批量 SIGSEGV #7366
Merged
zhangyujia77Add on 12 days ago
zhangyujia77
zhangyujia77Collaborators
12 days ago

现象

上游 run#7680(PR !7338,一个纯空行改动)的 PreSmoke_A5 750 例里 703 例失败,rc 分布 139(SIGSEGV) 358 个、134(SIGABRT) 200 个、1 141 个,连 add、matmul_basic_api 这种最基础的样例都挂。

根因

plog 第一现场(slog_dav-3510.tar.gz → log/debug/plog/):

GetVisibleDevices: set ASCEND_RT_VISIBLE_DEVICES:[6] error, input data range[0-4)
ChgUserDevIdToDeviceId: input userDevId:0 is err, userDeviceCnt:0, real deviceCnt:4
    isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[6], available visible devices:[]
rtSetDevice: ErrCode=107001, desc=[device id error]
ReportContextValidationError: No valid context is bound to the current thread. ErrorCode=EE1018
rtStreamCreateWithFlags: ErrCode=107002, desc=[the context is a null pointer]

ASCEND_RT_VISIBLE_DEVICES 要的是运行时可见设备集合里的下标,范围恒为 [0, deviceCnt)。而 resolve_npu_cards 的自动探测拿到的是宿主机物理编号 —— /dev/davinciN 的 N,或 npu-smi 的 NPU id。两者只在「卡从 0 开始连续透传」时才相等。

A5 runner(npu=4)是从整机里挑 4 个设备节点透进容器的。挑到 davinci4..7 就全挂,挑到 davinci0..3 就全绿 —— 所以这是个随机复现、约一半概率的坑。本次 status.txt 里是 multi_card_start cards=4 5 6 7。

修复

自动探测的结果折成 0..N-1。显式传 NPU_CARDS 的走 normalize_card_list,行为不变。

分片不受影响:load_fixed_shards 是按 enumerate(cards) 的下标去找 card_{index}.txt 的,不认卡号本身(本次失败的那轮分片计数仍然是正确的 207/181/181/181)。

验证

davinci4..7        -> 0 1 2 3
davinci2,davinci5  -> 0 1
davinci0..3        -> 0 1 2 3   (行为不变)
NPU_CARDS=1,3      -> 1 3       (显式值不折)

附带发现(本单不改)

同一轮里 hello_world 日志写着 [ERROR] [Bisheng] Get kernel function failure! ret 107002,rc 却是 0 → 记成 PASS。所以设备全废时 pass/fail 计数仍会给出 25 个「通过」。光看计数会被骗,另开单跟。

likedislike
Pull Request has been successfully merged by CANN-robot.
(Thank you for zhangyujia77's contribution.)
zhangyujia77zhangyujia77Collaborators
12 days ago Created pull request, commit ee7806dd
atomgit-bot
atomgit-bot
12 days ago Comments:

🤖 AI Code Review

⚠️ 审查未能完成

本次代码审查未能完成,可能是临时性故障。

你可以在评论区输入 /ai summary 重新发起审查。若多次失败,请联系管理员并附上下方追踪 ID。

追踪 ID: task-476545

likedislike
Incorrect?
atomgit-bot
atomgit-bot
12 days ago Comments:

🤖 AI Code Review

⚠️ 审查未能完成

本次代码审查未能完成,可能是临时性故障。

你可以在评论区输入 /ai review 重新发起审查。若多次失败,请联系管理员并附上下方追踪 ID。

追踪 ID: task-476546

likedislike
Incorrect?
zhangyujia77
zhangyujia77Collaborators
12 days ago Comments:

compile

likedislike
CANN-robotCANN-robotCollaborators
12 days ago Added label: cann-cla/yes
zhangyujia77zhangyujia77Collaborators
12 days ago Pre merge succeeded (commit_id: b37e94e00f36e1127e6cad8651f7abca7090c08a)
CANN-robot
CANN-robotCollaborators
12 days ago Comments:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here.
You can get sig-info at here.
You can self-configure the PR merge rules for this repository. For more details, please refer to Here.
For more, you also can visit HICANN.


PR Approval Progress

✅ Congratulations! All modules have met the lgtm and approve requirements.

Module Approval Details

module lgtm status approve status
scripts ✅ 诸葛洵, hujiawen_kaven (2/2) ✅ 诸葛洵 (1/1)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

zhangyujia77, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
CANN-robotCANN-robotCollaborators
12 days ago Set zengjuan,zhanghao_0689,suqwe,kevin365,wulinyu,kong0808,dengjing_aoe,YeZZzzz1,yangbinhua,zhudi22,chenyiyuan,xun_zhuge,XuebinYang,sjtulxh,bianque,houyuyang,Lycheeeee,YANXI_ZHAO,chentianyu19,shi_nanxiang,wuyang_hw,jiangxinyu3,ascendhjh,hujiawen_kaven,wangxiaozhi,DragonBornHD84,louyuxuan,hyolee,chenma,yukangchen as approver
CANN-robotCANN-robotCollaborators
12 days ago Set kevin365,xun_zhuge as approver
CANN-robot
CANN-robotCollaborators
12 days ago Comments:
🚀 CI 流水线已启动
📋 执行详情: 点击查看流水线
likedislike
CANN-robotCANN-robotCollaborators
12 days ago Added label: ci-pipeline-running
zhugexun_sjtu
zhugexun_sjtuCollaborators
12 days ago Comments:

/approve

likedislike
CANN-robotCANN-robotCollaborators
12 days ago Added label: approved
6 event messages are collapsed here。 See more
CANN-robotCANN-robotCollaborators
12 days ago Removed label: approved
CANN-robot
CANN-robotCollaborators
12 days ago Comments:

Notice

New code changes of the pull request are detected and remove these labels approved. 😳

likedislike
zhugexun_sjtu
zhugexun_sjtuCollaborators
12 days ago Comments:

/approve

likedislike
CANN-robotCANN-robotCollaborators
12 days ago Added label: approved
hujiawen_kaven
hujiawen_kavenCollaborators
12 days ago Comments:

/approve
/lgtm

likedislike
CANN-robot
CANN-robotCollaborators
12 days ago Comments:

The following users do not have permission to comment /lgtm or /approve on any module in this PR:
/approve: hujiawen_kaven

likedislike
CANN-robotCANN-robotCollaborators
12 days ago Added label: lgtm
CANN-robotCANN-robotCollaborators
12 days ago Removed label: ci-pipeline-running
CANN-robotCANN-robotCollaborators
12 days ago Added label: ci-pipeline-passed
CANN-robotCANN-robotCollaborators
12 days ago Merged pull request