Closed
[Requirement|需求建议]: Mc2 DFX需求,发生aic error后触发exception dump #5252
WangShuyingAdd on  Sep 12Closed at  Sep 12
WangShuyingCollaborators
Sep 12 Create

Thanks for sending an requirement! Please fill in the following template to help quickly solve your problem.

Backgroud(背景信息)

alltoallmatmul、alltoallmatmulv2、allgathermatmulv2、allgathermatmulv3、matmulreducescatterv2算子mx场景增加exception dump,获取xn/cke 地址 dump、tiling 数据/workspace 逐段 D2H 拷贝落盘、通信上下文与 peermem 窗口分段 dump和kernel 侧运行时进度打点信息

Benefit / Necessity (价值/作用)

支撑融合算子本身问题进一步的问题定位;

Design(设计方案)

算子DFX实现主要依赖CANN 自带的DFX能力,其提供了异常时的回调注册机制
tiling 数据结构中新增 workspace/peermem 段布局信息(WorkspaceLayoutInfo、PeermemLayoutInfo)与 quantMode/tilingKey 字段;host 侧新增基于 MC2_DFX_ENABLE 编译开关的异常 dump 函数(tiling 数据、workspace 各段、URMA 通信上下文与 peermem 窗口落盘),并通过 IMPL_OP(...).ExceptionDumpParseFunc(...) 为各算子注册异常回调;kernel 侧新增 Mc2Kernel::Mc2RuntimeInfo 运行时进度打点(HCCL commit/wait 次数、执行轮次与通信阶段),用于崩溃时定位卡死位置。

likedislike
WWangShuyingCollaborators
Sep 12 Added label: requirement
游震Collaborators
Sep 12 Comments:

/assign @WangShuying5

likedislike
CANN-robotCANN-robotCollaborators
Sep 12 Assign WangShuying5 as responsible person
WWangShuyingCollaborators
Sep 12 Associated pull request
CANN-robotCANN-robotCollaborators
Sep 12 Close issue
CANN-robotCANN-robotCollaborators
Sep 12 Added label: resolved
WWangShuyingCollaborators
Sep 24 Associated pull request