Closed
[Requirement|需求建议]: Mc2 DFX需求,发生aic error后触发exception dump #5252
WangShuyingAdd on Sep 12Closed at Sep 12
Sep 12 Added label: requirement
游震
Sep 12 Comments:
Sep 12 Comments:
/assign @WangShuying5


Sep 12 Assign WangShuying5 as responsible person
Sep 12 Associated pull request
Sep 12 Close issue
Sep 12 Added label: resolved
Sep 24 Associated pull request
Thanks for sending an requirement! Please fill in the following template to help quickly solve your problem.
Backgroud(背景信息)
alltoallmatmul、alltoallmatmulv2、allgathermatmulv2、allgathermatmulv3、matmulreducescatterv2算子mx场景增加exception dump,获取xn/cke 地址 dump、tiling 数据/workspace 逐段 D2H 拷贝落盘、通信上下文与 peermem 窗口分段 dump和kernel 侧运行时进度打点信息
Benefit / Necessity (价值/作用)
支撑融合算子本身问题进一步的问题定位;
Design(设计方案)
算子DFX实现主要依赖CANN 自带的DFX能力,其提供了异常时的回调注册机制
tiling 数据结构中新增 workspace/peermem 段布局信息(WorkspaceLayoutInfo、PeermemLayoutInfo)与 quantMode/tilingKey 字段;host 侧新增基于 MC2_DFX_ENABLE 编译开关的异常 dump 函数(tiling 数据、workspace 各段、URMA 通信上下文与 peermem 窗口落盘),并通过 IMPL_OP(...).ExceptionDumpParseFunc(...) 为各算子注册异常回调;kernel 侧新增 Mc2Kernel::Mc2RuntimeInfo 运行时进度打点(HCCL commit/wait 次数、执行轮次与通信阶段),用于崩溃时定位卡死位置。