已开启
feat(profiler): add ACLprof correlation diagnostics and validation #46284
feat(profiler): add ACLprof correlation diagnostics and validation #46284
已开启
denis_g创建于 9 天前
denis_g
denis_g
9 天前

This PR is blocked by feat(profiler): add CPU-to-NPU correlation for ACLprof

【合入来源】

如有社区issue,请关联issue链接 请勿携带内部流程信息(需求链接、问题单、内部issue等)

【修改方案】

请描述修改内容的具体实现,涉及哪些组件之间进行交互,可以用1、2、3、...进行罗列
如果是需求或者重构类的PR,需要补充详细设计文档(说明上下游组件关系、时序图、类图、DFX能力等内容)

This PR adds opt-in diagnostics and hardware validation support for ACLprof CPU↔NPU correlation in the Kineto profiler backend. The existing correlation behavior remains unchanged when diagnostics are disabled.

  1. Add opt-in ACLprof correlation diagnostics

    • Introduce diagnostic recording controlled by TORCH_NPU_ACLPROF_CORRELATION_DEBUG.
    • Keep diagnostics disabled by default so that normal profiling behavior is unaffected.
    • Record structured diagnostic events for correlation capture, task-queue processing, HostToDevice conversion, kernel resolution, matching, clock conversion, and final Kineto linkage.
    • Preserve diagnostic artifacts in JSONL format for post-run investigation.
  2. Add correlation and lifecycle counters

    • Track Kineto correlations captured at enqueue time.
    • Track enqueue/dequeue task-queue marks and HostToDevice records.
    • Track converted launch records and successful clock conversions.
    • Track matching outcomes including FOUND, NOT_FOUND, and AMBIGUOUS.
    • Track final CPU↔NPU links.
    • Emit per-session summaries to make correlation behavior observable without changing production matching semantics.
  3. Keep diagnostic work outside correlation-critical locking where possible

    • Defer diagnostic formatting and file I/O until after the correlation mutex is released.
    • Preserve the existing matching and linking logic while minimizing diagnostic overhead in correlation-critical paths.
  4. Improve correlation failure observability

    • Record clock reference and conversion results.
    • Record candidate matching information and final match results.
    • Record kernel-resolution and linkage decisions so missing or ambiguous CPU↔NPU associations can be investigated from retained artifacts.
  5. Add hardware-independent unit-test coverage

    • Verify diagnostics are inactive when debug mode is disabled.
    • Verify diagnostic state is reset correctly across profiling sessions.
    • Verify successful Kineto correlation capture and matching paths.
    • Cover negative and edge cases including ambiguous matching, missing matches, clock conversion failures, fallback behavior, diagnostic I/O failures, and profiling lifecycle transitions.
  6. Add a manual hardware validation tool

    • Add tools/kineto_aclprof_correlation_validation.py for validation on real Ascend hardware.
    • Run two consecutive ACLprof profiling sessions and validate the complete correlation chain from captured CPU correlation IDs through converted HostToDevice launches to final NPU kernel links.
    • Validate clock conversion, matching results, final links, and session isolation.
    • Retain diagnostic and profiling artifacts when validation fails to simplify investigation.
  7. Add correlation diagnostics documentation

    • Document prerequisites, invocation, environment configuration, PASS/FAIL criteria, diagnostic counters, and generated artifacts.
    • Clarify that the hardware validation script is a manual maintainer validation tool and is not registered as a CI test.

【资料变更】

请确认是否涉及资料变更。如涉及,需要在PR中体现,并简要说明修改内容。如不涉及,需填写“不涉及”

Documentation is updated.

Added docs/aclprof_correlation_diagnostics.md describing:

  • how to enable ACLprof correlation diagnostics;
  • how to run the hardware validation tool;
  • required runtime prerequisites;
  • automatically configured environment variables;
  • PASS and FAIL criteria;
  • interpretation of diagnostic counters;
  • retained diagnostic and profiler artifacts for failure analysis;
  • the manual, non-CI status of the hardware validation tool.

【接口变更】

请确认是否涉及跨代码仓或者客户面可见的接口变更。如涉及,需要详细说明接口以及对应的变更内容,同时需要在资料中体现。如不涉及,需填写“不涉及”

No cross-repository or customer-visible API changes.

The added diagnostic environment variables, diagnostic artifacts, and validation script are intended for profiler diagnostics and maintainer validation. Existing profiler APIs and the default profiling behavior are unchanged.

【功能验证】

说明测试场景,测试方法。如果本次测试方式与常规单元测试不同,请详细说明您的测试步骤
新增/变更内容是否已新增/适配UT测试用例看护,并补充测试自验证截图

The changes were validated with both hardware-independent unit tests and a manual hardware validation run on Ascend hardware.

  1. Profiler unit tests

    • ACLprof correlation diagnostics and lifecycle scenarios are covered by profiler GTests.
    • Tests cover diagnostics enabled/disabled behavior, session reset, successful correlation capture, matching results, ambiguity handling, clock conversion failures, fallback paths, diagnostic I/O failures, and lifecycle behavior.
    • The profiler test suite passed with 33/33 tests.
  2. Static/script validation

    • git diff --check passed.
    • python3 -m py_compile tools/kineto_aclprof_correlation_validation.py passed.
    • python3 tools/kineto_aclprof_correlation_validation.py --help passed.
  3. Real Ascend hardware validation

    • The validation tool was executed as:
mkdir -p ./kineto_aclprof_cor_output

python3 tools/kineto_aclprof_correlation_validation.py \
    --output-dir ./kineto_aclprof_cor_output \
    --device 0
  • Two consecutive ACLprof profiling sessions were validated.

  • Each session produced:

    • converted_launches = 24
    • successful_clock_conversions = 24
    • FOUND = 24
    • NOT_FOUND = 0
    • AMBIGUOUS = 0
    • final_links = 24
  • Both sessions completed successfully and the tool reported:

ACLprof correlation validation PASS

This confirms that CPU↔NPU correlation is correctly captured, converted, matched, linked, and reset across consecutive profiling sessions on real Ascend hardware.

【CheckList】

PR提交人对以下CheckList自检项进行全量自检,自检通过或不涉及,均修改 [ ] 为 [x]

likedislike
合并受阻
denis_gdenis_g
9 天前 创建了 pull request,commit fd80386c
atomgit-bot
atomgit-bot
9 天前 评论:

变更摘要

本 PR 为 Kineto profiler 后端新增可选的 ACLprof CPU↔NPU 相关性诊断与硬件校验能力,默认关闭诊断,不影响既有 profiling 行为。核心改动包括:新增 NPUActivityProfiler 抽象层与 NPUActivityProfilerFactory,依据 _NPUExperimentalConfig 注入的 custom_profiler_config(JSON {"backend":"aclprof"})在 AclprofActivityProfiler 与默认 MsptiActivityProfiler 之间选择后端,并移除旧 MsptiActivityProfilerPoc 的注册逻辑;AclprofActivityProfilerTORCH_NPU_ACLPROF_CORRELATION_DEBUG=1 时通过 AclprofDiagnostics 记录 correlation 捕获、任务队列标记、HostToDevice 转换、kernel 解析、时钟转换、匹配结果(FOUND/NOT_FOUND/AMBIGUOUS)与最终 Kineto 链接等结构化事件,以 JSONL 保留至 TORCH_NPU_ACLPROF_DIAGNOSTICS_DIR,并将诊断格式化与文件 I/O 推迟到相关性互斥锁释放之后;同时新增 AclrtGetPhyDevIdByUserDevId 接口、任务队列事件上报接入、硬件校验脚本 tools/kineto_aclprof_correlation_validation.py 及配套单测与诊断文档。

主要改动

  • 新增统一 NPU profiler 抽象与后端选择:新增 npu_activity_profiler.h/cpp,定义 NPUActivityProfiler 基类、NPUActivityProfilerFactory::create(解析 config.getCustomConfig() 中的 backend 字段,返回 AclprofActivityProfiler 或默认 MsptiActivityProfiler)、PrivateUse1ProfilerManagerNPUActivityProfilerPlugin 注册;同时从 mspti_activity_profiler.cpp/h 删除 MsptiActivityProfilerPoc 及其注册代码,MsptiActivityProfiler 改为继承 NPUActivityProfiler
  • 新增 ACLprof 相关性诊断与采集实现:新增 aclprof_activity_profiler.cpp/haclprof_activity_profiler_internal.hAclprofActivityProfiler 实现 ACLprof 时间线解析(parseAclprofTimelineconvertTimelineRecords)、任务队列时钟偏移加载与 correlation 解析(resolveCorrelations);AclprofDiagnosticsTORCH_NPU_ACLPROF_DIAGNOSTICS_DIR 输出 JSONL 诊断事件,通过 ACLPROF_DIAG_DEFERRED 将格式化与 I/O 推迟到 correlationMutex_ 释放后进行,并输出每会话 ACLPROF_CORRELATION_SUMMARY 计数(含 kineto_correlations_capturedfinal_links 等)。
  • 新增设备接口与任务队列事件上报AclInterface.cpp/h 新增 AclrtGetPhyDevIdByUserDevId 的动态加载与声明(用户设备号到物理设备号映射);npu_profiler.cppreportMarkDataToNpuProfiler 增加 recordAclprofTaskQueueEvent 调用并在常规上报未启用时提前返回;NpuUtils.cppProfReportMarkDataToNpuProfileraclprofTaskQueueCaptureEnabled() 为真时也执行标记数据上报。
  • 新增 Python 配置与硬件校验工具:新增 _kineto_config.py_NPUExperimentalConfig(继承 Kineto _ExperimentalConfig,将 {"backend": ...} 序列化到 custom_profiler_config)并在 torch_npu/profiler/__init__.py 导出;新增 tools/kineto_aclprof_correlation_validation.py,设置 TORCH_NPU_ACLPROF_CORRELATION_DEBUGTORCH_NPU_ACLPROF_DIAGNOSTICS_DIR 等环境变量,连续运行两次 ACLprof profiling 会话并校验 HostToDevice→KERNEL_RESOLVEMATCH_RESULTLINK 完整链路、会话隔离与产物完整性。
  • 补充测试覆盖:新增 test/cpp/api/profiler.cpp(GTest 覆盖诊断启停、会话重置、成功匹配、歧义/缺失匹配、时钟转换失败、回退路径、诊断 I/O 失败与生命周期场景);test/profiler/test_experimental_config.py 新增 test_kineto_backend_config,校验 pickle.dumps{"backend":"aclprof"} 仅出现在 _NPUExperimentalConfig 中。
likedislike
不准确?
atomgit-bot
atomgit-bot
9 天前 评论:

代码审查

✅ 未发现问题

likedislike
不准确?
ascend-robotascend-robot成员
9 天前 添加了label:stat/needs-squash
ascend-robotascend-robot成员
9 天前 添加了label:ascend-cla/yes
ascend-robotascend-robot成员
9 天前 添加了label:needs-issue
ascend-robot
ascend-robot成员
9 天前 评论:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here.
You can get sig-info at here.
You can self-configure the PR merge rules for this repository. For more details, please refer to Here.


PR Approval Progress

⚠️ This PR does not yet meet the following requirements:lgtm (requires ≥ 2 person(s) per module)、approve (requires ≥ 1 person(s) per module)

Module Approval Details

module lgtm status approve status
**/*.md ❌ (0/2)(You can also ask: LQ1206, zhenyu10, chujinjin, li_jing_hw, 楚浩田) ❌ (0/1)(You can also ask: liangsongwei, chenrayray, zhenyu10, kisnwang, wjlflyer)
repo-Ascend/pytorch ❌ (0/2)(You can also ask: wangqiang160, liangsongwei, zqwenn, hbhu_bin, zichun_ye) ❌ (0/1)(You can also ask: chengpeng25, 楚浩田, kisnwang, huangjingwei, wasd1111222)
test ❌ (0/2)(You can also ask: suhaibo, guoqi1024, wanglijun55, 陈豪, crazyDannyBoy) ❌ (0/1)(You can also ask: chujinjin, suhaibo, anyrenwei, liangsongwei, 王朝)
torch_npu/csrc/profiler ❌ (0/2)(You can also ask: 褚博宁, wjlflyer, liujunzhu, luochao60, huangyunlong2022) ❌ (0/1)(You can also ask: huangjingwei, 陈豪, chujinjin, wangqiang160, chengpeng25)
torch_npu/profiler ❌ (0/2)(You can also ask: rmch, kisnwang, luochao60, zqwenn, liangsongwei) ❌ (0/1)(You can also ask: liujunzhu, wangqiang160, chengpeng25, wjlflyer, zyw-hw)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

denis_g, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
ascend-robot
ascend-robot成员
9 天前 评论:

当前仓库存在以下 保护分支

Protected Branch Version Release
master
v2.12.0
v2.11.0
v2.10.0
v2.9.0
v2.7.1
v2.7.1-26.1.0
v2.9.0-26.1.0
v2.10.0-26.1.0
v2.11.0-26.1.0
v2.12.0-26.1.0
ci-test

评论 /sync <branch1> <branch2> ... 可将当前 PR 修改同步到其它分支(创建同步 PR):
a) 如果当前 PR 是 Open 状态,同步操作将延迟到 PR 被合并时执行
b) 如果当前 PR 已经 Merged,将立即执行同步操作

注意:

  1. /sync 命令可以指定同步到多个分支,仅最后一个 /sync 命令生效
  2. 如果创建的同步 PR 不正确,可通过向同步 PR 的源分支提交轻量级 PR 完善,或使用 /close 命令关闭
likedislike
ascend-robot
ascend-robot成员
9 天前 评论:

Linking Issue Notice

@denis_g , the pull request must be linked to at least one issue.
If an issue has already been linked, but the needs-issue label remains, you can remove the label by commenting /check-issue .

likedislike
AtlasAccountAtlasAccount成员
9 天前 添加了label:ci-pipeline-running
ascend-robot
ascend-robot成员
9 天前 评论:

ascend docs pipeline is running...

likedislike
ascend-robotascend-robot成员
9 天前 添加了label:docs-ci-pipeline-running
ascend-robot
ascend-robot成员
9 天前 评论:

✅ 文档门禁通过!

检查项 检查结果 详情
markdownlint ✅ 已通过 查看详情
link-validity-check ✅ 已通过 查看详情
resource-existence-check ✅ 已通过 查看详情
tag-closed-check ✅ 已通过 查看详情
likedislike
ascend-robotascend-robot成员
9 天前 删除了label:docs-ci-pipeline-running
此处折叠了9条事件消息 查看更多
AtlasAccountAtlasAccount成员
8 天前 添加了label:ci-pipeline-failed
AtlasAccount
AtlasAccount成员
8 天前 评论:
流水线 PR-pipeline_pytorch#67593 [ commitID:4cd177bc ] 运行失败
>>>代码风格自动修复执行成功(无修复内容)
阶段 任务名 状态 详情
编译构建 Build_X86 🟣 INIT >>>
Build_ARM 🟣 INIT >>>
Build_X86_torchair 🟣 INIT >>>
Build_ARM_torchair 🟣 INIT >>>
patch_test 🟣 INIT >>>
Build_X86_213 🟣 INIT >>>
Build_ARM_213 🟣 INIT >>>
恶意代码检查 Antipoison ✅ COMPLETED >>>
编码安全与规范检查 codecheck_pre-commit ❌ FAILED >>>
check_error ✅ COMPLETED >>>
lintrunner ❌ FAILED >>>
开源片段检查 SCA ✅ COMPLETED >>>
开发者测试 UT_ARM_A3_Part_01 🟣 INIT >>>
UT_ARM_A3_Part_02 🟣 INIT >>>
UT_ARM_A2_Part_01 🟣 INIT >>>
UT_ARM_A2_Part_02 🟣 INIT >>>
UT_ARM_A2_Part_03 🟣 INIT >>>
UT_inductor_Part_01 🟣 INIT >>>
UT_inductor_Part_02 🟣 INIT >>>
UT_inductor_Part_03 🟣 INIT >>>
UT_inductor_Part_04 🟣 INIT >>>
UT_DIST_ARM_Part_01 🟣 INIT >>>
UT_DIST_ARM_Part_02 🟣 INIT >>>
UT_DIST_ARM_Part_03 🟣 INIT >>>
UT_DIST_ARM_Part_04 🟣 INIT >>>
UT_ARM_A2_Select_Part_01 🟣 INIT >>>
UT_ARM_A2_Select_Part_02 🟣 INIT >>>
UT_inductor_Part_213 🟣 INIT >>>
流水线 PR-pipeline_pytorch ❌ FAILED >>>
此流水线已支持下列评论快捷指令,仅PR创建者和白名单成员[wujinyuan1, huangjingwei, liangsongwei, yashi999, culechan, Dring, wuyouqi1, L1919_snow, qq_52711437, WhiteNight12, nomiz, xiu_21, ffmh, wanglijun55, hss-shuai, husichao, smallsilly, lanshaozuishuai, jimmyisme1, lzy0920232, alpha-junh, Sunshine_Youngster, wei_zhuoyi, zhangyihuiben, zyw-hw, zzzkeke, rmch, yangch0324, LucciC, AACAES, renyujin, wjlflyer, senzhen-town, pengjingyou, qsc97, limuan, yule100, xiaoqi-zhou, kuhn7, chenxingying, hanye02, zichun_ye, anyrenwei, kkjocker, wangzili121, Lu_G, yvjc, puddingfjz, HandsoemLemon, bigprestigee1, huawuyi, zhenyu10, dairenjie, du-jin-hang, zou-jieyu, adelaideliu, TrHan, wanlinan, Windwindzzz, pengqihw, kisnwang, yuheng_wang, honghao_wang, jizewei, zhangguoguang, sunyu-xuan, chenrayray, hbhu_bin, liujunzhu, c_34, LiNuoh, maoyuanpeng1, zzhongmin, zhaoyu65, bellatan, jiabaolin, zhuofanshen, wencaiwen, lu_zhuge, caoshuyang, molly12, lyx324521, LQ1206, gitcode-chenjiao, cai-weiwei1989, CHDong, ogqin, yuanlipingGit, xuqinglin1, lqz2, zouwei1, chaoluoa, paradox325, jackzhang1116, yaoyao, akh, yujiacheng, dengjie0116, Hubert11111, Shine_Ws, wslhj555, longqiand, OYtao666, JiaqingQiang, luyyyy, Kingbelial, zhanghaiyu0101, wenxp1018, yanliu-luoluo, ksun_sekiro, liyong328, wgzheng, tangky, vivi_is_coding, aoiaoisola, weixin_44494597, wangmengmengwang65667, hid57809721, qq_35468730, comeonup, C547032, gcw_m5OQChA4, yao_yao_ling_xian, cnnbwcy, szqfes_12, cora_19, cann_lilin, can, shawnylee233, fanglanyue0916, hhz0, LiNuohang, taohuoquan, Jesse, WSs_321, SCh_zx]评论有效
  • compile、compile_inductor、compile_torchair : 运行流水线
  • retry : 重试流水线所有失败子任务
  • retry <任务名> : 仅重试指定失败子任务
  • stop : 停止流水线
likedislike
AtlasAccountAtlasAccount成员
8 天前 删除了label:ci-pipeline-failed
AtlasAccountAtlasAccount成员
8 天前 添加了label:ci-pipeline-running
AtlasAccountAtlasAccount成员
8 天前 删除了label:ci-pipeline-running
AtlasAccountAtlasAccount成员
8 天前 添加了label:ci-pipeline-failed
AtlasAccount
AtlasAccount成员
8 天前 评论:
流水线 PR-pipeline_pytorch#67593 (重试第1次) [ commitID:4cd177bc ] 运行失败
>>>代码风格自动修复执行成功(无修复内容)
阶段 任务名 状态 详情
编译构建 Build_X86 🟣 INIT >>>
Build_ARM 🟣 INIT >>>
Build_X86_torchair 🟣 INIT >>>
Build_ARM_torchair 🟣 INIT >>>
patch_test 🟣 INIT >>>
Build_X86_213 🟣 INIT >>>
Build_ARM_213 🟣 INIT >>>
恶意代码检查 Antipoison ✅ COMPLETED >>>
编码安全与规范检查 codecheck_pre-commit ❌ FAILED >>>
check_error ✅ COMPLETED >>>
lintrunner ❌ FAILED >>>
开源片段检查 SCA ✅ COMPLETED >>>
开发者测试 UT_ARM_A3_Part_01 🟣 INIT >>>
UT_ARM_A3_Part_02 🟣 INIT >>>
UT_ARM_A2_Part_01 🟣 INIT >>>
UT_ARM_A2_Part_02 🟣 INIT >>>
UT_ARM_A2_Part_03 🟣 INIT >>>
UT_inductor_Part_01 🟣 INIT >>>
UT_inductor_Part_02 🟣 INIT >>>
UT_inductor_Part_03 🟣 INIT >>>
UT_inductor_Part_04 🟣 INIT >>>
UT_DIST_ARM_Part_01 🟣 INIT >>>
UT_DIST_ARM_Part_02 🟣 INIT >>>
UT_DIST_ARM_Part_03 🟣 INIT >>>
UT_DIST_ARM_Part_04 🟣 INIT >>>
UT_ARM_A2_Select_Part_01 🟣 INIT >>>
UT_ARM_A2_Select_Part_02 🟣 INIT >>>
UT_inductor_Part_213 🟣 INIT >>>
流水线 PR-pipeline_pytorch ❌ FAILED >>>
此流水线已支持下列评论快捷指令,仅PR创建者和白名单成员[wujinyuan1, huangjingwei, liangsongwei, yashi999, culechan, Dring, wuyouqi1, L1919_snow, qq_52711437, WhiteNight12, nomiz, xiu_21, ffmh, wanglijun55, hss-shuai, husichao, smallsilly, lanshaozuishuai, jimmyisme1, lzy0920232, alpha-junh, Sunshine_Youngster, wei_zhuoyi, zhangyihuiben, zyw-hw, zzzkeke, rmch, yangch0324, LucciC, AACAES, renyujin, wjlflyer, senzhen-town, pengjingyou, qsc97, limuan, yule100, xiaoqi-zhou, kuhn7, chenxingying, hanye02, zichun_ye, anyrenwei, kkjocker, wangzili121, Lu_G, yvjc, puddingfjz, HandsoemLemon, bigprestigee1, huawuyi, zhenyu10, dairenjie, du-jin-hang, zou-jieyu, adelaideliu, TrHan, wanlinan, Windwindzzz, pengqihw, kisnwang, yuheng_wang, honghao_wang, jizewei, zhangguoguang, sunyu-xuan, chenrayray, hbhu_bin, liujunzhu, c_34, LiNuoh, maoyuanpeng1, zzhongmin, zhaoyu65, bellatan, jiabaolin, zhuofanshen, wencaiwen, lu_zhuge, caoshuyang, molly12, lyx324521, LQ1206, gitcode-chenjiao, cai-weiwei1989, CHDong, ogqin, yuanlipingGit, xuqinglin1, lqz2, zouwei1, chaoluoa, paradox325, jackzhang1116, yaoyao, akh, yujiacheng, dengjie0116, Hubert11111, Shine_Ws, wslhj555, longqiand, OYtao666, JiaqingQiang, luyyyy, Kingbelial, zhanghaiyu0101, wenxp1018, yanliu-luoluo, ksun_sekiro, liyong328, wgzheng, tangky, vivi_is_coding, aoiaoisola, weixin_44494597, wangmengmengwang65667, hid57809721, qq_35468730, comeonup, C547032, gcw_m5OQChA4, yao_yao_ling_xian, cnnbwcy, szqfes_12, cora_19, cann_lilin, can, shawnylee233, fanglanyue0916, hhz0, LiNuohang, taohuoquan, Jesse, WSs_321, SCh_zx]评论有效
  • compile、compile_inductor、compile_torchair : 运行流水线
  • retry : 重试流水线所有失败子任务
  • retry <任务名> : 仅重试指定失败子任务
  • stop : 停止流水线
likedislike