已开启
feat(profiler): add CPU-to-NPU correlation for ACLprof #46050
feat(profiler): add CPU-to-NPU correlation for ACLprof #46050
已开启
denis_g创建于 9月8日
denis_g
denis_g
9月8日

This PR is blocked by feat(profiler): implement ACLprof backend for Kineto NPU activities

【合入来源】

如有社区issue,请关联issue链接 请勿携带内部流程信息(需求链接、问题单、内部issue等)

【修改方案】

请描述修改内容的具体实现,涉及哪些组件之间进行交互,可以用1、2、3、...进行罗列
如果是需求或者重构类的PR,需要补充详细设计文档(说明上下游组件关系、时序图、类图、DFX能力等内容)

This PR adds CPU-to-NPU activity correlation for the ACLprof Kineto backend introduced by the parent ACLprof backend PR. It does not change ACLprof collection, backend selection, or the default MSPTI behavior.

The implementation connects ACLprof device kernels to their originating Kineto CPU activities through the task-queue and CANN profiling data path:

Kineto CPU activity K
        ↓
enqueue
        ↓ Q
task queue
        ↓
dequeue worker
        ↓
HostToDevice / CANN correlation
        ↓
ACLprof kernel
        ↓
linked_correlation_id = K

The main changes are:

  1. Capture Kineto correlation IDs through the NPU task queue

    • Extend task-queue profiler reporting so enqueue/dequeue marks are also collected while ACLprof capture is active.
    • Maintain the active Kineto correlation stack and capture the current Kineto correlation ID when an enqueue starts.
    • Associate the task-queue correlation ID Q with the originating Kineto correlation ID K.
    • Preserve logical nesting when an invalid correlation ID is encountered.
  2. Build enqueue/dequeue intervals for correlation

    • Record enqueue and dequeue start/end marks.
    • Build task-queue intervals using the worker thread ID and task-queue correlation ID.
    • Use the correlation ID from the closing task-queue mark.
    • Replace stale unmatched starts when a newer start for the same logical slot is observed, preventing malformed task-queue sequences from creating false intervals.
    • Use inclusive interval boundaries when matching launch timestamps.
  3. Resolve ACLprof kernels through HostToDevice activity

    • Parse HostToDevice flow records exported by ACLprof/CANN.
    • Read start_info and convert exported timeline timestamps to the CLOCK_MONOTONIC_RAW domain used by task-queue events.
    • Match HostToDevice launches to dequeue intervals using the worker TID and converted timestamp.
    • Do not require the exported process ID as a matching key.
    • Treat multiple matching physical candidates as ambiguous and fail closed instead of selecting one heuristically.
    • Keep unresolved dequeue intervals as physical candidates so they participate in ambiguity detection.
  4. Provide CANN correlation fallback

    • Resolve kernels through CANN correlation information when no HostToDevice flow is available.
    • Keep HostToDevice resolution authoritative when such flow is present.
    • Do not use CANN fallback to override an ambiguous HostToDevice result.
  5. Link ACLprof kernels to Kineto CPU activities

    • Resolve the final task-queue correlation Q to its captured Kineto correlation K.
    • Store resolved mappings in the ACLprof profiler correlation state.
    • Return the corresponding Kineto activity through getLinkedActivity().
    • Populate the emitted GenericTraceActivity::linked field so torch.profiler.events() exposes a valid linked_correlation_id for correlated ACLprof kernels.
    • Clear all correlation state between profiler sessions to prevent reuse across sequential ACLprof captures.
  6. Add hardware-free unit coverage

    Unit tests cover the complete correlation path and important fail-closed cases, including:

    • capturing the active Kineto correlation at enqueue start;
    • worker-thread-based dequeue matching;
    • one host launch linking multiple related kernels to the same CPU activity;
    • sequential sessions not reusing correlation state;
    • inclusive interval boundaries;
    • missing task-queue data;
    • missing or valid start_info;
    • ambiguous dequeue intervals;
    • unresolved physical dequeue candidates;
    • conflicting task-queue correlation ownership;
    • HostToDevice matching with different exported PIDs;
    • CANN correlation fallback;
    • ambiguous fallback remaining unresolved;
    • stale task-queue starts and extra closing marks.

The PR intentionally does not include structured correlation diagnostics or Stage-2 diagnostic tooling. Those are reserved for the follow-up diagnostics PR.

【资料变更】

请确认是否涉及资料变更。如涉及,需要在PR中体现,并简要说明修改内容。如不涉及,需填写“不涉及”

Not involved.

【接口变更】

请确认是否涉及跨代码仓或者客户面可见的接口变更。如涉及,需要详细说明接口以及对应的变更内容,同时需要在资料中体现。如不涉及,需填写“不涉及”

Not involved.

No new customer-facing API is introduced. The existing ACLprof backend selection remains unchanged. This PR extends the behavior of ACLprof-generated PrivateUse1 activities by linking device kernels to their originating CPU activities through the existing Kineto linked_correlation_id mechanism.

【功能验证】

说明测试场景,测试方法。如果本次测试方式与常规单元测试不同,请详细说明您的测试步骤
新增/变更内容是否已新增/适配UT测试用例看护,并补充测试自验证截图

The implementation was verified with both hardware-free unit tests and integration tests on a real Ascend device.

Hardware-free profiler unit tests

  • 27 profiler tests executed.
  • 27 passed, 0 failed.
  • This includes all 25 AclprofActivityProfilerTest cases and 17 PR2 correlation-specific test cases.
  1. ACLprof CPU-to-NPU correlation integration test

    Two sequential ACLprof profiler sessions were executed with task queue enabled:

    TASK_QUEUE_ENABLE=1
    ASCEND_LAUNCH_BLOCKING=0
    

    The workload contains separate MATMUL, RELU, ADD, and SIN CPU scopes. The second profiler session additionally inserts CPU-only events to change the CPU correlation-ID layout and verify that no correlation state is reused from the first session.

  2. ACLprof/MSPTI regression and backend-isolation validation

    The existing robust backend acceptance sequence was executed on the PR2 build:

    ACLprof → MSPTI → ACLprof
    

【CheckList】

PR提交人对以下CheckList自检项进行全量自检,自检通过或不涉及,均修改 [ ] 为 [x]

likedislike
合并受阻
denis_gdenis_g
9月8日 创建了 pull request,commit f0ddb271
ascend-robotascend-robot成员
9月8日 添加了label:stat/needs-squash
ascend-robotascend-robot成员
9月8日 添加了label:needs-issue
ascend-robotascend-robot成员
9月8日 添加了label:ascend-cla/yes
ascend-robot
ascend-robot成员
9月8日 评论:

Linking Issue Notice

@denis_g , the pull request must be linked to at least one issue.
If an issue has already been linked, but the needs-issue label remains, you can remove the label by commenting /check-issue .

likedislike
此处折叠了50条消息 查看更多
ascend-robotascend-robot成员
7 天前 删除了label:docs-ci-pipeline-success
ascend-robotascend-robot成员
7 天前 添加了label:docs-ci-pipeline-running
ascend-robot
ascend-robot成员
7 天前 评论:

✅ 跳过 docs ci 检查,没有需要检查的文档文件

likedislike
ascend-robotascend-robot成员
7 天前 删除了label:docs-ci-pipeline-running
ascend-robotascend-robot成员
7 天前 添加了label:docs-ci-pipeline-success