已开启
feat(profiler): implement ACLprof backend for Kineto NPU activities #45722
denis_g创建于 9月2日
feat(profiler): implement ACLprof backend for Kineto NPU activities #45722
已开启
denis_g创建于 9月2日
denis_g
denis_g
9月2日

This PR is blocked by Introduce PrivateUse1 Profiler Manager Architecture

【合入来源】

如有社区issue,请关联issue链接 请勿携带内部流程信息(需求链接、问题单、内部issue等)

【修改方案】

This PR implements the ACLprof-based NPU activity profiler backend for Kineto. The backend selection infrastructure and ACLprof profiler stub already exist in the base branch; this PR completes the ACLprof backend implementation without changing the default MSPTI behavior.

The implementation covers the following areas:

  1. ACLprof collection lifecycle

    • Implement AclprofActivityProfiler::start() and stop() using the CANN ACL profiling APIs.
    • Resolve the current NPU device before starting profiling.
    • Initialize ACLprof in an isolated temporary result directory.
    • Create the profiling configuration with ACL_PROF_TASK_TIME and ACL_PROF_ACL_API.
    • Start ACL profiling and register the device callback.
    • Stop profiling, destroy the profiling configuration, and finalize ACLprof after each profiling session.
    • Make cleanup safe for normal completion, partial initialization failures, start failures, teardown failures, and object destruction.
    • Reset profiler state between sessions to prevent ACLprof state from leaking into subsequent profiling sessions.
  2. Runtime physical-device resolution

    • Resolve the current user-visible NPU device directly to its physical device ID using aclrtGetPhyDevIdByUserDevId.
    • Add an internal wrapper for aclrtGetPhyDevIdByUserDevId.
    • Pass the current user-visible device ID to the ACLprof configuration while using the resolved physical device ID consistently for Kineto DeviceInfo, kernel activities, and stream ResourceInfo.
    • Treat device-resolution failures as profiler warnings and avoid emitting activities or resource information with an unresolved physical device ID.
    • Keep the runtime-resolved physical device authoritative; timeline metadata does not overwrite it.
  3. ACLprof data export

    • Discover the PROF_* directory produced by ACLprof.
    • Run msprof --export=on after collection finishes.
    • Discover the generated msprof*.json timeline files.
    • Report failures such as a missing msprof executable, failed export, missing profiling directory, or missing timeline output through the profiler error interface.
  4. ACLprof timeline parsing

    • Parse exported JSON timeline data and convert timestamps from microseconds to nanoseconds.
    • Identify device kernel records while excluding host/runtime records.
    • Support current timeline records identified by the Kernel category and legacy timeline shapes where device-side Task Type is the available discriminator.
    • Prefer structural kernel metadata and hardware process metadata over the legacy Task Type fallback when classifying device kernel records.
    • Preserve kernel name, task type, timestamp range, physical device, and stream ID.
    • Handle alternative stream metadata representations produced by supported msprof timeline formats.
  5. Kineto activity generation

    • Convert parsed NPU kernels into Kineto CONCURRENT_KERNEL GenericTraceActivity records.
    • Populate the physical device ID and stream ID.
    • Preserve ACLprof metadata such as Task Type and streamId.
    • Respect the Kineto trace capture window, including kernels overlapping the capture-window boundaries.
    • Skip activities with unresolved device IDs or invalid stream IDs and report the corresponding profiler errors.
    • Expose NPU device and stream resource information to Kineto.
    • Generate activities only when CONCURRENT_KERNEL collection is requested.
  6. Unit-test coverage

    • Replace the previous ACLprof stub test with tests covering the implemented backend.
    • Cover sequential profiling sessions and state cleanup.
    • Cover initialization, configuration creation, start, stop, destroy, and finalize failure paths.
    • Verify destructor cleanup and exactly-once teardown semantics.
    • Verify timeline parsing and kernel classification.
    • Verify structural Kernel classification, hardware process metadata classification, and legacy Task Type fallback behavior.
    • Verify the CONCURRENT_KERNEL activity gate.
    • Verify capture-window overlap and boundary behavior.
    • Verify runtime user-visible → physical device mapping.
    • Verify invalid device/stream handling and generated Kineto metadata/resource information.

Component interaction

The data flow introduced by this PR is:

torch.profiler / Kineto
        |
        | existing backend selection
        v
AclprofActivityProfiler
        |
        | aclprofInit / CreateConfig / Start
        v
CANN ACLprof
        |
        | profiling data
        v
PROF_* result directory
        |
        | msprof --export=on
        v
msprof timeline JSON
        |
        | parseAclprofTimeline()
        v
AclprofEvent
        |
        | processTrace()
        v
Kineto CONCURRENT_KERNEL
GenericTraceActivity
        |
        v
PrivateUse1 NPU events

The default MSPTI backend is not modified by this implementation. ACLprof is activated only through the existing backend-selection mechanism, and sequential ACLprof → MSPTI → ACLprof profiling sessions remain isolated.

DFX / error handling

  • ACLprof API failures are propagated through the profiler error list and TraceStatus::WARNING where appropriate.
  • Partial startup failures perform cleanup of resources that were already initialized.
  • Teardown continues attempting the remaining cleanup operations even if one cleanup step fails.
  • Temporary ACLprof output directories are removed after processing.
  • Invalid device and stream metadata are rejected instead of emitting malformed Kineto activities.
  • msprof execution and timeline-discovery failures are explicitly reported.

Scope

This PR implements ACLprof collection → msprof export → timeline parsing → Kineto NPU kernel activities.

CPU↔NPU correlation and population of linked_correlation_id are intentionally out of scope for this PR and will be handled separately.

【资料变更】

Not involved.

This PR does not change user-facing documentation or profiler APIs.

【接口变更】

Not involved.

There is no customer-visible or cross-repository API change.

An internal ACL runtime wrapper for aclrtGetPhyDevIdByUserDevId is added to resolve the physical NPU device ID required by the profiler backend. It is used internally by the ACLprof implementation and does not introduce a new customer-facing interface.

【功能验证】

1. C++ unit tests

Unit-test coverage was added to test/cpp/api/profiler.cpp for:

  • ACLprof lifecycle and sequential-session cleanup;
  • initialization and configuration failures;
  • start and teardown failures;
  • destructor cleanup;
  • timeline parsing and kernel classification;
  • structural kernel classification and hardware process metadata priority over legacy Task Type;
  • Kineto activity generation and metadata;
  • capture-window filtering;
  • activity-type filtering;
  • runtime physical-device resolution;
  • invalid device and stream handling.

These tests use an injected ACLprof API implementation where hardware access is not required, allowing lifecycle and failure paths to be validated deterministically.

All relevant hardware-free profiler GTests passed after the final cleanup and kernel-classification correction.

2. Ascend hardware integration validation

The backend was validated on an Ascend NPU with CANN 9.1.0 using an external robust integration test.

  • A fresh-process default MSPTI baseline produced 18 PrivateUse1 events and 18 Chrome kernel activities with no empty kernel names.
  • The ACLprof → MSPTI → ACLprof sequence completed successfully in a single process.
  • Both ACLprof sessions produced 38 PrivateUse1 events and 38 Chrome kernel activities, with Task Type metadata present on all ACLprof kernel activities.
  • The intermediate default MSPTI session preserved the expected 18 PrivateUse1 events and 18 Chrome kernel activities, with no ACLprof Task Type metadata leakage.
  • All sessions reported the expected physical NPU device.
  • A second ACLprof session after MSPTI completed successfully, validating backend lifecycle cleanup and session isolation.
  • PR2 CPU↔NPU correlation and linked_correlation_id behavior were intentionally not covered by this validation.

【CheckList】

PR提交人对以下CheckList自检项进行全量自检,自检通过或不涉及,均修改 [ ] 为 [x]

likedislike
合并受阻
denis_gdenis_g
9月2日 创建了 pull request,commit a7229f13
ascend-robotascend-robot成员
9月2日 添加了label:stat/needs-squash
ascend-robotascend-robot成员
9月2日 添加了label:ascend-cla/yes
ascend-robotascend-robot成员
9月2日 添加了label:needs-issue
ascend-robot
ascend-robot成员
9月2日 评论:

Thanks for your pull-request.
The full list of commands accepted by me can be found at here.
You can get sig-info at here.
You can self-configure the PR merge rules for this repository. For more details, please refer to Here.


PR Approval Progress

⚠️ This PR does not yet meet the following requirements:lgtm (requires ≥ 2 person(s) per module)、approve (requires ≥ 1 person(s) per module)

Module Approval Details

module lgtm status approve status
repo-Ascend/pytorch ❌ (0/2)(You can also ask: zyw-hw, wangqiang160, ffmh, chenrayray, medivh-x) ❌ (0/1)(You can also ask: zichun_ye, chujinjin, zhenyu10, 褚博宁, wanglijun55)
test ❌ (0/2)(You can also ask: senzhen-town, rain-666, liujunzhu, 王朝, rmch) ❌ (0/1)(You can also ask: kisnwang, suhaibo, liangsongwei, zichun_ye, crazyDannyBoy)
torch_npu/csrc/profiler ❌ (0/2)(You can also ask: chengpeng25, 陈豪, daifu1234567, rmch, kisnwang) ❌ (0/1)(You can also ask: li_jing_hw, 楚浩田, wangqiang160, fft1374, wasd1111222)
torch_npu/profiler ❌ (0/2)(You can also ask: chengpeng25, huangjingwei, chenrayray, fft1374, wasd1111222) ❌ (0/1)(You can also ask: liangsongwei, 陈豪, 楚浩田, zhenyu10, medivh-x)

💡 Tip:

  • Committer can comment /approve or /lgtm
  • Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

denis_g, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
此处折叠了123条消息 查看更多
ascend-robotascend-robot成员
10 天前 删除了label:docs-ci-pipeline-success
ascend-robotascend-robot成员
10 天前 添加了label:docs-ci-pipeline-running
ascend-robot
ascend-robot成员
10 天前 评论:

✅ 跳过 docs ci 检查,没有需要检查的文档文件

likedislike
ascend-robotascend-robot成员
10 天前 删除了label:docs-ci-pipeline-running
ascend-robotascend-robot成员
10 天前 添加了label:docs-ci-pipeline-success