feat(profiler): add ACLprof correlation diagnostics and validation-pytorch-AtomGit

Adocs/aclprof_correlation_diagnostics.md+157-0

@@ -0,0 +1,157 @@
	1	+# ACLprof correlation diagnostics
	2	+
	3	+Enable diagnostics before starting an ACLprof session:
	4	+
	5	+```bash
	6	+export TORCH_NPU_ACLPROF_CORRELATION_DEBUG=1
	7	+export TORCH_NPU_ACLPROF_DIAGNOSTICS_DIR=/path/to/output
	8	+```
	9	+
	10	+Only the exact debug value `1` enables diagnostics; it is read at session start. With debug OFF,
	11	+`diagnostics_` is null: no diagnostic session ID is allocated, no diagnostic counter changes, and no logging,
	12	+directory creation or artifact copying occurs. Enabled sessions receive process-local, monotonically
	13	+increasing IDs. ON → OFF → ON produces N and N+1. Each stderr JSON record includes `record` and `session`. The
	14	+diagnostics directory is optional; keep it outside the temporary ACLprof result root, which production cleanup
	15	+removes.
	16	+
	17	+\| Record \| Observation \|
	18	+\| --- \| --- \|
	19	+\| `ACLPROF_CLOCK_REFERENCE` \| Discovery, original references, accepted offset or failure reason \|
	20	+\| `ACLPROF_CLOCK_CONVERT` \| Source timestamp, reference, production conversion or failure \|
	21	+\| `ACLPROF_KINETO_STACK` \| Push/pop, TID and active K \|
	22	+\| `ACLPROF_TASK_QUEUE_MARK` \| Raw category, enqueue/dequeue, start/end, Q, captured K, TID, timestamp \|
	23	+\| `ACLPROF_ENQUEUE` \| Interval and final Q → K after conflicting owners are removed \|
	24	+\| `ACLPROF_DEQUEUE` \| Physical interval, Q, worker TID, resolved/unresolved K \|
	25	+\| `ACLPROF_KERNEL_RESOLVE` \| Kernel index/name, device, stream, CANN ID, HostToDevice/CANN fallback result \|
	26	+\| `ACLPROF_HOST_TO_DEVICE` \| Raw flow endpoints; `stage=converted_launch` identifies a converted launch \|
	27	+\| `ACLPROF_MATCH_BEGIN` \| Kernel/launch, worker TID, source/converted timestamp, exported PID \|
	28	+\| `ACLPROF_MATCH_CANDIDATE` \| Each visited production candidate, including unresolved candidates \|
	29	+\| `ACLPROF_MATCH_RESULT` \| FOUND/NOT_FOUND/AMBIGUOUS, selected Q and K \|
	30	+\| `ACLPROF_LINK` \| Kernel/launch/K, callback availability, actual linked correlation ID and result \|
	31	+\| `ACLPROF_CORRELATION_SUMMARY` \| Once per session after processTrace, or lifecycle fallback \|
	32	+
	33	+Reconstruct the chain using `(session, kernel, launch)` and Q/K from MATCH_RESULT. Record-named counters
	34	+count production observations; summary counters are explained below. `converted_launches` counts conversions
	35	+per kernel, not unique launches. Capture increments are snapshotted at the production decision and applied
	36	+after unlock; invalid stacks, enqueue-end and dequeue marks do not increment captures. Without a clock
	37	+reference, interval/matcher counters remain zero, following production.
	38	+
	39	+Artifacts use `session_<ID>_<unique suffix>/manifest.json`. The manifest contains the session ID and each
	40	+file's source path, relative `path`, `copied` and `error` fields. Original `start_info` and `msprof.json`
	41	+bytes and relative directory structure are preserved before cleanup. Unique directories prevent overwrites
	42	+across sessions/processes. All diagnostic I/O is best effort: allocation, formatting and copying failures must
	43	+not alter production results or `errors()`. Copy failures appear in the manifest; other I/O failures may emit
	44	+`ACLPROF_DIAGNOSTIC_WARNING`. Production errors remain visible.
	45	+
	46	+The production-invariance contract preserves captured K, Q, timestamps, operation order, queue semantics,
	47	+clock conversion, worker-TID matching, inclusive boundaries, unresolved candidate participation, ambiguity
	48	+fail-closed, and CANN fallback. The matcher stops at its second match; diagnostics do not run another matcher
	49	+or enumerate later candidates. `linkedCorrelations_`, `getLinkedActivity(K)`, emitted activities and public
	50	+APIs are unchanged. Under `correlationMutex_`, diagnostics only copy owned snapshots. JSON formatting, stderr
	51	+output and session destruction run after unlock. Concurrent diagnostic lines may interleave; use their
	52	+captured IDs and timestamps rather than stderr order.
	53	+
	54	+## Hardware-free validation
	55	+
	56	+Existing profiler GTests use `test/cpp/api/profiler.cpp`, registered in the existing `test_api` target with
	57	+`BUILD_GTEST`. They require no NPU, CANN runtime, msprof, local paths or manual runtime setup. In an
	58	+environment prepared for the repository's normal build:
	59	+
	60	+```bash
	61	+cmake --build build --target test_api -j2
	62	+build/gtest/test_api --gtest_filter='AclprofActivityProfilerTest.*'
	63	+```
	64	+
	65	+## Hardware correlation validation
	66	+
	67	+`tools/kineto_aclprof_correlation_validation.py` is a manual maintainer hardware validation tool.
	68	+It is not registered as an automated CI test: suitability of the existing hardware CI environment for
	69	+this ACLprof backend and `msprof` has not been demonstrated. MSPTI validation is outside its scope.
	70	+
	71	+### Running the validation
	72	+
	73	+Prerequisites: Linux, an available Ascend NPU, working compatible `torch` / `torch_npu` installations,
	74	+the CANN runtime, and `msprof` available in `PATH`. Run from the repository root:
	75	+
	76	+```bash
	77	+python3 tools/kineto_aclprof_correlation_validation.py \
	78	+ --output-dir ./aclprof_correlation_validation_output \
	79	+ --device 0
	80	+```
	81	+
	82	+`--device` selects the logical NPU index (default `0`). The script creates a unique
	83	+`aclprof_correlation_validation_<suffix>` run directory under `--output-dir` and prints effective settings.
	84	+Before importing `torch_npu` or initializing the NPU runtime, it sets `TASK_QUEUE_ENABLE=1`,
	85	+`ASCEND_LAUNCH_BLOCKING=0`, `TORCH_NPU_ACLPROF_CORRELATION_DEBUG=1`, and
	86	+`TORCH_NPU_ACLPROF_DIAGNOSTICS_DIR=<run-directory>`. No manual exports are needed for this script.
	87	+Two warm-up iterations precede two ACLprof sessions with two profiling iterations each: 128×128 and 192×192
	88	+matrices, `mm`, `relu`, `add_`, `sin`, `normal_`; shapes, memory, stacks, and FLOPs enabled.
	89	+
	90	+### Successful result
	91	+
	92	+A successful run ends approximately as follows and returns exit code `0`:
	93	+
	94	+```text
	95	+session <N> {...} artifacts <path>
	96	+session <N+1> {...} artifacts <path>
	97	+ACLprof correlation validation PASS: <run-directory>
	98	+```
	99	+
	100	+PASS means both consecutive ACLprof sessions completed and the script automatically checked:
	101	+
	102	+* Each session has CPU events, `PrivateUse1` kernels, and nonzero kernel links to actual CPU correlation IDs.
	103	+* All structured diagnostic record types listed above are present, with exactly one summary per session.
	104	+* At least one consistent `HostToDevice → MATCH_RESULT FOUND → ACLPROF_LINK FOUND` chain exists per session:
	105	+ the converted launch and HostToDevice resolution share `(kernel, launch)` with the match and link,
	106	+ and the matched Kineto ID equals the linked ID found in the profiler events.
	107	+* Clock conversion and task queue Kineto capture succeeded; capture, enqueue/dequeue, raw flow,
	108	+ kernel resolution, match attempt, `FOUND`, and `final_links` counters are positive.
	109	+* Session IDs are positive, distinct, and strictly increasing (consecutive numbering is not required).
	110	+* Each session has its own `manifest.json`, copied `start_info`, and `msprof.json` timeline files;
	111	+ every manifest entry was copied and exists. Session artifact paths are disjoint, and the first
	112	+ session's manifest and copied files retain their hashes after the second session.
	113	+
	114	+Exact counter values depend on the workload and runtime; they are not a fixed validation contract.
	115	+
	116	+### Failure and retained artifacts
	117	+
	118	+A validation exception returns exit code `1` and prints
	119	+`ACLprof correlation validation FAIL: artifacts retained at <run-directory>`.
	120	+
	121	+Inspect the retained data to investigate the failure. Native stderr and the Python traceback are captured
	122	+in `diagnostics.jsonl`, which may also contain non-JSON lines. An example layout is:
	123	+
	124	+```text
	125	+<run-directory>/
	126	+ diagnostics.jsonl
	127	+ trace_0.json
	128	+ trace_1.json
	129	+ session_<id>_<suffix>/
	130	+ manifest.json
	131	+ <original relative directories>/start_info*
	132	+ <original relative directories>/msprof*.json
	133	+```
	134	+
	135	+Each session directory preserves original relative paths listed in `manifest.json`; `trace_*.json` are Chrome
	136	+traces. Artifacts also remain after PASS; failure may leave partial output depending on where execution stopped.
	137	+Use `diagnostics.jsonl` to locate the break in `Kineto capture → enqueue/dequeue → clock conversion
	138	+→ HostToDevice matching → final link`, correlating records by session and kernel/launch IDs.
	139	+
	140	+### Interpreting summary counters
	141	+
	142	+\| Counter \| Meaning \|
	143	+\| --- \| --- \|
	144	+\| `kineto_correlations_captured` \| Successful captures of an active Kineto CPU correlation at enqueue-start; `> 0` proves capture occurred. \|
	145	+\| `enqueue_marks` \| Raw enqueue task queue marks collected; validation requires `> 0`. \|
	146	+\| `dequeue_marks` \| Raw dequeue worker marks collected; validation requires `> 0`. \|
	147	+\| `successful_clock_conversions` \| Successful clock-domain conversions per kernel; validation requires `> 0`. \|
	148	+\| `raw_host_to_device` \| Raw HostToDevice flow endpoints in the ACLprof export; validation requires `> 0`. \|
	149	+\| `FOUND` \| Match attempts that resolved a correlation candidate; validation requires `> 0`. \|
	150	+\| `NOT_FOUND` \| Unresolved match attempts. \|
	151	+\| `AMBIGUOUS` \| Attempts with ambiguous candidates, where the matcher fails closed without choosing a link. \|
	152	+\| `final_links` \| Actual final kernel links to CPU activities; validation requires `> 0`. \|
	153	+
	154	+`NOT_FOUND` or `AMBIGUOUS` being nonzero does not alone fail validation: the script requires a proven
	155	+positive end-to-end path and the checks above, not successful linkage of every kernel.
	156	+Record runtime versions, device, command, exit code, and output path in review. Without hardware, report
	157	+`Hardware correlation validation: NOT RUN`; real export, clocks, worker TIDs, and links remain unvalidated.

Mtest/cpp/api/CMakeLists.txt+4-1

	@@ -1,2 +1,5 @@
1	set(TORCH_API_TEST_DIR "${PROJECT_SOURCE_DIR}/test/cpp/api")	1	set(TORCH_API_TEST_DIR "${PROJECT_SOURCE_DIR}/test/cpp/api")
2	-set(TORCH_API_TEST_SOURCES ${TORCH_API_TEST_DIR}/modules.cpp PARENT_SCOPE)	2	+set(TORCH_API_TEST_SOURCES
		3	+ ${TORCH_API_TEST_DIR}/modules.cpp
		4	+ ${TORCH_API_TEST_DIR}/profiler.cpp
		5	+ PARENT_SCOPE)

Atest/cpp/api/profiler.cpp+1176-0

Mtest/profiler/test_experimental_config.py+9-0

	@@ -1,6 +1,9 @@
		1	+import pickle
1	import unittest	2	import unittest
2	from unittest.mock import patch	3	from unittest.mock import patch
3		4
		5	+from torch._C._profiler import _ExperimentalConfig as KinetoExperimentalConfig
		6	+from torch_npu.profiler import _NPUExperimentalConfig
4	from torch_npu.profiler.experimental_config import supported_ai_core_metrics	7	from torch_npu.profiler.experimental_config import supported_ai_core_metrics
5	from torch_npu.profiler.experimental_config import supported_profiler_level	8	from torch_npu.profiler.experimental_config import supported_profiler_level
6	from torch_npu.profiler.experimental_config import supported_export_type	9	from torch_npu.profiler.experimental_config import supported_export_type
	@@ -47,6 +50,12 @@ class TestExperimentalConfig(TestCase):
47	experimental_config = _ExperimentalConfig()	50	experimental_config = _ExperimentalConfig()
48	self.assertTrue(isinstance(experimental_config(), Cpp_ExperimentalConfig))	51	self.assertTrue(isinstance(experimental_config(), Cpp_ExperimentalConfig))
49		52
		53	+ def test_kineto_backend_config(self):
		54	+ standard_config = pickle.dumps(KinetoExperimentalConfig())
		55	+ aclprof_config = pickle.dumps(_NPUExperimentalConfig(backend="aclprof"))
		56	+ self.assertNotIn(b'{"backend":"aclprof"}', standard_config)
		57	+ self.assertIn(b'{"backend":"aclprof"}', aclprof_config)
		58	+
50	def test_mstx_domain_switches_will_reset_when_msproftx_and_mstx_not_enabled(self):	59	def test_mstx_domain_switches_will_reset_when_msproftx_and_mstx_not_enabled(self):
51	experimental_config = _ExperimentalConfig(msprof_tx=False,	60	experimental_config = _ExperimentalConfig(msprof_tx=False,
52	mstx=False,	61	mstx=False,

Atools/kineto_aclprof_correlation_validation.py+161-0

@@ -0,0 +1,161 @@
	1	+#!/usr/bin/env python3
	2	+"""Manual maintainer hardware validation; not an automated CI test."""
	3	+
	4	+import argparse
	5	+import hashlib
	6	+import json
	7	+import os
	8	+from pathlib import Path
	9	+import shutil
	10	+import sys
	11	+import tempfile
	12	+import traceback
	13	+
	14	+
	15	+REQUIRED = {"ACLPROF_" + name for name in (
	16	+ "CLOCK_REFERENCE CLOCK_CONVERT KINETO_STACK TASK_QUEUE_MARK ENQUEUE DEQUEUE "
	17	+ "KERNEL_RESOLVE HOST_TO_DEVICE MATCH_BEGIN MATCH_CANDIDATE MATCH_RESULT LINK CORRELATION_SUMMARY"
	18	+).split()}
	19	+
	20	+
	21	+def require(condition, message):
	22	+ if not condition:
	23	+ raise RuntimeError(message)
	24	+
	25	+
	26	+def read_records(path):
	27	+ records = []
	28	+ for line in path.read_text(errors="replace").splitlines():
	29	+ try:
	30	+ row = json.loads(line)
	31	+ except ValueError:
	32	+ continue
	33	+ if isinstance(row, dict) and row.get("record") in REQUIRED:
	34	+ records.append(row)
	35	+ return records
	36	+
	37	+
	38	+def validate(out, summary, rows, linked_ids):
	39	+ sid = summary["session"]
	40	+ require(REQUIRED <= {r["record"] for r in rows}, f"session {sid}: missing diagnostics")
	41	+
	42	+ def select(name, **fields):
	43	+ return [r for r in rows if r["record"] == "ACLPROF_" + name
	44	+ and all(r.get(k) == v for k, v in fields.items())]
	45	+
	46	+ launches = {(r["kernel"], r["launch"]) for r in select("HOST_TO_DEVICE", stage="converted_launch")}
	47	+ resolved = {(r["kernel"], r["launch"]) for r in select("KERNEL_RESOLVE", method="HostToDevice", result="FOUND")}
	48	+ matches = {(r["kernel"], r["launch"], r["kineto"]) for r in select("MATCH_RESULT", result="FOUND")}
	49	+ flow_kernels = launches & resolved
	50	+ require(any((r["kernel"], r["launch"]) in flow_kernels
	51	+ and (r["kernel"], r["launch"], r["kineto"]) in matches
	52	+ and r["kineto"] == r["linked_correlation_id"] and r["kineto"] in linked_ids
	53	+ for r in select("LINK", result="FOUND")),
	54	+ f"session {sid}: missing consistent HostToDevice/match/link chain")
	55	+ for key in ("raw_host_to_device", "successful_clock_conversions", "enqueue_marks", "dequeue_marks",
	56	+ "ACLPROF_KERNEL_RESOLVE", "ACLPROF_MATCH_BEGIN", "FOUND", "final_links",
	57	+ "kineto_correlations_captured"):
	58	+ require(summary["counters"].get(key, 0) > 0, f"session {sid}: zero {key}")
	59	+ manifests = list(out.glob(f"session_{sid}_*/manifest.json"))
	60	+ require(len(manifests) == 1, f"session {sid}: missing/duplicate artifacts")
	61	+ manifest = json.loads(manifests[0].read_text())
	62	+ require(manifest["session"] == sid, "manifest session mismatch")
	63	+ files = manifest["files"]
	64	+ names = [Path(f["path"]).name for f in files]
	65	+ require(any(n.startswith("start_info") for n in names), "missing start_info")
	66	+ require(any(n.startswith("msprof") and n.endswith(".json") for n in names), "missing timeline JSON")
	67	+ paths = [manifests[0]]
	68	+ for file in files:
	69	+ path = manifests[0].parent / file["path"]
	70	+ require(file["copied"] and path.is_file(), f"artifact copy failed: {path}")
	71	+ paths.append(path)
	72	+ print("session", sid, summary["counters"], "artifacts", manifests[0].parent, flush=True)
	73	+ return {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in paths}
	74	+
	75	+
	76	+def run(out, device):
	77	+ require(shutil.which("msprof"), "msprof is not on PATH")
	78	+ import torch
	79	+ import torch_npu # noqa: F401
	80	+ from torch.profiler import ProfilerActivity
	81	+ from torch_npu.profiler import _NPUExperimentalConfig
	82	+
	83	+ torch.npu.set_device(device)
	84	+ tensors = [torch.randn(n, n, device=f"npu:{device}") for n in (128, 192)]
	85	+
	86	+ def workload():
	87	+ for x in tensors:
	88	+ y = torch.mm(x, x).relu()
	89	+ y.add_(1)
	90	+ y.sin()
	91	+ x.normal_()
	92	+
	93	+ for _ in range(2):
	94	+ workload()
	95	+ torch.npu.synchronize()
	96	+ ids, preserved = [], {}
	97	+ for index in range(2):
	98	+ with torch.profiler.profile(
	99	+ activities=[ProfilerActivity.CPU, ProfilerActivity.PrivateUse1],
	100	+ experimental_config=_NPUExperimentalConfig(backend="aclprof"),
	101	+ record_shapes=True, profile_memory=True, with_stack=True, with_flops=True,
	102	+ ) as prof:
	103	+ for _ in range(2):
	104	+ workload()
	105	+ torch.npu.synchronize()
	106	+ prof.export_chrome_trace(str(out / f"trace_{index}.json"))
	107	+ events = prof.profiler.kineto_results.events()
	108	+ cpu_ids = {e.correlation_id() for e in events if e.device_type() == torch.autograd.DeviceType.CPU}
	109	+ kernels = [e for e in events if e.device_type() == torch.autograd.DeviceType.PrivateUse1]
	110	+ linked_ids = {e.linked_correlation_id() for e in kernels} & (cpu_ids - {0})
	111	+ require(cpu_ids and kernels and linked_ids, f"session {index}: missing CPU/kernels/valid links")
	112	+ records = read_records(out / "diagnostics.jsonl")
	113	+ summaries = [r for r in records if r["record"] == "ACLPROF_CORRELATION_SUMMARY"]
	114	+ require(len(summaries) == index + 1, "expected one summary per session")
	115	+ summary = summaries[-1]
	116	+ sid = summary["session"]
	117	+ require(sid > 0 and (not ids or sid > ids[-1]), "session IDs must increase")
	118	+ ids.append(sid)
	119	+ current = validate(out, summary, [r for r in records if r["session"] == sid], linked_ids)
	120	+ require(not preserved.keys() & current.keys(), "session artifacts share paths")
	121	+ for path, digest in preserved.items():
	122	+ require(path.is_file() and hashlib.sha256(path.read_bytes()).hexdigest() == digest,
	123	+ f"previous session artifact changed: {path}")
	124	+ preserved.update(current)
	125	+ require(len(list(out.glob("session_*/manifest.json"))) == 2, "expected two artifact directories")
	126	+
	127	+
	128	+def main():
	129	+ parser = argparse.ArgumentParser(description=__doc__)
	130	+ parser.add_argument("--output-dir", type=Path, required=True, help="Parent for a unique run directory")
	131	+ parser.add_argument("--device", type=int, default=0, help="Logical NPU device index (default: 0)")
	132	+ args = parser.parse_args()
	133	+ args.output_dir.mkdir(parents=True, exist_ok=True)
	134	+ out = Path(tempfile.mkdtemp(prefix="aclprof_correlation_validation_", dir=args.output_dir)).resolve()
	135	+ # Must precede torch_npu import and NPU initialization.
	136	+ os.environ["TASK_QUEUE_ENABLE"] = "1"
	137	+ os.environ["ASCEND_LAUNCH_BLOCKING"] = "0"
	138	+ os.environ["TORCH_NPU_ACLPROF_CORRELATION_DEBUG"] = "1"
	139	+ os.environ["TORCH_NPU_ACLPROF_DIAGNOSTICS_DIR"] = str(out)
	140	+ effective = {k: os.environ[k] for k in ("TASK_QUEUE_ENABLE", "ASCEND_LAUNCH_BLOCKING",
	141	+ "TORCH_NPU_ACLPROF_CORRELATION_DEBUG", "TORCH_NPU_ACLPROF_DIAGNOSTICS_DIR")}
	142	+ print(json.dumps({"device": args.device, **effective}), flush=True)
	143	+ with (out / "diagnostics.jsonl").open("w") as log:
	144	+ saved_stderr = os.dup(2)
	145	+ try:
	146	+ os.dup2(log.fileno(), 2) # Capture native fprintf(stderr) as well as Python errors.
	147	+ run(out, args.device)
	148	+ except Exception:
	149	+ traceback.print_exc()
	150	+ print(f"ACLprof correlation validation FAIL: artifacts retained at {out}", flush=True)
	151	+ return 1
	152	+ finally:
	153	+ sys.stderr.flush()
	154	+ os.dup2(saved_stderr, 2)
	155	+ os.close(saved_stderr)
	156	+ print(f"ACLprof correlation validation PASS: {out}")
	157	+ return 0
	158	+
	159	+
	160	+if __name__ == "__main__":
	161	+ sys.exit(main())

Mtorch_npu/csrc/core/npu/interface/AclInterface.cpp+11-0

	@@ -136,6 +136,7 @@ TORCH_NPU_LOAD_FUNC(aclrtMemset)
136	TORCH_NPU_LOAD_FUNC(aclrtMemsetAsync)	136	TORCH_NPU_LOAD_FUNC(aclrtMemsetAsync)
137	TORCH_NPU_LOAD_FUNC(aclmdlRICaptureThreadExchangeMode)	137	TORCH_NPU_LOAD_FUNC(aclmdlRICaptureThreadExchangeMode)
138	TORCH_NPU_LOAD_FUNC(aclrtGetLogicDevIdByUserDevId)	138	TORCH_NPU_LOAD_FUNC(aclrtGetLogicDevIdByUserDevId)
		139	+TORCH_NPU_LOAD_FUNC(aclrtGetPhyDevIdByUserDevId)
139	TORCH_NPU_LOAD_FUNC(aclrtBinaryLoadFromData)	140	TORCH_NPU_LOAD_FUNC(aclrtBinaryLoadFromData)
140	TORCH_NPU_LOAD_FUNC(aclrtBinaryGetFunction)	141	TORCH_NPU_LOAD_FUNC(aclrtBinaryGetFunction)
141	TORCH_NPU_LOAD_FUNC(aclrtBinaryUnLoad)	142	TORCH_NPU_LOAD_FUNC(aclrtBinaryUnLoad)
	@@ -1943,6 +1944,16 @@ aclError AclrtGetLogicDevIdByUserDevId(const int32_t userDevid, int32_t* const l
1943	TORCH_CHECK(func, "Failed to find function aclrtGetLogicDevIdByUserDevId", PTA_ERROR(ErrCode::NOT_FOUND));	1944	TORCH_CHECK(func, "Failed to find function aclrtGetLogicDevIdByUserDevId", PTA_ERROR(ErrCode::NOT_FOUND));
1944	return func(userDevid, logicDevId);	1945	return func(userDevid, logicDevId);
1945	}	1946	}
		1947	+
		1948	+aclError AclrtGetPhyDevIdByUserDevId(const int32_t userDevId, int32_t* const phyDevId) {
		1949	+ typedef aclError (AclrtGetPhyDevIdByUserDevId)(const int32_t, int32_t const);
		1950	+ static AclrtGetPhyDevIdByUserDevId func = nullptr;
		1951	+ if (func == nullptr) {
		1952	+ func = (AclrtGetPhyDevIdByUserDevId)TORCH_NPU_GET_FUNC(aclrtGetPhyDevIdByUserDevId);
		1953	+ }
		1954	+ TORCH_CHECK(func, "Failed to find function aclrtGetPhyDevIdByUserDevId", PTA_ERROR(ErrCode::NOT_FOUND));
		1955	+ return func(userDevId, phyDevId);
		1956	+}
1946	bool IsExistAclrtLaunchKernelWithHostArgs() {	1957	bool IsExistAclrtLaunchKernelWithHostArgs() {
1947	using Func = decltype(&aclrtLaunchKernelWithHostArgs);	1958	using Func = decltype(&aclrtLaunchKernelWithHostArgs);
1948	static Func func = (Func)TORCH_NPU_GET_FUNC(aclrtLaunchKernelWithHostArgs);	1959	static Func func = (Func)TORCH_NPU_GET_FUNC(aclrtLaunchKernelWithHostArgs);

Mtorch_npu/csrc/core/npu/interface/AclInterface.h+2-0

	@@ -442,6 +442,8 @@ aclError AclrtGetDeviceInfo(uint32_t deviceId, aclrtDevAttr attr, int64_t* value
442		442
443	aclError AclrtGetLogicDevIdByUserDevId(const int32_t userDevid, int32_t* const logicDevId);	443	aclError AclrtGetLogicDevIdByUserDevId(const int32_t userDevid, int32_t* const logicDevId);
444		444
		445	+aclError AclrtGetPhyDevIdByUserDevId(const int32_t userDevId, int32_t* const phyDevId);
		446	+
445	aclError AclrtMemSet(void* devPtr, size_t maxCount, int32_t value, size_t count);	447	aclError AclrtMemSet(void* devPtr, size_t maxCount, int32_t value, size_t count);
446		448
447	aclError AclrtMemSetAsync(void* devPtr, size_t maxCount, int32_t value, size_t count, aclrtStream stream);	449	aclError AclrtMemSetAsync(void* devPtr, size_t maxCount, int32_t value, size_t count, aclrtStream stream);

Mtorch_npu/csrc/framework/utils/NpuUtils.cpp+5-2

	@@ -17,6 +17,7 @@
17	#include "torch_npu/csrc/framework/utils/OpAdapter.h"	17	#include "torch_npu/csrc/framework/utils/OpAdapter.h"
18	#include "torch_npu/csrc/framework/utils/OpPreparation.h"	18	#include "torch_npu/csrc/framework/utils/OpPreparation.h"
19	#ifndef BUILD_LIBTORCH	19	#ifndef BUILD_LIBTORCH
		20	+#include "torch_npu/csrc/profiler/aclprof_activity_profiler.h"
20	#include "torch_npu/csrc/profiler/npu_profiler.h"	21	#include "torch_npu/csrc/profiler/npu_profiler.h"
21	#endif	22	#endif
22		23
	@@ -275,7 +276,8 @@ void NpuUtils::ProfReportMarkDataToNpuProfiler(uint32_t category, const std::str
275	if (data.empty()) {	276	if (data.empty()) {
276	return;	277	return;
277	}	278	}
278	- if (torch_npu::profiler::profDataReportEnable().load(std::memory_order_relaxed)) {	279	+ if (torch_npu::profiler::profDataReportEnable().load(std::memory_order_relaxed) \|\|
		280	+ torch_npu::profiler::aclprofTaskQueueCaptureEnabled()) {
279	torch_npu::profiler::reportMarkDataToNpuProfiler(category, data, correlation_id);	281	torch_npu::profiler::reportMarkDataToNpuProfiler(category, data, correlation_id);
280	}	282	}
281	}	283	}
	@@ -304,7 +306,8 @@ void NpuUtils::ProfReportMarkDataToNpuProfiler(uint32_t category, void* data, si
304	if (C10_UNLIKELY(!data)) {	306	if (C10_UNLIKELY(!data)) {
305	return;	307	return;
306	}	308	}
307	- if (torch_npu::profiler::profDataReportEnable().load(std::memory_order_relaxed)) {	309	+ if (torch_npu::profiler::profDataReportEnable().load(std::memory_order_relaxed) \|\|
		310	+ torch_npu::profiler::aclprofTaskQueueCaptureEnabled()) {
308	static const std::map<int64_t, DqueueCall> DEQUEUE_CALL_FUNC_MAP{	311	static const std::map<int64_t, DqueueCall> DEQUEUE_CALL_FUNC_MAP{
309	{c10_npu::queue::COMPILE_AND_EXECUTE, &DqueueCompileExcute},	312	{c10_npu::queue::COMPILE_AND_EXECUTE, &DqueueCompileExcute},
310	{c10_npu::queue::EXECUTE_OPAPI, &DqueueCompileExcuteOpApi},	313	{c10_npu::queue::EXECUTE_OPAPI, &DqueueCompileExcuteOpApi},

Atorch_npu/csrc/profiler/aclprof_activity_profiler.cpp+1287-0

Atorch_npu/csrc/profiler/aclprof_activity_profiler.h+112-0

@@ -0,0 +1,112 @@
	1	+#pragma once
	2	+
	3	+#include <cstdint>
	4	+#include <memory>
	5	+#include <mutex>
	6	+#include <optional>
	7	+#include <set>
	8	+#include <string>
	9	+#include <unordered_map>
	10	+#include <vector>
	11	+
	12	+#include "torch_npu/csrc/profiler/npu_activity_profiler.h"
	13	+
	14	+namespace torch_npu {
	15	+namespace profiler {
	16	+
	17	+namespace detail {
	18	+struct AclprofApi;
	19	+struct AclprofDiagnostics;
	20	+struct AclprofEvent;
	21	+} // namespace detail
	22	+
	23	+class AclprofActivityProfilerTestPeer;
	24	+
	25	+TORCH_NPU_API bool aclprofTaskQueueCaptureEnabled();
	26	+TORCH_NPU_API void recordAclprofTaskQueueEvent(
	27	+ uint32_t category,
	28	+ uint64_t taskQueueCorrelation,
	29	+ int64_t monotonicTimestampNs,
	30	+ int64_t pid,
	31	+ int64_t tid,
	32	+ const std::string& name);
	33	+
	34	+class TORCH_NPU_API AclprofActivityProfiler : public NPUActivityProfiler {
	35	+ public:
	36	+ explicit AclprofActivityProfiler(std::set<libkineto::ActivityType> activities);
	37	+ ~AclprofActivityProfiler() override;
	38	+
	39	+ const char* name() const override {
	40	+ return "aclprof";
	41	+ }
	42	+ void start() override;
	43	+ void stop() override;
	44	+ std::vector<std::string> errors() override;
	45	+ void processTrace(libkineto::ActivityLogger& logger) override;
	46	+ void processTrace(
	47	+ libkineto::ActivityLogger& logger,
	48	+ libkineto::getLinkedActivityCallback getLinkedActivity,
	49	+ int64_t startTime,
	50	+ int64_t endTime) override;
	51	+ std::unique_ptr<libkineto::DeviceInfo> getDeviceInfo() override;
	52	+ std::vector<libkineto::ResourceInfo> getResourceInfos() override;
	53	+ std::unique_ptr<libkineto::CpuTraceBuffer> getTraceBuffer() override;
	54	+ void pushCorrelationId(uint64_t id) override;
	55	+ void popCorrelationId() override;
	56	+
	57	+ private:
	58	+ struct TaskQueueMark {
	59	+ uint32_t category;
	60	+ uint64_t taskQueueCorrelation;
	61	+ int64_t timestampNs;
	62	+ int64_t tid;
	63	+ std::string name;
	64	+ std::optional<int32_t> kinetoCorrelation;
	65	+ };
	66	+
	67	+ AclprofActivityProfiler(std::set<libkineto::ActivityType> activities, std::shared_ptr<detail::AclprofApi> api);
	68	+ bool finishCollection(bool reportErrors);
	69	+ void cleanup();
	70	+ void resetCorrelationState();
	71	+ void beginCorrelationCapture();
	72	+ void endCorrelationCapture();
	73	+ void recordTaskQueueEvent(
	74	+ uint32_t category,
	75	+ uint64_t taskQueueCorrelation,
	76	+ int64_t monotonicTimestampNs,
	77	+ int64_t pid,
	78	+ int64_t tid,
	79	+ const std::string& name);
	80	+ void loadTaskQueueClockOffset(const std::string& profile);
	81	+ void resolveCorrelations();
	82	+ void summarizeDiagnostics();
	83	+
	84	+ friend class AclprofActivityProfilerTestPeer;
	85	+ friend void recordAclprofTaskQueueEvent(
	86	+ uint32_t category,
	87	+ uint64_t taskQueueCorrelation,
	88	+ int64_t monotonicTimestampNs,
	89	+ int64_t pid,
	90	+ int64_t tid,
	91	+ const std::string& name);
	92	+ std::set<libkineto::ActivityType> activities_;
	93	+ std::shared_ptr<detail::AclprofApi> api_;
	94	+ std::vector<detail::AclprofEvent> events_;
	95	+ std::vector<std::string> errors_;
	96	+ std::string root_;
	97	+ void* config_ = nullptr;
	98	+ int64_t device_ = -1;
	99	+ std::mutex correlationMutex_;
	100	+ std::shared_ptr<detail::AclprofDiagnostics> diagnostics_;
	101	+ std::unordered_map<int64_t, std::vector<std::optional<int32_t>>> activeCorrelations_;
	102	+ std::vector<TaskQueueMark> taskQueueMarks_;
	103	+ std::optional<int64_t> taskQueueClockOffsetNs_;
	104	+ std::vector<std::optional<int32_t>> linkedCorrelations_;
	105	+ bool initialized_ = false;
	106	+ bool started_ = false;
	107	+ bool captureCorrelations_ = false;
	108	+ bool correlationsResolved_ = false;
	109	+};
	110	+
	111	+} // namespace profiler
	112	+} // namespace torch_npu

Atorch_npu/csrc/profiler/aclprof_activity_profiler_internal.h+45-0

@@ -0,0 +1,45 @@
	1	+#pragma once
	2	+
	3	+#include <cstdint>
	4	+#include <functional>
	5	+#include <string>
	6	+#include <vector>
	7	+
	8	+namespace torch_npu::profiler::detail {
	9	+
	10	+struct AclprofEvent {
	11	+ std::string name;
	12	+ std::string taskType;
	13	+ int64_t startNs = 0;
	14	+ int64_t endNs = 0;
	15	+ int64_t device = 0;
	16	+ int64_t stream = 0;
	17	+ int64_t correlation = 0;
	18	+ uint64_t hostLaunchId = 0;
	19	+ int64_t flowStartNs = 0;
	20	+ int64_t flowStartPid = -1;
	21	+ int64_t flowStartTid = -1;
	22	+};
	23	+
	24	+struct AclprofDiagnostics;
	25	+
	26	+// physicalDevice is the runtime-resolved physical device for this single-device session.
	27	+std::vector<AclprofEvent> parseAclprofTimeline(
	28	+ const std::string& path,
	29	+ int64_t physicalDevice,
	30	+ std::vector<std::string>& errors,
	31	+ AclprofDiagnostics* diagnostics = nullptr);
	32	+
	33	+struct AclprofApi {
	34	+ std::function<int32_t(int32_t*)> getDevice;
	35	+ std::function<int32_t(int32_t, int32_t*)> getPhysicalDevice;
	36	+ std::function<int32_t(const std::string&)> init;
	37	+ std::function<void(uint32_t)> createConfig;
	38	+ std::function<int32_t()> start;
	39	+ std::function<int32_t()> registerDeviceCallback;
	40	+ std::function<int32_t()> stop;
	41	+ std::function<int32_t()> destroyConfig;
	42	+ std::function<int32_t()> finalize;
	43	+};
	44	+
	45	+} // namespace torch_npu::profiler::detail

Mtorch_npu/csrc/profiler/mspti_activity_profiler.cpp+0-40

	@@ -19,7 +19,6 @@
19	#include "torch_npu/csrc/framework/interface/MsptiInterface.h"	19	#include "torch_npu/csrc/framework/interface/MsptiInterface.h"
20	#include "torch_npu/csrc/core/npu/npu_log.h"	20	#include "torch_npu/csrc/core/npu/npu_log.h"
21	#include "torch_npu/csrc/toolkit/profiler/common/utils.h"	21	#include "torch_npu/csrc/toolkit/profiler/common/utils.h"
22	-#include <torch/csrc/profiler/standalone/privateuse1_profiler.h>
23		22
24	namespace torch_npu {	23	namespace torch_npu {
25	namespace profiler {	24	namespace profiler {
	@@ -47,12 +46,6 @@ bool g_sessionActive = false;
47		46
48	int64_t g_corrCounter = 1LL << 30; // 1,073,741,824 (< 2^32 - 1)	47	int64_t g_corrCounter = 1LL << 30; // 1,073,741,824 (< 2^32 - 1)
49		48
50	-bool privateUse1Requested(const std::set<libkineto::ActivityType>& types) {
51	- using AT = libkineto::ActivityType;
52	- return types.count(AT::CONCURRENT_KERNEL) != 0 \|\| types.count(AT::GPU_MEMCPY) != 0 \|\|
53	- types.count(AT::GPU_MEMSET) != 0 \|\| types.count(AT::GPU_USER_ANNOTATION) != 0;
54	-}
55	-
56	std::vector<msptiActivityKind> computeEnabledKinds(const std::set<libkineto::ActivityType>& acts) {	49	std::vector<msptiActivityKind> computeEnabledKinds(const std::set<libkineto::ActivityType>& acts) {
57	using AT = libkineto::ActivityType;	50	using AT = libkineto::ActivityType;
58	std::vector<msptiActivityKind> ks;	51	std::vector<msptiActivityKind> ks;
	@@ -426,38 +419,5 @@ std::unique_ptr<libkineto::CpuTraceBuffer> MsptiActivityProfiler::getTraceBuffer
426	return nullptr;	419	return nullptr;
427	}	420	}
428		421
429	-std::unique_ptr<libkineto::IActivityProfilerSession> MsptiActivityProfilerPoc::configure(
430	- const std::set<libkineto::ActivityType>& types,
431	- const libkineto::Config&) {
432	- if (!privateUse1Requested(types)) {
433	- return nullptr;
434	- }
435	- return std::make_unique<MsptiActivityProfiler>(types);
436	-}
437	-
438	-std::unique_ptr<libkineto::IActivityProfilerSession> MsptiActivityProfilerPoc::configure(
439	- int64_t,
440	- int64_t,
441	- const std::set<libkineto::ActivityType>& types,
442	- const libkineto::Config& cfg) {
443	- return configure(types, cfg);
444	-}
445	-
446	-// ------------------------------ Registration -------------------------------
447	-// Prefer the upstream macro (stores the factory in PrivateUse1ProfilerRegistry);
448	-#ifdef REGISTER_PRIVATEUSE1_PROFILER
449	-REGISTER_PRIVATEUSE1_PROFILER(MsptiActivityProfilerPoc);
450	-#else
451	-namespace {
452	-struct MsptiAutoRegister {
453	- MsptiAutoRegister() {
454	- libkineto::api().registerProfilerFactory(
455	- []() -> std::unique_ptr<libkineto::IActivityProfiler> { return std::make_unique<MsptiActivityProfilerPoc>(); });
456	- }
457	-};
458	-static MsptiAutoRegister g_mspti_auto_register;
459	-} // namespace
460	-#endif
461	-
462	} // namespace profiler	422	} // namespace profiler
463	} // namespace torch_npu	423	} // namespace torch_npu

Mtorch_npu/csrc/profiler/mspti_activity_profiler.h+6-30

	@@ -1,17 +1,12 @@
1	#pragma once	1	#pragma once
2	-// mspti NPU activity-profiler session + libkineto bridge.	2	+// MSPTI-specific NPU activity-profiler backend.
3	-//
4	-// MsptiActivityProfilerPoc (libkineto::IActivityProfiler) registers for
5	-// ProfilerActivity.PrivateUse1; configure() returns a MsptiActivityProfiler,
6	-// which is the session libkineto drives. The session enables mspti kinds from
7	-// the requested activity-type set and collects NPU kernel and host API records.
8		3
9	#include <memory>	4	#include <memory>
10	#include <set>	5	#include <set>
11	#include <string>	6	#include <string>
12	#include <vector>	7	#include <vector>
13		8
14	-#include "IActivityProfiler.h"	9	+#include "torch_npu/csrc/profiler/npu_activity_profiler.h"
15		10
16	namespace torch_npu {	11	namespace torch_npu {
17	namespace profiler {	12	namespace profiler {
	@@ -31,10 +26,13 @@ struct MsptiEvent {
31	int64_t streamId = 0;	26	int64_t streamId = 0;
32	};	27	};
33		28
34	-class MsptiActivityProfiler : public libkineto::IActivityProfilerSession {	29	+class MsptiActivityProfiler : public NPUActivityProfiler {
35	public:	30	public:
36	explicit MsptiActivityProfiler(std::set<libkineto::ActivityType> activities) : activities_(std::move(activities)) {}	31	explicit MsptiActivityProfiler(std::set<libkineto::ActivityType> activities) : activities_(std::move(activities)) {}
37		32
		33	+ const char* name() const override {
		34	+ return "mspti";
		35	+ }
38	void start() override;	36	void start() override;
39	void stop() override;	37	void stop() override;
40	std::vector<std::string> errors() override;	38	std::vector<std::string> errors() override;
	@@ -56,27 +54,5 @@ class MsptiActivityProfiler : public libkineto::IActivityProfilerSession {
56	std::vector<std::string> errors_;	54	std::vector<std::string> errors_;
57	};	55	};
58		56
59	-class MsptiActivityProfilerPoc : public libkineto::IActivityProfiler {
60	- public:
61	- const std::string& name() const override {
62	- return name_;
63	- }
64	- const std::set<libkineto::ActivityType>& availableActivities() const override {
65	- return avail_;
66	- }
67	- std::unique_ptr<libkineto::IActivityProfilerSession> configure(
68	- const std::set<libkineto::ActivityType>& activity_types,
69	- const libkineto::Config& config) override;
70	- std::unique_ptr<libkineto::IActivityProfilerSession> configure(
71	- int64_t ts_ms,
72	- int64_t duration_ms,
73	- const std::set<libkineto::ActivityType>& activity_types,
74	- const libkineto::Config& config) override;
75	-
76	- private:
77	- std::string name_{"mspti"};
78	- std::set<libkineto::ActivityType> avail_{libkineto::ActivityType::CONCURRENT_KERNEL};
79	-};
80	-
81	} // namespace profiler	57	} // namespace profiler
82	} // namespace torch_npu	58	} // namespace torch_npu

Atorch_npu/csrc/profiler/npu_activity_profiler.cpp+126-0

@@ -0,0 +1,126 @@
	1	+#include "torch_npu/csrc/profiler/npu_activity_profiler.h"
	2	+
	3	+#include <set>
	4	+#include <utility>
	5	+
	6	+#include <nlohmann/json.hpp>
	7	+
	8	+#include "libkineto.h"
	9	+#include "torch_npu/csrc/profiler/aclprof_activity_profiler.h"
	10	+#include "torch_npu/csrc/profiler/mspti_activity_profiler.h"
	11	+#include <torch/csrc/profiler/standalone/privateuse1_profiler.h>
	12	+
	13	+namespace torch_npu {
	14	+namespace profiler {
	15	+namespace {
	16	+
	17	+bool privateUse1Requested(const std::set<libkineto::ActivityType>& types) {
	18	+ using AT = libkineto::ActivityType;
	19	+ return types.count(AT::CONCURRENT_KERNEL) != 0 \|\| types.count(AT::GPU_MEMCPY) != 0 \|\|
	20	+ types.count(AT::GPU_MEMSET) != 0 \|\| types.count(AT::GPU_USER_ANNOTATION) != 0;
	21	+}
	22	+
	23	+bool useAclprofBackend(const std::string& customConfig) {
	24	+ const auto config = nlohmann::json::parse(customConfig, nullptr, false);
	25	+ if (!config.is_object()) {
	26	+ return false;
	27	+ }
	28	+ const auto backend = config.find("backend");
	29	+ return backend != config.end() && backend->is_string() && backend->get<std::string>() == "aclprof";
	30	+}
	31	+
	32	+} // namespace
	33	+
	34	+std::unique_ptr<NPUActivityProfiler> NPUActivityProfilerFactory::create(
	35	+ const std::set<libkineto::ActivityType>& activities,
	36	+ const libkineto::Config& config) {
	37	+ if (!privateUse1Requested(activities)) {
	38	+ return nullptr;
	39	+ }
	40	+ if (useAclprofBackend(config.getCustomConfig())) {
	41	+ return std::make_unique<AclprofActivityProfiler>(activities);
	42	+ }
	43	+ return std::make_unique<MsptiActivityProfiler>(activities);
	44	+}
	45	+
	46	+void PrivateUse1ProfilerManager::start() {
	47	+ profiler_->start();
	48	+ status_ = profiler_->status();
	49	+}
	50	+
	51	+void PrivateUse1ProfilerManager::stop() {
	52	+ profiler_->stop();
	53	+ status_ = profiler_->status();
	54	+}
	55	+
	56	+std::vector<std::string> PrivateUse1ProfilerManager::errors() {
	57	+ return profiler_->errors();
	58	+}
	59	+
	60	+void PrivateUse1ProfilerManager::processTrace(libkineto::ActivityLogger& logger) {
	61	+ profiler_->processTrace(logger);
	62	+}
	63	+
	64	+void PrivateUse1ProfilerManager::processTrace(
	65	+ libkineto::ActivityLogger& logger,
	66	+ libkineto::getLinkedActivityCallback getLinkedActivity,
	67	+ int64_t startTime,
	68	+ int64_t endTime) {
	69	+ profiler_->processTrace(logger, std::move(getLinkedActivity), startTime, endTime);
	70	+}
	71	+
	72	+std::unique_ptr<libkineto::DeviceInfo> PrivateUse1ProfilerManager::getDeviceInfo() {
	73	+ return profiler_->getDeviceInfo();
	74	+}
	75	+
	76	+std::vector<libkineto::ResourceInfo> PrivateUse1ProfilerManager::getResourceInfos() {
	77	+ return profiler_->getResourceInfos();
	78	+}
	79	+
	80	+std::unique_ptr<libkineto::CpuTraceBuffer> PrivateUse1ProfilerManager::getTraceBuffer() {
	81	+ return profiler_->getTraceBuffer();
	82	+}
	83	+
	84	+void PrivateUse1ProfilerManager::pushCorrelationId(uint64_t id) {
	85	+ profiler_->pushCorrelationId(id);
	86	+}
	87	+
	88	+void PrivateUse1ProfilerManager::popCorrelationId() {
	89	+ profiler_->popCorrelationId();
	90	+}
	91	+
	92	+std::unique_ptr<libkineto::IActivityProfilerSession> NPUActivityProfilerPlugin::configure(
	93	+ const std::set<libkineto::ActivityType>& activityTypes,
	94	+ const libkineto::Config& config) {
	95	+ auto profiler = NPUActivityProfilerFactory::create(activityTypes, config);
	96	+ if (profiler == nullptr) {
	97	+ return nullptr;
	98	+ }
	99	+ return std::make_unique<PrivateUse1ProfilerManager>(std::move(profiler));
	100	+}
	101	+
	102	+std::unique_ptr<libkineto::IActivityProfilerSession> NPUActivityProfilerPlugin::configure(
	103	+ int64_t,
	104	+ int64_t,
	105	+ const std::set<libkineto::ActivityType>& activityTypes,
	106	+ const libkineto::Config& config) {
	107	+ return configure(activityTypes, config);
	108	+}
	109	+
	110	+#ifdef REGISTER_PRIVATEUSE1_PROFILER
	111	+REGISTER_PRIVATEUSE1_PROFILER(NPUActivityProfilerPlugin);
	112	+#else
	113	+namespace {
	114	+struct NPUProfilerAutoRegister {
	115	+ NPUProfilerAutoRegister() {
	116	+ libkineto::api().registerProfilerFactory([]() -> std::unique_ptr<libkineto::IActivityProfiler> {
	117	+ return std::make_unique<NPUActivityProfilerPlugin>();
	118	+ });
	119	+ }
	120	+};
	121	+static NPUProfilerAutoRegister g_npu_profiler_auto_register;
	122	+} // namespace
	123	+#endif
	124	+
	125	+} // namespace profiler
	126	+} // namespace torch_npu

Atorch_npu/csrc/profiler/npu_activity_profiler.h+96-0

@@ -0,0 +1,96 @@
	1	+#pragma once
	2	+
	3	+#include <memory>
	4	+#include <set>
	5	+#include <string>
	6	+#include <utility>
	7	+#include <vector>
	8	+
	9	+#include "IActivityProfiler.h"
	10	+#include "torch_npu/csrc/core/npu/NPUMacros.h"
	11	+
	12	+namespace torch_npu {
	13	+namespace profiler {
	14	+
	15	+class NPUActivityProfiler {
	16	+ public:
	17	+ virtual ~NPUActivityProfiler() = default;
	18	+
	19	+ virtual const char* name() const = 0;
	20	+ virtual void start() = 0;
	21	+ virtual void stop() = 0;
	22	+ virtual std::vector<std::string> errors() = 0;
	23	+ virtual void processTrace(libkineto::ActivityLogger& logger) = 0;
	24	+ virtual void processTrace(
	25	+ libkineto::ActivityLogger& logger,
	26	+ libkineto::getLinkedActivityCallback getLinkedActivity,
	27	+ int64_t startTime,
	28	+ int64_t endTime) = 0;
	29	+ virtual std::unique_ptr<libkineto::DeviceInfo> getDeviceInfo() = 0;
	30	+ virtual std::vector<libkineto::ResourceInfo> getResourceInfos() = 0;
	31	+ virtual std::unique_ptr<libkineto::CpuTraceBuffer> getTraceBuffer() = 0;
	32	+ virtual void pushCorrelationId(uint64_t) {}
	33	+ virtual void popCorrelationId() {}
	34	+
	35	+ libkineto::TraceStatus status() const {
	36	+ return status_;
	37	+ }
	38	+
	39	+ protected:
	40	+ libkineto::TraceStatus status_ = libkineto::TraceStatus::READY;
	41	+};
	42	+
	43	+class TORCH_NPU_API NPUActivityProfilerFactory {
	44	+ public:
	45	+ static std::unique_ptr<NPUActivityProfiler> create(
	46	+ const std::set<libkineto::ActivityType>& activities,
	47	+ const libkineto::Config& config);
	48	+};
	49	+
	50	+class TORCH_NPU_API PrivateUse1ProfilerManager : public libkineto::IActivityProfilerSession {
	51	+ public:
	52	+ explicit PrivateUse1ProfilerManager(std::unique_ptr<NPUActivityProfiler> profiler) : profiler_(std::move(profiler)) {}
	53	+
	54	+ void start() override;
	55	+ void stop() override;
	56	+ std::vector<std::string> errors() override;
	57	+ void processTrace(libkineto::ActivityLogger& logger) override;
	58	+ void processTrace(
	59	+ libkineto::ActivityLogger& logger,
	60	+ libkineto::getLinkedActivityCallback getLinkedActivity,
	61	+ int64_t startTime,
	62	+ int64_t endTime) override;
	63	+ std::unique_ptr<libkineto::DeviceInfo> getDeviceInfo() override;
	64	+ std::vector<libkineto::ResourceInfo> getResourceInfos() override;
	65	+ std::unique_ptr<libkineto::CpuTraceBuffer> getTraceBuffer() override;
	66	+ void pushCorrelationId(uint64_t id) override;
	67	+ void popCorrelationId() override;
	68	+
	69	+ private:
	70	+ std::unique_ptr<NPUActivityProfiler> profiler_;
	71	+};
	72	+
	73	+class TORCH_NPU_API NPUActivityProfilerPlugin : public libkineto::IActivityProfiler {
	74	+ public:
	75	+ const std::string& name() const override {
	76	+ return name_;
	77	+ }
	78	+ const std::set<libkineto::ActivityType>& availableActivities() const override {
	79	+ return avail_;
	80	+ }
	81	+ std::unique_ptr<libkineto::IActivityProfilerSession> configure(
	82	+ const std::set<libkineto::ActivityType>& activityTypes,
	83	+ const libkineto::Config& config) override;
	84	+ std::unique_ptr<libkineto::IActivityProfilerSession> configure(
	85	+ int64_t tsMs,
	86	+ int64_t durationMs,
	87	+ const std::set<libkineto::ActivityType>& activityTypes,
	88	+ const libkineto::Config& config) override;
	89	+
	90	+ private:
	91	+ std::string name_{"mspti"};
	92	+ std::set<libkineto::ActivityType> avail_{libkineto::ActivityType::CONCURRENT_KERNEL};
	93	+};
	94	+
	95	+} // namespace profiler
	96	+} // namespace torch_npu

Mtorch_npu/csrc/profiler/npu_profiler.cpp+11-0

	@@ -8,6 +8,7 @@
8	#include "torch_npu/csrc/core/npu/npu_log.h"	8	#include "torch_npu/csrc/core/npu/npu_log.h"
9	#include "torch_npu/csrc/core/npu/NPUException.h"	9	#include "torch_npu/csrc/core/npu/NPUException.h"
10	#include "torch_npu/csrc/core/npu/interface/AclInterface.h"	10	#include "torch_npu/csrc/core/npu/interface/AclInterface.h"
		11	+#include "torch_npu/csrc/profiler/aclprof_activity_profiler.h"
11	#include "torch_npu/csrc/profiler/npu_profiler.h"	12	#include "torch_npu/csrc/profiler/npu_profiler.h"
12		13
13	#include "torch_npu/csrc/toolkit/profiler/common/utils.h"	14	#include "torch_npu/csrc/toolkit/profiler/common/utils.h"
	@@ -386,6 +387,16 @@ void finalizeNpuProfiler() {
386	}	387	}
387		388
388	void reportMarkDataToNpuProfiler(uint32_t category, const std::string& msg, uint64_t correlation_id) {	389	void reportMarkDataToNpuProfiler(uint32_t category, const std::string& msg, uint64_t correlation_id) {
		390	+ recordAclprofTaskQueueEvent(
		391	+ category,
		392	+ correlation_id,
		393	+ static_cast<int64_t>(Utils::GetClockMonotonicRawNs()),
		394	+ static_cast<int64_t>(Utils::GetPid()),
		395	+ static_cast<int64_t>(Utils::GetTid()),
		396	+ msg);
		397	+ if (!profDataReportEnable().load(std::memory_order_relaxed)) {
		398	+ return;
		399	+ }
389	ProfilerMgr::GetInstance()->UploadWithLock(std::make_unique<torch_npu::toolkit::profiler::OpMarkData>(	400	ProfilerMgr::GetInstance()->UploadWithLock(std::make_unique<torch_npu::toolkit::profiler::OpMarkData>(
390	static_cast<int64_t>(Utils::GetClockTime()), category, correlation_id, Utils::GetTid(), Utils::GetPid(), msg));	401	static_cast<int64_t>(Utils::GetClockTime()), category, correlation_id, Utils::GetTid(), Utils::GetPid(), msg));
391	}	402	}

Mtorch_npu/profiler/__init__.py+3-1

	@@ -9,8 +9,10 @@ from .scheduler import Schedule as schedule
9	from .scheduler import ProfilerAction	9	from .scheduler import ProfilerAction
10	from .experimental_config import _ExperimentalConfig, supported_profiler_level, supported_ai_core_metrics, \	10	from .experimental_config import _ExperimentalConfig, supported_profiler_level, supported_ai_core_metrics, \
11	supported_export_type, ProfilerLevel, AiCMetrics, ExportType, HostSystem	11	supported_export_type, ProfilerLevel, AiCMetrics, ExportType, HostSystem
		12	+from ._kineto_config import _NPUExperimentalConfig
12	from ._non_intrusive_profile import _NonIntrusiveProfile	13	from ._non_intrusive_profile import _NonIntrusiveProfile
13		14
14	__all__ = ["profile", "ProfilerActivity", "supported_activities", "tensorboard_trace_handler", "schedule",	15	__all__ = ["profile", "ProfilerActivity", "supported_activities", "tensorboard_trace_handler", "schedule",
15	- "ProfilerAction", "_ExperimentalConfig", "supported_profiler_level", "supported_ai_core_metrics",	16	+ "ProfilerAction", "_ExperimentalConfig", "supported_profiler_level",
		17	+ "supported_ai_core_metrics",
16	"supported_export_type", "ProfilerLevel", "AiCMetrics", "ExportType", "HostSystem"]	18	"supported_export_type", "ProfilerLevel", "AiCMetrics", "ExportType", "HostSystem"]

Atorch_npu/profiler/_kineto_config.py+15-0

@@ -0,0 +1,15 @@
	1	+import json
	2	+
	3	+from torch._C._profiler import _ExperimentalConfig as ExperimentalConfig
	4	+
	5	+
	6	+class _NPUExperimentalConfig(ExperimentalConfig):
	7	+ """Ascend-specific options for ``torch.profiler.profile``."""
	8	+
	9	+ def __init__(self, , backend: str = "mspti", *kwargs):
	10	+ super().__init__(
	11	+ custom_profiler_config=json.dumps(
	12	+ {"backend": backend}, separators=(",", ":")
	13	+ ),
	14	+ **kwargs,
	15	+ )