| @@ -25,7 +25,7 @@ The profiler comprehensively profiles performance data in PyTorch training and o |
| | |
| - Install the matching version of the CANN Toolkit development suite or operator package and configure environment variables. For details, see CANN Quick Installation Guide. | - Install the matching version of the CANN Toolkit development suite or operator package and configure environment variables. For details, see CANN Quick Installation Guide. |
| <!-- [CANN Quick Installation Guide](https://www.hiascend.com/en/cann/download) --> | <!-- [CANN Quick Installation Guide](https://www.hiascend.com/en/cann/download) --> |
| -- Prepare a training model developed with PyTorch 2.1.0 or later and the corresponding dataset, then migrate the model to the Ascend AI processor by following the [Model Migration](https://gitcode.com/Ascend/ModelZoo-PyTorch/blob/master/PyTorch/docs/zh/model_migration/README.md) section in the *PyTorch Model Migration and Tuning Guide*. | +- Prepare a training model developed with PyTorch 2.1.0 or later and the corresponding dataset, then migrate the model to the Ascend AI processor by following the [Model Migration](https://gitcode.com/Ascend/ModelZoo-PyTorch/blob/master/PyTorch/docs/en/model_migration/README.md) section in the *PyTorch Model Migration and Tuning Guide*. |
| | |
| ### Constraints | ### Constraints |
| | |
| @@ -330,11 +330,11 @@ The usage process is as follows: |
| | |
| 1. Add the Ascend PyTorch Profiler interface to the PyTorch script, and enable the custom marking function through the `mstx` parameter of [experimental_config](#experimental_config-parameter-description). | 1. Add the Ascend PyTorch Profiler interface to the PyTorch script, and enable the custom marking function through the `mstx` parameter of [experimental_config](#experimental_config-parameter-description). |
| 2. (Optional) Set the profiler_level switch to Level_none (you can configure the corresponding level based on actual collection needs) and use the domain filtering attributes mstx_domain_include or mstx_domain_exclude to collect marking data. | 2. (Optional) Set the profiler_level switch to Level_none (you can configure the corresponding level based on actual collection needs) and use the domain filtering attributes mstx_domain_include or mstx_domain_exclude to collect marking data. |
| -3. In the PyTorch script, call the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/zh/custom_APIs/torch_npu-npu/torch_npu-npu_list.md#table17382716193111) series APIs for the events you want to profile, to profile the duration of the corresponding events. | +3. In the PyTorch script, call the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/en/custom_APIs/torch_npu-npu/torch_npu-npu_list.md#table17382716193111) series APIs for the events you want to profile, to profile the duration of the corresponding events. |
| | |
| #### Precautions | #### Precautions |
| | |
| -If the user's code only involves pure CPU-side operations, there is no need to pass the stream parameter when calling the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/zh/custom_APIs/torch_npu-npu/torch_npu-npu_list.md#table17382716193111) series APIs. If the user's code includes device operations and it is necessary to observe the duration on the device side, the stream parameter must be passed. | +If the user's code only involves pure CPU-side operations, there is no need to pass the stream parameter when calling the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/en/custom_APIs/torch_npu-npu/torch_npu-npu_list.md#table17382716193111) series APIs. If the user's code includes device operations and it is necessary to observe the duration on the device side, the stream parameter must be passed. |
| | |
| #### Usage Example | #### Usage Example |
| | |
| @@ -1425,7 +1425,7 @@ This file is a table structure file. It is recommended to use the MindStudio Ins |
| | with_modules | Optional | Records Python call stacks at the modules level, i.e., call information at the framework layer, Bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>Takes effect when torch_npu.profiler.ProfilerActivity.CPU is enabled. <br/>Enabling this configuration introduces additional performance inflation. | | | with_modules | Optional | Records Python call stacks at the modules level, i.e., call information at the framework layer, Bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>Takes effect when torch_npu.profiler.ProfilerActivity.CPU is enabled. <br/>Enabling this configuration introduces additional performance inflation. | |
| | with_flops | Optional | Records operator floating-point operations (this parameter does not currently support parsing performance data). Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>Takes effect when torch_npu.profiler.ProfilerActivity.CPU is enabled. | | | with_flops | Optional | Records operator floating-point operations (this parameter does not currently support parsing performance data). Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>Takes effect when torch_npu.profiler.ProfilerActivity.CPU is enabled. | |
| | experimental_config | Optional | Extended parameter, used to configure common collection items for performance analysis tools. For supported collection items and detailed descriptions, see [experimental_config Parameter Description](#experimental_config-parameter-description). | | | experimental_config | Optional | Extended parameter, used to configure common collection items for performance analysis tools. For supported collection items and detailed descriptions, see [experimental_config Parameter Description](#experimental_config-parameter-description). | |
| -| custom_trace_id_callback | Optional | Generates a trace_id to identify each Profiler data file. <br>For a usage example, see [torch_npu.profiler.profile](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/zh/custom_APIs/torch_npu-profiler/torch_npu-profiler-profile.md) <br>The trace_id is output in the profiler\_metadata.json file. | | +| custom_trace_id_callback | Optional | Generates a trace_id to identify each Profiler data file. <br>For a usage example, see [torch_npu.profiler.profile](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/en/custom_APIs/torch_npu-profiler/torch_npu-profiler-profile.md) <br>The trace_id is output in the profiler\_metadata.json file. | |
| | |
| **Table 2** Description of `torch_npu.profiler.profile` and `torch_npu.profiler._KinetoProfile` methods | **Table 2** Description of `torch_npu.profiler.profile` and `torch_npu.profiler._KinetoProfile` methods |
| | |
| @@ -1528,15 +1528,15 @@ The experimental\_config parameters are all optional. The supported extended col |
| |gc_detect_threshold|GC detection threshold. Value range is a number greater than or equal to 0, in ms. When the threshold set by the user is a number, it indicates that GC detection is enabled, and only GC events exceeding the threshold are collected.<br/>When set to 0, all GC events are collected (which may result in an excessively large amount of collected data; configure with caution). It is recommended to set it to 1 ms.<br/>Default is null, indicating that the GC detection function is not enabled.<br/>**GC** is the memory reclamation of destroyed objects by the Python process.<br/>The parsing result of this parameter generates a GC layer in trace_view.json or a GC_RECORD table in ascend_pytorch_profiler_{Rank_ID}.db.| | |gc_detect_threshold|GC detection threshold. Value range is a number greater than or equal to 0, in ms. When the threshold set by the user is a number, it indicates that GC detection is enabled, and only GC events exceeding the threshold are collected.<br/>When set to 0, all GC events are collected (which may result in an excessively large amount of collected data; configure with caution). It is recommended to set it to 1 ms.<br/>Default is null, indicating that the GC detection function is not enabled.<br/>**GC** is the memory reclamation of destroyed objects by the Python process.<br/>The parsing result of this parameter generates a GC layer in trace_view.json or a GC_RECORD table in ascend_pytorch_profiler_{Rank_ID}.db.| |
| |data_simplification|Data simplification mode. When enabled, redundant data is deleted after exporting performance data, retaining only profiler_*.json files, the ASCEND_PROFILER_OUTPUT directory, raw performance data in the PROF_XXX directory, the FRAMEWORK directory, and the logs directory to save storage space. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Enabled by default.| | |data_simplification|Data simplification mode. When enabled, redundant data is deleted after exporting performance data, retaining only profiler_*.json files, the ASCEND_PROFILER_OUTPUT directory, raw performance data in the PROF_XXX directory, the FRAMEWORK directory, and the logs directory to save storage space. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Enabled by default.| |
| |record_op_args|Controls the operator information statistics function switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.<br/>When enabled, collected operator information files are output in the {worker_name}_{timestamp}_ascend_pt_op_args directory.<br/>This parameter is used for tuning in PyTorch training scenarios executed by the AOE tool, and it is not recommended to enable it simultaneously with other performance data collection interfaces. For detailed introduction, see AOE Tuning Tool User Guide.| | |record_op_args|Controls the operator information statistics function switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.<br/>When enabled, collected operator information files are output in the {worker_name}_{timestamp}_ascend_pt_op_args directory.<br/>This parameter is used for tuning in PyTorch training scenarios executed by the AOE tool, and it is not recommended to enable it simultaneously with other performance data collection interfaces. For detailed introduction, see AOE Tuning Tool User Guide.| |
| -<!-- [AOE Tuning Tool User Guide](https://www.hiascend.com/document/detail/zh/canncommercial/850/devaids/aoe/auxiliarydevtool_aoe_0001.html) --> | |
| |export_type|Sets the format of the exported performance data result file, List type. Values:<br/>• text: Indicates parsing into timeline and summary files in json and csv formats, as well as db format files (ascend_pytorch_profiler_{Rank_ID}.db, analysis.db) that aggregate all performance data.<br/>• db: Indicates parsing only into .db format files (ascend_pytorch_profiler_{Rank_ID}.db, analysis.db) that aggregate all performance data, displayed using the MindStudio Insight tool. Only supports export via the on_trace_ready interface and offline parsing export.<br/>If an invalid value is set or not configured, the default value text is used.<br/>For parsing result data, see [Output Result File Description](#output-result-file-description).| | |export_type|Sets the format of the exported performance data result file, List type. Values:<br/>• text: Indicates parsing into timeline and summary files in json and csv formats, as well as db format files (ascend_pytorch_profiler_{Rank_ID}.db, analysis.db) that aggregate all performance data.<br/>• db: Indicates parsing only into .db format files (ascend_pytorch_profiler_{Rank_ID}.db, analysis.db) that aggregate all performance data, displayed using the MindStudio Insight tool. Only supports export via the on_trace_ready interface and offline parsing export.<br/>If an invalid value is set or not configured, the default value text is used.<br/>For parsing result data, see [Output Result File Description](#output-result-file-description).| |
| |mstx or msprof_tx|Marker control switch, enabling custom marking functionality via the switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.<br/>For usage of this parameter, see [Profile and Parse mstx Data](#profile-and-parse-mstx-data).<br/>The original parameter name msprof_tx has been changed to mstx, and the new version remains compatible with the original parameter name msprof_tx.| | |mstx or msprof_tx|Marker control switch, enabling custom marking functionality via the switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.<br/>For usage of this parameter, see [Profile and Parse mstx Data](#profile-and-parse-mstx-data).<br/>The original parameter name msprof_tx has been changed to mstx, and the new version remains compatible with the original parameter name msprof_tx.| |
| |mstx_domain_include|Outputs the required domain data. When calling the torch_npu.npu.mstx series marking interfaces and using the default domain or a specified domain for marking, you can choose to output only the domain data configured by this parameter.<br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). Domain names are input using the List type.<br/>Mutually exclusive with the mstx_domain_exclude parameter. If both are configured, only mstx_domain_include takes effect.<br/>mstx=True must be configured.| | |mstx_domain_include|Outputs the required domain data. When calling the torch_npu.npu.mstx series marking interfaces and using the default domain or a specified domain for marking, you can choose to output only the domain data configured by this parameter.<br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). Domain names are input using the List type.<br/>Mutually exclusive with the mstx_domain_exclude parameter. If both are configured, only mstx_domain_include takes effect.<br/>mstx=True must be configured.| |
| |mstx_domain_exclude|Filters out unwanted domain data. When calling the torch_npu.npu.mstx series marking interfaces and using the default domain or a specified domain for marking, you can choose not to output the domain data configured by this parameter.<br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). Domain names are input using the List type.<br/>Mutually exclusive with the mstx_domain_include parameter. If both are configured, only mstx_domain_include takes effect.<br/>mstx=True must be configured.| | |mstx_domain_exclude|Filters out unwanted domain data. When calling the torch_npu.npu.mstx series marking interfaces and using the default domain or a specified domain for marking, you can choose not to output the domain data configured by this parameter.<br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). Domain names are input using the List type.<br/>Mutually exclusive with the mstx_domain_include parameter. If both are configured, only mstx_domain_include takes effect.<br/>mstx=True must be configured.| |
| -|host_sys|Host-side system data collection switch, List type. Not configured by default, indicating that Host-side system data collection is not enabled. Values:<br/>• cpu: Process-level CPU utilization.<br/>• mem: Process-level memory utilization.<br/>• disk: Process-level disk I/O utilization.<br/>• network: System-level network I/O utilization.<br/>• osrt: Process-level syscall and pthreadcall.<br/>Configuration example: host_sys: ["cpu", "disk"].<br/>• Collecting Host-side disk performance data requires installing the third-party open-source tool iotop. Collecting osrt performance data requires installing the third-party open-source tools perf and ltrace. For installation methods, see [Installing perf, iotop, and ltrace Tools](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/install_perf_iotop_ltrace.md). After installation, user permissions must be configured as described in [Configuring User Permissions](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/config_user_permission.md), and reconfiguration is required each time the CANN software package is reinstalled.<br/>• Using the open-source tool ltrace to collect osrt performance data will cause high CPU usage, which is related to the pthread locking/unlocking of the application project and will affect the process running speed.<br/>• The osrt parameter is supported on the KylinV10SP1 operating system with x86_64 architecture, but not on the KylinV10SP1 operating system with aarch64 architecture.<br/>• The network parameter is not supported on the virtualized environment Euler2.9 system.| | +|host_sys|Host-side system data collection switch, List type. Not configured by default, indicating that Host-side system data collection is not enabled. Values:<br/>• cpu: Process-level CPU utilization.<br/>• mem: Process-level memory utilization.<br/>• disk: Process-level disk I/O utilization.<br/>• network: System-level network I/O utilization.<br/>• osrt: Process-level syscall and pthreadcall.<br/>Configuration example: host_sys: ["cpu", "disk"].<br/>• Collecting Host-side disk performance data requires installing the third-party open-source tool iotop. Collecting osrt performance data requires installing the third-party open-source tools perf and ltrace. For installation methods, see Installing perf, iotop, and ltrace Tools. After installation, user permissions must be configured as described in Configuring User Permissions, and reconfiguration is required each time the CANN software package is reinstalled.<br/>• Using the open-source tool ltrace to collect osrt performance data will cause high CPU usage, which is related to the pthread locking/unlocking of the application project and will affect the process running speed.<br/>• The osrt parameter is supported on the KylinV10SP1 operating system with x86_64 architecture, but not on the KylinV10SP1 operating system with aarch64 architecture.<br/>• The network parameter is not supported on the virtualized environment Euler2.9 system.| |
| |sys_io|NIC, ROCE, MAC collection switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.| | |sys_io|NIC, ROCE, MAC collection switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.| |
| |sys_interconnection|Collective communication bandwidth data (HCCS), PCIe data collection switch, inter-chip transmission bandwidth information collection switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.| | |sys_interconnection|Collective communication bandwidth data (HCCS), PCIe data collection switch, inter-chip transmission bandwidth information collection switch. Values:<br/>• true: Enable.<br/>• false: Disable.<br/>Disabled by default.| |
| - | +<!-- [AOE Tuning Tool User Guide](https://www.hiascend.com/document/detail/zh/canncommercial/850/devaids/aoe/auxiliarydevtool_aoe_0001.html) 1530行--> |
| +<!-- Collecting Host-side disk performance data requires installing the third-party open-source tool iotop. Collecting osrt performance data requires installing the third-party open-source tools perf and ltrace. For installation methods, see [Installing perf, iotop, and ltrace Tools](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/install_perf_iotop_ltrace.md). After installation, user permissions must be configured as described in [Configuring User Permissions](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/config_user_permission.md), and reconfiguration is required each time the CANN software package is reinstalled. --> |
| ### experimental_config Parameter Description | ### experimental_config Parameter Description |
| | |
| The experimental\_config parameters are all optional and support the following extended collection items: | The experimental\_config parameters are all optional and support the following extended collection items: |
| @@ -1548,19 +1548,19 @@ The experimental\_config parameters are all optional and support the following e |
| |export_type|Sets the format of the exported performance data result file, of List type. Values: <br/>• torch_npu.profiler.ExportType.Text: Indicates parsing into timeline and summary files in .json and .csv formats, as well as .db format files (ascend_pytorch_profiler\_{Rank_ID}.db, analysis.db) that aggregate all performance data. <br/>• torch_npu.profiler.ExportType.Db: Indicates parsing only into .db format files (ascend_pytorch_profiler_{Rank_ID}.db, analysis.db) that aggregate all performance data, for display using the MindStudio Insight tool. Only supported for export via the on_trace_ready interface and [offline parsing](#offline-parsing). <br/>If an invalid value is set or no configuration is provided, the default value torch_npu.profiler.ExportType.Text is used. <br/>For details on the parsed result data, see [output result file description](#output-result-file-description).| | |export_type|Sets the format of the exported performance data result file, of List type. Values: <br/>• torch_npu.profiler.ExportType.Text: Indicates parsing into timeline and summary files in .json and .csv formats, as well as .db format files (ascend_pytorch_profiler\_{Rank_ID}.db, analysis.db) that aggregate all performance data. <br/>• torch_npu.profiler.ExportType.Db: Indicates parsing only into .db format files (ascend_pytorch_profiler_{Rank_ID}.db, analysis.db) that aggregate all performance data, for display using the MindStudio Insight tool. Only supported for export via the on_trace_ready interface and [offline parsing](#offline-parsing). <br/>If an invalid value is set or no configuration is provided, the default value torch_npu.profiler.ExportType.Text is used. <br/>For details on the parsed result data, see [output result file description](#output-result-file-description).| |
| |profiler_level|The Level of collection, of Enum type. Values: <br/>• torch_npu.profiler.ProfilerLevel.Level_none: Does not collect data controlled by any Level hierarchy, i.e., disables profiler_level. <br/>• torch_npu.profiler.ProfilerLevel.Level0: Collects upper-layer application data, lower-layer NPU data, and information on operators executed on the NPU. When this parameter is configured, only partial data is collected, and some operator information is not collected. For details, see the description regarding task_time being l0 in [op_summary (Operator Details)](https://gitcode.com/Ascend/msprof/blob/26.1.0/docs/en/user_guide/profile_data_file_references.md#op_summary-operator-details). <br/>• torch_npu.profiler.ProfilerLevel.Level1: In addition to Level0, collects CANN layer AscendCL data and AI Core performance metric information executed on the NPU, enables aic_metrics=torch_npu.profiler.AiCMetrics.PipeUtilization, and generates communication.json, communication_matrix.json, and api_statistic.csv files for communication operators. <br/>• torch_npu.profiler.ProfilerLevel.Level2: In addition to Level1, collects CANN layer Runtime data and AI CPU (data_preprocess.csv file) data. <br/>The default value is torch_npu.profiler.ProfilerLevel.Level0.| | |profiler_level|The Level of collection, of Enum type. Values: <br/>• torch_npu.profiler.ProfilerLevel.Level_none: Does not collect data controlled by any Level hierarchy, i.e., disables profiler_level. <br/>• torch_npu.profiler.ProfilerLevel.Level0: Collects upper-layer application data, lower-layer NPU data, and information on operators executed on the NPU. When this parameter is configured, only partial data is collected, and some operator information is not collected. For details, see the description regarding task_time being l0 in [op_summary (Operator Details)](https://gitcode.com/Ascend/msprof/blob/26.1.0/docs/en/user_guide/profile_data_file_references.md#op_summary-operator-details). <br/>• torch_npu.profiler.ProfilerLevel.Level1: In addition to Level0, collects CANN layer AscendCL data and AI Core performance metric information executed on the NPU, enables aic_metrics=torch_npu.profiler.AiCMetrics.PipeUtilization, and generates communication.json, communication_matrix.json, and api_statistic.csv files for communication operators. <br/>• torch_npu.profiler.ProfilerLevel.Level2: In addition to Level1, collects CANN layer Runtime data and AI CPU (data_preprocess.csv file) data. <br/>The default value is torch_npu.profiler.ProfilerLevel.Level0.| |
| |mstx or msprof_tx|Mark control switch. Enables the custom mark function via this switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>For usage of this parameter, see [Profile and Parse mstx Data](#profile-and-parse-mstx-data). The original parameter name msprof_tx has been changed to mstx, but the new version remains compatible with the original parameter name msprof_tx.| | |mstx or msprof_tx|Mark control switch. Enables the custom mark function via this switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>For usage of this parameter, see [Profile and Parse mstx Data](#profile-and-parse-mstx-data). The original parameter name msprof_tx has been changed to mstx, but the new version remains compatible with the original parameter name msprof_tx.| |
| -|mstx_domain_include|Outputs the required domain data. When calling the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/zh/custom_APIs/torch_npu-npu/torch_npu-npu-mstx.md) series of mark interfaces and using the default domain or a specified domain for marking, you can choose to output only the domain data configured by this parameter. <br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). The domain name is input as a List type. <br/>This parameter is mutually exclusive with mstx_domain_exclude. If both are configured, only mstx_domain_include takes effect. <br/>mstx=True must be configured.| | +|mstx_domain_include|Outputs the required domain data. When calling the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/en/custom_APIs/torch_npu-npu/torch_npu-npu-mstx.md) series of mark interfaces and using the default domain or a specified domain for marking, you can choose to output only the domain data configured by this parameter. <br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). The domain name is input as a List type. <br/>This parameter is mutually exclusive with mstx_domain_exclude. If both are configured, only mstx_domain_include takes effect. <br/>mstx=True must be configured.| |
| -|mstx_domain_exclude|Filters out unwanted domain data. When calling the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/zh/custom_APIs/torch_npu-npu/torch_npu-npu-mstx.md) series of mark interfaces and using the default domain or a specified domain for marking, you can choose not to output the domain data configured by this parameter. <br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). The domain name is input as a List type. <br/>This parameter is mutually exclusive with mstx_domain_include. If both are configured, only mstx_domain_include takes effect. <br/>mstx=True must be configured.| | +|mstx_domain_exclude|Filters out unwanted domain data. When calling the [torch_npu.npu.mstx](https://gitcode.com/Ascend/op-plugin/blob/26.1.0/docs/en/custom_APIs/torch_npu-npu/torch_npu-npu-mstx.md) series of mark interfaces and using the default domain or a specified domain for marking, you can choose not to output the domain data configured by this parameter. <br/>The domain name is the domain passed in by the user when calling the torch_npu.npu.mstx series interfaces or the default domain ('default'). The domain name is input as a List type. <br/>This parameter is mutually exclusive with mstx_domain_include. If both are configured, only mstx_domain_include takes effect. <br/>mstx=True must be configured.| |
| |aic_metrics|AI Core performance metric collection items. Values: <br/>The result data for the following collection items will be displayed in Kernel View. <br/>For the meaning of the result data for the following collection items, see [op_summary (Operator Details)](https://gitcode.com/Ascend/msprof/blob/26.1.0/docs/en/user_guide/profile_data_file_references.md#op_summary-operator-details), but the specific collection results are subject to actual conditions. <br/>• AiCoreNone: Disables AI Core performance metric collection. <br/>• PipeUtilization: The proportion of time consumed by compute units and data transfer units. <br/>• ArithmeticUtilization: Statistics on the proportion of various compute-related metrics. <br/>• Memory: The proportion of external memory read/write instructions. <br/>• MemoryL0: The proportion of internal L0 memory read/write instructions. <br/>• ResourceConflictRatio: The proportion of pipeline queue instructions. <br/>• MemoryUB: The proportion of internal UB memory read/write instructions. <br/>• L2Cache: The number of read/write cache hits and reallocations after misses. <br/>• MemoryAccess: The bandwidth data volume of memory access on the core by the operator. <br/>When profiler_level is set to torch_npu.profiler.ProfilerLevel.Level_none or torch_npu.profiler.ProfilerLevel.Level0, the default value is AiCoreNone; when profiler_level is set to torch_npu.profiler.ProfilerLevel.Level1 or torch_npu.profiler.ProfilerLevel.Level2, the default value is PipeUtilization.| | |aic_metrics|AI Core performance metric collection items. Values: <br/>The result data for the following collection items will be displayed in Kernel View. <br/>For the meaning of the result data for the following collection items, see [op_summary (Operator Details)](https://gitcode.com/Ascend/msprof/blob/26.1.0/docs/en/user_guide/profile_data_file_references.md#op_summary-operator-details), but the specific collection results are subject to actual conditions. <br/>• AiCoreNone: Disables AI Core performance metric collection. <br/>• PipeUtilization: The proportion of time consumed by compute units and data transfer units. <br/>• ArithmeticUtilization: Statistics on the proportion of various compute-related metrics. <br/>• Memory: The proportion of external memory read/write instructions. <br/>• MemoryL0: The proportion of internal L0 memory read/write instructions. <br/>• ResourceConflictRatio: The proportion of pipeline queue instructions. <br/>• MemoryUB: The proportion of internal UB memory read/write instructions. <br/>• L2Cache: The number of read/write cache hits and reallocations after misses. <br/>• MemoryAccess: The bandwidth data volume of memory access on the core by the operator. <br/>When profiler_level is set to torch_npu.profiler.ProfilerLevel.Level_none or torch_npu.profiler.ProfilerLevel.Level0, the default value is AiCoreNone; when profiler_level is set to torch_npu.profiler.ProfilerLevel.Level1 or torch_npu.profiler.ProfilerLevel.Level2, the default value is PipeUtilization.| |
| |l2_cache|Controls the L2 Cache data collection switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>This collection item generates an l2_cache.csv file in ASCEND_PROFILER_OUTPUT. For an introduction to the result fields, see [l2_cache (L2 Cache Hit Rate)](https://gitcode.com/Ascend/msprof/blob/26.1.0/docs/en/user_guide/profile_data_file_references.md#l2_cache-l2-cache-hit-ratio).| | |l2_cache|Controls the L2 Cache data collection switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>This collection item generates an l2_cache.csv file in ASCEND_PROFILER_OUTPUT. For an introduction to the result fields, see [l2_cache (L2 Cache Hit Rate)](https://gitcode.com/Ascend/msprof/blob/26.1.0/docs/en/user_guide/profile_data_file_references.md#l2_cache-l2-cache-hit-ratio).| |
| |op_attr|Controls the switch for collecting operator attribute information. Currently, only supports collecting aclnn operators, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>The performance data collected by this parameter only takes effect for db format files; when torch_npu.profiler.ProfilerLevel.Level_none is set, this parameter does not take effect.| | |op_attr|Controls the switch for collecting operator attribute information. Currently, only supports collecting aclnn operators, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>The performance data collected by this parameter only takes effect for db format files; when torch_npu.profiler.ProfilerLevel.Level_none is set, this parameter does not take effect.| |
| |data_simplification|Data simplification mode. When enabled, redundant data will be deleted after exporting performance data, retaining only profiler_*.json files, the ASCEND_PROFILER_OUTPUT directory, the original performance data in the PROF_XXX directory, the FRAMEWORK directory, and the logs directory to save storage space, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Enabled by default.| | |data_simplification|Data simplification mode. When enabled, redundant data will be deleted after exporting performance data, retaining only profiler_*.json files, the ASCEND_PROFILER_OUTPUT directory, the original performance data in the PROF_XXX directory, the FRAMEWORK directory, and the logs directory to save storage space, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Enabled by default.| |
| |record_op_args|Controls the switch for the operator information statistics function, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>When enabled, the collected operator information files will be output in the {worker_name}\_{timestamp}_ascend_pt_op_args directory. <br/>This parameter is used when the AOE tool performs tuning in PyTorch training scenarios, and it is not recommended to enable it simultaneously with other performance data collection interfaces. For detailed introduction, see AOE Tuning Tool User Guide.| | |record_op_args|Controls the switch for the operator information statistics function, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default. <br/>When enabled, the collected operator information files will be output in the {worker_name}\_{timestamp}_ascend_pt_op_args directory. <br/>This parameter is used when the AOE tool performs tuning in PyTorch training scenarios, and it is not recommended to enable it simultaneously with other performance data collection interfaces. For detailed introduction, see AOE Tuning Tool User Guide.| |
| -<!-- [AOE Tuning Tool User Guide](https://www.hiascend.com/document/detail/en/canncommercial/850/devaids/aoe/auxiliarydevtool_aoe_0001.html) --> | |
| |gc_detect_threshold|GC detection threshold, of float type. The value range is a number greater than or equal to 0, in ms. When the threshold set by the user is a number, it indicates that GC detection is enabled, and only GC events exceeding the threshold are collected. <br/>When configured as 0, it indicates collecting all GC events (which may result in an excessively large amount of collected data; please configure with caution). It is recommended to set it to 1 ms. <br/>The default is None, indicating that the GC detection function is not enabled. <br/>**GC** is the memory reclamation of destroyed objects by the Python process. <br/>The parsing result of this parameter is the generation of a GC layer in trace_view.json or a GC_RECORD table in ascend_pytorch_profiler_{Rank_ID}.db.| | |gc_detect_threshold|GC detection threshold, of float type. The value range is a number greater than or equal to 0, in ms. When the threshold set by the user is a number, it indicates that GC detection is enabled, and only GC events exceeding the threshold are collected. <br/>When configured as 0, it indicates collecting all GC events (which may result in an excessively large amount of collected data; please configure with caution). It is recommended to set it to 1 ms. <br/>The default is None, indicating that the GC detection function is not enabled. <br/>**GC** is the memory reclamation of destroyed objects by the Python process. <br/>The parsing result of this parameter is the generation of a GC layer in trace_view.json or a GC_RECORD table in ascend_pytorch_profiler_{Rank_ID}.db.| |
| -|host_sys|Host-side system data collection switch, of List type. By default, it is not configured, indicating that Host-side system data collection is not enabled. Values: <br/>• torch_npu.profiler.HostSystem.CPU: Process-level CPU utilization. <br/>• torch_npu.profiler.HostSystem.MEM: Process-level memory utilization. <br/>• torch_npu.profiler.HostSystem.DISK: Process-level disk I/O utilization. <br/>• torch_npu.profiler.HostSystem.NETWORK: System-level network I/O utilization. <br/>• torch_npu.profiler.HostSystem.OSRT: Process-level syscall and pthreadcall. <br/>Configuration example: host_sys=[torch_npu.profiler.HostSystem.CPU, torch_npu.profiler.HostSystem.MEM] <br/>• Collecting Host-side disk performance data requires installing the third-party open-source tool iotop, and collecting osrt performance data requires installing the third-party open-source tools perf and ltrace. For installation methods, see [Installing perf, iotop, and ltrace Tools](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/install_perf_iotop_ltrace.md), you must complete user permission configuration as described in [Configuring User Permissions](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/config_user_permission.md), and this configuration must be redone each time the CANN software package is reinstalled. <br/>• Using the open-source tool ltrace to collect osrt performance data will cause high CPU usage, which is related to the pthread locking and unlocking of the application project and will affect the process running speed. <br/>• The torch_npu.profiler.HostSystem.OSRT parameter is supported on the KylinV10SP1 operating system with x86_64 architecture, but not on the KylinV10SP1 operating system with aarch64 architecture. <br/>• The torch_npu.profiler.HostSystem.NETWORK parameter is not supported on the virtualized environment Euler2.9 system.| | +|host_sys|Host-side system data collection switch, of List type. By default, it is not configured, indicating that Host-side system data collection is not enabled. Values: <br/>• torch_npu.profiler.HostSystem.CPU: Process-level CPU utilization. <br/>• torch_npu.profiler.HostSystem.MEM: Process-level memory utilization. <br/>• torch_npu.profiler.HostSystem.DISK: Process-level disk I/O utilization. <br/>• torch_npu.profiler.HostSystem.NETWORK: System-level network I/O utilization. <br/>• torch_npu.profiler.HostSystem.OSRT: Process-level syscall and pthreadcall. <br/>Configuration example: host_sys=[torch_npu.profiler.HostSystem.CPU, torch_npu.profiler.HostSystem.MEM] <br/>• Collecting Host-side disk performance data requires installing the third-party open-source tool iotop, and collecting osrt performance data requires installing the third-party open-source tools perf and ltrace. For installation methods, see Installing perf, iotop, and ltrace Tools, you must complete user permission configuration as described in Configuring User Permissions, and this configuration must be redone each time the CANN software package is reinstalled. <br/>• Using the open-source tool ltrace to collect osrt performance data will cause high CPU usage, which is related to the pthread locking and unlocking of the application project and will affect the process running speed. <br/>• The torch_npu.profiler.HostSystem.OSRT parameter is supported on the KylinV10SP1 operating system with x86_64 architecture, but not on the KylinV10SP1 operating system with aarch64 architecture. <br/>• The torch_npu.profiler.HostSystem.NETWORK parameter is not supported on the virtualized environment Euler2.9 system.| |
| |sys_io|NIC, ROCE, and MAC collection switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default.| | |sys_io|NIC, ROCE, and MAC collection switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default.| |
| |sys_interconnection|Collective communication bandwidth data (HCCS), PCIe data collection switch, and inter-chip transmission bandwidth information collection switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default.| | |sys_interconnection|Collective communication bandwidth data (HCCS), PCIe data collection switch, and inter-chip transmission bandwidth information collection switch, of bool type. Values: <br/>• True: Enable. <br/>• False: Disable. <br/>Disabled by default.| |
| - | +<!-- [AOE Tuning Tool User Guide](https://www.hiascend.com/document/detail/en/canncommercial/850/devaids/aoe/auxiliarydevtool_aoe_0001.html) 1558行修改--> |
| +<!-- Collecting Host-side disk performance data requires installing the third-party open-source tool iotop, and collecting osrt performance data requires installing the third-party open-source tools perf and ltrace. For installation methods, see [Installing perf, iotop, and ltrace Tools](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/install_perf_iotop_ltrace.md), you must complete user permission configuration as described in [Configuring User Permissions](https://gitcode.com/cann/oam-tools/blob/master/docs/zh/profiling/appendices/config_user_permission.md), and this configuration must be redone each time the CANN software package is reinstalled. 1559行修改--> |
| ### torch_npu.profiler.schedule Class Parameter Description | ### torch_npu.profiler.schedule Class Parameter Description |
| | |
| The torch_npu.profiler.schedule class is used to set the collection behavior at different steps during the collection process. The interface prototype is: | The torch_npu.profiler.schedule class is used to set the collection behavior at different steps during the collection process. The interface prototype is: |
| |