MindStudio Profiler Parsing Tool

Overview

The MindStudio Profiler (msProf) command-line tool is encapsulated using msprof.py and supports general parsing of profile data.

Tool Usage Process

• Automatic parsing: Generally, running the msprof command to collect profile data parses and exports profile data files by default.
• Offline parsing:
  • If automatic parsing is not supported or re-parsing is required, proceed to Parsing and Exporting Profile Data.
  • (Optional) If a profile data file fails to be parsed, re-parse the file by referring to Parsing Profile Data, then proceed to Parsing and Exporting Profile Data.
  • (Optional) To specify an iteration ID and model ID for parsing, refer to Querying Profile Data File Information or Parsing Profile Data to query the total iteration count and model ID. Then, select the required iteration ID and model ID to proceed to Parsing and Exporting Profile Data.
  • For profile data in communication scenarios, proceed to Parsing Communication Profile Data to parse the exported results after completing Parsing and Exporting Profile Data.

Preparations

Environment Setup

1. Install the matching CANN Toolkit and ops operator packages, and configure CANN environment variables. For details, see the CANN Installation Guide.

   In Ascend EP scenarios, the msProf tool is located in ${INSTALL_DIR}/tools/profiler/bin. Replace ${INSTALL_DIR} with the actual CANN installation directory. If the Toolkit is installed by the root user, the default installation directory is /usr/local/Ascend/cann.

   In Ascend RC scenarios, the msProf tool is stored in /var.

2. Set Python environment variables.

   If multiple Python 3 series versions exist, specify your Python installation path according to your service needs. The following takes Python 3.7.5 as an example.

   export PATH=/usr/local/python3.7.5/bin:$PATH
   ## Set the library path for Python 3.7.5.
   export LD_LIBRARY_PATH=/usr/local/python3.7.5/lib:$LD_LIBRARY_PATH
   

Constraints

Before using this tool, understand the following constraints:

• Permission constraints

  • Adhere to the principle of least privilege (for example, prohibit write access for others and avoid setting file permissions to 666 or 777).
  • Before using msProf, ensure the user's umask is set to 0027 or a more restrictive value. Otherwise, the generated profile data directories and files will have excessive permissions.
  • To ensure security and adhere to the principle of least privilege, you are advised to run this tool as a common user rather than a high-privilege user (such as root).
  • This tool is a development and debugging tool. You are advised not to use it in the production environment.
  • Ensure that the profile data is stored in the current user directory that does not contain soft links. Otherwise, security issues may occur.

• Data flush constraints

  During profile data parsing, if the configured drive or user directory space is full, the parsing fails or the file cannot be flushed to the drive. In this case, clear the drive or user directory space.

• Compatibility and scenario constraints

  Python 3.7.5 or later is required.

• Other constraints

  Path parameters for this tool must not contain the following special characters:

  "\n", "\\n", "\f", "\\f", "\r", "\\r", "\b", "\\b", "\t", "\\t", "\v", "\\v", "\u007F", "\\u007F", "\"", "\\\"", "'", "\'", "\\", "\\\\", "%", "\\%", ">", "\\>", "<", "\\<", "|", "\\|", "&", "\\&", "$", "\\$", ";", "\\;", "`", "\\`"
  

Parsing and Exporting Profile Data

Supported Products

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	√
Atlas inference products	√
Atlas training products	√

Function

Parses and exports profile data.

Precautions

• Complete preparations first.
• Complete profile data collection first.
• Direct parsing on the device is not supported for the following products. The generated PROF_XXX directory must be copied to an environment with the Toolkit package installed.
  • Ascend RC scenario for Atlas 200I/500 A2 inference products

Syntax

msprof --export=on --output=<dir> [--type=<type>] [--reports=<reports_sample_config.json>] [--model-id=<number>] [--iteration-id=<number>] [--summary-format=<csv/json>] [--clear=on]

Command-line Options

Table 1 Command-line options

Option	Mandatory (Yes/No)	Description
--export	Yes	Parses and exports profile data. Valid values: on or off (default). • on: enables this option. • off: disables this option. To export data of a specific model (model ID) or iteration (iteration ID), run the msprof --export command again to configure the --model-id and --iteration-id options after msProf profile data collection. For unparsed PROF_XXX files, this option performs automatic parsing before exporting. Example: msprof --export=on --output=/home/HwHiAiUser
--output	Yes	Specifies the profile data directory. The value must be a PROF_XXX directory or its parent directory, such as /home/profiler_data/PROF_XXX.
--type	No	Sets the format of profile data parsing result files to be exported. This option determines the format of automatic parsing result files generated after the execution of the msprof command. Valid values: • text: parses data into JSON/CSV timeline and summary files plus a DB file (msprof_timestamp.db). For details, see Profile Data File References. It supports profile data parsing with CANN 7.0.0 and later. • db: parses data into a single DB file (msprof_timestamp.db) for display in MindStudio Insight. Currently, this format differs in information volume from text, so text is recommended. When the type is set to db, only the --output option of the msprof --export command is supported. Other options of the command are invalid. Default value: text.
--reports	No	Specifies a custom reports_sample_config.json configuration file to export the corresponding profile data based on the scope specified in the file. For details, see Example (--reports Option). Currently, the Atlas 350 accelerator card does not support this option.
--model-id	No	Specifies the model ID. The value must be a positive integer. This option must be specified in combination with --iteration-id to export the profile data of a specified compute iteration in the model. If neither --model-id nor --iteration-id is specified, all profile data is exported by default. • For Atlas A2 training products/Atlas A2 inference products as well as Atlas A3 training products/Atlas A3 inference products, --model-id can be set to 4294967295, which specifies the step mode. That is, the value of --iteration-id specifies parsing by step. Only profile data of the MindSpore framework (version 2.3 or later) can be parsed. • If --model-id is set to other values, this option specifies the iteration ID for graph-based statistics collection. The iteration ID is incremented by 1 each time a graph is executed. When a script is compiled into multiple graphs, the iteration ID is different from the step ID at the script layer.
--iteration-id	No	Specifies the iteration ID. The value must be a positive integer. This option must be specified in combination with --model-id to export the profile data of a specified compute iteration in the model. If neither --model-id nor --iteration-id is specified, all profile data is exported by default. • For Atlas A2 training products/Atlas A2 inference products, as well as Atlas A3 training products/Atlas A3 inference products, --model-id can be set to 4294967295, which specifies the iteration ID for step-based statistics collection. The iteration ID is incremented by 1 each time a step is executed. Only profile data of the MindSpore framework (version 2.3 or later) can be parsed. • If --model-id is set to other values, this option specifies the iteration ID for graph-based statistics collection. The iteration ID is incremented by 1 each time a graph is executed. When a script is compiled into multiple graphs, the iteration ID is different from the step ID at the script layer.
--summary-format	No	Specifies the summary data file export format. Valid values: • json: exports the summary data file is in JSON format. • csv (default): exports the summary data file in CSV format. This option is supported only when --type is set to text.
--python-path	No	Specifies the path to the Python interpreter used for parsing. The Python version must be 3.7.5 or later.
--clear	No	Sets the data clearance mode. After this option is enabled, the sqlite directory in PROF_XXX/device_{id} is deleted (after profile data is exported) to save storage space. Valid values: on or off (default).

Example

Specify the /home/profiler_data/PROF_XXX directory as the profile data directory and run the msprof --export command:

msprof --export=on --output=/home/profiler_data/PROF_XXX

Example (--reports Option)

Specify the /home/profiler_data/PROF_XXX directory as the profile data directory, pass the custom reports_sample_config.json configuration file, and run the msprof --export command:

msprof --export=on --output=/home/profiler_data/PROF_XXX --reports=${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msconfig/reports_sample_config.json

Replace ${INSTALL_DIR} with the actual CANN installation directory. If the Toolkit is installed by the root user, the default installation directory is /usr/local/Ascend/cann.

By default, the reports_sample_config.json file is stored in the ${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msconfig/ directory. The file content is as follows:

(You can create a reports_sample_config.json file in any directory with read and write permissions.)

{
    "json_process": {
        "ascend": true,
        "acc_pmu": true,
        "cann": true,
        "ddr": true,
        "stars_chip_trans": true,
        "hbm": true,
        "communication": true,
        "hccs": true,
        "os_runtime_api": true,
        "network_usage": true,
        "disk_usage": true,
        "memory_usage": true,
        "cpu_usage": true,
        "msproftx": true,
        "npu_mem": true,
        "overlap_analyse": true,
        "pcie": true,
        "sio": true,
        "stars_soc": true,
        "step_trace": true,
        "freq": true,
        "llc": true,
        "nic": true,
        "roce": true,
        "qos": true,
        "device_tx": true
    }
}

The preceding configuration items are switches for controlling specific profile data. You can set them to true (to enable the fields) or false (to disable or delete the fields). The controlled profile data includes the timeline tracks (including CANN, Ascend Hardware, AI Core Freq, on-chip memory, Communication, Overlap Analysis, and NPU_MEM tracks) in msprof_*.json.

Output Description

Executing the msprof --export command generates the mindstudio_profiler_output directory within the PROF_XXX directory.

The structure of the generated profile data directory is as follows:

• Single-process collection

  └── PROF_XXX
        ├── device_0
        │    └── data
        ├── device_1
        │    └── data
        ├── host
        │    └── data
        ├── msprof_*.db
        └── mindstudio_profiler_output
              ├── msprof_*.json
              ├── step_trace_*.json
              ├── xx_*.csv
               ...
              └── README.txt
  

• Multi-process collection

  └── PROF_XXX1
        ├── device_0
        │    └── data
        ├── host
        │    └── data
        ├── msprof_*.db
        └── mindstudio_profiler_output
              ├── msprof_*.json
              ├── step_trace_*.json
              ├── xx_*.csv
               ...
              └── README.txt
  └── PROF_XXX2
        ├── device_1
        │    └── data
        ├── host
        │    └── data
        ├── msprof_*.db
        └── mindstudio_profiler_output
              ├── msprof_*.json
              ├── step_trace_*.json
              ├── xx_*.csv
               ...
              └── README.txt
  

Querying Profile Data File Information

Supported Products

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	√
Atlas inference products	√
Atlas training products	√

Function

Queries profile data file information. Ensure that the model ID and iteration ID are specified during export.

During profile data parsing, the profile data file information is automatically displayed on the screen. Therefore, this function is optional in data parsing and is mainly used to query information about profile data files in the historical parsed PROF_XXX directories.

Precautions

• Complete preparations first.
• Complete profile data collection first.
• Direct parsing on the device is not supported for the following products. The generated PROF_XXX directory must be copied to an environment with the Toolkit package installed.
  • Ascend RC scenario for Atlas 200I/500 A2 inference products

Syntax

msprof --query=on --output=<dir>

Command-line Options

Table 1 Command-line options

Option	Mandatory (Yes/No)	Description
--query	Yes	Queries profile data file information. Valid values: on or off (default). • on: enables this option. • off: disables this option. After data parsing, you can use this option to query the basic information about profile data files.
--output	Yes	Specifies the directory for storing parsed profile data files. The value must be a PROF_XXX directory or its parent directory, such as /home/profiler_data/PROF_XXX.

Example

Set the directory for storing parsed profile data files to /home/profiler_data/PROF_XXX and enable the feature of querying profile data files.

msprof --query=on --output=/home/profiler_data/PROF_XXX

Output Description

The following table describes information obtained by the query feature (msprof --query) of msProf.

Table 2 Profile data file fields

Field	Description
Job Info	Job name
Device ID	Device ID
Dir Name	Directory name
Collection Time	Data collection time
Model ID	Model ID
Iteration Number	Total number of iterations
Top Time Iteration	Top five iterations with the longest durations
Rank ID	Node ID in the cluster scenario

Parsing Profile Data

Supported Products

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	√
Atlas inference products	√
Atlas training products	√

Function

Performs profile data parsing and does not export profile data files. For details about how to export profile data files, see Parsing and Exporting Profile Data.

Generally, the profile data parsing option does not need to be used independently. It is used in the following scenarios:

• If profile data file parsing fails (for example, residual files exist from a failed initial parsing), run msprof --parse to re-parse the files before running msprof --export.
• If the --iteration-id and --model-id options need to be added to msprof --export, you can run the msprof --parse command to parse and print the iteration ID and model ID, and then specify the required iteration ID and model ID for export.

Precautions

• Complete preparations first.
• Complete profile data collection first.
• Direct parsing on the device is not supported for the following products. The generated PROF_XXX directory must be copied to an environment with the Toolkit package installed.
  • Ascend RC scenario for Atlas 200I/500 A2 inference products

Syntax

msprof --parse=on --output=<dir>

Command-line Options

Table 1 Command-line options

Option	Mandatory (Yes/No)	Description
--parse	Yes	Parses the raw profile data files. Valid values: on or off (default). • on: enables this option. • off: disables this option.
--output	Yes	Specifies the raw profile data file directory. The value must be a PROF_XXX directory or its parent directory, such as /home/profiler_data/PROF_XXX.
--python-path	No	Specifies the path to the Python interpreter used for parsing. The Python version must be 3.7.5 or later.

Example

Parse raw profile data files by specifying /home/profiler_data/PROF_XXX as the raw profile data file directory.

msprof --parse=on --output=/home/profiler_data/PROF_XXX

Output Description

After running the preceding command, the profile data file information is displayed, and a sqlite directory containing db files is generated in the device_{id} and host directories of PROF_XXX.

To export the final result timeline data or db files, see Parsing and Exporting Profile Data.

Parsing Communication Profile Data

Supported Products

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	x
Atlas inference products	√
Atlas training products	√

Function

Parses communication profile data into statistics on segment duration, copy operations, and bandwidth for communication data analysis. Communication data exists only in multi-rank, multi-node, or cluster scenarios.

Precautions

• Complete preparations first.

• Parse the PROF_XXX directory and perform the export by using the msprof command and disable data clearance mode. The command example is as follows:

  msprof --export=on --output=/home/xxx/profiler_data/PROF_XXX --clear=off
  

• Direct parsing on the device is not supported for the following products. The generated PROF_XXX directory must be copied to an environment with the Toolkit package installed.

  • Ascend RC scenario for Atlas 200I/500 A2 inference products

Syntax

msprof command-line mode:

msprof --analyze=on [--type=<type>] [--rule=communication] --output=<dir> [--clear=on]

msprof.py script mode:

python3 msprof.py analyze [--type <type>] --rule communication -dir <dir> [--clear]

Command-line Options and Parameters

Table 1 Command-line options (msprof command-line mode)

Option	Mandatory (Yes/No)	Description
--analyze	Yes	Analyzes profile data files. Valid values: on or off (default). • on: enables this option. • off: disables this option.
--type	No	Sets the format of profile data parsing result files. This option determines the format of automatic parsing result files generated after the execution of the msprof command. Valid values: text: parses the profile data into JSON and communication_analyzer.db files. db: parses the profile data into a communication_analyzer.db file. Default value: text.
--rule	No	Specifies the analysis rule. Valid values: • communication: analyzes communication data. ‣ If --type is set to text, the communication.json file is generated in the PROF_XXX/analyze directory to display details such as the communication duration and bandwidth of all communication operators on a single rank, as shown in Figure 4. The communication_analyzer.db file is also generated. ‣ If --type is set to db, only the communication_analyzer.db file is generated in the PROF_XXX/analyze directory to store the CommAnalyzerTime (communication duration) and CommAnalyzerBandwidth (communication bandwidth) tables. • communication_matrix: analyzes communication matrix data. ‣ If --type is set to text, the communication_matrix.json file is generated in the PROF_XXX/analyze directory to display basic information about small communication operators, including the communication size, bandwidth, and rank information used to analyze communication details, as shown in Figure 5. The communication_analyzer.db file is also generated. ‣ If --type is set to db, only the communication_analyzer.db file is generated in PROF_XXX/analyze to store the CommAnalyzerMatrix (communication matrix) table. Both values can be set simultaneously, separated by a comma (,), such as --rule=communication,communication_matrix. By default, they are both set.
--output	Yes	Specifies the profile data directory. The directory must be a PROF_XXX directory, such as /home/HwHiAiUser/profiler_data/PROF_XXX.
--clear	No	Sets the data clearance mode. After this option is enabled, the sqlite directory in PROF_XXX is deleted (after profile data is exported) to save storage space. Valid values: on or off (default).

Table 2 Command-line options and parameters (msprof.py script mode)

Option/Parameter	Mandatory (Yes/No)	Description
analyze	Yes	Analyzes the profile data files.
--type	No	Sets the format of profile data parsing result files. This option determines the format of automatic parsing result files generated after the execution of msprof.py. Valid values: • text: parses the profile data into JSON and communication_analyzer.db files. • db: parses the profile data into the communication_analyzer.db file. Default value: text.
-r or --rule	Yes	Specifies the analysis rule. Valid values: • communication: analyzes communication data. ‣ If --type is set to text, the communication.json file is generated in the PROF_XXX/analyze directory to display details such as the communication duration and bandwidth of all communication operators on a single rank, as shown in Figure 4. The communication_analyzer.db file is also generated. ‣ If --type is set to db, only the communication_analyzer.db file is generated in the PROF_XXX/analyze directory to store the CommAnalyzerTime (communication duration) and CommAnalyzerBandwidth (communication bandwidth) tables. • communication_matrix: analyzes communication matrix data. ‣ If --type is set to text, the communication_matrix.json file is generated in the PROF_XXX/analyze directory to display basic information about small communication operators, including the communication size, bandwidth, and rank information used to analyze communication details, as shown in Figure 5. The communication_analyzer.db file is also generated. ‣ If --type is set to db, only the communication_analyzer.db file is generated in PROF_XXX/analyze to store the CommAnalyzerMatrix (communication matrix) table. You can set either or both of these two values. If you set both of them, use a comma (,) to separate them, such as --rule communication,communication_matrix.
-dir or --collection-dir	Yes	Specifies the profile data directory. The directory must be a PROF_XXX directory, such as /home/profiler_data/PROF_XXX.
--clear	No	Sets the data clearance mode. After this option is enabled, the sqlite directory in PROF_XXX is deleted (after profile data is exported) to save storage space. Specifying this option enables the data clearance mode, which is disabled if not specified.

Example (msprof Command-line Mode)

Specify the /home/profiler_data/PROF_XXX directory as the profile data file directory and enable the communication profile data parsing option.

msprof --analyze=on --output=/home/profiler_data/PROF_XXX

Example (msprof.py Script Mode)

1. Log in to the development environment as the Toolkit installation user.

2. Go to the directory of msprof.py.

   ${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msprof. Replace ${INSTALL_DIR} with the CANN installation directory. If the Toolkit is installed by the root user, the default installation directory is /usr/local/Ascend/cann.

3. Specify the /home/profiler_data/PROF_XXX directory as the profile data file directory and run the parsing command:

   python3 msprof.py analyze --rule communication -dir /home/profiler_data/PROF_XXX
   

Output File Description

• Scenario: --type=text or --type=db and --rule=communication

Figure 1 CommAnalyzerTime

Table 3 CommAnalyzerTime

Field	Description
hccl_op_name	Communication operator name.
group_name	Name of the communication operator group.
start_timestamp	Communication start timestamp.
elapse_time	Total operator communication duration (ms).
transit_time	Communication duration (ms). If the communication duration is too long, a link may be faulty.
wait_time	Wait duration (ms). Nodes must fully synchronize before communication.
synchronization_time	Synchronization duration (ms). It is the duration required for synchronization between nodes.
idle_time	Idle duration (ms). Idle duration (idle_time) = Total communication duration of the operator (elapse_time) – Communication duration (transit_time) – Wait duration (wait_time)

Figure 2 CommAnalyzerBandwidth

Table 4 CommAnalyzerBandwidth

Field	Description
hccl_op_name	Communication operator name.
group_name	Name of the communication operator group.
transport_type	Communication transmission type, including LOCAL, SDMA, RDMA, PCIE, HCCS, and SIO.
transit_size	Communication data volume (MB).
transit_time	Communication duration (ms). If the communication duration is too long, a link may be faulty.
bandwidth	Communication bandwidth (GB/s).
large_packet_ratio	Percentage of large packets in communication data.
package_size	Size of a communication data packet transmitted at a time (MB).
count	Number of communication transmissions.
total_duration	Total data transmission duration (ms).

• Scenario: --type=text or --type=db and --rule=communication_matrix

Figure 3 CommAnalyzerMatrix

Table 5 CommAnalyzerMatrix

Field	Description
hccl_op_name	Communication operator name.
group_name	Name of the communication operator group.
src_rank	Communication source rank.
dst_rank	Communication destination rank.
transport_type	Communication transmission type, including LOCAL, SDMA, RDMA, PCIE, HCCS, and SIO.
transit_size	Communication data volume (MB).
transit_time	Communication duration (ms). If the communication duration is too long, a link may be faulty.
bandwidth	Communication bandwidth (GB/s).

• Scenario: --type=text and --rule=communication

Figure 4 communication.json

• Scenario: --type=text and --rule=communication_matrix

Figure 5 communication_matrix.json