Operator Performance Comparison Skill

This skill tests the performance of a specified API implementation using Ascend msprof profiling tool.

Features

• Single API Testing: Test any API by specifying its import path
• Flexible Input Modes: Random generation or load from .pth files
• JSON Parameter Format: Easy-to-use JSON format for optional parameters
• Tensor Parameter Support: Generate tensor parameters with specified shape and dtype
• Fixed Random Seed: Reproducible results with configurable seed
• msprof Integration: Uses Ascend's official profiling tool for accurate performance measurement

Quick Start

1. Basic Usage (Random Mode)

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [10,10] x float32 \
     --input [10,10] y float16

2. File Mode (Load from .pth files)

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input-mode file \
     --input path/to/x.pth x \
     --input path/to/y.pth y

3. With Optional Parameters

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [10,10] x float32 \
     --input [10,10] y float32 \
     --other-optional-params '{"alpha": 0.5}'

4. Fixed Random Seed

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [10,10] x float32 \
     --input [10,10] y float32 \
     --seed 123

Command Line Arguments

Argument	Short	Description	Default	Example
--api	-a	Full API import path	Required	"from mindspeed_ops.api.triton.add import add"
--input-mode		Input data source: random or file	random	random, file
--input	-i	Input tensor specification	Required	-i [10,10] x float32
--other-optional-params		Optional parameters (JSON format)	Optional	'{"alpha": 0.5}'
--seed		Random seed for reproducibility	42	123
--iterations	-I	Number of benchmark iterations	100	500
--output-format	-f	Output format	text	text, json, table
--output-file	-O	Output file path	stdout	report.txt
--output-dir	-D	Directory for output files	current dir	./results/
--project-root	-p	Project root directory	current dir	/path/to/MindSpeed-Ops

API Path Format

The --api parameter requires a full import path in the following format:

"from <module_path> import <function_name>"

Examples

Triton Implementation:

--api "from mindspeed_ops.api.triton.add import add"

ACLNN Implementation:

--api "from mindspeed_ops.api.aclnn.matmul import matmul"

TileLang Implementation:

--api "from mindspeed_ops.api.tilelang.custom_op import custom_op"

Input Modes

Random Mode (Default)

Generate random tensors with specified shape and dtype:

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input-mode random \
     --input [10,10] x float32 \
     --input [10,10] y float16

File Mode

Load tensors from .pth files:

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input-mode file \
     --input path/to/x.pth x \
     --input path/to/y.pth y

Optional Parameters

JSON Format

All optional parameters use JSON format:

--other-optional-params '{"alpha": 0.5, "block_size": 512}'

Parameter Types

1. Simple Values

{
  "alpha": 0.5,
  "block_size": 512,
  "inplace": true
}

2. Tensor Parameters

{
  "bias": {
    "is_tensor": true,
    "shape": [10, 10],
    "dtype": "float32"
  }
}

3. Mixed Parameters

{
  "alpha": 0.5,
  "bias": {
    "is_tensor": true,
    "shape": [10, 10],
    "dtype": "float32"
  }
}

Random Seed

The tool uses a fixed random seed to ensure reproducible results:

• Default seed: 42
• Custom seed: Use --seed parameter
• Effect: Ensures same random tensors are generated each run

# Use default seed (42)
python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [10,10] x float32

# Use custom seed
python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [10,10] x float32 \
     --seed 123

Advanced Usage

1. Complex API with Multiple Parameters

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.aclnn.matmul import matmul" \
     --input [256,256] a float32 \
     --input [256,256] b float32 \
     --other-optional-params '{"transpose_a": false, "transpose_b": false}'

2. Tensor Parameters

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.custom_op import custom_op" \
     --input [10,10] x float32 \
     --other-optional-params '{
       "alpha": 0.5,
       "bias": {"is_tensor": true, "shape": [10, 10], "dtype": "float32"}
     }'

3. File Mode with Custom Seed

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input-mode file \
     --input data/x.pth x \
     --input data/y.pth y \
     --seed 999

Output Formats

Text Format (Default)

============================================================
OP_STATISTIC CSV DATA (Filtered Columns)
============================================================
Note: Showing only OP_Type, Core_Type, MinTime, Avg_Time, Max_Time, Count columns
============================================================

API: from mindspeed_ops.api.triton.add import add
Input specifications:
  x: shape=[10, 10], dtype=float32
  y: shape=[10, 10], dtype=float16

Random seed: 42

Data Type: float32_float16
Source file: op_statistic_0.csv
Shape: 5 rows � 6 columns
------------------------------------------------------------
OP Type         Core Type  Min Time(us)  Avg Time(us)  Max Time(us)  Count
----------------------------------------------------------------------
Add             AI Core    12.45         15.23         18.91         100

JSON Format

{
  "api_path": "from mindspeed_ops.api.triton.add import add",
  "module": "mindspeed_ops.api.triton.add",
  "function": "add",
  "seed": 42,
  "implementations": {
    "float32_float16": {
      "source_file": "op_statistic_0.csv",
      "is_op_statistic": true,
      "columns": ["OP Type", "Core Type", "Min Time(us)", "Avg Time(us)", "Max Time(us)", "Count"],
      "data": [...]
    }
  }
}

Generated Files

File Structure

./
������ benchmark_api/
�?  ������ benchmark_<func_name>_<input_info>.py
������ msprof_<func_name>_<input_info>/
�?  ������ ... (profiling results)
������ performance_report.txt

Example

# Run benchmark
python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [10,10] x float32 \
     --input [10,10] y float16

# Generated files
./
������ benchmark_api/
�?  ������ benchmark_add_x_y.py
������ msprof_add_10_10_float32_10_10_float16/
�?  ������ ... (profiling data)
������ performance_report.txt

How It Works

1. API Import

• Parse API path
• Dynamically import module and function
• Validate import success

2. Input Processing

• Random mode: Generate tensors with torch.randn() using fixed seed
• File mode: Load tensors with torch.load()
• Tensor parameters: Generate additional tensors as needed

3. Parameter Handling

• Parse JSON format parameters
• Handle simple values and tensor parameters
• Create parameter tuple for API call

4. Benchmark Execution

• No warmup runs (direct benchmarking)
• Measures execution time across iterations
• Collects performance metrics via msprof

5. Results Analysis

• Parses profiling data
• Generates comprehensive reports
• Shows OP_STATISTIC CSV data

Prerequisites

1. Environment

# Install dependencies
pip install torch torch_npu

# Install MindSpeed-Ops
cd /path/to/MindSpeed-Ops
pip install -e .

2. Ascend Setup

• CANN toolkit installed
• NPU drivers configured
• msprof tool available

Troubleshooting

Common Issues

1. API import failed

   Error: Failed to import module 'mindspeed_ops.api.triton.add'
   

   Solution: Check module path and ensure module exists.

2. Function not found

   Error: Function 'add' not found in module
   

   Solution: Check function name and ensure it's exported.

3. Invalid API path format

   Error: Invalid API path format
   

   Solution: Use format: "from module.path import function"

4. File not found

   Error: Input file not found: path/to/x.pth
   

   Solution: Check file path and ensure file exists.

Best Practices

1. Use fixed seed: Ensure reproducible results
2. Start small: Use small shapes for quick testing
3. Scale up: Use production-level shapes after validation
4. Multiple iterations: Increase iterations for stable results
5. Save test data: Use file mode to save inputs for reproduction

Examples

Example 1: Basic Add API

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input [1024,1024] x float32 \
     --input [1024,1024] y float16

Example 2: Matmul with Parameters

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.aclnn.matmul import matmul" \
     --input [256,256] a float32 \
     --input [256,256] b float32 \
     --other-optional-params '{"transpose_a": false}'

Example 3: File Mode

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.add import add" \
     --input-mode file \
     --input data/x.pth x \
     --input data/y.pth y

Example 4: Tensor Parameters

python operator_performance_profile.py \
     --api "from mindspeed_ops.api.triton.custom_op import custom_op" \
     --input [10,10] x float32 \
     --other-optional-params '{
       "alpha": 0.5,
       "bias": {"is_tensor": true, "shape": [10, 10], "dtype": "float32"}
     }'

License

This skill is part of the MindSpeed-Ops project and follows the same licensing terms.

Support

For issues or questions:

1. Check the documentation
2. Verify API path is correct
3. Ensure proper environment configuration
4. Contact skill maintainer for assistance