Performance Optimization Overview

This document provides an overview of the full Ascend performance tuning process, along with the corresponding tools and methods for performance data collection, analysis, and tuning.

Performance Tuning Process

Before tuning, first refer to the corresponding tuning process based on the training framework you use. Different frameworks use different tuning methods and tools. Rec SDK supports PyTorch and TensorFlow. You can refer to the following tuning processes at the AI framework layer:

• PyTorch
• TensorFlow

Tool Introduction

The tuning process usually involves performance data collection and data visualization to identify performance bottlenecks. This section lists commonly used tools for reference.

Data Collection

By calling framework-layer interfaces in the model code, you can obtain framework-layer operator information, CANN-layer operator information, underlying NPU operator information, and operator memory usage information. The data obtained by these tools can help analyze operator performance and identify host, device, communication, and I/O bottlenecks:

• Ascend PyTorch Profiler
• TensorFlow Profiling API

Performance Visualization

MindStudio Insight is a visual tuning tool for Ascend AI developers. It visualizes real hardware and software runtime data, analyzes performance bottlenecks from multiple dimensions, and supports visual cluster performance analysis at scales from hundreds to thousands of cards and beyond. The tool provides timeline views, memory analysis, operator duration analysis, communication bottleneck analysis, and other functions to help developers quickly locate model performance bottlenecks:

• MindStudio Insight

Performance Comparison (Optional)

compare_tools supports comparing performance differences between GPU and NPU, and between different NPUs. By comparing training duration and memory usage, it pinpoints specific degraded operators and helps improve tuning efficiency. The tool breaks training duration down into three dimensions: computation, communication, and scheduling. It performs operator-level comparisons for computation and communication separately. It also breaks down the total memory used for training into operator-level memory usage for comparison.

• compare_tools

Performance Data Analysis Examples

The examples listed in this section include the specific process of tool usage, data analysis, and tuning. They are for reference only:

• MindStudio Insight example
• Rec SDK example

Performance Tuning Methods

Tuning methods are mainly divided into host-side and device-side methods. Some methods are framework-specific. You are advised to first refer to the following tuning strategies, identify the specific tuning scenario, and then choose the appropriate optimization method to apply.

Overall Strategy

• General tuning strategy

• PyTorch tuning strategy

Host Optimization

General Methods

• Automatic CPU core binding
• OS optimization
  • Use a high-performance memory library
  • Large page memory

PyTorch Host Optimization

• PyTorch core binding optimization Avoid thread preemption, improve cache hit rates, avoid memory access across NUMA (non-uniform memory access architecture) nodes, reduce task scheduling overhead, and improve task execution efficiency.
• Operator dispatch pipeline optimization
• Compilation optimization

Device Optimization

Communication Optimization

• Communication overview and optimization methods

PyTorch Device Optimization

• Data I/O optimization
• Fused operator replacement
  • RotaryMul & RotaryMulGrad
  • RmsNorm & RmsNormGrad
  • ScaledMaskedSoftmax & ScaledMaskedSoftmaxGrad
  • MatmulAllReduce
  • FlashAttentionScore
  • SwiGlu
• Fused optimizer replacement
• Affinity operator replacement
  • Matrix indexing IndexPut
  • Nonzero
  • where
• Memory optimization

TensorFlow Device Optimization

• Mixed-precision training
• Affinity operator replacement
• Training iteration loop sinking
• AOE automatic tuning