torch_extension APIs
How to Use
To simplify operator calling, the project provides a set of APIs compatible with the native PyTorch style. This API uses the JIT mechanism (torch.utils.cpp_extension.load) of PyTorch to compile the C++ Kernel Wrapper in real time when it is called for the first time and bridge the PyTorch function to the aclnn API of CANN. In addition, GE Converter supports the TorchAir graph mode, helping developers build models and applications.
-
Software Package Description
Before calling a torch_extension API, ensure that the CANN Toolkit, ops-transformer, and Ascend for PyTorch packages have been installed.
-
Calling Method
When the torch_extension API is called, the
cann-ops-transformermodule is required, which is defined in${INSTALL_DIR}/python/sitepackage/cann-ops-transformer.${INSTALL_DIR}indicates the file path after CANN installation.import torch import torch_npu import cann_ops_transformer -
Version Evolution
Note that some APIs have multiple versions. Select the latest version. (APIs of a later version are compatible with all capabilities of an earlier version.)
API List
Note
- Operator features: Before calling APIs, learn the basic knowledge of operators, including deterministic algorithms, batch consistency, and common quantization modes. For details, see Basic Concepts.
- Symbol description: The hyphen (-) in the table indicates that the interface does not support the current product.
| API Name | Description | Deterministic Computation (A2/A3) | Deterministic Computation (Ascend 950) |
|---|---|---|---|
| block_attn_res_prepare | Completes the first phase of the two-phase calculation of Attention Residuals historical residuals, and returns the softmax weighted numerator and statistics. | - | Supported by default |
| block_attn_res_update | Accumulates delta in place into partial_block, calculates the RMSNorm score of the updated partial_block, combines it with the historical online softmax intermediate state, and returns the current layer result h. |
- | Supported by default |
| block_attention_residuals | Concatenate partial_block and block_res, perform RMS, projection scoring, and Softmax weighted fusion, and output hidden_states. |
Supported by default | Supported by default |
| block_sparse_attention | Call BlockSparseAttention to complete block-level sparse attention computation. | - | - |
| apply_rotary_pos_emb | Fuses the query and key rotation position encoding calculation and returns the query and key output tensors after rotation position encoding. | Supported by default | Supported by default |
| apply_rotary_pos_emb_grad | Performs reverse computation of dual-channel rotation position encoding, and combines the gradient computation of the query and key into one kernel call. | - | Supported by default |
| attention_to_ffn | Sends token data from the Attention node to the FFN node. | - | Supported by default |
| all_gather_quant_matmul | Performs the fusion calculation of AllGather communication and MX quantization matrix multiplication. | - | Supported by default |
| all_to_all_quant_matmul | Performs the fusion calculation of all-to-all communication and MX quantization matrix multiplication. | - | Supported by default |
| causal_conv1d_fn | Performs causal one-dimensional convolution forward computation (prefill/chunk-prefill) and encapsulates aclnnCausalConv1dFn. | - | Deterministic computing is supported by default. |
| causal_conv1d_update | 1D convolutional state update (decode/update), encapsulated with aclnnCausalConv1dUpdate. | - | Deterministic computing is supported by default. |
| compressor | The KV cache of every four or 128 tokens is compressed into one, and then DSA calculation is performed on each token and the compressed KV cache. | Deterministic computing is supported by default. | Deterministic computing is supported by default. |
| ElasticBuffer | Unified distributed Engram storage and MoE dispatch/combine communication buffer management, matching get_engram_storage_size_hint and engram_fetch/engram_fetch_grad interfaces. | - | - |
| dense_lightning_indexer_softmax_lse | In the dense scenario, the DenseLightningIndexerGradKlLoss operator is a branch operator for calculating the Softmax input. Compressed Attention is supported, and the metadata front-end operator can be used for core-based load balancing. It must be used together with dense_lightning_indexer_softmax_lse_metadata. | - | Supported by default |
| dense_lightning_indexer_softmax_lse_metadata | Frontend interface of the dense_lightning_indexer_softmax_lse interface, which is used to calculate the load balancing of dense_lightning_indexer_softmax_lse. | - | Supported by default |
| flash_attn | The FlashAttn operator is called to complete non-quantized attention calculation of the shared KV (the key and value use the same input) and normalize the training and inference. It must be used together with flash_attn_metadata. |
- | Deterministic computing is supported by default. |
| flash_attn_grad | Calls the FlashAttnGrad operator to compute the backward gradient of Flash Attention and calculates dq, dk and dv based on the forward softmax_lse, attn_out, and upstream gradient dout. It must be used together with flash_attn_metadata (is_grad_enabled=True). |
- | Deterministic computing is supported by default. |
| ffn_to_attention | An FFN node in a communication domain sends data to an Attention node and writes a status bit to detect whether a communication link is normal. | - | Deterministic computing is supported by default. |
| fused_causal_conv1d | Performs causal one-dimensional convolution on the sequence. Padding is performed on the header of each sequence along the sequence dimension (the length is the width of the convolution kernel minus 1) to ensure that the output depends on the current and historical inputs. After the convolution is complete, some data of the current sequence is updated to the cache. Based on the causal one-dimensional convolution output, the original input is added to the output to implement residual connection. Supports features such as automatic prefix caching (APC), speculative decoding (MTP), and residual connection. | - | Supported by default |
| fused_causal_conv1d_ | Performs causal one-dimensional convolution on the sequence. Padding is performed on the header of each sequence along the sequence dimension (the length is the width of the convolution kernel minus 1) to ensure that the output depends on the current and historical inputs. After the convolution is complete, some data of the current sequence is updated to the cache. Based on the causal one-dimensional convolution output, the original input is added to the output to implement residual connection. Supports features such as automatic prefix caching (APC), speculative decoding (MTP), residual connection, and in-place update. | - | Supported by default |
| get_low_latency_ccl_buffer_size | HCCL communication buffer_size (unit: MB) required for calculating low_latency_dispatch/low_latency_combine. It is a static method of MoeDistributeBuffer and can be called before initialization. | Supported by default | Supported by default |
| generic_block_sparse_attention | Call GenericBlockSparseAttention to complete sparse attention calculation for blocks of any granularity. | - | - |
| generic_block_sparse_attention_grad | Call GenericBlockSparseAttentionGrad to complete the general block sparse attention inverse computation. | - | - |
| grouped_matmul_activation_quant | Fuses the GMM, activation function, and quantization operator to complete matrix multiplication, activation, and quantization computation by group, and outputs the quantization result and quantization factors. | - | Supported by default |
| indexer_quant_cache | In the Epilog phase of the indexer attention mechanism, the KV cache is dynamically quantized, compressed, and updated, and aclnnIndexerQuantCache is encapsulated. | - | Supported by default |
| inplace_partial_rotary_mul | Performs the inplace computation for single-channel rotation position encoding. The input tensor is directly modified, and no new output tensor is generated. | Supported by default | Supported by default |
| inplace_partial_rotary_mul_backward | Performs backward computation of inplace_partial_rotary_mul, performs inplace update on the input gradient tensor, and replaces the tensor within the slice with the RoPE gradient, while keeping the tensor outside the slice unchanged. |
- | Supported by default |
| key_pool | Key compression operator in the inference scenario: K/Gate projection is performed on the input token, and the token is grouped by cmp_ratio and weighted pooled into keys. The status of uncompleted groups is saved by state_cache. | - | - |
| kv_compress_epilog | In the Epilog phase of the KV cache, the cache is quantized, compressed, and updated in place, and aclnnKvCompressEpilog is encapsulated. | Supported by default | - |
| kv_quant_sparse_flash_attention | Call KvQuantSparseFlashAttentionV2 to complete quantization and attention computation in sparse scenarios. The Per-Token-Head-Tile-128 quantization input is supported. | - | Supported by default |
| lightning_indexer | Top-k locations corresponding to each token are obtained based on a series of operations. KV compression is supported. | Supported by default | - |
| lightning_indexer_metadata | Front interface of the lightning_indexer interface, which is used to calculate the load balancing of the lightning_indexer interface. | Supported by default | Supported by default |
| lightning_indexer_kl_loss | Calculate the KL divergence loss between the teacher distribution and student distribution in Lightning Indexer. | By default, it is non-deterministic. You can call torch.use_deterministic_algorithms to enable deterministic computation. |
- |
| low_latency_dispatch | Implements low-latency dispatch of tokens in MoE parallel deployment and supports dynamic quantization and alltoallv communication in the EP domain. This function must be used together with low_latency_combine. | Supported by default | Supported by default |
| low_latency_combine | This field is used with low_latency_dispatch. The low-latency combine reverse aggregation of the token returned by dispatch is performed based on the original path. If topk_weights is not empty, the value is multiplied by the route weight and then added. If topk_weights is None, the value is directly added. | Supported by default | Supported by default |
| mega_moe | MoE E2E general computing fusion operator, which fuses Dispatch+GroupMatmul1+SwiGLUQuant+GroupMatmul2+Combine into a single operator and is used together with get_mega_moe_ccl_buffer_size and get_symm_buffer_for_mega_moe. | - | Supported by default |
| mhc_post | Implements forward computation of the MHC Post component, which is used in the post-processing phase of multi-layer residual join in the Transformer model. This operator fuses the residual matrix transformation and output state projection into a single computation, avoiding extra overhead caused by multiple independent operator calls. | Supported by default | - |
| mhc_pre_sinkhorn | Based on a series of calculations, the Hres′\mathbf{H}'_{\text{res}} and Hpost\mathbf{H}_{\text{post}} projection matrices of the hidden layer in the MHC architecture and the input matrix hin\mathbf{h}_{\text{in}} of the Attention or MLP layer are obtained. Performs Sinkhorn iterative normalization on the Hres′\mathbf{H}'_{\text{res}} matrix to obtain the double random matrix Hres\mathbf{H}_{\text{res}}. The intermediate calculation result can be output for backward gradient calculation. | Supported by default | Supported by default |
| mixed_quant_sparse_flash_mla | In quantization scenarios, MixedQuantSparseFlashMla sparse attention computation is performed based on the shared KV. It must be used together with mixed_quant_sparse_flash_mla_metadata. |
Supported by default | Supported by default |
| mla_prolog | Call MlaPrologV4WeightNz to preprocess the query/KV in the MLA decoder. | Supported by default | Supported by default |
| moe_finalize_routing | The output results of each expert FFN are weighted and combined based on the route weight to restore the original token sequence. | Deterministic computing is supported by default. | Deterministic computing is supported by default. |
| moe_finalize_routing_grad | Backward API of moe_finalize_routing, which encapsulates aclnnMoeFinalizeRoutingV2Grad. |
Supported by default | Supported by default |
| moe_init_routing | Performs mixture of experts (MoE) routing based on the computation results of moe_gating_top_k_softmax. Non-quantization, static quantization, and dynamic quantization configurations are supported. | Supported by default | Supported by default |
| moe_init_routing_grad | Backward interface of moe_init_routing, which encapsulates aclnnMoeInitRoutingV2Grad. | Supported by default | Supported by default |
| moe_re_routing | Rearranges tokens by expert order in the Mixture of Experts (MoE) network after AlltoAll communication across ranks. The topkWeight can be rearranged. | Supported by default | Supported by default |
| moe_token_permute | Extends and sorts tokens based on the expert index. | Supported by default | - |
| msa_index_score | Calculates the block score of the MSA index branch, and performs matmul+maxpool on each query token and KV sparse block to obtain the importance score. | Supported by default | Supported by default |
| pool_key_indexer | Multiple consecutive tokens are packed into a pool, and the attention correlation score is calculated based on the pool and the top-k positions are selected. In this way, the index overhead is reduced while the advantages of sparse attention are retained. | Supported by default | Supported by default |
| qkv_rms_norm_rope_cache_with_k_scale | Integrates Q/K/V splitting, Q/K RMSNorm, RoPE/M-RoPE, quantization, and KV cache update, and supports in-place and functional interfaces and M-RoPE MX scenarios. | - | Supported by default |
| quant_all_reduce | Implements AllReduce communication for low-bit data, dequantizes data during communication, and outputs communication results. | - | - |
| quant_compressor | A quantized version of Compressor. It compresses every four or 128 tokens into one KV cache, and then performs DSA calculation on each token and the compressed KV cache. | - | Supported by default |
| quant_flash_attn | The QuantFlashAttn operator is called to complete full-quantized attention computation in the MxFP8/HiF8/MxFP4 quantization scenario and normalize training and inference. | Supported by default | - |
| quant_flash_attn_grad | Call the QuantFlashAttnGrad operator to perform full-quantized attention computation and training inference normalization in the HiF8 quantization scenario. | Supported by default | - |
| quant_lightning_indexer | The top-k locations corresponding to each token are obtained based on a series of operations. | Supported by default | - |
| quant_reduce_scatter | Implements quant + reduceScatter fusion computation. | - | - |
| quant_sparse_flash_mla | Call the QuantSparseFlashMla operator to complete sparse attention calculation of the shared KV (the key and value use the same input). | Supported by default | - |
| recurrent_kda | Performs recursive forward computation of Kimi Delta Attention (KDA) in the decode and MTP short sequence scenarios. | Supported by default | Supported by default |
| scatter_pa_kv_cache_with_k_scale | Updates the key and value at a specified position in KvCache and updates the scale value of the key in training scenarios. | - | Supported by default |
| sparse_flash_mla | SparseFlashMla sparse attention calculation is performed based on the shared KV. It must be used together with sparse_flash_mla_metadata. |
Supported by default | Supported by default |
| sparse_flash_mla_grad | Computes the backward output of attention in the SparseFlashMla training scenario. Sliding Window Attention, Compressed Attention, and Sparse Compressed Attention are supported. It must be used together with sparse_flash_mla_grad_metadata. |
- | Supported by default |
| sparse_lightning_indexer_kl_loss_grad | Backward output in the Lightning Indexer KL Loss training scenario. It must be used together with sparse_lightning_indexer_kl_loss_grad_metadata. |
Supported by default | Supported by default |
| stem_oam_prep_varlen_q | Performs pre-processing on the Q side in Stem OAM block-sparse attention and converts the variable-length Q tensor into the flattened qFlat output grouped by stem block. | - | Supported by default |
| dense_lightning_indexer_kl_loss_grad | Backward output of Lightning Indexer KL Loss training in the dense scenario. It must be used together with dense_lightning_indexer_kl_loss_grad_metadata. | - | Supported by default |
| sparse_flash_mla_softmax_l1_norm | Preprocessing API of dense_lightning_indexer_kl_loss_grad, which generates attn_softmax_l1_norm. It must be used together with sparse_flash_mla_softmax_l1_norm_metadata. |
- | Supported by default |
| stem_oam_prep_paged_kv | Pre-scoring module of the dynamic sparse attention mechanism for foundation model inference, which is the pre-scoring module of block-sparse-attention. | - | Supported by default |
| und_gen_qkv_rms_norm_rope_cache | Indirect addressing and stitching of the und and gen QKV segments, Q/K RMSNorm, MRoPE, and paged KV cache update are integrated. Q is returned, and k_cache/v_cache are updated in place. | - | Supported by default |