torch_extension APIs

How to Use

To simplify operator calling, the project provides a set of APIs compatible with the native PyTorch style. This API uses the JIT mechanism (torch.utils.cpp_extension.load) of PyTorch to compile the C++ Kernel Wrapper in real time when it is called for the first time and bridge the PyTorch function to the aclnn API of CANN. In addition, GE Converter supports the TorchAir graph mode, helping developers build models and applications.

  • Software Package Description

    Before calling a torch_extension API, ensure that the CANN Toolkit, ops-transformer, and Ascend for PyTorch packages have been installed.

  • Calling Method

    When the torch_extension API is called, the cann-ops-transformer module is required, which is defined in ${INSTALL_DIR}/python/sitepackage/cann-ops-transformer. ${INSTALL_DIR} indicates the file path after CANN installation.

    import torch
    import torch_npu
    import cann_ops_transformer
    
  • Version Evolution

    Note that some APIs have multiple versions. Select the latest version. (APIs of a later version are compatible with all capabilities of an earlier version.)

API List

Note

  • Operator features: Before calling APIs, learn the basic knowledge of operators, including deterministic algorithms, batch consistency, and common quantization modes. For details, see Basic Concepts.
  • Symbol description: The hyphen (-) in the table indicates that the interface does not support the current product.
API Name Description Deterministic Computation (A2/A3) Deterministic Computation (Ascend 950)
block_attn_res_prepare Completes the first phase of the two-phase calculation of Attention Residuals historical residuals, and returns the softmax weighted numerator and statistics. - Supported by default
block_attn_res_update Accumulates delta in place into partial_block, calculates the RMSNorm score of the updated partial_block, combines it with the historical online softmax intermediate state, and returns the current layer result h. - Supported by default
block_attention_residuals Concatenate partial_block and block_res, perform RMS, projection scoring, and Softmax weighted fusion, and output hidden_states. Supported by default Supported by default
block_sparse_attention Call BlockSparseAttention to complete block-level sparse attention computation. - -
apply_rotary_pos_emb Fuses the query and key rotation position encoding calculation and returns the query and key output tensors after rotation position encoding. Supported by default Supported by default
apply_rotary_pos_emb_grad Performs reverse computation of dual-channel rotation position encoding, and combines the gradient computation of the query and key into one kernel call. - Supported by default
attention_to_ffn Sends token data from the Attention node to the FFN node. - Supported by default
all_gather_quant_matmul Performs the fusion calculation of AllGather communication and MX quantization matrix multiplication. - Supported by default
all_to_all_quant_matmul Performs the fusion calculation of all-to-all communication and MX quantization matrix multiplication. - Supported by default
causal_conv1d_fn Performs causal one-dimensional convolution forward computation (prefill/chunk-prefill) and encapsulates aclnnCausalConv1dFn. - Deterministic computing is supported by default.
causal_conv1d_update 1D convolutional state update (decode/update), encapsulated with aclnnCausalConv1dUpdate. - Deterministic computing is supported by default.
compressor The KV cache of every four or 128 tokens is compressed into one, and then DSA calculation is performed on each token and the compressed KV cache. Deterministic computing is supported by default. Deterministic computing is supported by default.
ElasticBuffer Unified distributed Engram storage and MoE dispatch/combine communication buffer management, matching get_engram_storage_size_hint and engram_fetch/engram_fetch_grad interfaces. - -
dense_lightning_indexer_softmax_lse In the dense scenario, the DenseLightningIndexerGradKlLoss operator is a branch operator for calculating the Softmax input. Compressed Attention is supported, and the metadata front-end operator can be used for core-based load balancing. It must be used together with dense_lightning_indexer_softmax_lse_metadata. - Supported by default
dense_lightning_indexer_softmax_lse_metadata Frontend interface of the dense_lightning_indexer_softmax_lse interface, which is used to calculate the load balancing of dense_lightning_indexer_softmax_lse. - Supported by default
flash_attn The FlashAttn operator is called to complete non-quantized attention calculation of the shared KV (the key and value use the same input) and normalize the training and inference. It must be used together with flash_attn_metadata. - Deterministic computing is supported by default.
flash_attn_grad Calls the FlashAttnGrad operator to compute the backward gradient of Flash Attention and calculates dq, dk and dv based on the forward softmax_lse, attn_out, and upstream gradient dout. It must be used together with flash_attn_metadata (is_grad_enabled=True). - Deterministic computing is supported by default.
ffn_to_attention An FFN node in a communication domain sends data to an Attention node and writes a status bit to detect whether a communication link is normal. - Deterministic computing is supported by default.
fused_causal_conv1d Performs causal one-dimensional convolution on the sequence. Padding is performed on the header of each sequence along the sequence dimension (the length is the width of the convolution kernel minus 1) to ensure that the output depends on the current and historical inputs. After the convolution is complete, some data of the current sequence is updated to the cache. Based on the causal one-dimensional convolution output, the original input is added to the output to implement residual connection. Supports features such as automatic prefix caching (APC), speculative decoding (MTP), and residual connection. - Supported by default
fused_causal_conv1d_ Performs causal one-dimensional convolution on the sequence. Padding is performed on the header of each sequence along the sequence dimension (the length is the width of the convolution kernel minus 1) to ensure that the output depends on the current and historical inputs. After the convolution is complete, some data of the current sequence is updated to the cache. Based on the causal one-dimensional convolution output, the original input is added to the output to implement residual connection. Supports features such as automatic prefix caching (APC), speculative decoding (MTP), residual connection, and in-place update. - Supported by default
get_low_latency_ccl_buffer_size HCCL communication buffer_size (unit: MB) required for calculating low_latency_dispatch/low_latency_combine. It is a static method of MoeDistributeBuffer and can be called before initialization. Supported by default Supported by default
generic_block_sparse_attention Call GenericBlockSparseAttention to complete sparse attention calculation for blocks of any granularity. - -
generic_block_sparse_attention_grad Call GenericBlockSparseAttentionGrad to complete the general block sparse attention inverse computation. - -
grouped_matmul_activation_quant Fuses the GMM, activation function, and quantization operator to complete matrix multiplication, activation, and quantization computation by group, and outputs the quantization result and quantization factors. - Supported by default
indexer_quant_cache In the Epilog phase of the indexer attention mechanism, the KV cache is dynamically quantized, compressed, and updated, and aclnnIndexerQuantCache is encapsulated. - Supported by default
inplace_partial_rotary_mul Performs the inplace computation for single-channel rotation position encoding. The input tensor is directly modified, and no new output tensor is generated. Supported by default Supported by default
inplace_partial_rotary_mul_backward Performs backward computation of inplace_partial_rotary_mul, performs inplace update on the input gradient tensor, and replaces the tensor within the slice with the RoPE gradient, while keeping the tensor outside the slice unchanged. - Supported by default
key_pool Key compression operator in the inference scenario: K/Gate projection is performed on the input token, and the token is grouped by cmp_ratio and weighted pooled into keys. The status of uncompleted groups is saved by state_cache. - -
kv_compress_epilog In the Epilog phase of the KV cache, the cache is quantized, compressed, and updated in place, and aclnnKvCompressEpilog is encapsulated. Supported by default -
kv_quant_sparse_flash_attention Call KvQuantSparseFlashAttentionV2 to complete quantization and attention computation in sparse scenarios. The Per-Token-Head-Tile-128 quantization input is supported. - Supported by default
lightning_indexer Top-k locations corresponding to each token are obtained based on a series of operations. KV compression is supported. Supported by default -
lightning_indexer_metadata Front interface of the lightning_indexer interface, which is used to calculate the load balancing of the lightning_indexer interface. Supported by default Supported by default
lightning_indexer_kl_loss Calculate the KL divergence loss between the teacher distribution and student distribution in Lightning Indexer. By default, it is non-deterministic. You can call torch.use_deterministic_algorithms to enable deterministic computation. -
low_latency_dispatch Implements low-latency dispatch of tokens in MoE parallel deployment and supports dynamic quantization and alltoallv communication in the EP domain. This function must be used together with low_latency_combine. Supported by default Supported by default
low_latency_combine This field is used with low_latency_dispatch. The low-latency combine reverse aggregation of the token returned by dispatch is performed based on the original path. If topk_weights is not empty, the value is multiplied by the route weight and then added. If topk_weights is None, the value is directly added. Supported by default Supported by default
mega_moe MoE E2E general computing fusion operator, which fuses Dispatch+GroupMatmul1+SwiGLUQuant+GroupMatmul2+Combine into a single operator and is used together with get_mega_moe_ccl_buffer_size and get_symm_buffer_for_mega_moe. - Supported by default
mhc_post Implements forward computation of the MHC Post component, which is used in the post-processing phase of multi-layer residual join in the Transformer model. This operator fuses the residual matrix transformation and output state projection into a single computation, avoiding extra overhead caused by multiple independent operator calls. Supported by default -
mhc_pre_sinkhorn Based on a series of calculations, the Hres′\mathbf{H}'_{\text{res}} and Hpost\mathbf{H}_{\text{post}} projection matrices of the hidden layer in the MHC architecture and the input matrix hin\mathbf{h}_{\text{in}} of the Attention or MLP layer are obtained. Performs Sinkhorn iterative normalization on the Hres′\mathbf{H}'_{\text{res}} matrix to obtain the double random matrix Hres\mathbf{H}_{\text{res}}. The intermediate calculation result can be output for backward gradient calculation. Supported by default Supported by default
mixed_quant_sparse_flash_mla In quantization scenarios, MixedQuantSparseFlashMla sparse attention computation is performed based on the shared KV. It must be used together with mixed_quant_sparse_flash_mla_metadata. Supported by default Supported by default
mla_prolog Call MlaPrologV4WeightNz to preprocess the query/KV in the MLA decoder. Supported by default Supported by default
moe_finalize_routing The output results of each expert FFN are weighted and combined based on the route weight to restore the original token sequence. Deterministic computing is supported by default. Deterministic computing is supported by default.
moe_finalize_routing_grad Backward API of moe_finalize_routing, which encapsulates aclnnMoeFinalizeRoutingV2Grad. Supported by default Supported by default
moe_init_routing Performs mixture of experts (MoE) routing based on the computation results of moe_gating_top_k_softmax. Non-quantization, static quantization, and dynamic quantization configurations are supported. Supported by default Supported by default
moe_init_routing_grad Backward interface of moe_init_routing, which encapsulates aclnnMoeInitRoutingV2Grad. Supported by default Supported by default
moe_re_routing Rearranges tokens by expert order in the Mixture of Experts (MoE) network after AlltoAll communication across ranks. The topkWeight can be rearranged. Supported by default Supported by default
moe_token_permute Extends and sorts tokens based on the expert index. Supported by default -
msa_index_score Calculates the block score of the MSA index branch, and performs matmul+maxpool on each query token and KV sparse block to obtain the importance score. Supported by default Supported by default
pool_key_indexer Multiple consecutive tokens are packed into a pool, and the attention correlation score is calculated based on the pool and the top-k positions are selected. In this way, the index overhead is reduced while the advantages of sparse attention are retained. Supported by default Supported by default
qkv_rms_norm_rope_cache_with_k_scale Integrates Q/K/V splitting, Q/K RMSNorm, RoPE/M-RoPE, quantization, and KV cache update, and supports in-place and functional interfaces and M-RoPE MX scenarios. - Supported by default
quant_all_reduce Implements AllReduce communication for low-bit data, dequantizes data during communication, and outputs communication results. - -
quant_compressor A quantized version of Compressor. It compresses every four or 128 tokens into one KV cache, and then performs DSA calculation on each token and the compressed KV cache. - Supported by default
quant_flash_attn The QuantFlashAttn operator is called to complete full-quantized attention computation in the MxFP8/HiF8/MxFP4 quantization scenario and normalize training and inference. Supported by default -
quant_flash_attn_grad Call the QuantFlashAttnGrad operator to perform full-quantized attention computation and training inference normalization in the HiF8 quantization scenario. Supported by default -
quant_lightning_indexer The top-k locations corresponding to each token are obtained based on a series of operations. Supported by default -
quant_reduce_scatter Implements quant + reduceScatter fusion computation. - -
quant_sparse_flash_mla Call the QuantSparseFlashMla operator to complete sparse attention calculation of the shared KV (the key and value use the same input). Supported by default -
recurrent_kda Performs recursive forward computation of Kimi Delta Attention (KDA) in the decode and MTP short sequence scenarios. Supported by default Supported by default
scatter_pa_kv_cache_with_k_scale Updates the key and value at a specified position in KvCache and updates the scale value of the key in training scenarios. - Supported by default
sparse_flash_mla SparseFlashMla sparse attention calculation is performed based on the shared KV. It must be used together with sparse_flash_mla_metadata. Supported by default Supported by default
sparse_flash_mla_grad Computes the backward output of attention in the SparseFlashMla training scenario. Sliding Window Attention, Compressed Attention, and Sparse Compressed Attention are supported. It must be used together with sparse_flash_mla_grad_metadata. - Supported by default
sparse_lightning_indexer_kl_loss_grad Backward output in the Lightning Indexer KL Loss training scenario. It must be used together with sparse_lightning_indexer_kl_loss_grad_metadata. Supported by default Supported by default
stem_oam_prep_varlen_q Performs pre-processing on the Q side in Stem OAM block-sparse attention and converts the variable-length Q tensor into the flattened qFlat output grouped by stem block. - Supported by default
dense_lightning_indexer_kl_loss_grad Backward output of Lightning Indexer KL Loss training in the dense scenario. It must be used together with dense_lightning_indexer_kl_loss_grad_metadata. - Supported by default
sparse_flash_mla_softmax_l1_norm Preprocessing API of dense_lightning_indexer_kl_loss_grad, which generates attn_softmax_l1_norm. It must be used together with sparse_flash_mla_softmax_l1_norm_metadata. - Supported by default
stem_oam_prep_paged_kv Pre-scoring module of the dynamic sparse attention mechanism for foundation model inference, which is the pre-scoring module of block-sparse-attention. - Supported by default
und_gen_qkv_rms_norm_rope_cache Indirect addressing and stitching of the und and gen QKV segments, Q/K RMSNorm, MRoPE, and paged KV cache update are integrated. Q is returned, and k_cache/v_cache are updated in place. - Supported by default