ops-transformer_4703:基于 CANN 算子生态的 Transformer 大模型计算算子库项目

本项目是CANN提供的transformer类大模型算子库,实现网络在NPU上加速计算。

Branch19Tags0

ops-transformer

English | 简体中文

CANNcommunity repo license contributing SIG Zread sample


🔥Latest News

🚀Overview

ops-transformer is an advanced operator library in the CANN (Compute Architecture for Neural Networks) operator library that provides computation for transformer-type large models. It includes the attention, moe, and mc2 operator categories, covering various scenarios such as attention computation, MoE computation, and communication-computation fusion. The position of the operator library in the architecture diagram is as follows.

Architecture diagram

📌Version Compatibility

The source code of this project is released along with CANN software versions. For the mapping between CANN software versions and tags of this project, refer to the corresponding version descriptions in the release repository. Note that to ensure smooth customized development of your source code, select a compatible CANN version and the source code of the corresponding Gitcode tag. Using the master branch may cause version mismatch risks.

🛠️Environment Preparation

Environment deployment is the prerequisite for experiencing the capabilities of this project. Complete the NPU driver and CANN package installation first to ensure that the environment works properly.

⬇️Source Code Download

After the environment is ready, download the branch source code that matches the CANN version. The general command is as follows, where ${tag_version} is replaced with the branch tag name. The following uses downloading the 9.0.0 branch source code as an example:

# General command: git clone -b ${tag_version} https://atomgit.com/cann/ops-transformer.git
git clone -b 9.0.0 https://atomgit.com/cann/ops-transformer.git

Description: If the compatible branch source code already exists in the environment, you can skip this step. For example, CANNLab provides the source code corresponding to the latest CANN version by default.

📖Learning Tutorials

  • Quick Start: Quickly experience the core basic capabilities of the project from scratch, covering source code compilation, operator invocation, development, debugging, and other operations.
  • Advanced Tutorials: If you need to learn more about project compilation and deployment, operator invocation, development, debugging and tuning, and other capabilities, see the documentation center for detailed guidance.

📘Sample Practice

To learn high-performance practical cases in the operator domain, see the cann-samples Performance directory.

Operator category Sample Description
attention full_quant_fused_infer_attention_score_story Provides a per-block full-quantization example for the FIA (Fused Infer Attention Score) operator, including input data generation, operator execution, and result verification.
mc2 moe_dispatch_and_combine_story Performance optimization practice for the moe dispatch/combine communication operators, including build and run commands, test data generation, and precision verification.
moe moe_init_routing_story End-to-end tuning guide for the MoeInitRoutingV3 operator, covering multi-core parallelism, memory bandwidth optimization, in-core pipeline scheduling, SIMT programming, and hardware feature adaptation.

PS: The features and documentation of this project are continuously being updated and improved. Stay tuned for the latest version.

  • Issue feedback: Submit issues through GitCode Issues.
  • Community interaction: Join discussions through GitCode Discussions.
  • Technical column: Get technical articles (such as serialized tutorials and best practices) through GitCode Wiki.

​

Introduction

本项目是CANN提供的transformer类大模型算子库,实现网络在NPU上加速计算。

Customize your domain