Matrix Multiplication Template Summary

• Theoretical template: Common
• Engineering optimization: Pipeline optimization (Multi-Buffer)
• Key deliverables
  • host: 00_basic_matmul
  • kernel: basic_matmul.hpp
  • blockMmad: block_mmad_pingpong.hpp
• dispatchPolicy: MmadAtlasA2Pingpong

• Theoretical template: Common
• Engineering optimization:
  • Pipeline optimization (Multi-Buffer)
  • Read bandwidth optimization (padding) - PaddingMatrixND
• Key deliverables
  • host: 04_padding_matmul
  • kernel: padding_matmul.hpp
  • blockMmad: block_mmad_pingpong.hpp
• dispatchPolicy: MmadAtlasA2Pingpong

• Theoretical template: Common
• Engineering optimization:
  • Pipeline optimization (Multi-Buffer)
  • Pipeline optimization (Preload)
  • Read bandwidth optimization (padding) - PaddingMatrixNZ
  • Read bandwidth optimization (ShuffleK)
  • Read bandwidth optimization (instruction replacement in small-M scenarios) (requires modifying the sample to enable)
• Key deliverables
  • host: 06_optimized_matmul
  • kernel: optimized_matmul.hpp
  • Padding prologue component: padding_matmul.hpp
  • blockMmad: block_mmad_preload.hpp
• dispatchPolicy: MmadAtlasA2Preload
• ⚠️ Note: Even without PaddingMatrixNZ prologue, there is still overhead from MIX operator compilation and CV1:0 launch (greater than the overhead of launching only AIC).

• Theoretical template: MultiCoreSplitK
• Engineering optimization: Pipeline optimization (Multi-Buffer)
• Key deliverables
  • host: 09_splitk_matmul
  • kernel: splitk_matmul.hpp
  • blockMmad: block_mmad_pingpong.hpp
• dispatchPolicy: MmadAtlasA2Pingpong

(This sample primarily demonstrates the adaptation method for NZ-layout inputs. It can also be adapted to ND-layout inputs, with no MIX operator compilation and launch overhead.)

• Theoretical template: Common
• Engineering optimization:
  • Pipeline optimization (Multi-Buffer)
  • Pipeline optimization (Preload)
  • Read bandwidth optimization (ShuffleK)
• Key deliverables
  • host: 21_basic_matmul_preload_zN
  • kernel: basic_matmul_preload.hpp
  • blockMmad: block_mmad_preload.hpp
• dispatchPolicy: MmadAtlasA2Preload

• Theoretical template: MultiCoreSplitK
• Engineering optimization:
  • Pipeline optimization (Multi-Buffer)
  • Read bandwidth optimization (padding) - PaddingMatrixND
• Key deliverables
  • host: 22_padding_splitk_matmul
  • kernel: padding_splitk_matmul.hpp
  • Padding prologue component: padding_matmul.hpp
  • SplitkReduceAdd epilogue component: splitk_matmul.hpp
  • blockMmad: block_mmad_pingpong.hpp
• dispatchPolicy: MmadAtlasA2Pingpong

(This sample and its related components only implement full loading of matrix A. To implement full loading of matrix B, refer to the key deliverables for self-development.)

• Theoretical template: Common
• Engineering optimization:
  • Pipeline optimization (Multi-Buffer) (The fully loaded matrix A does not use multi-buffering in L1.)
  • Read bandwidth optimization (L1 residency)
• Key deliverables
  • host: 25_matmul_full_loadA
  • kernel: matmul_full_loadA.hpp
  • blockMmad: block_mmad_pingpong_full_loadA.hpp
• dispatchPolicy: MmadAtlasA2FullLoadA
• BlockScheduler: GemmIdentityBlockSwizzleL1FullLoad

• Theoretical template: <idp:inline displayname="code" id="code85531551151311">Common</idp:inline>
• Engineering optimization:
  • <idp:inline displayname="code" id="code191623240184">Pipeline optimization (Multi-Buffer)</idp:inline>
  • Scalar overhead reduction
• Key deliverables
  • host: 31_small_matmul
  • kernel: small_matmul.hpp
  • blockMmad: block_mmad_small.hpp
• dispatchPolicy: MmadAtlasA2Small
• BlockScheduler: not actually used in the kernel

• Theoretical template: SingleCoreSplitK
• Engineering optimization:
  • <idp:inline displayname="code" id="code416252411819">Pipeline optimization (Multi-Buffer)</idp:inline>
  • <idp:inline displayname="code" id="code11507244191919">Read bandwidth optimization (padding) - PaddingMatrixNZ</idp:inline>
  • Write bandwidth optimization
• Key deliverables
  • host: 34_single_core_splitk_matmul
  • kernel: single_core_slicek_matmul.hpp
  • Padding prologue component and RemovePaddingNDAndCast epilogue component: padding_matmul.hpp
  • blockMmad: block_mmad_single_core_splitk.hpp
• dispatchPolicy: MmadAtlasA2SingleCoreSplitk
• BlockScheduler: SingleCoreSplitkGemmIdentityBlockSwizzle

Tiling Modeling

The figure shows a conventional FP16 matrix multiplication (accumulation in FP32 on L0C). Define the following parameters:

• Problem shape: $M$, $N$, $K$
• TileShape when moving data into L1Cache: $m_1$, $n_1$, $k_1$
• TileShape when moving data into L0A/L0B and out of L0C: $m_0$, $n_0$, $k_0$

Core distribution is performed along the $M$ and $N$ directions. Tiling is done by $m_1$ and $n_1$. This produces $\frac{MN}{m_1{n}_1}$ basic task blocks, which are assigned to AIC cores for data movement and computation. Each basic task block needs to move $m_{1}K+K{n}_1$ data elements, compute $m_1{n}_1$ results, and move them out. This yields the following constraints:

• $m_1{k}_1\ast L1Stag{e}_A+n_1{k}_1\ast L1Stag{e}_B<=L1Size/2Byte$
• $m_0{k}_0\ast L0AStage<=L0ASize/2Byte$
• $n_0{k}_0\ast L0BStage<=L0BSize/2Byte$
• $m_0{n}_0\ast L0CStage<=L0CSize/4Byte$
• $m_0=m_1$
• $n_0=n_1$

Amount of Data Read

Each basic task block needs to move $m_{1}K+K{n}_1$ elements of data. The total amount of data read is:

$2Byte\ast [m_{1}K+K{n}_1]\ast \frac{MN}{m_1{n}_1}=2Byte\ast MNK\ast [\frac{1}{m_1}+\frac{1}{n_1}]$

Amount of Data Written

Each basic task block computes and moves out $m_1{n}_1$ results. The total amount of data written is:

$2Byte\ast MN$

Amount of Computation

Each data point in the output matrix C requires $K$ multiply-add operations. The total amount of computation is fixed at:

$2MNK$

In most cases, computation time is rigid and depends only on the number of participating AIC cores. It is the same across all theoretical templates and will not be discussed further below.

Tiling Modeling

The figure shows a conventional FP16 matrix multiplication (accumulation in FP32 on L0C). A total of 12 basic task blocks are tiled along the $MN$ direction. Assume that there are 24 physical AIC cores. In this case, the load is unbalanced. Therefore, the $K$ axis is tiled into two $k$ segments, producing 24 basic task blocks and achieving load balancing across the AIC cores. Define the following parameters:

• Problem shape: $M$, $N$, $K$
• TileShape when moving data into L1Cache: $m_1$, $n_1$, $k_1$
• TileShape when moving data into L0A/L0B and out of L0C: $m_0$, $n_0$, $k_0$
• Compared to the Common template, a new tiling length $k$ along the $K$ direction is added.

When $m_1$ and $n_1$ are large, load imbalance may occur. That is, the number of task blocks tiled along the M and N directions is far less than the number of AIC cores, resulting in low read bandwidth (insufficient number of cores). Therefore, cores can be distributed along the K direction. This produces $\frac{MNK}{m_1{n}_{1}k}$ basic task blocks, which are assigned to AIC cores for data movement and computation. Each basic task block needs to move $m_{1}k+k{n}_1$ data elements, compute $m_1{n}_1$ results, and move them out. Hardware constraints are the same as those for the Common template.

Amount of Data Read

Each basic task block needs to move $m_{1}K+K{n}_1$ elements of data. The total amount of data read is the same as that for the Common template:

$2Byte\ast [m_{1}k+k{n}_1]\ast \frac{MNK}{m_1{n}_{1}k}=2Byte\ast MNK\ast [\frac{1}{m_1}+\frac{1}{n_1}]$

Amount of Data Written

Each basic task block computes and moves out $m_1{n}_1$ results. It requires $\frac{K}{k}$ basic blocks to accumulate to obtain the final output for the $m_1{n}_1$ blocks of the output matrix C. The total amount of data written is:

$2Byte\ast MNK/k$

Qualitative Analysis

Compared to the Common template, the amount of data read remains unchanged, the amount of data written increases, and there is overhead from ReduceAdd (including MIX operator compilation and launch overhead). However, more basic blocks are tiled, making load balancing easier.

Tiling Modeling

The figure shows a conventional FP16 matrix multiplication (accumulation in FP32 on L0C). Define the following parameters:

• Problem shape: $M$, $N$, $K$
• TileShape when moving data into L1Cache: $m_1$, $n_1$, $k_1$
• TileShape when moving data into L0A/L0B and out of L0C: $m_0$, $n_0$, $k_0$

Compared to the Common template, to reduce the amount of data read and further increase $m_1$ and $n_1$ abstractly, consider directly computing the $m_1{k}_1$ tile with all corresponding $k_1{n}_1$ tiles (equivalent to scaling $n_1$ up to $N$). In this case, the output $m_0{n}_0$ tile cannot be resident in $L0C$ for accumulation and must be moved out in a timely manner, accumulating in GM via atomicAdd. Hardware constraints are as follows:

• $m_1{k}_1\ast L1Stag{e}_A+n_1{k}_1\ast L1Stag{e}_B<=L1Size/2Byte$
• $m_0{k}_0\ast L0AStage<=L0ASize/2Byte$
• $n_0{k}_0\ast L0BStage<=L0BSize/2Byte$
• $m_0{n}_0\ast L0CStage<=L0CSize/4Byte$
• $m_0<=m_1$
• $n_0<=n_1$

Amount of Data Read

Using the data read formula in the Common template, scale $n_1$ to $N$. Alternatively, from the perspective of tiling basic task blocks from matrix A, get $\frac{MK}{m_1{k}_1}$ basic blocks tiled. Each basic block moves in this matrix A tile and the corresponding entire B matrix tile. The amount of data moved is $m_1{k}_1+k_{1}N$:

$2Byte\ast [m_1{k}_1+k_{1}N]\ast \frac{MK}{m_1{k}_1}=2Byte\ast MNK\ast [\frac{1}{m_1}+\frac{1}{N}]$

Amount of Data Written

Tile matrix A into basic task blocks, producing $\frac{MK}{m_1{k}_1}$ basic blocks. Each basic task block computes and moves out $m_{1}N$ results. The total amount of data written is:

$2Byte\ast MNK/k_1$

Qualitative Analysis

Compared to the Common template, the amount of data moved in decreases, the amount of data written out increases, and there is no dependency on AIV.

Problem Analysis

The following figure shows a simple scenario under a Common template. For a single AIC to process a basic task block C, the required matrix A/B tiles are small and can fit entirely in L1. When moving matrix A/B from L1 to L0, four times of partitioning are required.

An example of the instruction pipeline diagram for each pipe is as follows:

If, when loading data tiles into AIC's L1/L0A/L0B/L0C, you always try to fill all the space, it leads to serialized pipelines across different pipes, resulting in low efficiency.

Optimization Solution

Use the conventional optimization technique Multi-Buffer. Enable multi-buffering in L1/L0A/L0B/L0C to make pipelines as parallel as possible to improve efficiency. This strategy is illustrated in the following figure:

An example of the instruction pipeline diagram for each pipe is as follows. 0 and 1 on MTE1 instructions indicate ping-pong pipelining:

⚠️ Note that when combined with L1 residency optimization, disable multi-buffering for the resident tiles of matrix A/B.

Code Location for the Feature

Since this is a conventional optimization technique, it is enabled in all blockMmad components.

Problem Analysis

Through simulation pipeline analysis, issues were found with the ping-pong policy in blockMmad:

• On the MTE2 pipeline, there is a bubble between loading the last matrix A (matrix B) tile for the current matrix C basic block computation and loading the first matrix A (matrix B) tile for the next matrix C basic block computation.

Optimization Solution

For the GM->L1 process, when reading $m_1{k}_1$ ($k_1{n}_1$) of the current round, compute the data read in the previous round (assuming one round of preloading and PRELOAD_STAGES = 1). The pseudocode for the steps is as follows:

for ... {
    // Load data for the current round.
    copyGM2L1A
    copyGM2L1B
    preload_count++
    for (preload_count == PRELOAD_STAGES) {
        // Compute data from the previous PRELOAD_STAGES rounds.
        copyL12L0A
        copyL12L0B
        Mmad
    }
}

The following figure shows a simple scenario under the Common template. For a single AIC to process two basic task blocks C1 and C2, the required matrix A/B tiles need to be partitioned twice to fit in L1, and the matrix A/B L1 tiles need to be partitioned four times when moving from L1 to L0. (You can refer to the Pipeline optimization (Multi-Buffer) section above for a better understanding.) Below is a comparison of instructions for MmadAtlasA2Pingpong, MmadAtlasA2Preload, and MmadAtlasA2PreloadAsync:

• In MmadAtlasA2Pingpong, two blockMmad calls complete the computation of C1 and C2 separately.
• In MmadAtlasA2Preload, two blockMmad calls also complete the computation of C1 and C2 separately, but the GmToL1 movement of A3/B3 is advanced to the first blockMmad call.
• In MmadAtlasA2PreloadAsync, two blockMmad calls and one SynchronizeBlock call are used. The L1ToL0 movement, tileMmad, and C1 movement of A2/B2 are postponed from the first blockMmad call to the second. The L1ToL0 movement, tileMmad, and C2 movement of A4/B4 are postponed from the second blockMmad call to the SynchronizeBlock call.

An example of the instruction pipeline diagram for each pipe is as follows. Ultimately, the GmToL1 movement of A3/B3 blocks is advanced, reducing the movement bubble on the MTE2 pipeline:

Code Location for the Feature

• block_mmad_preload.hpp, corresponding to dispatchPolicy: MmadAtlasA2Preload. You need to manually compute the information about the next block of preloaded data in the kernel.
• block_mmad_preload_async.hpp, corresponding to dispatchPolicy: MmadAtlasA2PreloadAsync. With asynchronous control in use, you do not need to manually compute the information about the next block of preloaded data. Callback can be passed after Mmad computation completes.
• block_mmad_preload_async_with_callback.hpp, corresponding to dispatchPolicy: MmadAtlasA2PreloadAsyncWithCallback. With asynchronous control in use, you do not need to manually compute the information about the next block of preloaded data. Callback can be passed before and after blockMmad computation.

Problem Analysis

When data read is the main pipeline, optimizing read bandwidth can yield performance gains. Using the FP16 matrix A as an example, the following low-bandwidth scenarios currently exist:

• Low bandwidth due to Stride not being 512-byte aligned When the movement parameter srcDValue (see DataCopy-ND2NZ Movement with Channel Conversion) is not 512-byte aligned, the bandwidth decreases significantly.
• Low bandwidth due to movement instruction restrictions. For ND2NZ movement instructions, the srcDValue parameter is of type uint16, with a maximum value of 65535. When K > 65535, the movement instruction must be called repeatedly in the M direction with ndNum=1, reducing read bandwidth.
• ND2NZ with channel conversion has bandwidth loss compared to ND2ND (no layout conversion).

Optimization Solution

For the above scenarios, an AIV helps rearrange the data format (a preprocessing action). When the rearrangement overhead is lower than the bandwidth loss, there is a performance gain. Based on complexity and the addressable scenarios, there are three different rearrangement methods.

PaddingMatrixND

Aligns the Stride direction to 512 bytes. This has the lowest implementation complexity and can handle bandwidth decreases caused by Stride misalignment.

PaddingMatrixBlockND

Rearranges data by $m_1\ast k_1$ as the "block" granularity. Within a block, it is row-major. Between blocks, it is also row-major. $k_1$ is 512-byte aligned. This has moderate implementation complexity and can handle bandwidth decreases caused by Stride misalignment and Stride exceeding 65535.

PaddingMatrixNZ

Rearranges data into the zN format, which poses the highest implementation complexity (among the padding strategies described). This is because the data layout is consistent with that in L1, resulting in the highest movement bandwidth. It can handle bandwidth decreases caused by Stride misalignment, Stride exceeding 65535, and ND2NZ with channel conversion.

In practice, different padding methods work for different cases. There is no globally go-to padding solution.

Code Location for the Feature

• padding_matmul.hpp contains the padding prologue component.
• For details about the actual adaptation, see 06_optimized_matmul. Use PaddingTag and PaddingBuilder to assemble the padding prologues for matrix A and matrix B.

Problem Analysis

Typically, all AIC cores start moving data from the first tile along the $K$ direction. This can result in multiple cores reading data from the same location in GM simultaneously, creating data read conflicts and reducing read bandwidth.

Optimization Solution

Using the Common template as an example:

• $CoreX$ in $matC$ indicates that the basic block is assigned to the $X$-th AIC core for computation.
• $Aj$ indicates the $j$-th L1Tile basic block of matrix A partitioned along the $K$ axis under this $m_1$.
• $Bij$ indicates the $j$-th L1Tile basic block of matrix B partitioned along the $K$ axis under this $n_1$.
• The core distribution for the matC basic blocks uses Swizzle<2, 1>. For details, see swizzle_explanation.

In the original solution shown on the left, both $Core2$ and $Core3$ move matrix A in the order of $A0$ -> $A1$ -> $A2$ -> $A3$, resulting in a data read conflict.

In the ShuffleK solution shown in the figure, the starting movement index $j$ is offset based on $CoreIdx$. $Core2$ moves matrix A in the order of $A2$ -> $A3$ -> $A0$ -> $A1$, and correspondingly moves matrix B in the order of $B02$ -> $B03$ -> $B00$ -> $B01$. $Core3$ moves matrix A in the order of $A3$ -> $A0$ -> $A1$ -> $A2$, and correspondingly moves matrix B in the order of $B13$ -> $B10$ -> $B11$ -> $B12$. This staggers the timing, avoiding access conflicts to the same addresses.

Code Location for the Feature

• block_mmad_preload.hpp
• block_mmad_preload_async.hpp
• block_mmad_preload_async_with_callback.hpp

The above block-layer implementation code achieves staggering by setting the starting L1 index to CoreIdx/kTileCount:

kTileCount = CeilDiv<L1TileShape::K>(actualShape.k());
startTileIdx = AscendC::GetBlockIdx();
firstTileIdx = startTileIdx % kTileCount;

Problem Analysis

In matrix computation, when $M$ is small (for example, $M$ < 8), using DataCopy with ND2NZ with channel conversion (see DataCopy - ND2NZ Movement with Channel Conversion) is not efficient.

Optimization Solution

Use a for loop to move one row at a time, and call DataCopy for each row to perform multiple movements (using common strided movements).

Code Location for the Feature

• CopyGmToL1IntervalDataCopy
• For details about the actual adaptation, see 06_optimized_matmul. Manually replace using CopyGmToL1A = Gemm::Tile::CopyGmToL1IntervalDataCopy<ArchTag, AType>; in struct TileCopyOpt instead of using the default using CopyGmToL1A = typename Base::CopyGmToL1A;.

struct TileCopyOpt : public Catlass::Gemm::Tile::TileCopy<ArchTag, AType, BType, CType, BiasType> {
    ...

-   // using CopyGmToL1A = Gemm::Tile::CopyGmToL1IntervalDataCopy<ArchTag, AType>;
+   using CopyGmToL1A = Gemm::Tile::CopyGmToL1IntervalDataCopy<ArchTag, AType>;

-   using CopyGmToL1A = typename Base::CopyGmToL1A;
+   // using CopyGmToL1A = typename Base::CopyGmToL1A;
    ...
};

Optimization Solution

In practice, a tile can be made resident in L1, reducing repeated reads of tile data. This effectively improves read bandwidth. This feature needs to be implemented in conjunction with different theoretical templates.

Code Location for the Feature

• For details about the Common template, see 25_matmul_full_loadA and related deliverables. This example optimizes performance in specific scenarios by fully loading matrix A on a single core or multiple cores along the M axis, and uses a dedicated swizzle policy to increase the reuse frequency of the fully loaded matrix A block in L1.
  • kernel: matmul_full_loadA.hpp
  • blockMmad: block_mmad_pingpong_full_loadA.hpp
  • dispatchPolicy: <idp:inline displayname="code" id="code413194620281">MmadAtlasA2FullLoadA</idp:inline>
  • BlockScheduler: <idp:inline displayname="code" id="code13616124752811">GemmIdentityBlockSwizzleL1FullLoad</idp:inline>
• The single-core split-K template (34_single_core_splitk_matmul) considers the optimization of L1Tile block residency in its theoretical design.
  • kernel: single_core_slicek_matmul.hpp
  • blockMmad: block_mmad_single_core_splitk.hpp
  • dispatchPolicy: <idp:inline displayname="code" id="code94861649152910">MmadAtlasA2SingleCoreSplitk</idp:inline>
  • BlockScheduler: <idp:inline displayname="code" id="code17850135052918">SingleCoreSplitkGemmIdentityBlockSwizzle</idp:inline>

Problem Analysis

In small-shape scenarios, for example, in the Common template:

• The number of cores in the $M$ and $N$ directions is less than the actual number of physical cores. Each AIC physical core processes a maximum of one basic task block.
• $k_1$ >= $K$. No partitioning is needed along the $K$ direction when moving from GM to L1.

In this case, the total kernel execution time is relatively small, and scalar overhead has a significant impact on performance.

Optimization Solution

Reduce redundant scalar computations.

• Do not use BlockScheduler inside the kernel to assign task blocks to physical cores. Manually calculate the task block corresponding to each physical core.
• Eliminate the basic block loop inside the kernel (each AIC processes only one task block).
• Simplify offset-related calculations inside the kernel.
• Eliminate the $m$ and $n$ loops of L1A/L1B inside blockMmad.
• Simplify offset-related calculations inside blockMmad.

Code Location for the Feature

• For details, see 31_small_matmul. You can compare it with the deliverables of 00_basic_matmul to deepen your understanding.
  • kernel: small_matmul.hpp
  • blockMmad: block_mmad_small.hpp

Problem Analysis

When data write is the main pipeline, optimizing the write bandwidth can yield performance gains.

• When dstStride is not 512-byte aligned during write, the bandwidth decreases significantly.
• Writing with NZ2ND with channel conversion incurs bandwidth loss.

Optimization Solution

For the above scenarios, an AIV can be used to rearrange the data format. When the rearrangement overhead is lower than the bandwidth loss, there is a performance gain. Four rearrangement methods are provided below.

(↑) Method 1: Use a local workspace. Write ND to GM with 512-byte alignment. Then rearrange data in UB at block granularity before writing it back to GM.

(↑) Method 2: Use a full workspace. Write ND to GM with 512-byte alignment. After the entire result is written, start rearranging data in UB and write it to GM.

(↑) Method 3: Use a local workspace. Write NZ to GM with 512-byte alignment. Then rearrange data in UB at block granularity before writing ND back to GM.

(↑) Method 4: Use a full workspace. Write NZ to GM with 512-byte alignment. After the entire result is written, start rearranging data in UB and write ND back to GM.

Code Location for the Feature

• padding_matmul.hpp implements RemovePaddingNDAndCast epilogue component that includes Method 2.
• For details about actual adaptation, see 34_single_core_splitk_matmul.

First, try to tune TileShape based on 00_basic_matmul and obtain a performance baseline. For details, see Template Library Optimization Guide.

Then, identify whether the scenario fits each template and compare with the performance baseline:

• 31_small_matmul:

  • Calculate the number of current basic task blocks taskBlocks.

    taskBlocks = CeilDiv(M, m1) * CeilDiv(N, n1);
    

  • The number of basic task blocks is less than the number of AIC cores: $taskBlocks<aicCoreNum$

  • The $K$ axis is small: $K<=k_1$

• 09_splitk_matmul or 22_padding_splitk_matmul (with padding prologue)

  • Select $m_1$, $n_1$, and $k_1$.

    • Set $m_1=128$, $n_1=256$, and $k_1=256$.
    • If either of the following conditions is met, change to $m_1=256$ and $n_1=128$.
      • Both matrices A and B are column-major.
      • Matrix A is column-major, matrix B is row-major, and $M>N$.

  • Calculate the number of current basic task blocks taskBlocks.

    taskBlocks = CeilDiv(M, m1) * CeilDiv(N, n1);
    

  • The following two scenarios are met:

    • The number of basic task blocks is less than half of the number of AIC cores, and the $K$ axis is sufficiently large: $taskBlocks<aicCoreNum/2,K>5120$
    • The number of basic task blocks is less than 3, and the $K$ axis is not small: $taskBlocks<=2,K>1024$
• 06_optimized_matmul (with padding prologue) and 21_basic_matmul_preload_zN (manually change to ND input)
  • These feature stronger generalization and are suitable for the remaining scenarios.
  • If padding is not required, you are advised to use the 21_basic_matmul_preload_zN template to reduce the overhead of MIX operator compilation and launch.

⚠️ The applicable scenarios for the full-load feature in 25_matmul_full_loadA and the single-core split-K solution in 34_single_core_splitk_matmul are yet to refine.

Consider padding prologue when Stride is not 512-byte aligned. However, the overhead introduced by padding and the MIX operator compilation and launch overhead must be considered (the small-shape method in 31_small_matmul is not recommended for additional padding adaptation).

The applicable scenarios for PaddingMatrixND, PaddingMatrixBlockND, and PaddingMatrixNZ are yet to refine. In terms of generalization, PaddingMatrixNZ has more advantages.