Merged
docs(quant): 规范量化矩阵乘与参数转换算子的中文文档 #13862
panzhijaAdd on 15 days ago
docs(quant): 规范量化矩阵乘与参数转换算子的中文文档 #13862
Merged
Merged fromsmdbha/ops-nnmasterintocann/ops-nnmaster
Total: 48 files changed+6527-6708
| @@ -3,134 +3,75 @@ | |||
| 3 | ## 产品支持情况 | 3 | ## 产品支持情况 |
| 4 | 4 | ||
| 5 | | 产品 | 是否支持 | | 5 | | 产品 | 是否支持 | |
| 6 | -| ---- | :----:| | 6 | +| --- | :---: | |
| 7 | -| Ascend 950PR&950DT系列产品 | √ | | 7 | +| <term>Ascend 950PR&950DT系列产品</term> | √ | |
| 8 | -| Atlas A3系列产品 | × | | 8 | +| <term>Atlas A3系列产品</term> | × | |
| 9 | -| Atlas A2系列产品 | × | | 9 | +| <term>Atlas A2系列产品</term> | × | |
| 10 | -| Atlas 200I/500 A2推理产品 | × | | 10 | +| <term>Atlas 200I/500 A2推理产品</term> | × | |
| 11 | -| Atlas推理系列产品 | × | | 11 | +| <term>Atlas推理系列产品</term> | × | |
| 12 | -| Atlas训练系列产品 | × | | 12 | +| <term>Atlas训练系列产品</term> | × | |
| 13 | 13 | ||
| 14 | ## 功能说明 | 14 | ## 功能说明 |
| 15 | 15 | ||
| 16 | -- 算子功能:在micro-batch训练场景中,需要做micro-batch的梯度累积,会存在大量QuantBatchMatmul后接InplaceAdd的融合场景。QuantBatchMatmulInplaceAdd算子将上述算子融合起来,提高网络性能。实现量化矩阵乘计算和加法计算,基本功能为矩阵乘和加法的组合。 | 16 | +- 算子功能:融合量化矩阵乘和原地加法,将矩阵乘结果累加到已有的y中,适用于微批次(micro-batch)训练中的梯度累积。 |
| 17 | 17 | ||
| 18 | -- 计算公式: | 18 | +- 计算公式:支持MX(Microscaling,微缩放)量化和HIFLOAT8的T-T(pertensor-pertensor,逐张量对逐张量)量化,模式说明见[量化模式介绍](../../docs/zh/context/quant_mode_introduction.md)。 |
| 19 | 19 | ||
| 20 | - - **MX量化:** | 20 | + 以下公式中的$X_1$、$X_2$表示按转置属性解释后的矩阵,形状分别为$(M,K)$、$(K,N)$。M、N为结果矩阵的行数、列数,K为归约维度;$Y_{\mathrm{in}}$、$Y_{\mathrm{out}}$分别表示原地累加前后的y。 |
| 21 | + | ||
| 22 | + - MX量化: | ||
| 21 | 23 | ||
| 22 | $$ | 24 | $$ |
| 23 | - y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n] | 25 | + Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+ |
| 26 | + \sum_{j=0}^{L-1}\left[ | ||
| 27 | + \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right) | ||
| 28 | + S_{1,m,j}S_{2,j,n}\right] | ||
| 24 | $$ | 29 | $$ |
| 25 | 30 | ||
| 26 | - 其中,$gsK$ 代表K轴的量化的block size即32,$x1Slice$ 代表 $x1$ 第m行长度为 $gsK$ 的向量,$x2Slice$ 代表 $x2$ 第n列长度为 $gsK$ 的向量,K轴均从 $j*gsK$ 起始切片,j的取值范围 [0, kLoops),kLoops=ceil($K_i$ / $gsK$),支持最后的切片长度不足 $gsK$。 | 31 | + 其中,$g_K=32$,$L=\lceil K/g_K\rceil$,$g_j=\min(g_K,K-jg_K)$。$S_1$、$S_2$为对应K分组的输入scale,最后一组可以不足32个元素。 |
| 27 | 32 | ||
| 28 | - - **HIFLOAT8 T-T量化:** | 33 | + - HIFLOAT8的T-T量化: |
| 29 | 34 | ||
| 30 | $$ | 35 | $$ |
| 31 | - y[m,n] = (\sum_{k=0}^{K_i-1} (x1[m,k] * x2[k,n])) * (scale1 * scale2) + y[m,n] | 36 | + Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+ |
| 37 | + \left(\sum_{k=0}^{K-1}X_{1,m,k}X_{2,k,n}\right)S_1S_2 | ||
| 32 | $$ | 38 | $$ |
| 33 | 39 | ||
| 34 | - 其中,$scale1$ 和 $scale2$ 分别对应 $x1Scale$ 和 $x2Scale$,均为标量(shape为 $(1)$);当设置`transposeX1`/`transposeX2`时,公式中的 $x1$ 和 $x2$ 按转置后的视图参与计算。 | 40 | + 其中,$S_1$、$S_2$分别对应x1_scale、x2_scale,均为形状为$(1,)$的缩放因子。 |
| 35 | 41 | ||
| 36 | ## 参数说明 | 42 | ## 参数说明 |
| 37 | 43 | ||
| 38 | -<table class="tg"><thead> | 44 | +| 参数名 | 输入/输出/属性 | 描述 | 数据类型 | 数据格式 | |
| 39 | - <tr> | 45 | +| --- | --- | --- | --- | --- | |
| 40 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">参数名</span></th> | 46 | +| x1 | 输入 | 矩阵乘的左矩阵,对应公式中的X₁。 | FLOAT8_E4M3FN、FLOAT8_E5M2、HIFLOAT8、FLOAT4_E2M1 | ND | |
| 41 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">输入/输出/属性</span></th> | 47 | +| x2 | 输入 | 矩阵乘的右矩阵,对应公式中的X₂。 | FLOAT8_E4M3FN、FLOAT8_E5M2、HIFLOAT8、FLOAT4_E2M1 | ND | |
| 42 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">描述</span></th> | 48 | +| x2_scale | 输入 | x2的量化缩放因子,对应公式中的S₂。MX量化使用FLOAT8_E8M0,HIFLOAT8的T-T量化使用FLOAT32。 | FLOAT8_E8M0、FLOAT32 | ND | |
| 43 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据类型</span></th> | 49 | +| y | 输入 | 参与原地累加的已有结果,对应公式中的Y_in。 | FLOAT32 | ND | |
| 44 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据格式</span></th> | 50 | +| x1_scale | 可选输入 | x1的量化缩放因子,对应公式中的S₁。当前支持的MX和HIFLOAT8的T-T量化场景均须提供,数据类型与x2_scale一致。 | FLOAT8_E8M0、FLOAT32 | ND | |
| 45 | - </tr></thead> | 51 | +| y | 输出 | 原地累加后的结果,对应公式中的Y_out,与输入y共用存储。 | FLOAT32 | ND | |
| 46 | -<tbody> | 52 | +| transpose_x1 | 可选属性 | <ul><li>是否转置x1的最后两维:false表示不转置,true表示转置。</li><li>默认值为false;当前支持的计算场景须设为true。</li></ul> | BOOL | - | |
| 47 | - <tr> | 53 | +| transpose_x2 | 可选属性 | <ul><li>是否转置x2的最后两维:false表示不转置,true表示转置。</li><li>默认值为false;当前支持的计算场景须保持false。</li></ul> | BOOL | - | |
| 48 | - <td class="tg-22a9">x1</td> | 54 | +| group_size | 可选属性 | <ul><li>M、N、K方向量化分组大小的编码值,各分量占16位;支持的分组组合见<a href="#约束说明">约束说明</a>。</li><li>默认值为0。</li></ul> | INT64 | - | |
| 49 | - <td class="tg-22a9">输入</td> | ||
| 50 | - <td class="tg-22a9">矩阵乘运算中的左矩阵,公式中的输入x1。</td> | ||
| 51 | - <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td> | ||
| 52 | - <td class="tg-22a9">ND</td> | ||
| 53 | - </tr> | ||
| 54 | - <tr> | ||
| 55 | - <td class="tg-22a9">x2</td> | ||
| 56 | - <td class="tg-22a9">输入</td> | ||
| 57 | - <td class="tg-22a9">矩阵乘运算中的右矩阵,公式中的输入x2。</td> | ||
| 58 | - <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td> | ||
| 59 | - <td class="tg-22a9">ND</td> | ||
| 60 | - </tr> | ||
| 61 | - <tr> | ||
| 62 | - <td class="tg-22a9">x1Scale</td> | ||
| 63 | - <td class="tg-22a9">可选输入</td> | ||
| 64 | - <td class="tg-22a9">量化参数中由x1量化引入的缩放因子,对应公式的scale1。</td> | ||
| 65 | - <td class="tg-22a9">FLOAT8_E8M0(MX)、FLOAT32(HIFLOAT8 TT)</td> | ||
| 66 | - <td class="tg-22a9">ND</td> | ||
| 67 | - </tr> | ||
| 68 | - <tr> | ||
| 69 | - <td class="tg-22a9">x2Scale</td> | ||
| 70 | - <td class="tg-22a9">输入</td> | ||
| 71 | - <td class="tg-22a9">量化参数中由x2量化引入的缩放因子,对应公式的scale2。</td> | ||
| 72 | - <td class="tg-22a9">FLOAT8_E8M0(MX)、FLOAT32(HIFLOAT8 TT)</td> | ||
| 73 | - <td class="tg-22a9">ND</td> | ||
| 74 | - </tr> | ||
| 75 | - <tr> | ||
| 76 | - <td class="tg-22a9">yRef</td> | ||
| 77 | - <td class="tg-22a9">输入输出</td> | ||
| 78 | - <td class="tg-22a9">矩阵乘运算与加法累加后的结果,对应公式中的输入输出y。</td> | ||
| 79 | - <td class="tg-22a9">FLOAT32</td> | ||
| 80 | - <td class="tg-22a9">ND</td> | ||
| 81 | - </tr> | ||
| 82 | - <tr> | ||
| 83 | - <td class="tg-22a9">transposeX1</td> | ||
| 84 | - <td class="tg-22a9">输入</td> | ||
| 85 | - <td class="tg-22a9">表示x1的输入shape是否转置。</td> | ||
| 86 | - <td class="tg-22a9">bool</td> | ||
| 87 | - <td class="tg-22a9">-</td> | ||
| 88 | - </tr> | ||
| 89 | - <tr> | ||
| 90 | - <td class="tg-22a9">transposeX2</td> | ||
| 91 | - <td class="tg-22a9">输入</td> | ||
| 92 | - <td class="tg-22a9">表示x2的输入shape是否转置。</td> | ||
| 93 | - <td class="tg-22a9">bool</td> | ||
| 94 | - <td class="tg-22a9">-</td> | ||
| 95 | - </tr> | ||
| 96 | - <tr> | ||
| 97 | - <td class="tg-22a9">groupSize</td> | ||
| 98 | - <td class="tg-22a9">输入</td> | ||
| 99 | - <td class="tg-22a9">整数型参数,用于输入m、n、k方向上的量化分组大小。</td> | ||
| 100 | - <td class="tg-22a9">INT64</td> | ||
| 101 | - <td class="tg-22a9">-</td> | ||
| 102 | - </tr> | ||
| 103 | -</tbody></table> | ||
| 104 | - | ||
| 105 | -- <term>Ascend 950PR&950DT系列产品</term>: | ||
| 106 | - - x1、x2支持FLOAT8_E4M3FN、FLOAT8_E5M2、HIFLOAT8、FLOAT4_E2M1数据类型。 | ||
| 107 | - - x1Scale、x2Scale支持FLOAT8_E8M0(MX)和FLOAT32(HIFLOAT8 TT)数据类型。 | ||
| 108 | - - yRef只支持FLOAT32数据类型。 | ||
| 109 | - - 当前仅支持transposeX1=true且transposeX2=false。 | ||
| 110 | - - groupSize由groupSizeM、groupSizeN、groupSizeK拼接组成:groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32。 | ||
| 111 | 55 | ||
| 112 | ## 约束说明 | 56 | ## 约束说明 |
| 113 | 57 | ||
| 114 | -- 支持连续tensor,[非连续tensor](../../docs/zh/context/non_contiguous_tensor.md)支持转置场景。 | 58 | +- 当前仅支持transpose_x1=true且transpose_x2=false。输入x1、x2的逻辑形状分别为`(K, M)`、`(K, N)`,y为`(M, N)`。 |
| 115 | -- groupSize相关约束:传入的groupSize内部会按公式分解得到groupSizeM、groupSizeN、groupSizeK,当其中有1个或多个为0时,会根据x1/x2/x1Scale/x2Scale输入shape重新推断。 | 59 | +- group_size按`group_size_k | (group_size_n << 16) | (group_size_m << 32)`编码。分量为0时由对应输入及scale的形状推导;推导后须满足对应量化模式的分组要求。 |
| 116 | -- 动态量化(MX量化)场景约束: | 60 | +- MX量化: |
| 117 | - - 输入和输出支持以下数据类型组合:x1、x2为FLOAT8_E5M2/FLOAT8_E4M3FN,x1Scale、x2Scale为FLOAT8_E8M0,yRef为FLOAT32。 | 61 | + - x1、x2可分别为FLOAT8_E4M3FN或FLOAT8_E5M2,也可同时为FLOAT4_E2M1;不支持FP4与FP8混合输入,不支持FLOAT4_E1M2。 |
| 118 | - - 输入和输出还支持以下数据类型组合:x1、x2均为FLOAT4_E2M1,x1Scale、x2Scale均为FLOAT8_E8M0,yRef为FLOAT32。不支持FLOAT4_E1M2输入,不支持FP4与FP8混合输入。 | 62 | + - x1_scale、x2_scale均为FLOAT8_E8M0,形状分别为`(ceil(K/64), M, 2)`、`(ceil(K/64), N, 2)`,其中ceil表示向上取整。 |
| 119 | - - x1 shape为(k, m),x2 shape为(k, n),x1Scale shape为(ceil(k/64), m, 2),x2Scale shape为(ceil(k/64), n, 2),yRef shape为(m, n),[gsM, gsN, gsK] 为 [1, 1, 32],groupSize为32。 | 63 | + - 最终分组大小为`[group_size_m, group_size_n, group_size_k]=[1, 1, 32]`。可显式设置group_size=4295032864;设置为32时,M、N方向的分组大小由形状推导。 |
| 120 | - - MXFP4量化场景额外约束: | 64 | + - MXFP4场景下,M、N须为非负偶数。x1、x2采用4位打包存储,每字节保存两个FLOAT4_E2M1元素;连续输入对应的字节存储形状分别为`(K, M/2)`、`(K, N/2)`。 |
| 121 | - - m、n必须为非负偶数。 | 65 | + - MXFP4场景下,当M、N、K均大于0时,还须同时满足$M\times K>2$、$N\times K>2$。 |
| 122 | - - 上述shape以逻辑元素数表示。x1、x2采用4-bit打包存储,一个字节保存两个FLOAT4_E2M1编码。连续输入x1、x2的逻辑shape分别为(k, m)、(k, n),对应的字节存储shape分别为(k, m/2)、(k, n/2)。 | 66 | +- HIFLOAT8的T-T量化: |
| 123 | - - 当m、n、k均大于0时,必须同时满足m × k > 2、n × k > 2。 | 67 | + - x1、x2均为HIFLOAT8,x1_scale、x2_scale均为FLOAT32且形状为`(1,)`。 |
| 124 | -- HIFLOAT8 T-T场景约束: | 68 | + - group_size须为0。 |
| 125 | - - 输入和输出支持以下数据类型组合:x1、x2为HIFLOAT8,x1Scale、x2Scale为FLOAT32,yRef为FLOAT32。 | ||
| 126 | - - x1 shape为(k, m),x2 shape为(k, n),yRef shape为(m, n)。 | ||
| 127 | - - x1Scale shape为(1),x2Scale shape为(1),groupSize必须为0。 | ||
| 128 | - - 不支持bias、offset参数。 | ||
| 129 | 69 | ||
| 130 | ## 调用说明 | 70 | ## 调用说明 |
| 131 | 71 | ||
| 132 | | 调用方式 | 样例代码 | 说明 | | 72 | | 调用方式 | 样例代码 | 说明 | |
| 133 | -| ---------------- | --------------------------- | --------------------------------------------------- | | 73 | +| --- | --- | --- | |
| 134 | -| aclnn接口 | [test_aclnn_quant_batch_matmul_inplace_add_mxfp8](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | 展示mxFP8量化场景的调用方式,其中`x1/x2`为FLOAT8、`x1Scale/x2Scale`为FLOAT8_E8M0,`groupSize=32`。 | | 74 | +| aclnn API | [MXFP8样例](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | 通过[aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd.md)调用,展示MXFP8量化。 | |
| 135 | -| aclnn接口 | <a href="examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp">test_aclnn_quant_batch_matmul_inplace_add_mxfp4</a> | 展示MXFP4量化场景的调用方式,其中x1、x2为打包存储的FLOAT4_E2M1,x1Scale、x2Scale为FLOAT8_E8M0,yRef为FLOAT32,groupSize=32。 | | 75 | +| aclnn API | [MXFP4样例](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp) | 通过[aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd.md)调用,展示打包存储的FLOAT4_E2M1输入。 | |
| 136 | -| aclnn接口 | [test_aclnn_quant_batch_matmul_inplace_add_TT](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | 展示当前支持组合`transposeX1=true`、`transposeX2=false`下HIFLOAT8 TT场景的调用方式。 | | 76 | +| aclnn API | [HIFLOAT8的T-T样例](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | 通过[aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd.md)调用,展示逐张量量化。 | |
| 77 | +| GE图模式 | - | 通过[QuantBatchMatmulInplaceAdd算子IR](op_graph/quant_batch_matmul_inplace_add_proto.h)构图调用。 | | ||
| @@ -5,134 +5,75 @@ English | [简体中文](./README.md) | |||
| 5 | ## Applicable Products | 5 | ## Applicable Products |
| 6 | 6 | ||
| 7 | | Product | Support | | 7 | | Product | Support | |
| 8 | -| ---- | :----:| | 8 | +| --- | :---: | |
| 9 | -| Ascend 950PR&950DT products | √ | | 9 | +| <term>Ascend 950PR&950DT products</term> | √ | |
| 10 | -| Atlas A3 products | × | | 10 | +| <term>Atlas A3 products</term> | × | |
| 11 | -| Atlas A2 products | × | | 11 | +| <term>Atlas A2 products</term> | × | |
| 12 | -| Atlas 200I/500 A2 inference products | × | | 12 | +| <term>Atlas 200I/500 A2 inference products</term> | × | |
| 13 | -| Atlas inference products | × | | 13 | +| <term>Atlas inference products</term> | × | |
| 14 | -| Atlas training products | × | | 14 | +| <term>Atlas training products</term> | × | |
| 15 | 15 | ||
| 16 | ## Function | 16 | ## Function |
| 17 | 17 | ||
| 18 | -- Operator function: in the micro-batch training scenario, gradient accumulation of micro-batches is required, and a large number of fusion scenarios where QuantBatchMatmul is followed by InplaceAdd exist. The QuantBatchMatmulInplaceAdd operator fuses the preceding operators to improve network performance. It implements quantized matrix multiplication and addition, and its basic function is a combination of matrix multiplication and addition. | 18 | +- Operator function: Fuses quantized matrix multiplication and in-place addition, accumulating the matrix product into an existing y. This supports gradient accumulation during micro-batch training. |
| 19 | 19 | ||
| 20 | -- Formula: | 20 | +- Formulas: Supports MX (Microscaling) quantization and HIFLOAT8 T-T (per-tensor–per-tensor) quantization. See [Introduction to Quantization Modes](../../docs/en/context/quant_mode_introduction.md). |
| 21 | 21 | ||
| 22 | - - **MX quantization:** | 22 | + In the following formulas, $X_1$ and $X_2$ are the matrices interpreted according to the transpose attributes, with shapes $(M,K)$ and $(K,N)$. M and N are the result matrix dimensions, and K is the reduction dimension. $Y_{\mathrm{in}}$ and $Y_{\mathrm{out}}$ denote y before and after in-place accumulation. |
| 23 | + | ||
| 24 | + - MX quantization: | ||
| 23 | 25 | ||
| 24 | $$ | 26 | $$ |
| 25 | - y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n] | 27 | + Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+ |
| 28 | + \sum_{j=0}^{L-1}\left[ | ||
| 29 | + \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right) | ||
| 30 | + S_{1,m,j}S_{2,j,n}\right] | ||
| 26 | $$ | 31 | $$ |
| 27 | 32 | ||
| 28 | - where $gsK$ denotes the quantization block size of the K axis, that is, 32; $x1Slice$ denotes the vector of length $gsK$ in row m of $x1$; $x2Slice$ denotes the vector of length $gsK$ in column n of $x2$. Slicing on the K axis starts from $j*gsK$, the value range of j is [0, kLoops), kLoops=ceil($K_i$ / $gsK$), and the case where the length of the last slice is less than $gsK$ is supported. | 33 | + Here, $g_K=32$, $L=\lceil K/g_K\rceil$, and $g_j=\min(g_K,K-jg_K)$. $S_1$ and $S_2$ are the input scales for the corresponding K group. The last group may contain fewer than 32 elements. |
| 29 | 34 | ||
| 30 | - - **HIFLOAT8 T-T quantization:** | 35 | + - HIFLOAT8 T-T quantization: |
| 31 | 36 | ||
| 32 | $$ | 37 | $$ |
| 33 | - y[m,n] = (\sum_{k=0}^{K_i-1} (x1[m,k] * x2[k,n])) * (scale1 * scale2) + y[m,n] | 38 | + Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+ |
| 39 | + \left(\sum_{k=0}^{K-1}X_{1,m,k}X_{2,k,n}\right)S_1S_2 | ||
| 34 | $$ | 40 | $$ |
| 35 | 41 | ||
| 36 | - where $scale1$ and $scale2$ correspond to $x1Scale$ and $x2Scale$ respectively, both being scalars (shape $(1)$). When `transposeX1`/`transposeX2` is set, the $x1$ and $x2$ in the formula participate in the computation with the transposed view. | 42 | + Here, $S_1$ and $S_2$ correspond to x1_scale and x2_scale, respectively, and both have shape $(1,)$. |
| 37 | 43 | ||
| 38 | ## Parameters | 44 | ## Parameters |
| 39 | 45 | ||
| 40 | -<table class="tg"><thead> | 46 | +| Parameter | Input/Output/Attribute | Description | Data Type | Data Format | |
| 41 | - <tr> | 47 | +| --- | --- | --- | --- | --- | |
| 42 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Parameter name</span></th> | 48 | +| x1 | Input | Left matrix, corresponding to X₁ in the formulas. | FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1 | ND | |
| 43 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Input/Output/Attribute</span></th> | 49 | +| x2 | Input | Right matrix, corresponding to X₂ in the formulas. | FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1 | ND | |
| 44 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Description</span></th> | 50 | +| x2_scale | Input | Quantization scale for x2, corresponding to S₂ in the formulas. Uses FLOAT8_E8M0 for MX quantization and FLOAT32 for HIFLOAT8 T-T quantization. | FLOAT8_E8M0, FLOAT32 | ND | |
| 45 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data type</span></th> | 51 | +| y | Input | Existing result used for in-place accumulation, corresponding to Y_in in the formulas. | FLOAT32 | ND | |
| 46 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data format</span></th> | 52 | +| x1_scale | Optional input | Quantization scale for x1, corresponding to S₁ in the formulas. Required for both currently supported modes, MX and HIFLOAT8 T-T. Its data type must match x2_scale. | FLOAT8_E8M0, FLOAT32 | ND | |
| 47 | - </tr></thead> | 53 | +| y | Output | Result after in-place accumulation, corresponding to Y_out in the formulas. Shares storage with input y. | FLOAT32 | ND | |
| 48 | -<tbody> | 54 | +| transpose_x1 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x1: false disables transposition and true enables it.</li><li>Defaults to false; must be true for currently supported computations.</li></ul> | BOOL | - | |
| 49 | - <tr> | 55 | +| transpose_x2 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x2: false disables transposition and true enables it.</li><li>Defaults to false; must remain false for currently supported computations.</li></ul> | BOOL | - | |
| 50 | - <td class="tg-22a9">x1</td> | 56 | +| group_size | Optional attribute | <ul><li>Encodes quantization group sizes along M, N, and K, with 16 bits per component. For supported combinations, see <a href="#constraints">Constraints</a>.</li><li>Defaults to 0.</li></ul> | INT64 | - | |
| 51 | - <td class="tg-22a9">Input</td> | ||
| 52 | - <td class="tg-22a9">Left matrix in the matrix multiplication, corresponding to input x1 in the formula.</td> | ||
| 53 | - <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td> | ||
| 54 | - <td class="tg-22a9">ND</td> | ||
| 55 | - </tr> | ||
| 56 | - <tr> | ||
| 57 | - <td class="tg-22a9">x2</td> | ||
| 58 | - <td class="tg-22a9">Input</td> | ||
| 59 | - <td class="tg-22a9">Right matrix in the matrix multiplication, corresponding to input x2 in the formula.</td> | ||
| 60 | - <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td> | ||
| 61 | - <td class="tg-22a9">ND</td> | ||
| 62 | - </tr> | ||
| 63 | - <tr> | ||
| 64 | - <td class="tg-22a9">x1Scale</td> | ||
| 65 | - <td class="tg-22a9">Optional input</td> | ||
| 66 | - <td class="tg-22a9">Scaling factor introduced by the quantization of x1 in the quantization parameters, corresponding to scale1 in the formula.</td> | ||
| 67 | - <td class="tg-22a9">FLOAT8_E8M0 (MX), FLOAT32 (HIFLOAT8 TT)</td> | ||
| 68 | - <td class="tg-22a9">ND</td> | ||
| 69 | - </tr> | ||
| 70 | - <tr> | ||
| 71 | - <td class="tg-22a9">x2Scale</td> | ||
| 72 | - <td class="tg-22a9">Input</td> | ||
| 73 | - <td class="tg-22a9">Scaling factor introduced by the quantization of x2 in the quantization parameters, corresponding to scale2 in the formula.</td> | ||
| 74 | - <td class="tg-22a9">FLOAT8_E8M0 (MX), FLOAT32 (HIFLOAT8 TT)</td> | ||
| 75 | - <td class="tg-22a9">ND</td> | ||
| 76 | - </tr> | ||
| 77 | - <tr> | ||
| 78 | - <td class="tg-22a9">yRef</td> | ||
| 79 | - <td class="tg-22a9">Input and output</td> | ||
| 80 | - <td class="tg-22a9">Result after the matrix multiplication and the addition accumulation, corresponding to input and output y in the formula.</td> | ||
| 81 | - <td class="tg-22a9">FLOAT32</td> | ||
| 82 | - <td class="tg-22a9">ND</td> | ||
| 83 | - </tr> | ||
| 84 | - <tr> | ||
| 85 | - <td class="tg-22a9">transposeX1</td> | ||
| 86 | - <td class="tg-22a9">Input</td> | ||
| 87 | - <td class="tg-22a9">Indicates whether the input shape of x1 is transposed.</td> | ||
| 88 | - <td class="tg-22a9">bool</td> | ||
| 89 | - <td class="tg-22a9">-</td> | ||
| 90 | - </tr> | ||
| 91 | - <tr> | ||
| 92 | - <td class="tg-22a9">transposeX2</td> | ||
| 93 | - <td class="tg-22a9">Input</td> | ||
| 94 | - <td class="tg-22a9">Indicates whether the input shape of x2 is transposed.</td> | ||
| 95 | - <td class="tg-22a9">bool</td> | ||
| 96 | - <td class="tg-22a9">-</td> | ||
| 97 | - </tr> | ||
| 98 | - <tr> | ||
| 99 | - <td class="tg-22a9">groupSize</td> | ||
| 100 | - <td class="tg-22a9">Input</td> | ||
| 101 | - <td class="tg-22a9">Integer parameter used for the quantization group sizes along the m, n, and k directions of the input.</td> | ||
| 102 | - <td class="tg-22a9">INT64</td> | ||
| 103 | - <td class="tg-22a9">-</td> | ||
| 104 | - </tr> | ||
| 105 | -</tbody></table> | ||
| 106 | - | ||
| 107 | -- <term>Ascend 950PR&950DT products</term>: | ||
| 108 | - - x1 and x2 support the FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, and FLOAT4_E2M1 data types. | ||
| 109 | - - x1Scale and x2Scale support the FLOAT8_E8M0 (MX) and FLOAT32 (HIFLOAT8 TT) data types. | ||
| 110 | - - yRef supports only the FLOAT32 data type. | ||
| 111 | - - Currently only transposeX1=true and transposeX2=false are supported. | ||
| 112 | - - groupSize is composed by concatenating groupSizeM, groupSizeN, and groupSizeK: groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32. | ||
| 113 | 57 | ||
| 114 | ## Constraints | 58 | ## Constraints |
| 115 | 59 | ||
| 116 | -- Contiguous tensors are supported, and [non-contiguous tensors](../../docs/en/context/non_contiguous_tensor.md) support the transposition scenario. | 60 | +- Only transpose_x1=true and transpose_x2=false are currently supported. The logical input shapes of x1 and x2 are `(K, M)` and `(K, N)`, respectively, and y has shape `(M, N)`. |
| 117 | -- groupSize-related constraints: the passed-in groupSize is internally decomposed according to the formula to obtain groupSizeM, groupSizeN, and groupSizeK. When one or more of them is 0, the values are inferred again according to the input shapes of x1/x2/x1Scale/x2Scale. | 61 | +- group_size is encoded as `group_size_k | (group_size_n << 16) | (group_size_m << 32)`. A zero component is inferred from the corresponding input and scale shapes. The inferred sizes must satisfy the selected quantization mode. |
| 118 | -- Constraints of the dynamic quantization (MX quantization) scenario: | 62 | +- MX quantization: |
| 119 | - - The inputs and outputs support the following data type combinations: x1 and x2 are FLOAT8_E5M2/FLOAT8_E4M3FN, x1Scale and x2Scale are FLOAT8_E8M0, and yRef is FLOAT32. | 63 | + - x1 and x2 may each be FLOAT8_E4M3FN or FLOAT8_E5M2, or both may be FLOAT4_E2M1. Mixed FP4/FP8 inputs and FLOAT4_E1M2 are not supported. |
| 120 | - - The inputs and outputs also support the following data type combination: x1 and x2 are both FLOAT4_E2M1, x1Scale and x2Scale are both FLOAT8_E8M0, and yRef is FLOAT32. FLOAT4_E1M2 inputs and mixed FP4 and FP8 inputs are not supported. | 64 | + - x1_scale and x2_scale are FLOAT8_E8M0, with shapes `(ceil(K/64), M, 2)` and `(ceil(K/64), N, 2)`, respectively, where ceil rounds upward. |
| 121 | - - The shape of x1 is (k, m), the shape of x2 is (k, n), the shape of x1Scale is (ceil(k/64), m, 2), the shape of x2Scale is (ceil(k/64), n, 2), the shape of yRef is (m, n), [gsM, gsN, gsK] is [1, 1, 32], and groupSize is 32. | 65 | + - The resulting group sizes must be `[group_size_m, group_size_n, group_size_k]=[1, 1, 32]`. Set group_size=4295032864 explicitly, or set it to 32 to infer the M and N group sizes from the shapes. |
| 122 | - - Additional constraints for MXFP4 quantization: | 66 | + - For MXFP4, M and N must be nonnegative even numbers. x1 and x2 use packed 4-bit storage, with two FLOAT4_E2M1 elements per byte. For contiguous inputs, the byte-storage shapes are `(K, M/2)` and `(K, N/2)`, respectively. |
| 123 | - - m and n must be non-negative even numbers. | 67 | + - For MXFP4, when M, N, and K are all positive, both $M\times K>2$ and $N\times K>2$ must also hold. |
| 124 | - - The preceding shapes are expressed in logical element counts. x1 and x2 use 4-bit packed storage, with each byte storing two FLOAT4_E2M1 codes. For contiguous inputs, the logical shapes of x1 and x2 are (k, m) and (k, n), respectively, and the corresponding byte storage shapes are (k, m/2) and (k, n/2). | 68 | +- HIFLOAT8 T-T quantization: |
| 125 | - - When m, n, and k are all greater than 0, both m × k > 2 and n × k > 2 must hold. | 69 | + - x1 and x2 are HIFLOAT8. x1_scale and x2_scale are FLOAT32 and both have shape `(1,)`. |
| 126 | -- Constraints of the HIFLOAT8 T-T scenario: | 70 | + - group_size must be 0. |
| 127 | - - The inputs and outputs support the following data type combinations: x1 and x2 are HIFLOAT8, x1Scale and x2Scale are FLOAT32, and yRef is FLOAT32. | ||
| 128 | - - The shape of x1 is (k, m), the shape of x2 is (k, n), and the shape of yRef is (m, n). | ||
| 129 | - - The shape of x1Scale is (1), the shape of x2Scale is (1), and groupSize must be 0. | ||
| 130 | - - The bias and offset parameters are not supported. | ||
| 131 | 71 | ||
| 132 | ## Invocation Description | 72 | ## Invocation Description |
| 133 | 73 | ||
| 134 | | Invocation method | Sample code | Description | | 74 | | Invocation method | Sample code | Description | |
| 135 | -| ---------------- | --------------------------- | --------------------------------------------------- | | 75 | +| --- | --- | --- | |
| 136 | -| aclnn interface | [test_aclnn_quant_batch_matmul_inplace_add_mxfp8](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | Shows the invocation method of the mxFP8 quantization scenario, where `x1/x2` are FLOAT8, `x1Scale/x2Scale` are FLOAT8_E8M0, and `groupSize=32`. | | 76 | +| aclnn API | [MXFP8 example](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | Calls [aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd_en.md) to demonstrate MXFP8 quantization. | |
| 137 | -| aclnn interface | <a href="examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp">test_aclnn_quant_batch_matmul_inplace_add_mxfp4</a> | Shows the invocation method of the MXFP4 quantization scenario, where x1 and x2 are packed FLOAT4_E2M1, x1Scale and x2Scale are FLOAT8_E8M0, yRef is FLOAT32, and groupSize=32. | | 77 | +| aclnn API | [MXFP4 example](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp) | Calls [aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd_en.md) with packed FLOAT4_E2M1 inputs. | |
| 138 | -| aclnn interface | [test_aclnn_quant_batch_matmul_inplace_add_TT](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | Shows the invocation method of the HIFLOAT8 TT scenario under the currently supported combination `transposeX1=true` and `transposeX2=false`. | | 78 | +| aclnn API | [HIFLOAT8 T-T example](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | Calls [aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd_en.md) to demonstrate per-tensor quantization. | |
| 79 | +| GE graph mode | - | Builds a graph using the [QuantBatchMatmulInplaceAdd operator IR](op_graph/quant_batch_matmul_inplace_add_proto.h). | | ||
| @@ -37,13 +37,13 @@ | |||
| 37 | 37 | ||
| 38 | ### 保留Bitcast的转置融合 | 38 | ### 保留Bitcast的转置融合 |
| 39 | 39 | ||
| 40 | -对于“转置节点→`Bitcast`→目标算子”的结构,融合转置并保留`Bitcast`。`x1`和`x1_scale`可以分别包含一个`Bitcast`,不要求两路同时包含。`x2`和`x2_scale`支路保持不变。此模式的共享节点限制见使用约束。 | 40 | +对于“转置节点→`Bitcast`→目标算子”的结构,融合转置并保留`Bitcast`。`x1`和`x1_scale`可以分别包含一个`Bitcast`,不要求两路同时包含。`x2`和`x2_scale`支路保持不变。此模式的共享节点限制见[使用约束](#使用约束)。 |
| 41 | 41 | ||
| 42 |  | 42 |  |
| 43 | 43 | ||
| 44 | ### 保留Bitcast的x1_scale Reshape融合 | 44 | ### 保留Bitcast的x1_scale Reshape融合 |
| 45 | 45 | ||
| 46 | -`x1_scale`支路可以为“`Reshape`→`Bitcast`→目标算子”。当`Reshape`满足等价转置条件,且带`Bitcast`的模式被识别时,融合该`Reshape`并保留`Bitcast`。图中展示静态MX量化场景;动态shape组合的限制见使用约束。 | 46 | +`x1_scale`支路可以为“`Reshape`→`Bitcast`→目标算子”。当`Reshape`满足等价转置条件,且带`Bitcast`的模式被识别时,融合该`Reshape`并保留`Bitcast`。图中展示静态MX量化场景;动态shape组合的限制见[使用约束](#使用约束)。 |
| 47 | 47 | ||
| 48 |  | 48 |  |
| 49 | 49 | ||
| @@ -79,10 +79,10 @@ | |||
| 79 | - 含`Bitcast`的模式仅用于待融合转换节点不被其他计算支路共享的图;共享支路的数值正确性不在该模式的保证范围内。 | 79 | - 含`Bitcast`的模式仅用于待融合转换节点不被其他计算支路共享的图;共享支路的数值正确性不在该模式的保证范围内。 |
| 80 | - 带`Bitcast`的融合模式须通过目标平台的能力校验;不支持该模式的平台保持原图。此限制不影响不带`Bitcast`的模式。 | 80 | - 带`Bitcast`的融合模式须通过目标平台的能力校验;不支持该模式的平台保持原图。此限制不影响不带`Bitcast`的模式。 |
| 81 | - 每个目标节点根据自身`x1`、`x2`的shape独立识别动态shape,不受此前处理的节点或图影响。动态shape的“`Reshape`→`Bitcast`”组合仍须满足平台能力及其他融合条件。此处动态shape指含未知维度,与MX动态量化不是同一概念。 | 81 | - 每个目标节点根据自身`x1`、`x2`的shape独立识别动态shape,不受此前处理的节点或图影响。动态shape的“`Reshape`→`Bitcast`”组合仍须满足平台能力及其他融合条件。此处动态shape指含未知维度,与MX动态量化不是同一概念。 |
| 82 | -- 实际数据类型、量化模式、shape和转置属性须符合[QuantBatchMatmulInplaceAdd算子说明](../README.md)。该算子当前要求`transposeX1=true`、`transposeX2=false`;图示中的属性取反不扩展算子的支持范围。 | 82 | +- 实际数据类型、量化模式、shape和转置属性须符合[QuantBatchMatmulInplaceAdd算子说明](../README.md)。该算子当前要求`transpose_x1=true`、`transpose_x2=false`;图示中的属性取反不扩展算子的支持范围。 |
| 83 | 83 | ||
| 84 | ## 支持的型号 | 84 | ## 支持的型号 |
| 85 | 85 | ||
| 86 | <!-- npu="950" id1 --> | 86 | <!-- npu="950" id1 --> |
| 87 | -Ascend 950PR&950DT系列产品 | 87 | +- <term>Ascend 950PR&950DT系列产品</term> |
| 88 | <!-- end id1 --> | 88 | <!-- end id1 --> |
Amatmul/quant_batch_matmul_inplace_add/docs/QuantBatchMatmulInplaceAddTransposeFusionPass_en.md+308-0
| @@ -0,0 +1,308 @@ | |||
| 1 | +# QuantBatchMatmulInplaceAddTransposeFusionPass | ||
| 2 | + | ||
| 3 | +## Fusion Patterns | ||
| 4 | + | ||
| 5 | +This fusion rule folds a `Transpose` or `TransposeD` before the `x1` input of `QuantBatchMatmulInplaceAdd` into the operator and negates `transpose_x1`. When the conditions are met, a transpose or equivalent `Reshape` before `x1_scale` is also fused. An intermediate `Bitcast` is preserved. `x2`, `x2_scale`, and `transpose_x2` remain unchanged. | ||
| 6 | + | ||
| 7 | +Diagram conventions: | ||
| 8 | + | ||
| 9 | +- The inputs are `x1`, `x2`, `x1_scale`, `x2_scale`, and `yRef`. `yRef` is the accumulation input and corresponds to the operator input `y`. Output `y` preserves the in-place accumulation relationship. | ||
| 10 | +- `t1` and `t2` denote `transpose_x1` and `transpose_x2` before fusion, respectively; `!t1` denotes logical negation. This rule applies only when `t1=false` and `t2=false`. After fusion, `transpose_x1=true` and `transpose_x2=false`. | ||
| 11 | +- `M` and `N` are the numbers of rows and columns in the result matrix, and `K` is the reduction dimension of the matrix multiplication. Shapes are shown in square brackets. A singleton dimension has size 1. | ||
| 12 | +- Inputs specifying the transpose permutation and the target Reshape shape are omitted. The diagrams illustrate the transformation; the actual inputs and fused result must satisfy the operator specifications. | ||
| 13 | + | ||
| 14 | +### Fusing Only the x1 Transpose | ||
| 15 | + | ||
| 16 | +The `Transpose` or `TransposeD` before `x1` is folded into the target operator, negating `transpose_x1`. A qualifying transpose before `x1_scale` is fused at the same time. The connections of `x2` and `x2_scale` are preserved. | ||
| 17 | + | ||
| 18 | +```mermaid | ||
| 19 | +flowchart LR | ||
| 20 | + subgraph before["Before fusion"] | ||
| 21 | + direction TB | ||
| 22 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 23 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 24 | + before_x2["x2"] --> before_op | ||
| 25 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Transpose or TransposeD"] --> before_op | ||
| 26 | + before_x2_scale["x2_scale"] --> before_op | ||
| 27 | + before_yRef["yRef"] --> before_op | ||
| 28 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 29 | + end | ||
| 30 | + subgraph after["After fusion"] | ||
| 31 | + direction TB | ||
| 32 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 33 | + after_x1["x1"] --> after_op | ||
| 34 | + after_x2["x2"] --> after_op | ||
| 35 | + after_x1_scale["x1_scale"] --> after_op | ||
| 36 | + after_x2_scale["x2_scale"] --> after_op | ||
| 37 | + after_yRef["yRef"] --> after_op | ||
| 38 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 39 | + end | ||
| 40 | +``` | ||
| 41 | + | ||
| 42 | +### No Fusion When Only x2 Has a Transpose | ||
| 43 | + | ||
| 44 | +Transformations only on the `x2` or `x2_scale` branch do not trigger this rule. The graph remains unchanged. | ||
| 45 | + | ||
| 46 | +```mermaid | ||
| 47 | +flowchart LR | ||
| 48 | + subgraph before["Before fusion"] | ||
| 49 | + direction TB | ||
| 50 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 51 | + before_x1["x1"] --> before_op | ||
| 52 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 53 | + before_x1_scale["x1_scale"] --> before_op | ||
| 54 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Transpose or TransposeD"] --> before_op | ||
| 55 | + before_yRef["yRef"] --> before_op | ||
| 56 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 57 | + end | ||
| 58 | + subgraph after["Unchanged"] | ||
| 59 | + direction TB | ||
| 60 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 61 | + after_x1["x1"] --> after_op | ||
| 62 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 63 | + after_x1_scale["x1_scale"] --> after_op | ||
| 64 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Transpose or TransposeD"] --> after_op | ||
| 65 | + after_yRef["yRef"] --> after_op | ||
| 66 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 67 | + end | ||
| 68 | +``` | ||
| 69 | + | ||
| 70 | +### Fusing Only x1 When Both Sides Have Transposes | ||
| 71 | + | ||
| 72 | +When both data inputs have preceding transposes, only the transformations of `x1` and a qualifying `x1_scale` are fused. `Transpose`, `TransposeD`, `Reshape`, and `Bitcast` nodes before `x2` and `x2_scale`, and their connections, are preserved. | ||
| 73 | + | ||
| 74 | +```mermaid | ||
| 75 | +flowchart LR | ||
| 76 | + subgraph before["Before fusion"] | ||
| 77 | + direction TB | ||
| 78 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 79 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 80 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 81 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Transpose or TransposeD"] --> before_op | ||
| 82 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Transpose or TransposeD"] --> before_op | ||
| 83 | + before_yRef["yRef"] --> before_op | ||
| 84 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 85 | + end | ||
| 86 | + subgraph after["After fusion"] | ||
| 87 | + direction TB | ||
| 88 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 89 | + after_x1["x1"] --> after_op | ||
| 90 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 91 | + after_x1_scale["x1_scale"] --> after_op | ||
| 92 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Transpose or TransposeD"] --> after_op | ||
| 93 | + after_yRef["yRef"] --> after_op | ||
| 94 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 95 | + end | ||
| 96 | +``` | ||
| 97 | + | ||
| 98 | +### Fusing x1_scale Reshape in MX Quantization | ||
| 99 | + | ||
| 100 | +For static shapes, a `Reshape` of a three-dimensional FLOAT8_E8M0 `x1_scale` is treated as an equivalent transpose when it swaps the first two dimensions and at least one of those dimensions has size 1. For example, the shape changes from `[1, M, 2]` to `[M, 1, 2]`. This `Reshape` can be fused together with the `x1` transpose. A `Reshape` before `x2_scale` is preserved. | ||
| 101 | + | ||
| 102 | +```mermaid | ||
| 103 | +flowchart LR | ||
| 104 | + subgraph before["Before fusion"] | ||
| 105 | + direction TB | ||
| 106 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 107 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 108 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 109 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape"] --> before_op | ||
| 110 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_op | ||
| 111 | + before_yRef["yRef"] --> before_op | ||
| 112 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 113 | + end | ||
| 114 | + subgraph after["After fusion"] | ||
| 115 | + direction TB | ||
| 116 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 117 | + after_x1["x1"] --> after_op | ||
| 118 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 119 | + after_x1_scale["x1_scale"] --> after_op | ||
| 120 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_op | ||
| 121 | + after_yRef["yRef"] --> after_op | ||
| 122 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 123 | + end | ||
| 124 | +``` | ||
| 125 | + | ||
| 126 | +### Transpose Fusion Retaining Bitcast | ||
| 127 | + | ||
| 128 | +For a transpose → `Bitcast` → target operator sequence, the transpose is fused and `Bitcast` is preserved. `x1` and `x1_scale` may each have one `Bitcast`; both branches need not contain one. The `x2` and `x2_scale` branches remain unchanged. For restrictions on shared nodes, see [Usage Constraints](#constraints). | ||
| 129 | + | ||
| 130 | +```mermaid | ||
| 131 | +flowchart LR | ||
| 132 | + subgraph before["Before fusion"] | ||
| 133 | + direction TB | ||
| 134 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 135 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_x1_1["Bitcast"] --> before_op | ||
| 136 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_x2_1["Bitcast"] --> before_op | ||
| 137 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Transpose or TransposeD"] --> before_x1_scale_1["Bitcast"] --> before_op | ||
| 138 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Transpose or TransposeD"] --> before_x2_scale_1["Bitcast"] --> before_op | ||
| 139 | + before_yRef["yRef"] --> before_op | ||
| 140 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 141 | + end | ||
| 142 | + subgraph after["After fusion"] | ||
| 143 | + direction TB | ||
| 144 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 145 | + after_x1["x1"] --> after_x1_0["Bitcast"] --> after_op | ||
| 146 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_x2_1["Bitcast"] --> after_op | ||
| 147 | + after_x1_scale["x1_scale"] --> after_x1_scale_0["Bitcast"] --> after_op | ||
| 148 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Transpose or TransposeD"] --> after_x2_scale_1["Bitcast"] --> after_op | ||
| 149 | + after_yRef["yRef"] --> after_op | ||
| 150 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 151 | + end | ||
| 152 | +``` | ||
| 153 | + | ||
| 154 | +### Fusing x1_scale Reshape While Preserving Bitcast | ||
| 155 | + | ||
| 156 | +The `x1_scale` branch may have a `Reshape` → `Bitcast` → target operator sequence. If `Reshape` is equivalent to a transpose and the pattern containing `Bitcast` is recognized, `Reshape` is fused and `Bitcast` is preserved. The diagram shows static-shape MX quantization. For restrictions on dynamic-shape combinations, see [Usage Constraints](#constraints). | ||
| 157 | + | ||
| 158 | +```mermaid | ||
| 159 | +flowchart LR | ||
| 160 | + subgraph before["Before fusion"] | ||
| 161 | + direction TB | ||
| 162 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 163 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_x1_1["Bitcast"] --> before_op | ||
| 164 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_x2_1["Bitcast"] --> before_op | ||
| 165 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape"] --> before_x1_scale_1["Bitcast"] --> before_op | ||
| 166 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_x2_scale_1["Bitcast"] --> before_op | ||
| 167 | + before_yRef["yRef"] --> before_op | ||
| 168 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 169 | + end | ||
| 170 | + subgraph after["After fusion"] | ||
| 171 | + direction TB | ||
| 172 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 173 | + after_x1["x1"] --> after_x1_0["Bitcast"] --> after_op | ||
| 174 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_x2_1["Bitcast"] --> after_op | ||
| 175 | + after_x1_scale["x1_scale"] --> after_x1_scale_0["Bitcast"] --> after_op | ||
| 176 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_x2_scale_1["Bitcast"] --> after_op | ||
| 177 | + after_yRef["yRef"] --> after_op | ||
| 178 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 179 | + end | ||
| 180 | +``` | ||
| 181 | + | ||
| 182 | +### Preserving Connections for Single-Element Scales | ||
| 183 | + | ||
| 184 | +For HIFLOAT8 per-tensor quantization, for example, `x1_scale` and `x2_scale` are one-dimensional FLOAT32 tensors of shape `[1]`, each containing one quantization coefficient. If neither scale has a preceding transformation, their connections are preserved and only the `x1` transpose is fused. Here, `[1]` is distinct from the zero-dimensional tensor shape `[]`. | ||
| 185 | + | ||
| 186 | +```mermaid | ||
| 187 | +flowchart LR | ||
| 188 | + subgraph before["Before fusion"] | ||
| 189 | + direction TB | ||
| 190 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 191 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 192 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 193 | + before_x1_scale["x1_scale<br/>FLOAT32, shape [1]"] --> before_op | ||
| 194 | + before_x2_scale["x2_scale<br/>FLOAT32, shape [1]"] --> before_op | ||
| 195 | + before_yRef["yRef"] --> before_op | ||
| 196 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 197 | + end | ||
| 198 | + subgraph after["After fusion"] | ||
| 199 | + direction TB | ||
| 200 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 201 | + after_x1["x1"] --> after_op | ||
| 202 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 203 | + after_x1_scale["x1_scale<br/>FLOAT32, shape [1]"] --> after_op | ||
| 204 | + after_x2_scale["x2_scale<br/>FLOAT32, shape [1]"] --> after_op | ||
| 205 | + after_yRef["yRef"] --> after_op | ||
| 206 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 207 | + end | ||
| 208 | +``` | ||
| 209 | + | ||
| 210 | +### Fusing x1_scale Reshape with Dynamic Shapes | ||
| 211 | + | ||
| 212 | +When a data input shape contains an unknown dimension, a `Reshape` before `x1_scale` can be recognized without the singleton-dimension condition required for static shapes. The input graph must ensure that this `Reshape` is equivalent to the required transpose. The diagram contains no `Bitcast`. This describes a fusion pattern and does not imply support for arbitrary unknown input dimensions. | ||
| 213 | + | ||
| 214 | +```mermaid | ||
| 215 | +flowchart LR | ||
| 216 | + subgraph before["Before fusion"] | ||
| 217 | + direction TB | ||
| 218 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 219 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 220 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 221 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape (dynamic)"] --> before_op | ||
| 222 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape (dynamic)"] --> before_op | ||
| 223 | + before_yRef["yRef"] --> before_op | ||
| 224 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 225 | + end | ||
| 226 | + subgraph after["After fusion"] | ||
| 227 | + direction TB | ||
| 228 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 229 | + after_x1["x1"] --> after_op | ||
| 230 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 231 | + after_x1_scale["x1_scale"] --> after_op | ||
| 232 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape (dynamic)"] --> after_op | ||
| 233 | + after_yRef["yRef"] --> after_op | ||
| 234 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 235 | + end | ||
| 236 | +``` | ||
| 237 | + | ||
| 238 | +### Other Reshape Patterns with Singleton Dimensions | ||
| 239 | + | ||
| 240 | +For static shapes, a preceding `Reshape` can also be recognized when `x1_scale` has at least two dimensions and at least one of its last two dimensions has size 1, for example, a shape change from `[8, 1]` to `[1, 8]`. The input graph must ensure equivalence to a transpose. The quantization mode and parameter combination must still satisfy the operator specifications. | ||
| 241 | + | ||
| 242 | +```mermaid | ||
| 243 | +flowchart LR | ||
| 244 | + subgraph before["Before fusion"] | ||
| 245 | + direction TB | ||
| 246 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 247 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 248 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 249 | + before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape"] --> before_op | ||
| 250 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_op | ||
| 251 | + before_yRef["yRef"] --> before_op | ||
| 252 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 253 | + end | ||
| 254 | + subgraph after["After fusion"] | ||
| 255 | + direction TB | ||
| 256 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 257 | + after_x1["x1"] --> after_op | ||
| 258 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 259 | + after_x1_scale["x1_scale"] --> after_op | ||
| 260 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_op | ||
| 261 | + after_yRef["yRef"] --> after_op | ||
| 262 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 263 | + end | ||
| 264 | +``` | ||
| 265 | + | ||
| 266 | +### No Transformation Before x1_scale | ||
| 267 | + | ||
| 268 | +If `x1_scale` connects directly to the target operator, has no recognizable preceding transformation, or is an unconnected optional input, that branch remains unchanged and does not prevent fusion of the `x1` transpose. `x2_scale` must retain a valid input connection. | ||
| 269 | + | ||
| 270 | +```mermaid | ||
| 271 | +flowchart LR | ||
| 272 | + subgraph before["Before fusion"] | ||
| 273 | + direction TB | ||
| 274 | + before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"] | ||
| 275 | + before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op | ||
| 276 | + before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op | ||
| 277 | + before_x1_scale["x1_scale"] --> before_op | ||
| 278 | + before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_op | ||
| 279 | + before_yRef["yRef"] --> before_op | ||
| 280 | + before_op --> before_y["y (in-place accumulation)"] | ||
| 281 | + end | ||
| 282 | + subgraph after["After fusion"] | ||
| 283 | + direction TB | ||
| 284 | + after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"] | ||
| 285 | + after_x1["x1"] --> after_op | ||
| 286 | + after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op | ||
| 287 | + after_x1_scale["x1_scale"] --> after_op | ||
| 288 | + after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_op | ||
| 289 | + after_yRef["yRef"] --> after_op | ||
| 290 | + after_op --> after_y["y (in-place accumulation)"] | ||
| 291 | + end | ||
| 292 | +``` | ||
| 293 | + | ||
| 294 | +## Constraints | ||
| 295 | + | ||
| 296 | +- Before fusion, `transpose_x1=false` and `transpose_x2=false` are required so that the resulting attributes meet the backend requirements. Other attribute combinations leave the graph unchanged. | ||
| 297 | +- A recognizable `Transpose` or `TransposeD` must precede `x1`, connecting directly to the target operator or through one `Bitcast`. A `Reshape` alone or a transpose only on the `x2` branch cannot trigger this rule. | ||
| 298 | +- Data transposes must match matrix-multiplication transpose semantics, and scale transformations must preserve the correspondence between quantization coefficients and data. Arbitrary dimension permutations or Reshape operations are not equivalent transposes. | ||
| 299 | +- Patterns containing `Bitcast` apply only when the transformation being fused is not shared with another computation branch. Numerical correctness on shared branches is not guaranteed by these patterns. | ||
| 300 | +- Patterns containing `Bitcast` must pass the target platform's capability check. Unsupported platforms leave the graph unchanged. This restriction does not apply to patterns without `Bitcast`. | ||
| 301 | +- Each target node determines whether shapes are dynamic from its own `x1` and `x2` shapes, independently of previously processed nodes or graphs. A dynamic-shape `Reshape` → `Bitcast` combination must still satisfy platform capability and other fusion requirements. Dynamic shape here means a shape with unknown dimensions and is distinct from MX dynamic quantization. | ||
| 302 | +- Data types, quantization modes, shapes, and transpose attributes must satisfy the [QuantBatchMatmulInplaceAdd specifications](../README_en.md). The operator currently requires `transpose_x1=true` and `transpose_x2=false`. Negating the attribute in a diagram does not extend operator support. | ||
| 303 | + | ||
| 304 | +## Applicable Products | ||
| 305 | + | ||
| 306 | +<!-- npu="950" id1 --> | ||
| 307 | +- <term>Ascend 950PR&950DT products</term> | ||
| 308 | +<!-- end id1 --> | ||
| @@ -27,21 +27,25 @@ | |||
| 27 | 27 | ||
| 28 | - 计算公式: | 28 | - 计算公式: |
| 29 | 29 | ||
| 30 | - - **MX量化:** | 30 | + $X_1$、$X_2$表示应用转置属性后的矩阵,shape分别为$(M,K)$、$(K,N)$。$Y_{\mathrm{in}}$为调用前的`yRef`,$Y_{\mathrm{out}}$为原地更新后的结果;两者的shape均为$(M,N)$。 |
| 31 | + | ||
| 32 | + - MX(Microscaling,微缩放)量化: | ||
| 31 | 33 | ||
| 32 | $$ | 34 | $$ |
| 33 | - y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n] | 35 | + Y_{\mathrm{out}}[m,n]=Y_{\mathrm{in}}[m,n]+\sum_{j=0}^{L-1} |
| 36 | + \left(\sum_{t=0}^{g_j-1}X_1[m,jg_K+t]X_2[jg_K+t,n]\right)S_1[m,j]S_2[j,n] | ||
| 34 | $$ | 37 | $$ |
| 35 | 38 | ||
| 36 | - 其中,$gsK$ 代表K轴的量化的block size即32,$x1Slice$代表$x1$第m行长度为 $gsK$ 的向量,$x2Slice$代表$x2$第n列长度为 $gsK$ 的向量,K轴均从$j*gsK$起始切片,j的取值范围[0, kLoops), kLoops=ceil($K_i$ / $gsK$),支持最后的切片长度不足 $gsK$。 | 39 | + 其中,$g_K=32$,$L=\lceil K/g_K\rceil$,$g_j=\min(g_K,K-jg_K)$。$S_1$、$S_2$分别为`x1ScaleOptional`、`x2Scale`中对应分组的scale,存储shape见[约束说明](#约束说明)。 |
| 37 | 40 | ||
| 38 | - - **HIFLOAT8 T-T量化:** | 41 | + - HIFLOAT8 T-T量化: |
| 39 | 42 | ||
| 40 | $$ | 43 | $$ |
| 41 | - y[m,n] = (\sum_{k=0}^{K_i-1} (x1[m,k] * x2[k,n])) * (scale1 * scale2) + y[m,n] | 44 | + Y_{\mathrm{out}}[m,n]=Y_{\mathrm{in}}[m,n]+ |
| 45 | + \left(\sum_{k=0}^{K-1}X_1[m,k]X_2[k,n]\right)s_1s_2 | ||
| 42 | $$ | 46 | $$ |
| 43 | 47 | ||
| 44 | - 其中,$scale1$和$scale2$分别对应$x1Scale$和$x2Scale$,均为标量(shape为 $(1)$);当设置transposeX1/transposeX2时,公式中的$x1$和$x2$按转置后的视图参与计算。 | 48 | + $s_1$、$s_2$分别对应`x1ScaleOptional`、`x2Scale`,shape均为$(1,)$。 |
| 45 | 49 | ||
| 46 | ## 函数原型 | 50 | ## 函数原型 |
| 47 | 51 | ||
| @@ -1,7 +1,5 @@ | |||
| 1 | # aclnnQuantBatchMatmulInplaceAdd | 1 | # aclnnQuantBatchMatmulInplaceAdd |
| 2 | 2 | ||
| 3 | -<!-- md-trans-meta sourceCommit=unknown translatedAt=2026-09-22T18:26:21.508Z --> | ||
| 4 | - | ||
| 5 | ## Applicable Products | 3 | ## Applicable Products |
| 6 | 4 | ||
| 7 | <!-- npu="950" id1 --> | 5 | <!-- npu="950" id1 --> |
| @@ -3,182 +3,104 @@ | |||
| 3 | ## 产品支持情况 | 3 | ## 产品支持情况 |
| 4 | 4 | ||
| 5 | | 产品 | 是否支持 | | 5 | | 产品 | 是否支持 | |
| 6 | -| ---- | :----:| | 6 | +| --- | :---: | |
| 7 | -|Ascend 950PR&950DT系列产品|√| | 7 | +| <term>Ascend 950PR&950DT系列产品</term> | √ | |
| 8 | -|Atlas A3系列产品|√| | 8 | +| <term>Atlas A3系列产品</term> | √ | |
| 9 | -|Atlas A2系列产品|√| | 9 | +| <term>Atlas A2系列产品</term> | √ | |
| 10 | -|Atlas 200I/500 A2推理产品|x| | 10 | +| <term>Atlas 200I/500 A2推理产品</term> | × | |
| 11 | -|Atlas推理系列产品|√| | 11 | +| <term>Atlas推理系列产品</term> | √ | |
| 12 | -|Atlas训练系列产品|x| | 12 | +| <term>Atlas训练系列产品</term> | × | |
| 13 | -|Kirin X90处理器系列产品|√| | 13 | +| <term>KirinX90处理器系列产品</term> | √ | |
| 14 | -|Kirin 9030处理器系列产品|√| | 14 | +| <term>Kirin9030处理器系列产品</term> | √ | |
| 15 | 15 | ||
| 16 | ## 功能说明 | 16 | ## 功能说明 |
| 17 | 17 | ||
| 18 | -- 算子功能:完成量化的矩阵乘计算,最小支持输入维度为2维,最大支持输入维度为6维。 | 18 | +- 算子功能:完成量化矩阵乘计算,支持缩放、累加偏置和输出类型转换,输入的逻辑维数为2~6。支持的逐张量、逐token、逐通道及分组量化模式随产品和数据类型而异,模式定义见[量化模式介绍](../../docs/zh/context/quant_mode_introduction.md)。 |
| 19 | -- 计算公式: | ||
| 20 | 19 | ||
| 21 | - - <term>Atlas A2系列产品</term>、<term>Atlas A3系列产品</term>、<term>Ascend 950PR&950DT系列产品</term>: | 20 | +- 版本关系:QuantBatchMatmulV3是算子名称,与aclnn接口的版本号分别定义。[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4.md)相较[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3.md)增加了左矩阵的pertoken_scale输入;[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)提供更多量化参数,并按输入组合调用本算子或[QuantBatchMatmulV4](../quant_batch_matmul_v4/README.md)。 |
| 22 | - - 无pertoken无bias: | ||
| 23 | 21 | ||
| 24 | - $$ | 22 | +- 计算公式:以下省略batch下标,$X_1$、$X_2$表示按transpose_x1、transpose_x2解释后的矩阵,形状分别为$(M,K)$、$(K,N)$。M、N为输出矩阵的行数、列数,K为归约维度;$S_1$、$S_2$分别对应pertoken_scale、scale。未提供pertoken_scale、bias、offset时,分别按1、0、0处理。 |
| 25 | - out = x1@x2 * scale + offset | ||
| 26 | - $$ | ||
| 27 | 23 | ||
| 28 | - - bias INT32: | 24 | + - 非分组量化、无bias: |
| 29 | 25 | ||
| 30 | - $$ | 26 | + $$ |
| 31 | - out = (x1@x2 + bias) * scale + offset | 27 | + y=(X_1@X_2)\odot S_2\odot S_1+offset |
| 32 | - $$ | 28 | + $$ |
| 33 | 29 | ||
| 34 | - - bias BFLOAT16/FLOAT32(此场景无offset): | 30 | + - INT8或INT4输入且bias为INT32: |
| 35 | 31 | ||
| 36 | - $$ | 32 | + $$ |
| 37 | - out = x1@x2 * scale + bias | 33 | + y=(X_1@X_2+bias)\odot S_2\odot S_1+offset |
| 38 | - $$ | 34 | + $$ |
| 39 | 35 | ||
| 40 | - - pertoken无bias: | 36 | + - INT8或INT4输入且bias为BFLOAT16、FLOAT16或FLOAT32: |
| 41 | 37 | ||
| 42 | - $$ | 38 | + $$ |
| 43 | - out = x1@x2 * scale * pertokenScaleOptional | 39 | + y=(X_1@X_2)\odot S_2\odot S_1+bias |
| 44 | - $$ | 40 | + $$ |
| 45 | 41 | ||
| 46 | - - pertoken, bias INT32(此场景无offset): | 42 | + - FLOAT8_E4M3FN、FLOAT8_E5M2或HIFLOAT8的非分组量化,bias为FLOAT32: |
| 47 | 43 | ||
| 48 | - $$ | 44 | + $$ |
| 49 | - out = (x1@x2 + bias) * scale * pertokenScaleOptional | 45 | + y=(X_1@X_2+bias)\odot S_2\odot S_1 |
| 50 | - $$ | 46 | + $$ |
| 51 | 47 | ||
| 52 | - - pertoken, bias BFLOAT16/FLOAT16/FLOAT32(此场景无offset): | 48 | + - INT32输出:直接输出整数累加结果,不执行缩放。 |
| 53 | 49 | ||
| 54 | - $$ | 50 | + $$ |
| 55 | - out = x1@x2 * scale * pertokenScaleOptional + bias | 51 | + y=X_1@X_2+bias |
| 56 | - $$ | 52 | + $$ |
| 57 | 53 | ||
| 58 | - - <term>Atlas推理系列产品</term>: | 54 | + - MX(Microscaling,微缩放)、G-B和B-B分组量化:按K轴分组计算,再累加各组结果。 |
| 59 | - - 无pertoken、无bias: | ||
| 60 | 55 | ||
| 61 | - $$ | 56 | + $$ |
| 62 | - out = x1@x2 * scale + offset | 57 | + y_{m,n}=\sum_{j=0}^{L-1}\left[ |
| 63 | - $$ | 58 | + \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right) |
| 59 | + S_{1,m,j}S_{2,j,n}\right]+bias_n | ||
| 60 | + $$ | ||
| 64 | 61 | ||
| 65 | - - 无pertoken、bias INT32: | 62 | + 其中,$g_K$为K方向分组大小,$L=\lceil K/g_K\rceil$,$g_j=\min(g_K,K-jg_K)$;$S_1$、$S_2$表示按对应量化分组广播后的scale。G-B和B-B模式不支持bias,公式中取$bias_n=0$。 |
| 66 | 63 | ||
| 67 | - $$ | 64 | + 其中,$@$表示矩阵乘,$\odot$表示按量化参数形状广播的逐元素乘。offset仅用于支持的INT8输出组合,不能与pertoken_scale同时使用。上述公式表示量化计算关系,实际结果还需转换为dtype指定的数据类型。 |
| 68 | - out = (x1@x2 + bias) * scale + offset | ||
| 69 | - $$ | ||
| 70 | - | ||
| 71 | - - 有pertoken、无bias | ||
| 72 | - | ||
| 73 | - $$ | ||
| 74 | - out = x1@x2 * scale * pertokenScaleOptional | ||
| 75 | - $$ | ||
| 76 | - | ||
| 77 | - - 有pertoken、bias INT32 | ||
| 78 | - | ||
| 79 | - $$ | ||
| 80 | - out = (x1@x2 + bias) * scale * pertokenScaleOptional | ||
| 81 | - $$ | ||
| 82 | - | ||
| 83 | - - 有pertoken、bias FLOAT32 | ||
| 84 | - | ||
| 85 | - $$ | ||
| 86 | - out = x1@x2 * scale * pertokenScaleOptional + bias | ||
| 87 | - $$ | ||
| 88 | 65 | ||
| 89 | ## 参数说明 | 66 | ## 参数说明 |
| 90 | 67 | ||
| 91 | -<table class="tg"><thead> | 68 | +| 参数名 | 输入/输出/属性 | 描述 | 数据类型 | 数据格式 | |
| 92 | - <tr> | 69 | +| --- | --- | --- | --- | --- | |
| 93 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">参数名</span></th> | 70 | +| x1 | 输入 | 矩阵乘的左矩阵,对应公式中的X₁。 | INT8、INT4、HIFLOAT8、FLOAT8_E5M2、FLOAT8_E4M3FN、FLOAT4_E2M1 | ND、FRACTAL_NZ | |
| 94 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">输入/输出/属性</span></th> | 71 | +| x2 | 输入 | 矩阵乘的右矩阵,对应公式中的X₂。 | INT8、INT4、HIFLOAT8、FLOAT8_E5M2、FLOAT8_E4M3FN、FLOAT4_E2M1 | ND、FRACTAL_NZ | |
| 95 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">描述</span></th> | 72 | +| scale | 输入 | 右矩阵的量化缩放参数。UINT64、INT64类型存储编码后的量化参数,须按解码后的scale参与公式计算。 | UINT64、FLOAT32、INT64、BFLOAT16、FLOAT8_E8M0 | ND、FRACTAL_NZ | |
| 96 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据类型</span></th> | 73 | +| offset | 可选输入 | 输出量化的偏置因子;未提供时按0处理,仅用于支持的INT8输出组合。 | FLOAT32 | ND | |
| 97 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据格式</span></th> | 74 | +| bias | 可选输入 | 矩阵乘的累加偏置;未提供时不加偏置。累加位置由输入及bias的数据类型决定。 | INT32、BFLOAT16、FLOAT16、FLOAT32 | ND | |
| 98 | - </tr></thead> | 75 | +| pertoken_scale | 可选输入 | 左矩阵的量化缩放参数;未提供时按1处理。逐token及分组量化场景须提供对应的缩放参数。 | FLOAT32、FLOAT8_E8M0 | ND | |
| 99 | -<tbody> | 76 | +| y | 输出 | 矩阵乘及量化处理后的结果,数据类型由dtype指定。 | FLOAT16、INT8、BFLOAT16、INT32、FLOAT32 | ND、FRACTAL_NZ | |
| 100 | - <tr> | 77 | +| dtype | 属性 | 输出y的数据类型枚举,支持0(FLOAT32)、1(FLOAT16)、2(INT8)、3(INT32)、27(BFLOAT16),具体取值受产品及输入组合限制。 | INT64 | - | |
| 101 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">x1</span></td> | 78 | +| transpose_x1 | 可选属性 | <ul><li>是否转置x1的最后两维:false表示不转置,true表示转置。</li><li>默认值为false。</li></ul> | BOOL | - | |
| 102 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输入</span></td> | 79 | +| transpose_x2 | 可选属性 | <ul><li>是否转置x2的最后两维:false表示不转置,true表示转置。</li><li>默认值为false。</li></ul> | BOOL | - | |
| 103 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">矩阵乘运算中的左矩阵。</span></td> | 80 | +| group_size | 可选属性 | <ul><li>M、N、K方向量化分组大小的编码值,各分量占16位;支持的分组组合见<a href="#约束说明">约束说明</a>。</li><li>默认值为0。</li></ul> | INT64 | - | |
| 104 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td> | ||
| 105 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND, FRACTAL_NZ</span></td> | ||
| 106 | - </tr> | ||
| 107 | - <tr> | ||
| 108 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">x2</span></td> | ||
| 109 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">输入</span></td> | ||
| 110 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">矩阵乘运算中的右矩阵。</span></td> | ||
| 111 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td> | ||
| 112 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND, FRACTAL_NZ</span></td> | ||
| 113 | - </tr> | ||
| 114 | - <tr> | ||
| 115 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">scale</span></td> | ||
| 116 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输入</span></td> | ||
| 117 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">量化参数的缩放因子,对应公式的scale。</span></td> | ||
| 118 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">UINT64, FLOAT32, INT64, BF16, FLOAT8_E8M0</span></td> | ||
| 119 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td> | ||
| 120 | - </tr> | ||
| 121 | - <tr> | ||
| 122 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">offset</span></td> | ||
| 123 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">输入</span></td> | ||
| 124 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">量化参数的偏置因子,对应公式的offset。</span></td> | ||
| 125 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32</span></td> | ||
| 126 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td> | ||
| 127 | - </tr> | ||
| 128 | - <tr> | ||
| 129 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">bias</span></td> | ||
| 130 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输入</span></td> | ||
| 131 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">矩阵乘运算后累加的偏置,对应公式中的bias。</span></td> | ||
| 132 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT32, BF16, FLOAT16, FLOAT32</span></td> | ||
| 133 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td> | ||
| 134 | - </tr> | ||
| 135 | - <tr> | ||
| 136 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">pertoken_scale</span></td> | ||
| 137 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">输入</span></td> | ||
| 138 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">量化参数的缩放因子,对应公式中的pertokenScaleOptional。</span></td> | ||
| 139 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32, FLOAT8_E8M0</span></td> | ||
| 140 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td> | ||
| 141 | - </tr> | ||
| 142 | - <tr> | ||
| 143 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">y</span></td> | ||
| 144 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输出</span></td> | ||
| 145 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">矩阵乘运算的计算结果。</span></td> | ||
| 146 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">FLOAT16, INT8, BF16, INT32, FLOAT32</span></td> | ||
| 147 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td> | ||
| 148 | - </tr> | ||
| 149 | -</tbody></table> | ||
| 150 | 81 | ||
| 151 | -- <term>Atlas A2系列产品</term>、<term>Atlas A3系列产品</term>: | 82 | +- <term>Atlas A3系列产品</term>、<term>Atlas A2系列产品</term>:x1、x2不支持HIFLOAT8、FLOAT8_E5M2、FLOAT8_E4M3FN、FLOAT4_E2M1;scale不支持FLOAT8_E8M0;pertoken_scale仅支持FLOAT32;y不支持FLOAT32。 |
| 152 | - - x1只支持INT8、INT4数据类型。 | 83 | +- <term>Atlas推理系列产品</term>:x1、x2仅支持INT8;scale支持UINT64、INT64,逐token场景还支持FLOAT32;bias支持INT32,逐token场景还支持FLOAT32;pertoken_scale仅支持FLOAT32;y支持FLOAT16、INT8,逐token场景仅支持FLOAT16。逐token场景的y使用FRACTAL_NZ,其余场景使用ND。 |
| 153 | - - x2只支持INT8、INT4数据类型。 | 84 | +- <term>KirinX90处理器系列产品</term>、<term>Kirin9030处理器系列产品</term>:x1、x2仅支持INT8,x1使用ND,x2使用FRACTAL_NZ;scale仅支持UINT64、INT64;bias仅支持INT32;pertoken_scale仅支持FLOAT32;y仅支持FLOAT16、INT8。 |
| 154 | - - scale只支持UINT64、FLOAT32、INT64、BF16数据类型。 | 85 | +- <term>Ascend 950PR&950DT系列产品</term>:scale的FRACTAL_NZ格式仅用于MX全量化的ScaleNZ场景,要求x2也使用FRACTAL_NZ,且x1、x2类型相同,为FLOAT8_E4M3FN或FLOAT4_E2M1。布局和转换要求见[ScaleNZ约束](docs/aclnnQuantMatmulWeightNz.md#scalenz约束)。其他产品的scale使用ND。 |
| 155 | - - bias只支持INT32、BFLOAT16、FLOAT16、FLOAT32数据类型。 | ||
| 156 | - - offset只支持FLOAT32数据类型。 | ||
| 157 | - - pertoken_scale只支持FLOAT32数据类型。 | ||
| 158 | - - y只支持FLOAT16和BFLOAT16数据类型。 | ||
| 159 | -- <term>Atlas推理系列产品</term>: | ||
| 160 | - - x1只支持INT8数据类型。 | ||
| 161 | - - x2只支持INT8数据类型。 | ||
| 162 | - - scale只支持UINT64、FLOAT32数据类型。 | ||
| 163 | - - bias只支持INT32数据类型,pertoken场景额外支持FLOAT32数据类型。 | ||
| 164 | - - pertoken_scale只支持FLOAT32数据类型。 | ||
| 165 | - - y只支持FLOAT16和INT32数据类型。 | ||
| 166 | -- Kirin X90/Kirin 9030处理器系列产品: | ||
| 167 | - - x1、x2只支持INT8数据类型。 | ||
| 168 | - - scale只支持UINT64、INT64数据类型。 | ||
| 169 | - - offset只支持FLOAT32数据类型。 | ||
| 170 | - - bias只支持INT32数据类型。 | ||
| 171 | - - pertoken_scale只支持FLOAT32数据类型。 | ||
| 172 | - - y只支持FLOAT16和INT8数据类型。 | ||
| 173 | 86 | ||
| 174 | ## 约束说明 | 87 | ## 约束说明 |
| 175 | 88 | ||
| 176 | -- 不支持空tensor。 | 89 | +- 输入矩阵的K维必须一致,batch维须满足[广播关系](../../docs/zh/context/broadcast_relationship.md)。具体形状和产品限制见[算子IR说明](docs/ascendir_QuantBatchMatmulV3.md)。 |
| 177 | -- 支持连续tensor,[非连续tensor](../../docs/zh/context/non_contiguous_tensor.md)只支持转置场景。 | 90 | +- INT4输入仅支持二维,transpose_x1须为false,输入的最内维须为偶数。 |
| 91 | +- group_size按`group_size_k | (group_size_n << 16) | (group_size_m << 32)`编码。分组分量为0时由输入形状推导,推导时对应维度须满足整除要求。MX、G-B、B-B模式的最终分组大小分别为`[1, 1, 32]`、`[1, 128, 128]`、`[128, 128, 128]`;其他量化模式的group_size为0。 | ||
| 92 | +- MX模式下,pertoken_scale、scale均须提供且类型为FLOAT8_E8M0。FLOAT4_E2M1输入的最内维须为偶数,且K须大于2。 | ||
| 93 | +- <term>Atlas A3系列产品</term>、<term>Atlas A2系列产品</term>:输入矩阵逻辑形状的最后一维不能超过65535。 | ||
| 94 | +- <term>Ascend 950PR&950DT系列产品</term>:x2使用ND时支持M=0或N=0并输出空Tensor;x2使用FRACTAL_NZ时支持M=0并输出空Tensor。其他场景不支持维度大小为0的输入。 | ||
| 178 | 95 | ||
| 179 | ## 调用说明 | 96 | ## 调用说明 |
| 180 | 97 | ||
| 181 | -| 调用方式 | 样例代码 | 说明 | | 98 | +| 调用方式 | 样例代码 | 说明 | |
| 182 | -| ---------------- | --------------------------- | --------------------------------------------------- | | 99 | +| --- | --- | --- | |
| 183 | -| GEIR接口 | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | 通过[QuantBatchMatmulV3算子IR](docs/ascendir_QuantBatchMatmulV3.md)构图并调用算子。 | | 100 | +| GE图模式 | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | 通过[QuantBatchMatmulV3算子IR](docs/ascendir_QuantBatchMatmulV3.md)构图调用。 | |
| 184 | -| aclnn接口 | [test_aclnn_quant_matmul_v3](examples/test_aclnn_quant_matmul_v3_at.cpp) | 通过<br>[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3.md)<br>[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4.md)<br>[aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz.md)<br>[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)<br>等方式调用QuantBatchMatmulV3算子。 | | 101 | +| aclnn API | [test_aclnn_quant_matmul_v3_at](examples/test_aclnn_quant_matmul_v3_at.cpp) | 通过[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3.md)调用。 | |
| 102 | +| aclnn API | [test_aclnn_quant_matmul_v4_at2_at3](examples/test_aclnn_quant_matmul_v4_at2_at3.cpp) | 通过[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4.md)调用,支持左矩阵逐token缩放。 | | ||
| 103 | +| aclnn API | [test_aclnn_quant_matmul_weight_nz](examples/test_aclnn_quant_matmul_weight_nz.cpp) | 通过[aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz.md)调用,使用FRACTAL_NZ格式的权重。 | | ||
| 104 | +| aclnn API | [test_aclnn_quant_matmul_v5_hif8hif8](../quant_batch_matmul_v4/examples/arch35/test_aclnn_quant_matmul_v5_hif8hif8.cpp) | 通过[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)调用,样例展示由QuantBatchMatmulV3执行的HIFLOAT8量化场景。 | | ||
| 105 | + | ||
| 106 | +aclnnQuantMatmulV3、aclnnQuantMatmulV4已在接口声明中标记为待废弃,迁移说明见对应接口文档。 | ||
| @@ -5,182 +5,104 @@ English | [简体中文](./README.md) | |||
| 5 | ## Applicable Products | 5 | ## Applicable Products |
| 6 | 6 | ||
| 7 | | Product | Support | | 7 | | Product | Support | |
| 8 | -| ---- | :----:| | 8 | +| --- | :---: | |
| 9 | -|Ascend 950PR&950DT products|√| | 9 | +| <term>Ascend 950PR&950DT products</term> | √ | |
| 10 | -|Atlas A3 products|√| | 10 | +| <term>Atlas A3 products</term> | √ | |
| 11 | -|Atlas A2 products|√| | 11 | +| <term>Atlas A2 products</term> | √ | |
| 12 | -|Atlas 200I/500 A2 inference products|x| | 12 | +| <term>Atlas 200I/500 A2 inference products</term> | × | |
| 13 | -|Atlas inference products|√| | 13 | +| <term>Atlas inference products</term> | √ | |
| 14 | -|Atlas training products|x| | 14 | +| <term>Atlas training products</term> | × | |
| 15 | -|Kirin X90 processor series products|√| | 15 | +| <term>Kirin X90 processor products</term> | √ | |
| 16 | -|Kirin 9030 processor series products|√| | 16 | +| <term>Kirin 9030 processor products</term> | √ | |
| 17 | 17 | ||
| 18 | ## Function | 18 | ## Function |
| 19 | 19 | ||
| 20 | -- Operator function: Performs quantized matrix multiplication. The input supports a minimum of 2 dimensions and a maximum of 6 dimensions. | 20 | +- Operator function: Performs quantized matrix multiplication with scaling, bias accumulation, and output type conversion. The logical input rank is 2–6. Supported per-tensor, per-token, per-channel, and grouped modes depend on the product and data type. See [Introduction to Quantization Modes](../../docs/en/context/quant_mode_introduction.md). |
| 21 | -- Formula: | ||
| 22 | 21 | ||
| 23 | - - <term>Atlas A2 products</term>, <term>Atlas A3 products</term>, <term>Ascend 950PR&950DT products</term>: | 22 | +- Version relationship: QuantBatchMatmulV3 is the operator name; ACLNN interface versions are defined separately. [aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4_en.md) adds left-matrix pertoken_scale compared with [aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3_en.md). [aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5_en.md) provides additional quantization parameters and dispatches to this operator or [QuantBatchMatmulV4](../quant_batch_matmul_v4/README_en.md) according to the input combination. |
| 24 | - - No pertoken and no bias: | ||
| 25 | 23 | ||
| 26 | - $$ | 24 | +- Formulas: Batch indices are omitted below. $X_1$ and $X_2$ are the matrices interpreted according to transpose_x1 and transpose_x2, with shapes $(M,K)$ and $(K,N)$. M and N are the result matrix dimensions, and K is the reduction dimension. $S_1$ and $S_2$ correspond to pertoken_scale and scale. Omitted pertoken_scale, bias, and offset are treated as 1, 0, and 0, respectively. |
| 27 | - out = x1@x2 * scale + offset | ||
| 28 | - $$ | ||
| 29 | 25 | ||
| 30 | - - bias INT32: | 26 | + - Non-grouped quantization without bias: |
| 31 | 27 | ||
| 32 | - $$ | 28 | + $$ |
| 33 | - out = (x1@x2 + bias) * scale + offset | 29 | + y=(X_1@X_2)\odot S_2\odot S_1+offset |
| 34 | - $$ | 30 | + $$ |
| 35 | 31 | ||
| 36 | - - bias BFLOAT16/FLOAT32 (no offset in this scenario): | 32 | + - INT8 or INT4 inputs with INT32 bias: |
| 37 | 33 | ||
| 38 | - $$ | 34 | + $$ |
| 39 | - out = x1@x2 * scale + bias | 35 | + y=(X_1@X_2+bias)\odot S_2\odot S_1+offset |
| 40 | - $$ | 36 | + $$ |
| 41 | 37 | ||
| 42 | - - pertoken and no bias: | 38 | + - INT8 or INT4 inputs with BFLOAT16, FLOAT16, or FLOAT32 bias: |
| 43 | 39 | ||
| 44 | - $$ | 40 | + $$ |
| 45 | - out = x1@x2 * scale * pertokenScaleOptional | 41 | + y=(X_1@X_2)\odot S_2\odot S_1+bias |
| 46 | - $$ | 42 | + $$ |
| 47 | 43 | ||
| 48 | - - pertoken and bias INT32 (no offset in this scenario): | 44 | + - Non-grouped quantization with FLOAT8_E4M3FN, FLOAT8_E5M2, or HIFLOAT8 inputs and FLOAT32 bias: |
| 49 | 45 | ||
| 50 | - $$ | 46 | + $$ |
| 51 | - out = (x1@x2 + bias) * scale * pertokenScaleOptional | 47 | + y=(X_1@X_2+bias)\odot S_2\odot S_1 |
| 52 | - $$ | 48 | + $$ |
| 53 | 49 | ||
| 54 | - - pertoken and bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario): | 50 | + - INT32 output: Returns the integer accumulation result directly, without scaling. |
| 55 | 51 | ||
| 56 | - $$ | 52 | + $$ |
| 57 | - out = x1@x2 * scale * pertokenScaleOptional + bias | 53 | + y=X_1@X_2+bias |
| 58 | - $$ | 54 | + $$ |
| 59 | 55 | ||
| 60 | - - <term>Atlas inference products</term>: | 56 | + - MX (Microscaling), G-B, and B-B grouped quantization: Compute by K groups and sum the group results. |
| 61 | - - No pertoken and no bias: | ||
| 62 | 57 | ||
| 63 | - $$ | 58 | + $$ |
| 64 | - out = x1@x2 * scale + offset | 59 | + y_{m,n}=\sum_{j=0}^{L-1}\left[ |
| 65 | - $$ | 60 | + \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right) |
| 61 | + S_{1,m,j}S_{2,j,n}\right]+bias_n | ||
| 62 | + $$ | ||
| 66 | 63 | ||
| 67 | - - No pertoken and bias INT32: | 64 | + Here, $g_K$ is the group size along K, $L=\lceil K/g_K\rceil$, and $g_j=\min(g_K,K-jg_K)$. $S_1$ and $S_2$ are the scales broadcast according to the quantization groups. G-B and B-B do not support bias, so $bias_n=0$ in the formula. |
| 68 | 65 | ||
| 69 | - $$ | 66 | + Here, $@$ denotes matrix multiplication, and $\odot$ denotes elementwise multiplication with broadcasting according to the quantization-parameter shapes. offset is used only in supported INT8-output combinations and cannot be combined with pertoken_scale. These formulas describe the quantized computation; the result is also converted to the type specified by dtype. |
| 70 | - out = (x1@x2 + bias) * scale + offset | ||
| 71 | - $$ | ||
| 72 | - | ||
| 73 | - - pertoken and no bias | ||
| 74 | - | ||
| 75 | - $$ | ||
| 76 | - out = x1@x2 * scale * pertokenScaleOptional | ||
| 77 | - $$ | ||
| 78 | - | ||
| 79 | - - pertoken and bias INT32 | ||
| 80 | - | ||
| 81 | - $$ | ||
| 82 | - out = (x1@x2 + bias) * scale * pertokenScaleOptional | ||
| 83 | - $$ | ||
| 84 | - | ||
| 85 | - - pertoken and bias FLOAT32 | ||
| 86 | - | ||
| 87 | - $$ | ||
| 88 | - out = x1@x2 * scale * pertokenScaleOptional + bias | ||
| 89 | - $$ | ||
| 90 | 67 | ||
| 91 | ## Parameters | 68 | ## Parameters |
| 92 | 69 | ||
| 93 | -<table class="tg"><thead> | 70 | +| Parameter | Input/Output/Attribute | Description | Data Type | Data Format | |
| 94 | - <tr> | 71 | +| --- | --- | --- | --- | --- | |
| 95 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Parameter name</span></th> | 72 | +| x1 | Input | Left matrix, corresponding to X₁ in the formulas. | INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1 | ND, FRACTAL_NZ | |
| 96 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Input/Output/Attribute</span></th> | 73 | +| x2 | Input | Right matrix, corresponding to X₂ in the formulas. | INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1 | ND, FRACTAL_NZ | |
| 97 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Description</span></th> | 74 | +| scale | Input | Right-matrix quantization scale. UINT64 and INT64 store encoded quantization parameters; use the decoded scale in the formulas. | UINT64, FLOAT32, INT64, BFLOAT16, FLOAT8_E8M0 | ND, FRACTAL_NZ | |
| 98 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data type</span></th> | 75 | +| offset | Optional input | Output quantization offset. Treated as 0 when omitted; used only in supported INT8-output combinations. | FLOAT32 | ND | |
| 99 | - <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data format</span></th> | 76 | +| bias | Optional input | Matrix-product accumulation bias. Omitted bias adds nothing. Its accumulation position depends on the input and bias data types. | INT32, BFLOAT16, FLOAT16, FLOAT32 | ND | |
| 100 | - </tr></thead> | 77 | +| pertoken_scale | Optional input | Left-matrix quantization scale. Treated as 1 when omitted. The corresponding scales are required for per-token and grouped quantization. | FLOAT32, FLOAT8_E8M0 | ND | |
| 101 | -<tbody> | 78 | +| y | Output | Result after matrix multiplication and quantization, with data type specified by dtype. | FLOAT16, INT8, BFLOAT16, INT32, FLOAT32 | ND, FRACTAL_NZ | |
| 102 | - <tr> | 79 | +| dtype | Attribute | Output y data type: 0 (FLOAT32), 1 (FLOAT16), 2 (INT8), 3 (INT32), or 27 (BFLOAT16), subject to product and input-combination restrictions. | INT64 | - | |
| 103 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">x1</span></td> | 80 | +| transpose_x1 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x1: false disables transposition and true enables it.</li><li>Defaults to false.</li></ul> | BOOL | - | |
| 104 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Input</span></td> | 81 | +| transpose_x2 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x2: false disables transposition and true enables it.</li><li>Defaults to false.</li></ul> | BOOL | - | |
| 105 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Left matrix in the matrix multiplication operation.</span></td> | 82 | +| group_size | Optional attribute | <ul><li>Encodes quantization group sizes along M, N, and K, with 16 bits per component. For supported combinations, see <a href="#constraints">Constraints</a>.</li><li>Defaults to 0.</li></ul> | INT64 | - | |
| 106 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td> | ||
| 107 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND, FRACTAL_NZ</span></td> | ||
| 108 | - </tr> | ||
| 109 | - <tr> | ||
| 110 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">x2</span></td> | ||
| 111 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Input</span></td> | ||
| 112 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Right matrix in the matrix multiplication operation.</span></td> | ||
| 113 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td> | ||
| 114 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND, FRACTAL_NZ</span></td> | ||
| 115 | - </tr> | ||
| 116 | - <tr> | ||
| 117 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">scale</span></td> | ||
| 118 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Input</span></td> | ||
| 119 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Scale factor of the quantization parameter, corresponding to scale in the formula.</span></td> | ||
| 120 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">UINT64, FLOAT32, INT64, BF16, FLOAT8_E8M0</span></td> | ||
| 121 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td> | ||
| 122 | - </tr> | ||
| 123 | - <tr> | ||
| 124 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">offset</span></td> | ||
| 125 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Input</span></td> | ||
| 126 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Offset factor of the quantization parameter, corresponding to offset in the formula.</span></td> | ||
| 127 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32</span></td> | ||
| 128 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td> | ||
| 129 | - </tr> | ||
| 130 | - <tr> | ||
| 131 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">bias</span></td> | ||
| 132 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Input</span></td> | ||
| 133 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Bias accumulated after the matrix multiplication operation, corresponding to bias in the formula.</span></td> | ||
| 134 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT32, BF16, FLOAT16, FLOAT32</span></td> | ||
| 135 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td> | ||
| 136 | - </tr> | ||
| 137 | - <tr> | ||
| 138 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">pertoken_scale</span></td> | ||
| 139 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Input</span></td> | ||
| 140 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Scale factor of the quantization parameter, corresponding to pertokenScaleOptional in the formula.</span></td> | ||
| 141 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32, FLOAT8_E8M0</span></td> | ||
| 142 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td> | ||
| 143 | - </tr> | ||
| 144 | - <tr> | ||
| 145 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">y</span></td> | ||
| 146 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Output</span></td> | ||
| 147 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Computation result of the matrix multiplication operation.</span></td> | ||
| 148 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">FLOAT16, INT8, BF16, INT32, FLOAT32</span></td> | ||
| 149 | - <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td> | ||
| 150 | - </tr> | ||
| 151 | -</tbody></table> | ||
| 152 | 83 | ||
| 153 | -- <term>Atlas A2 products</term>, <term>Atlas A3 products</term>: | 84 | +- <term>Atlas A3 products</term>, <term>Atlas A2 products</term>: x1 and x2 do not support HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, or FLOAT4_E2M1. scale does not support FLOAT8_E8M0; pertoken_scale supports only FLOAT32; y does not support FLOAT32. |
| 154 | - - x1 supports only the INT8 and INT4 data types. | 85 | +- <term>Atlas inference products</term>: x1 and x2 support only INT8. scale supports UINT64 and INT64, and also FLOAT32 for per-token quantization. bias supports INT32, and also FLOAT32 for per-token quantization. pertoken_scale supports only FLOAT32. y supports FLOAT16 and INT8, but only FLOAT16 for per-token quantization. y uses FRACTAL_NZ for per-token quantization and ND otherwise. |
| 155 | - - x2 supports only the INT8 and INT4 data types. | 86 | +- <term>Kirin X90 processor products</term>, <term>Kirin 9030 processor products</term>: x1 and x2 support only INT8; x1 uses ND and x2 uses FRACTAL_NZ. scale supports only UINT64 and INT64; bias only INT32; pertoken_scale only FLOAT32; y only FLOAT16 and INT8. |
| 156 | - - scale supports only the UINT64, FLOAT32, INT64, and BF16 data types. | 87 | +- <term>Ascend 950PR&950DT products</term>: FRACTAL_NZ scale is used only for ScaleNZ in full MX quantization. x2 must also use FRACTAL_NZ, and x1 and x2 must share the same type, FLOAT8_E4M3FN or FLOAT4_E2M1. For layout and conversion requirements, see [ScaleNZ Constraints](docs/aclnnQuantMatmulWeightNz_en.md#scalenz-constraints). scale uses ND on other products. |
| 157 | - - bias supports only the INT32, BFLOAT16, FLOAT16, and FLOAT32 data types. | ||
| 158 | - - offset supports only the FLOAT32 data type. | ||
| 159 | - - pertoken_scale supports only the FLOAT32 data type. | ||
| 160 | - - y supports only the FLOAT16 and BFLOAT16 data types. | ||
| 161 | -- <term>Atlas inference products</term>: | ||
| 162 | - - x1 supports only the INT8 data type. | ||
| 163 | - - x2 supports only the INT8 data type. | ||
| 164 | - - scale supports only the UINT64 and FLOAT32 data types. | ||
| 165 | - - bias supports only the INT32 data type, and the pertoken scenario additionally supports the FLOAT32 data type. | ||
| 166 | - - pertoken_scale supports only the FLOAT32 data type. | ||
| 167 | - - y supports only the FLOAT16 and INT32 data types. | ||
| 168 | -- Kirin X90/Kirin 9030 processor series products: | ||
| 169 | - - x1 and x2 support only the INT8 data type. | ||
| 170 | - - scale supports only the UINT64 and INT64 data types. | ||
| 171 | - - offset supports only the FLOAT32 data type. | ||
| 172 | - - bias supports only the INT32 data type. | ||
| 173 | - - pertoken_scale supports only the FLOAT32 data type. | ||
| 174 | - - y supports only the FLOAT16 and INT8 data types. | ||
| 175 | 88 | ||
| 176 | ## Constraints | 89 | ## Constraints |
| 177 | 90 | ||
| 178 | -- Empty tensors are not supported. | 91 | +- The input K dimensions must match, and batch dimensions must satisfy the [broadcasting rules](../../docs/en/context/broadcast_relationship.md). For shapes and product-specific restrictions, see the [Operator IR Description](docs/ascendir_QuantBatchMatmulV3_en.md). |
| 179 | -- Contiguous tensors are supported. [Non-contiguous tensors](../../docs/en/context/non_contiguous_tensor.md) support only the transpose scenario. | 92 | +- INT4 inputs support only two dimensions, require transpose_x1=false, and must have an even innermost dimension. |
| 93 | +- group_size is encoded as `group_size_k | (group_size_n << 16) | (group_size_m << 32)`. Zero components are inferred from input shapes, with divisibility required for the corresponding dimensions. The resulting sizes for MX, G-B, and B-B must be `[1, 1, 32]`, `[1, 128, 128]`, and `[128, 128, 128]`, respectively. Other modes use group_size=0. | ||
| 94 | +- MX requires both pertoken_scale and scale as FLOAT8_E8M0. FLOAT4_E2M1 inputs must have an even innermost dimension, and K must be greater than 2. | ||
| 95 | +- <term>Atlas A3 products</term>, <term>Atlas A2 products</term>: The last logical dimension of each input matrix must not exceed 65535. | ||
| 96 | +- <term>Ascend 950PR&950DT products</term>: ND x2 supports M=0 or N=0 and produces an empty tensor. FRACTAL_NZ x2 supports M=0 and produces an empty tensor. Other cases do not support inputs with zero-sized dimensions. | ||
| 180 | 97 | ||
| 181 | ## Invocation Description | 98 | ## Invocation Description |
| 182 | 99 | ||
| 183 | -| Invocation method | Sample code | Description | | 100 | +| Invocation method | Sample code | Description | |
| 184 | -| ---------------- | --------------------------- | --------------------------------------------------- | | 101 | +| --- | --- | --- | |
| 185 | -| aclnn interface | [test_aclnn_quant_matmul_v3](examples/test_aclnn_quant_matmul_v3_at.cpp) | Calls the QuantBatchMatmulV3 operator through <br>[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3_en.md)<br>[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4_en.md)<br>[aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz_en.md)<br>[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5_en.md)<br>and other interfaces. | | 102 | +| GE graph mode | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | Builds a graph using the [QuantBatchMatmulV3 operator IR](docs/ascendir_QuantBatchMatmulV3_en.md). | |
| 186 | -| GEIR interface | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | Builds the graph and invokes the operator through the [QuantBatchMatmulV3 operator IR](docs/ascendir_QuantBatchMatmulV3.md). | | 103 | +| aclnn API | [test_aclnn_quant_matmul_v3_at](examples/test_aclnn_quant_matmul_v3_at.cpp) | Invokes [aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3_en.md). | |
| 104 | +| aclnn API | [test_aclnn_quant_matmul_v4_at2_at3](examples/test_aclnn_quant_matmul_v4_at2_at3.cpp) | Invokes [aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4_en.md) with left-matrix per-token scaling. | | ||
| 105 | +| aclnn API | [test_aclnn_quant_matmul_weight_nz](examples/test_aclnn_quant_matmul_weight_nz.cpp) | Invokes [aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz_en.md) with FRACTAL_NZ weights. | | ||
| 106 | +| aclnn API | [test_aclnn_quant_matmul_v5_hif8hif8](../quant_batch_matmul_v4/examples/arch35/test_aclnn_quant_matmul_v5_hif8hif8.cpp) | Invokes [aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5_en.md). The example demonstrates HIFLOAT8 quantization executed by QuantBatchMatmulV3. | | ||
| 107 | + | ||
| 108 | +aclnnQuantMatmulV3 and aclnnQuantMatmulV4 are marked for future deprecation in their interface declarations. See the corresponding interface documents for migration information. | ||
| @@ -64,7 +64,7 @@ | |||
| 64 | ### 输入与转置要求 | 64 | ### 输入与转置要求 |
| 65 | 65 | ||
| 66 | - `x1`和`x2`的原始`shape`维数均不得小于2,至少一路数据输入存在`Transpose`或`TransposeD`,否则不触发融合。数据输入前仅有`Reshape`不能触发融合。 | 66 | - `x1`和`x2`的原始`shape`维数均不得小于2,至少一路数据输入存在`Transpose`或`TransposeD`,否则不触发融合。数据输入前仅有`Reshape`不能触发融合。 |
| 67 | -- `x1`、`x2`的数据类型均支持`INT8`、`INT4`、`FLOAT8_E4M3FN`、`FLOAT8_E5M2`、`HIFLOAT8`、`FLOAT4_E2M1`;输出数据类型支持`INT8`、`FLOAT16`、`BF16`、`INT32`、`FLOAT32`。实际组合还需满足目标产品的算子约束。 | 67 | +- `x1`、`x2`的数据类型均支持`INT8`、`INT4`、`FLOAT8_E4M3FN`、`FLOAT8_E5M2`、`HIFLOAT8`、`FLOAT4_E2M1`;输出数据类型支持`INT8`、`FLOAT16`、`BFLOAT16`、`INT32`、`FLOAT32`。实际组合还需满足目标产品的算子约束。 |
| 68 | - 支持动态`shape`;`x1`或`x2`的`shape`存在未知维度时,按动态场景处理。 | 68 | - 支持动态`shape`;`x1`或`x2`的`shape`存在未知维度时,按动态场景处理。 |
| 69 | - 数据输入的转置必须只交换最后两维,其他维度的顺序保持不变。例如,形状为`(B, M, K)`的输入转置为`(B, K, M)`。交换批次维度等其他转置方式不属于本规则的适用范围。对应的`scale`转换也须保持量化系数与数据的对应关系,具体要求见下一节。 | 69 | - 数据输入的转置必须只交换最后两维,其他维度的顺序保持不变。例如,形状为`(B, M, K)`的输入转置为`(B, K, M)`。交换批次维度等其他转置方式不属于本规则的适用范围。对应的`scale`转换也须保持量化系数与数据的对应关系,具体要求见下一节。 |
| 70 | - `pertoken_scale`是可选输入,未连接时不处理该路`scale`。 | 70 | - `pertoken_scale`是可选输入,未连接时不处理该路`scale`。 |
| @@ -102,8 +102,8 @@ | |||
| 102 | 102 | ||
| 103 | | 产品 | 普通规则 | Limit补充规则 | | 103 | | 产品 | 普通规则 | Limit补充规则 | |
| 104 | | ---- | ---- | ---- | | 104 | | ---- | ---- | ---- | |
| 105 | -| Ascend 950PR&950DT系列产品 | `x1`、`x2`均不受本节65535维度阈值限制,满足其他使用约束时执行转置融合 | 跳过本补充规则,转置融合由普通规则处理 | | 105 | +| <term>Ascend 950PR&950DT系列产品</term> | `x1`、`x2`均不受本节65535维度阈值限制,满足其他使用约束时执行转置融合 | 跳过本补充规则,转置融合由普通规则处理 | |
| 106 | -| Atlas A2训练系列产品/Atlas A2推理系列产品、Atlas A3训练系列产品/Atlas A3推理系列产品 | `x1`要求`outer` ≤ 65535;`x2`要求`outer` ≤ 65535或格式为`FRACTAL_NZ` | 分别检查`x1`、`x2`;仅处理满足0 < `outer` ≤ 65535且`inner` > 65535的支路 | | 106 | +| <term>Atlas A2系列产品</term>、<term>Atlas A3系列产品</term> | `x1`要求`outer` ≤ 65535;`x2`要求`outer` ≤ 65535或格式为`FRACTAL_NZ` | 分别检查`x1`、`x2`;仅处理满足0 < `outer` ≤ 65535且`inner` > 65535的支路 | |
| 107 | 107 | ||
| 108 | Limit规则的维度条件同样适用于`FRACTAL_NZ`格式,不因格式而放宽。参与判断的`outer`或`inner`为未知值`-1`时,该支路不满足Limit规则的维度条件。 | 108 | Limit规则的维度条件同样适用于`FRACTAL_NZ`格式,不因格式而放宽。参与判断的`outer`或`inner`为未知值`-1`时,该支路不满足Limit规则的维度条件。 |
| 109 | 109 | ||
| @@ -117,13 +117,13 @@ Limit规则的维度条件同样适用于`FRACTAL_NZ`格式,不因格式而放 | |||
| 117 | 产品的数据类型与格式支持范围参见[QuantBatchMatmulV3算子说明](../README.md)。本规则按平台指令能力判断融合条件,算子支持某产品并不表示该产品支持本规则的全部融合模式。 | 117 | 产品的数据类型与格式支持范围参见[QuantBatchMatmulV3算子说明](../README.md)。本规则按平台指令能力判断融合条件,算子支持某产品并不表示该产品支持本规则的全部融合模式。 |
| 118 | 118 | ||
| 119 | <!-- npu="910b" id1 --> | 119 | <!-- npu="910b" id1 --> |
| 120 | -Atlas A2训练系列产品/Atlas A2推理系列产品 | 120 | +- <term>Atlas A2系列产品</term> |
| 121 | <!-- end id1 --> | 121 | <!-- end id1 --> |
| 122 | 122 | ||
| 123 | <!-- npu="A3" id2 --> | 123 | <!-- npu="A3" id2 --> |
| 124 | -Atlas A3训练系列产品/Atlas A3推理系列产品 | 124 | +- <term>Atlas A3系列产品</term> |
| 125 | <!-- end id2 --> | 125 | <!-- end id2 --> |
| 126 | 126 | ||
| 127 | <!-- npu="950" id3 --> | 127 | <!-- npu="950" id3 --> |
| 128 | -Ascend 950PR&950DT系列产品 | 128 | +- <term>Ascend 950PR&950DT系列产品</term> |
| 129 | <!-- end id3 --> | 129 | <!-- end id3 --> |
| @@ -1,81 +1,79 @@ | |||
| 1 | # QuantBatchMatmulV3TransposeFusionPass | 1 | # QuantBatchMatmulV3TransposeFusionPass |
| 2 | 2 | ||
| 3 | -<!-- md-trans-meta sourceCommit=unknown translatedAt=2026-10-12T17:10:14.145Z pushedAt=2026-10-13T01:47:29.772Z --> | ||
| 4 | - | ||
| 5 | ## Fusion Patterns | 3 | ## Fusion Patterns |
| 6 | 4 | ||
| 7 | -This fusion rule folds the explicit transpose in the `x1` or `x2` input branch of `QuantBatchMatmulV3` into the matmul computation, and can also process both branches at the same time. The recognized transpose node types are `Transpose` and `TransposeD`. When fusing the `x1` branch, invert the `transpose_x1` attribute of the `QuantBatchMatmulV3` node; when fusing the `x2` branch, invert the `transpose_x2` attribute of the node. Inverting means changing `false` to `true`, or `true` to `false`. | 5 | +This fusion rule folds explicit transposes on the `x1` or `x2` input branch of `QuantBatchMatmulV3` into matrix multiplication, and can process both branches together. It recognizes `Transpose` and `TransposeD` nodes. Fusing the `x1` branch inverts the node's `transpose_x1` attribute; fusing the `x2` branch inverts `transpose_x2`. Inversion changes `false` to `true` or `true` to `false`. |
| 8 | 6 | ||
| 9 | -In this document, "bypass" refers to the following connection adjustment: connect the input of a conversion node directly to the successor node of that conversion node, so that the current branch no longer passes through the conversion node. After the bypass, keep the conversion node if it still has other data consumers; otherwise, delete it. The successor node can be `QuantBatchMatmulV3` or the retained `Bitcast`. | 7 | +In this document, "bypass" means connecting a conversion node's input directly to its successor so that the current branch no longer passes through that conversion node. Retain the bypassed node if other data consumers still use it; otherwise, remove it. The successor can be `QuantBatchMatmulV3` or a retained `Bitcast`. |
| 10 | 8 | ||
| 11 | -The following figures assume that the [Input and Transpose Requirements](#input-and-transpose-requirements) and [Dimension and Platform Restrictions](#dimension-and-platform-constraints) are met. The variables in the figures are described as follows: | 9 | +The diagrams below assume that the [Input and Transpose Requirements](#input-and-transpose-requirements) and [Dimension and Platform Constraints](#dimension-and-platform-constraints) are satisfied. Variables used in the diagrams are defined below: |
| 12 | 10 | ||
| 13 | -- **Input Mapping**: Shows `x1`, `x2`, and the two `scale` inputs respectively. `x1_scale` denotes the input `pertoken_scale` (IR index 5), and `x2_scale` denotes the input `scale` (IR index 2); these two names are aliases used in the figures, not new operator parameters or operator attributes. | 11 | +- **Input mapping**: The diagrams show `x1`, `x2`, and their two `scale` branches. `x1_scale` denotes input `pertoken_scale` (IR index 5), and `x2_scale` denotes input `scale` (IR index 2). These names are diagram aliases, not additional operator parameters or attributes. |
| 14 | -- **Symbol Definition**: `t1` and `t2` denote the values of the `transpose_x1` and `transpose_x2` attributes of `QuantBatchMatmulV3` before fusion, respectively; `!` denotes logical inversion. | 12 | +- **Symbols**: `t1` and `t2` denote the `transpose_x1` and `transpose_x2` attributes of `QuantBatchMatmulV3` before fusion. `!` denotes logical negation. |
| 15 | -- **Source Tensor**: A source tensor with the same name before and after fusion refers to the same tensor. | 13 | +- **Source tensors**: A source tensor with the same name before and after fusion denotes the same tensor. |
| 16 | -- **Omitted Content**: The figures omit `bias`, `offset`, the input specifying the transpose dimension order, and the input specifying the shape after reshape, and show only the data connections relevant to fusion. | 14 | +- **Omitted items**: The diagrams omit `bias`, `offset`, transpose-permutation inputs, and target-shape inputs. Only connections relevant to fusion are shown. |
| 17 | 15 | ||
| 18 | -### Fusion-Only x1 Side Transpose | 16 | +### Fusing Only the x1 Transpose |
| 19 | 17 | ||
| 20 | -When there exists one transpose node in each of the `x1` branch and the `x1_scale` branch, bypass respectively the two transpose nodes, and connect their inputs directly to the `x1` and `pertoken_scale` input ports of `QuantBatchMatmulV3`. Meanwhile, invert the `transpose_x1` attribute of the `QuantBatchMatmulV3` node. The connections of the `x2` and `scale` inputs and the `transpose_x2` attribute of the node remain unchanged. | 18 | +When the `x1` and `x1_scale` branches each contain a transpose node, bypass both transpose nodes and connect their inputs directly to the `x1` and `pertoken_scale` ports of `QuantBatchMatmulV3`, respectively. Invert the node's `transpose_x1` attribute. Keep the `x2` and `scale` connections and the `transpose_x2` attribute unchanged. |
| 21 | 19 | ||
| 22 | - | 20 | + |
| 23 | 21 | ||
| 24 | -### Fusion-only x2 Side Transpose | 22 | +### Fusing Only the x2 Transpose |
| 25 | 23 | ||
| 26 | -When there exists one transpose node each in the `x2` branch and the `x2_scale` branch, bypass the two transpose nodes respectively, and connect their inputs directly to the `x2` and `scale` input ports of `QuantBatchMatmulV3`. Also, invert the `transpose_x2` attribute of the `QuantBatchMatmulV3` node. Keep the connections of the `x1` and `pertoken_scale` inputs and the `transpose_x1` attribute of the node unchanged. | 24 | +When the `x2` and `x2_scale` branches each contain a transpose node, bypass both transpose nodes and connect their inputs directly to the `x2` and `scale` ports of `QuantBatchMatmulV3`, respectively. Invert the node's `transpose_x2` attribute. Keep the `x1` and `pertoken_scale` connections and the `transpose_x1` attribute unchanged. |
| 27 | 25 | ||
| 28 | - | 26 | + |
| 29 | 27 | ||
| 30 | -### Simultaneously Fusing Transposes on Both Sides | 28 | +### Fusing Transposes on Both Sides |
| 31 | 29 | ||
| 32 | -When there exists one transpose node in each of the four branches of `x1`, `x2`, `x1_scale`, and `x2_scale`, and both the `x1` and `x2` sides meet the usage constraints listed at the beginning of this section, the four transpose nodes are bypassed respectively. The inputs of the four transpose nodes connect to the `x1`, `x2`, `pertoken_scale`, and `scale` input ports of `QuantBatchMatmulV3`, respectively; the `transpose_x1` and `transpose_x2` attributes of the `QuantBatchMatmulV3` node are both inverted. Matching only the structure in the figure is insufficient to trigger fusion. Constraints such as data type, dimension, and platform capability must also be satisfied. | 30 | +When each of the `x1`, `x2`, `x1_scale`, and `x2_scale` branches contains a transpose node, and both data branches meet the constraints listed at the start of this section, bypass all four transpose nodes. Connect their inputs to the `x1`, `x2`, `pertoken_scale`, and `scale` ports of `QuantBatchMatmulV3`, respectively, and invert both `transpose_x1` and `transpose_x2`. Matching the illustrated structure alone is insufficient: the data type, dimension, and platform capability constraints must also be satisfied. |
| 33 | 31 | ||
| 34 | - | 32 | + |
| 35 | 33 | ||
| 36 | -### Reshape Fusion for Normal Scale | 34 | +### Reshape Fusion for Normal scale |
| 37 | 35 | ||
| 38 | -When there exists one transpose node in each of the `x1` and `x2` input branches, and one `Reshape` equivalent to the required transpose in each of the two `scale` branches, bypass the two transpose nodes and the two `Reshape` nodes respectively. The four branches connect to their corresponding `QuantBatchMatmulV3` input ports, and the `transpose_x1` and `transpose_x2` attributes of the `QuantBatchMatmulV3` node are both inverted. For identification requirements of static normal `scale`, see [Reshape Requirements for scale](#reshape-requirements-for-scale). The figure shows the case where both sides can be fused. You can also fuse only the side that satisfies the constraints. | 36 | +When each data branch contains a transpose node and each `scale` branch contains a `Reshape` equivalent to the required transpose, bypass both transpose nodes and both `Reshape` nodes. Connect the four branches to their corresponding `QuantBatchMatmulV3` input ports and invert both `transpose_x1` and `transpose_x2`. For static normal `scale` requirements, see [Reshape Requirements for scale](#reshape-requirements-for-scale). The figure shows fusion on both sides; only the side meeting the constraints can also be fused. |
| 39 | 37 | ||
| 40 | - | 38 | + |
| 41 | 39 | ||
| 42 | -### Reshape Fusion of MX Scale | 40 | +### Reshape Fusion for MX scale |
| 43 | 41 | ||
| 44 | -Taking the `x2` side as an example: the `x2` input is preceded by `Transpose` or `TransposeD`, `x2_scale` is of the `FLOAT8_E8M0` type, and its `Reshape` changes (1, N, 2) to (N, 1, 2). During fusion, the transpose node in the `x2` branch and the `Reshape` in the `x2_scale` branch are bypassed, and their inputs are connected to the `x2` and `scale` input ports of `QuantBatchMatmulV3` respectively; the `transpose_x2` attribute of the `QuantBatchMatmulV3` node is inverted. The connections of `x1` and `x1_scale` remain unchanged. The same processing logic applies when the `x1` side meets the corresponding conditions. | 42 | +For example, on the `x2` side, the data input is preceded by `Transpose` or `TransposeD`, `x2_scale` has type `FLOAT8_E8M0`, and its `Reshape` changes (1, N, 2) to (N, 1, 2). Bypass the data transpose and scale `Reshape`, connect their inputs to the `x2` and `scale` ports of `QuantBatchMatmulV3`, respectively, and invert `transpose_x2`. Keep the `x1` and `x1_scale` connections unchanged. Apply the same logic on the `x1` side when its corresponding conditions are met. |
| 45 | 43 | ||
| 46 | - | 44 | + |
| 47 | 45 | ||
| 48 | ### Transpose Fusion Retaining Bitcast | 46 | ### Transpose Fusion Retaining Bitcast |
| 49 | 47 | ||
| 50 | -`Bitcast` reinterprets the bit representation of data as the target data type. When each of the four-way inputs is connected as "transpose node → `Bitcast` → `QuantBatchMatmulV3`", the fusion directly connects the inputs of the four transpose nodes to the subsequent `Bitcast` respectively, retains the four `Bitcast` nodes and their connections to `QuantBatchMatmulV3`, and performs attribute inversion on the `transpose_x1` and `transpose_x2` attributes of the `QuantBatchMatmulV3` node. After fusion, the data type of the `Bitcast` output remains unchanged. | 48 | +`Bitcast` reinterprets the data bits as the target data type. When each of the four branches follows "transpose node → `Bitcast` → `QuantBatchMatmulV3`", connect each transpose node's input directly to its following `Bitcast`. Retain all four `Bitcast` nodes and their connections to `QuantBatchMatmulV3`, and invert both `transpose_x1` and `transpose_x2`. The output data type of each `Bitcast` remains unchanged. |
| 51 | 49 | ||
| 52 | - | 50 | + |
| 53 | 51 | ||
| 54 | -The input branches are not required to contain `Bitcast` at the same time. A transpose node can connect to the matmul directly or through a `Bitcast`; fusion across multiple consecutive `Bitcast` nodes is not supported. For the `scale` branch, the node before the `Bitcast` can also be a `Reshape` that meets the requirements described below. For platform restrictions and the applicability boundary of dynamic `Reshape`, see the usage constraints. | 52 | +The input branches do not all have to contain `Bitcast`. A transpose can connect to matrix multiplication directly or through one `Bitcast`; fusion across multiple consecutive `Bitcast` nodes is unsupported. On a `scale` branch, the node preceding `Bitcast` can also be a `Reshape` meeting the requirements below. For platform restrictions and the scope of dynamic `Reshape` support, see Constraints. |
| 55 | 53 | ||
| 56 | -### Reshape Fusion for Dynamic Scale | 54 | +### Reshape Fusion for Dynamic scale |
| 57 | 55 | ||
| 58 | -If `x1` or `x2` contains unknown dimensions, handle it as a dynamic scenario. Taking the `x2` side as an example, the fusion bypasses the transpose node on the `x2` branch and the `Reshape` on the `x2_scale` branch, connects their inputs to the `x2` and `scale` input ports of `QuantBatchMatmulV3` respectively, and will invert the `transpose_x2` attribute of the `QuantBatchMatmulV3` node. `x1` and `x1_scale` keep their original connections. The dynamic scenario does not perform the dimension condition check for a static `Reshape`, but the input graph must still ensure transpose equivalence between the `Reshape` and the required `scale`. | 56 | +Treat the scenario as dynamic if `x1` or `x2` contains an unknown dimension. For example, on the `x2` side, bypass the data transpose and `x2_scale` `Reshape`, connect their inputs to the `x2` and `scale` ports of `QuantBatchMatmulV3`, respectively, and invert `transpose_x2`. Keep `x1` and `x1_scale` unchanged. Static `Reshape` dimension checks are not performed in dynamic scenarios, but the input graph must still ensure that the `Reshape` is equivalent to the required scale transpose. |
| 59 | 57 | ||
| 60 | - | 58 | + |
| 61 | 59 | ||
| 62 | -The `scale` conversion nodes in each figure are the specific structures of the illustrated scenario. A conversion node is not required on every `scale` branch. A `scale` branch is processed only when the `Transpose` or `TransposeD` of the corresponding data input is fused; when `pertoken_scale` is not connected, that input is not processed. A bypassed node is retained if it still has other data consumers; otherwise, it is removed from the graph. | 60 | +The scale conversion nodes shown in each figure are specific to that example. A conversion node is not required on every `scale` branch. Process a `scale` branch only when the corresponding data input's `Transpose` or `TransposeD` is fused; do not process `pertoken_scale` if it is unconnected. Retain a bypassed node if other data consumers still use it; otherwise, remove it from the graph. |
| 63 | 61 | ||
| 64 | ## Constraints | 62 | ## Constraints |
| 65 | 63 | ||
| 66 | ### Input and Transpose Requirements | 64 | ### Input and Transpose Requirements |
| 67 | 65 | ||
| 68 | -- The original `shape` of both `x1` and `x2` must have at least 2 dimensions. At least one data input must connect to `Transpose` or `TransposeD`; otherwise, fusion is not triggered. A data input preceded only by `Reshape` does not trigger fusion. | 66 | +- The original shapes of `x1` and `x2` must each have at least two dimensions. At least one data input must have a preceding `Transpose` or `TransposeD`; otherwise, fusion is not triggered. A `Reshape` alone before a data input cannot trigger fusion. |
| 69 | -- The data types of both `x1` and `x2` support `INT8`, `INT4`, `FLOAT8_E4M3FN`, `FLOAT8_E5M2`, `HIFLOAT8`, and `FLOAT4_E2M1`. The output data types support `INT8`, `FLOAT16`, `BF16`, `INT32`, and `FLOAT32`. The actual combinations must also meet the operator constraints of the target product. | 67 | +- `x1` and `x2` each support `INT8`, `INT4`, `FLOAT8_E4M3FN`, `FLOAT8_E5M2`, `HIFLOAT8`, and `FLOAT4_E2M1`. The output supports `INT8`, `FLOAT16`, `BFLOAT16`, `INT32`, and `FLOAT32`. The actual combination must also satisfy the target product's operator constraints. |
| 70 | -- Dynamic `shape` is supported. If the `shape` of `x1` or `x2` contains unknown dimensions, it is processed as a dynamic scenario. | 68 | +- Dynamic shapes are supported. If the shape of `x1` or `x2` contains an unknown dimension, the case is treated as dynamic. |
| 71 | -- The transpose of a data input must only swap the last two dimensions, with the order of other dimensions unchanged. For example, an input of shape `(B, M, K)` is transposed to `(B, K, M)`. Other transposes, such as swapping the batch dimension, are out of the scope of this rule. The corresponding `scale` conversion must also maintain the correspondence between quantization coefficients and data. For details, see the next section. | 69 | +- A data transpose must swap only the last two dimensions and preserve the order of all other dimensions, for example, `(B, M, K)` → `(B, K, M)`. Other permutations, such as swapping batch dimensions, are outside this rule's scope. The corresponding scale transformation must also preserve the correspondence between quantization coefficients and data. See the next section. |
| 72 | -- `pertoken_scale` is an optional input. If it is not connected, the corresponding `scale` is not processed. | 70 | +- `pertoken_scale` is optional. Its scale branch is not processed when unconnected. |
| 73 | 71 | ||
| 74 | -### Restrictions in the Bitcast Scenario | 72 | +### Restrictions for Bitcast Patterns |
| 75 | 73 | ||
| 76 | -On some platforms, the matrix multiplication instruction supports mixed INT8 and INT4 inputs. On such platforms, if one of the four-way inputs matches the structure in the following table, this rule retains the entire `QuantBatchMatmulV3` node and all of its input branches, and does not perform this transpose fusion. This restriction applies even if the current node does not use mixed INT8 and INT4 inputs. | 74 | +Some platforms support mixed INT8 and INT4 matrix multiplication inputs. On those platforms, if any of the four input branches matches a pattern in the table below, retain the entire `QuantBatchMatmulV3` node and all its input branches without performing this transpose fusion. This restriction applies even when the current node does not use mixed INT8 and INT4 inputs. |
| 77 | 75 | ||
| 78 | -| Input Branch | Connection Structure That Triggers the Preceding Constraint | | 76 | +| Input branch | Connection pattern that triggers the restrictions above | |
| 79 | | ---- | ---- | | 77 | | ---- | ---- | |
| 80 | | `x1`, `x2`, or either of the two `scale` branches | `Transpose` or `TransposeD` → `Bitcast` → `QuantBatchMatmulV3` | | 78 | | `x1`, `x2`, or either of the two `scale` branches | `Transpose` or `TransposeD` → `Bitcast` → `QuantBatchMatmulV3` | |
| 81 | | Either of the two `scale` branches | `Reshape` that meets the fusion requirements in the next section → `Bitcast` → `QuantBatchMatmulV3` | | 79 | | Either of the two `scale` branches | `Reshape` that meets the fusion requirements in the next section → `Bitcast` → `QuantBatchMatmulV3` | |
| @@ -104,8 +102,8 @@ Let `outer` denote the penultimate dimension of the original `shape` of the data | |||
| 104 | 102 | ||
| 105 | | Product | General Rule | Limit Supplementary Rule | | 103 | | Product | General Rule | Limit Supplementary Rule | |
| 106 | | ---- | ---- | ---- | | 104 | | ---- | ---- | ---- | |
| 107 | -| <term>Ascend 950PR&950DT products</term> | `x1` and `x2` are not subject to the 65535 dimension threshold in this section. Transpose fusion is performed when other usage constraints are met. | This supplementary rule is skipped, and transpose fusion is handled by the general rule. | | 105 | +| <term>Ascend 950PR&950DT products</term> | Neither `x1` nor `x2` is subject to this section's 65535-dimension threshold. Transpose fusion runs when the other constraints are met. | This supplementary rule is skipped; transpose fusion is handled by the regular rule. | |
| 108 | -| <term>Atlas A2 products</term> and <term>Atlas A3 products</term> | `x1` requires `outer` ≤ 65535; `x2` requires `outer` ≤ 65535 or the `FRACTAL_NZ` format. | Checks `x1` and `x2` respectively; processes only branches that meet 0 < `outer` ≤ 65535 and `inner` > 65535. | | 106 | +| <term>Atlas A2 products</term>, <term>Atlas A3 products</term> | `x1` requires `outer` ≤ 65535. `x2` requires `outer` ≤ 65535 or `FRACTAL_NZ` format. | Checks `x1` and `x2` independently and processes only branches with 0 < `outer` ≤ 65535 and `inner` > 65535. | |
| 109 | 107 | ||
| 110 | The dimension conditions of the Limit rule also apply to the `FRACTAL_NZ` format; they are not relaxed due to the format. If the `outer` or `inner` value involved in the check is the unknown value `-1`, the branch does not satisfy the dimension conditions of the Limit rule. | 108 | The dimension conditions of the Limit rule also apply to the `FRACTAL_NZ` format; they are not relaxed due to the format. If the `outer` or `inner` value involved in the check is the unknown value `-1`, the branch does not satisfy the dimension conditions of the Limit rule. |
| 111 | 109 | ||
| @@ -116,16 +114,16 @@ The dimension conditions of the Limit rule also apply to the `FRACTAL_NZ` format | |||
| 116 | 114 | ||
| 117 | ## Applicable Products | 115 | ## Applicable Products |
| 118 | 116 | ||
| 119 | -See [QuantBatchMatmulV3 Operator Description](../README.md) for the data types and formats supported by the product. This rule determines fusion conditions based on platform instruction capabilities. Operator support for a product does not mean that the product supports all fusion modes of this rule. | 117 | +See [QuantBatchMatmulV3 Operator Description](../README_en.md) for the data types and formats supported by the product. This rule determines fusion conditions based on platform instruction capabilities. Operator support for a product does not mean that the product supports all fusion modes of this rule. |
| 120 | 118 | ||
| 121 | <!-- npu="910b" id1 --> | 119 | <!-- npu="910b" id1 --> |
| 122 | -<term>Atlas A2 products</term> | 120 | +- <term>Atlas A2 products</term> |
| 123 | <!-- end id1 --> | 121 | <!-- end id1 --> |
| 124 | 122 | ||
| 125 | <!-- npu="A3" id2 --> | 123 | <!-- npu="A3" id2 --> |
| 126 | -<term>Atlas A3 products</term> | 124 | +- <term>Atlas A3 products</term> |
| 127 | <!-- end id2 --> | 125 | <!-- end id2 --> |
| 128 | 126 | ||
| 129 | <!-- npu="950" id3 --> | 127 | <!-- npu="950" id3 --> |
| 130 | -<term>Ascend 950PR&950DT products</term> | 128 | +- <term>Ascend 950PR&950DT products</term> |
| 131 | <!-- end id3 --> | 129 | <!-- end id3 --> |
| @@ -2,13 +2,13 @@ | |||
| 2 | 2 | ||
| 3 | ## 融合模式 | 3 | ## 融合模式 |
| 4 | 4 | ||
| 5 | -该融合将MatMul/MatMulV2/MatMulV3、BatchMatMul/BatchMatMulV2/BatchMatMulV3算子转换为QuantBatchMatMulV3算子,并插入scale为1.0的常量作为scale输入,bias输入、转置属性及输出y的数据类型保持不变。 | 5 | +该融合将MatMul/MatMulV2/MatMulV3、BatchMatMul/BatchMatMulV2/BatchMatMulV3算子转换为QuantBatchMatmulV3算子,并插入scale为1.0的常量作为scale输入,bias输入、转置属性及输出y的数据类型保持不变。 |
| 6 | 6 | ||
| 7 | - | 7 | + |
| 8 | 8 | ||
| 9 | ## 使用约束 | 9 | ## 使用约束 |
| 10 | 10 | ||
| 11 | -- MatMul类型包括BatchMatMul/BatchMatMulV2/MatMul/MatMulV2。 | 11 | +- MatMul类型包括MatMul、MatMulV2、MatMulV3、BatchMatMul、BatchMatMulV2、BatchMatMulV3。 |
| 12 | - 输入的数据类型支持HIFLOAT8。 | 12 | - 输入的数据类型支持HIFLOAT8。 |
| 13 | <!-- npu="950" id1 --> | 13 | <!-- npu="950" id1 --> |
| 14 | - <term>Ascend 950PR&950DT系列产品</term>:支持输入为INT8且输出为INT32的组合。 | 14 | - <term>Ascend 950PR&950DT系列产品</term>:支持输入为INT8且输出为INT32的组合。 |
| @@ -17,5 +17,5 @@ | |||
| 17 | ## 支持的型号 | 17 | ## 支持的型号 |
| 18 | 18 | ||
| 19 | <!-- npu="950" id2 --> | 19 | <!-- npu="950" id2 --> |
| 20 | -Ascend 950PR&950DT系列产品 | 20 | +- <term>Ascend 950PR&950DT系列产品</term> |
| 21 | <!-- end id2 --> | 21 | <!-- end id2 --> |
| @@ -1,17 +1,15 @@ | |||
| 1 | # ZZMatMulTOQBMMV3FusionPass | 1 | # ZZMatMulTOQBMMV3FusionPass |
| 2 | 2 | ||
| 3 | -<!-- md-trans-meta sourceCommit=unknown translatedAt=2026-10-12T17:10:34.905Z pushedAt=2026-10-13T01:47:29.775Z --> | 3 | +## Fusion Patterns |
| 4 | 4 | ||
| 5 | -## Fusion Pattern | 5 | +This fusion converts MatMul/MatMulV2/MatMulV3 and BatchMatMul/BatchMatMulV2/BatchMatMulV3 into QuantBatchMatmulV3, inserting a constant scale of 1.0. The bias input, transpose attributes, and output y data type are preserved. |
| 6 | 6 | ||
| 7 | -This fusion converts the MatMul/MatMulV2/MatMulV3 and BatchMatMul/BatchMatMulV2/BatchMatMulV3 operators into the QuantBatchMatMulV3 operator, and inserts a constant with a scale of 1.0 as the scale input. The bias input, transpose attribute, and data type of output y remain unchanged. | 7 | + |
| 8 | - | ||
| 9 | - | ||
| 10 | 8 | ||
| 11 | ## Constraints | 9 | ## Constraints |
| 12 | 10 | ||
| 13 | -- The MatMul type includes BatchMatMul/BatchMatMulV2/MatMul/MatMulV2. | 11 | +- MatMul types include MatMul, MatMulV2, MatMulV3, BatchMatMul, BatchMatMulV2, and BatchMatMulV3. |
| 14 | -- The input supports the HIFLOAT8 data type. | 12 | +- The supported input data type is HIFLOAT8. |
| 15 | <!-- npu="950" id1 --> | 13 | <!-- npu="950" id1 --> |
| 16 | - <term>Ascend 950PR&950DT products</term>: Support the combination of INT8 input and INT32 output. | 14 | - <term>Ascend 950PR&950DT products</term>: Support the combination of INT8 input and INT32 output. |
| 17 | <!-- end id1 --> | 15 | <!-- end id1 --> |
| @@ -19,5 +17,5 @@ This fusion converts the MatMul/MatMulV2/MatMulV3 and BatchMatMul/BatchMatMulV2/ | |||
| 19 | ## Applicable Products | 17 | ## Applicable Products |
| 20 | 18 | ||
| 21 | <!-- npu="950" id2 --> | 19 | <!-- npu="950" id2 --> |
| 22 | -<term>Ascend 950PR&950DT products</term> | 20 | +- <term>Ascend 950PR&950DT products</term> |
| 23 | <!-- end id2 --> | 21 | <!-- end id2 --> |
| @@ -1,6 +1,6 @@ | |||
| 1 | # aclnnQuantMatmulV3 | 1 | # aclnnQuantMatmulV3 |
| 2 | 2 | ||
| 3 | -**须知:该接口后续版本会废弃,请使用最新aclnnQuantMatmulV5接口。** | 3 | +**须知:该接口后续版本会废弃,请使用[aclnnQuantMatmulV5](../../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)接口。** |
| 4 | 4 | ||
| 5 | ## 产品支持情况 | 5 | ## 产品支持情况 |
| 6 | 6 | ||
| @@ -1,6 +1,6 @@ | |||
| 1 | # aclnnQuantMatmulV4 | 1 | # aclnnQuantMatmulV4 |
| 2 | 2 | ||
| 3 | -**须知:该接口后续版本会废弃,请使用最新aclnnQuantMatmulV5接口。** | 3 | +**须知:该接口后续版本会废弃,请使用[aclnnQuantMatmulV5](../../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)接口。** |
| 4 | 4 | ||
| 5 | ## 产品支持情况 | 5 | ## 产品支持情况 |
| 6 | 6 | ||
Mmatmul/quant_matmul_activation_quant/tests/README_en.md+40-30File content is under review, please try again later