Merged
docs(quant): 规范量化矩阵乘与参数转换算子的中文文档 #13862
panzhijaAdd on 15 days ago
docs(quant): 规范量化矩阵乘与参数转换算子的中文文档 #13862
Merged
panzhijaAdd on 15 days ago
Total: 48 files changed+6527-6708
@@ -3,134 +3,75 @@
3## 产品支持情况3## 产品支持情况
4 4 
5| 产品 | 是否支持 |5| 产品 | 是否支持 |
6-| ---- | :----:|6+| --- | :---: |
7-| Ascend 950PR&950DT系列产品 | √ |7+| <term>Ascend 950PR&950DT系列产品</term> | √ |
8-| Atlas A3系列产品 | × |8+| <term>Atlas A3系列产品</term> | × |
9-| Atlas A2系列产品 | × |9+| <term>Atlas A2系列产品</term> | × |
10-| Atlas 200I/500 A2推理产品 | × |10+| <term>Atlas 200I/500 A2推理产品</term> | × |
11-| Atlas推理系列产品 | × |11+| <term>Atlas推理系列产品</term> | × |
12-| Atlas训练系列产品 | × |12+| <term>Atlas训练系列产品</term> | × |
13 13 
14## 功能说明14## 功能说明
15 15 
16-- 算子功能:在micro-batch训练场景中,需要做micro-batch的梯度累积,会存在大量QuantBatchMatmul后接InplaceAdd的融合场景。QuantBatchMatmulInplaceAdd算子将上述算子融合起来,提高网络性能。实现量化矩阵乘计算和加法计算,基本功能为矩阵乘和加法的组合。16+- 算子功能:融合量化矩阵乘和原地加法,将矩阵乘结果累加到已有的y中,适用于微批次(micro-batch)训练中的梯度累积。
17 17 
18-- 计算公式:18+- 计算公式:支持MX(Microscaling,微缩放)量化和HIFLOAT8的T-T(pertensor-pertensor,逐张量对逐张量)量化,模式说明见[量化模式介绍](../../docs/zh/context/quant_mode_introduction.md)。
19 19 
20- - **MX量化:**20+ 以下公式中的$X_1$、$X_2$表示按转置属性解释后的矩阵,形状分别为$(M,K)$、$(K,N)$。M、N为结果矩阵的行数、列数,K为归约维度;$Y_{\mathrm{in}}$、$Y_{\mathrm{out}}$分别表示原地累加前后的y。
21+ 
22+ - MX量化:
21 23 
22 $$24 $$
23- y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n]25+ Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+
26+ \sum_{j=0}^{L-1}\left[
27+ \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right)
28+ S_{1,m,j}S_{2,j,n}\right]
24 $$29 $$
25 30 
26- 其中,$gsK$ 代表K轴的量化的block size即32,$x1Slice$ 代表 $x1$ 第m行长度为 $gsK$ 的向量,$x2Slice$ 代表 $x2$ 第n列长度为 $gsK$ 的向量,K轴均从 $j*gsK$ 起始切片,j的取值范围 [0, kLoops),kLoops=ceil($K_i$ / $gsK$),支持最后的切片长度不足 $gsK$。31+ 其中,$g_K=32$,$L=\lceil K/g_K\rceil$,$g_j=\min(g_K,K-jg_K)$。$S_1$、$S_2$为对应K分组的输入scale,最后一组可以不足32个元素。
27 32 
28- - **HIFLOAT8 T-T量化:**33+ - HIFLOAT8的T-T量化:
29 34 
30 $$35 $$
31- y[m,n] = (\sum_{k=0}^{K_i-1} (x1[m,k] * x2[k,n])) * (scale1 * scale2) + y[m,n]36+ Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+
37+ \left(\sum_{k=0}^{K-1}X_{1,m,k}X_{2,k,n}\right)S_1S_2
32 $$38 $$
33 39 
34- 其中,$scale1$ 和 $scale2$ 分别对应 $x1Scale$ 和 $x2Scale$,均为标量(shape为 $(1)$);当设置`transposeX1`/`transposeX2`时,公式中的 $x1$ 和 $x2$ 按转置后的视图参与计算。40+ 其中,$S_1$、$S_2$分别对应x1_scale、x2_scale,均为形状为$(1,)$的缩放因子。
35 41 
36## 参数说明42## 参数说明
37 43 
38-<table class="tg"><thead>44+| 参数名 | 输入/输出/属性 | 描述 | 数据类型 | 数据格式 |
39- <tr>45+| --- | --- | --- | --- | --- |
40- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">参数名</span></th>46+| x1 | 输入 | 矩阵乘的左矩阵,对应公式中的X₁。 | FLOAT8_E4M3FN、FLOAT8_E5M2、HIFLOAT8、FLOAT4_E2M1 | ND |
41- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">输入/输出/属性</span></th>47+| x2 | 输入 | 矩阵乘的右矩阵,对应公式中的X₂。 | FLOAT8_E4M3FN、FLOAT8_E5M2、HIFLOAT8、FLOAT4_E2M1 | ND |
42- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">描述</span></th>48+| x2_scale | 输入 | x2的量化缩放因子,对应公式中的S₂。MX量化使用FLOAT8_E8M0,HIFLOAT8的T-T量化使用FLOAT32。 | FLOAT8_E8M0、FLOAT32 | ND |
43- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据类型</span></th>49+| y | 输入 | 参与原地累加的已有结果,对应公式中的Y_in。 | FLOAT32 | ND |
44- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据格式</span></th>50+| x1_scale | 可选输入 | x1的量化缩放因子,对应公式中的S₁。当前支持的MX和HIFLOAT8的T-T量化场景均须提供,数据类型与x2_scale一致。 | FLOAT8_E8M0、FLOAT32 | ND |
45- </tr></thead>51+| y | 输出 | 原地累加后的结果,对应公式中的Y_out,与输入y共用存储。 | FLOAT32 | ND |
46-<tbody>52+| transpose_x1 | 可选属性 | <ul><li>是否转置x1的最后两维:false表示不转置,true表示转置。</li><li>默认值为false;当前支持的计算场景须设为true。</li></ul> | BOOL | - |
47- <tr>53+| transpose_x2 | 可选属性 | <ul><li>是否转置x2的最后两维:false表示不转置,true表示转置。</li><li>默认值为false;当前支持的计算场景须保持false。</li></ul> | BOOL | - |
48- <td class="tg-22a9">x1</td>54+| group_size | 可选属性 | <ul><li>M、N、K方向量化分组大小的编码值,各分量占16位;支持的分组组合见<a href="#约束说明">约束说明</a>。</li><li>默认值为0。</li></ul> | INT64 | - |
49- <td class="tg-22a9">输入</td>
50- <td class="tg-22a9">矩阵乘运算中的左矩阵,公式中的输入x1。</td>
51- <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td>
52- <td class="tg-22a9">ND</td>
53- </tr>
54- <tr>
55- <td class="tg-22a9">x2</td>
56- <td class="tg-22a9">输入</td>
57- <td class="tg-22a9">矩阵乘运算中的右矩阵,公式中的输入x2。</td>
58- <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td>
59- <td class="tg-22a9">ND</td>
60- </tr>
61- <tr>
62- <td class="tg-22a9">x1Scale</td>
63- <td class="tg-22a9">可选输入</td>
64- <td class="tg-22a9">量化参数中由x1量化引入的缩放因子,对应公式的scale1。</td>
65- <td class="tg-22a9">FLOAT8_E8M0(MX)、FLOAT32(HIFLOAT8 TT)</td>
66- <td class="tg-22a9">ND</td>
67- </tr>
68- <tr>
69- <td class="tg-22a9">x2Scale</td>
70- <td class="tg-22a9">输入</td>
71- <td class="tg-22a9">量化参数中由x2量化引入的缩放因子,对应公式的scale2。</td>
72- <td class="tg-22a9">FLOAT8_E8M0(MX)、FLOAT32(HIFLOAT8 TT)</td>
73- <td class="tg-22a9">ND</td>
74- </tr>
75- <tr>
76- <td class="tg-22a9">yRef</td>
77- <td class="tg-22a9">输入输出</td>
78- <td class="tg-22a9">矩阵乘运算与加法累加后的结果,对应公式中的输入输出y。</td>
79- <td class="tg-22a9">FLOAT32</td>
80- <td class="tg-22a9">ND</td>
81- </tr>
82- <tr>
83- <td class="tg-22a9">transposeX1</td>
84- <td class="tg-22a9">输入</td>
85- <td class="tg-22a9">表示x1的输入shape是否转置。</td>
86- <td class="tg-22a9">bool</td>
87- <td class="tg-22a9">-</td>
88- </tr>
89- <tr>
90- <td class="tg-22a9">transposeX2</td>
91- <td class="tg-22a9">输入</td>
92- <td class="tg-22a9">表示x2的输入shape是否转置。</td>
93- <td class="tg-22a9">bool</td>
94- <td class="tg-22a9">-</td>
95- </tr>
96- <tr>
97- <td class="tg-22a9">groupSize</td>
98- <td class="tg-22a9">输入</td>
99- <td class="tg-22a9">整数型参数,用于输入m、n、k方向上的量化分组大小。</td>
100- <td class="tg-22a9">INT64</td>
101- <td class="tg-22a9">-</td>
102- </tr>
103-</tbody></table>
104- 
105-- <term>Ascend 950PR&950DT系列产品</term>:
106- - x1、x2支持FLOAT8_E4M3FN、FLOAT8_E5M2、HIFLOAT8、FLOAT4_E2M1数据类型。
107- - x1Scale、x2Scale支持FLOAT8_E8M0(MX)和FLOAT32(HIFLOAT8 TT)数据类型。
108- - yRef只支持FLOAT32数据类型。
109- - 当前仅支持transposeX1=true且transposeX2=false。
110- - groupSize由groupSizeM、groupSizeN、groupSizeK拼接组成:groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32。
111 55 
112## 约束说明56## 约束说明
113 57 
114-- 支持连续tensor,[非连续tensor](../../docs/zh/context/non_contiguous_tensor.md)支持转置场景。58+- 当前仅支持transpose_x1=true且transpose_x2=false。输入x1、x2的逻辑形状分别为`(K, M)`、`(K, N)`,y为`(M, N)`。
115-- groupSize相关约束:传入的groupSize内部会按公式分解得到groupSizeM、groupSizeN、groupSizeK,当其中有1个或多个为0时,会根据x1/x2/x1Scale/x2Scale输入shape重新推断。59+- group_size按`group_size_k | (group_size_n << 16) | (group_size_m << 32)`编码。分量为0时由对应输入及scale的形状推导;推导后须满足对应量化模式的分组要求。
116-- 动态量化(MX量化)场景约束:60+- MX量化:
117- - 输入和输出支持以下数据类型组合:x1、x2为FLOAT8_E5M2/FLOAT8_E4M3FN,x1Scale、x2Scale为FLOAT8_E8M0,yRef为FLOAT32。61+ - x1、x2可分别为FLOAT8_E4M3FN或FLOAT8_E5M2,也可同时为FLOAT4_E2M1;不支持FP4与FP8混合输入,不支持FLOAT4_E1M2。
118- - 输入和输出还支持以下数据类型组合:x1、x2均为FLOAT4_E2M1,x1Scale、x2Scale均为FLOAT8_E8M0,yRef为FLOAT32。不支持FLOAT4_E1M2输入,不支持FP4与FP8混合输入。62+ - x1_scale、x2_scale均为FLOAT8_E8M0,形状分别为`(ceil(K/64), M, 2)`、`(ceil(K/64), N, 2)`,其中ceil表示向上取整。
119- - x1 shape为(k, m),x2 shape为(k, n),x1Scale shape为(ceil(k/64), m, 2),x2Scale shape为(ceil(k/64), n, 2),yRef shape为(m, n),[gsM, gsN, gsK] 为 [1, 1, 32],groupSize为32。63+ - 最终分组大小为`[group_size_m, group_size_n, group_size_k]=[1, 1, 32]`。可显式设置group_size=4295032864;设置为32时,M、N方向的分组大小由形状推导。
120- - MXFP4量化场景额外约束:64+ - MXFP4场景下,M、N须为非负偶数。x1、x2采用4位打包存储,每字节保存两个FLOAT4_E2M1元素;连续输入对应的字节存储形状分别为`(K, M/2)`、`(K, N/2)`。
121- - m、n必须为非负偶数。65+ - MXFP4场景下,当M、N、K均大于0时,还须同时满足$M\times K>2$、$N\times K>2$。
122- - 上述shape以逻辑元素数表示。x1、x2采用4-bit打包存储,一个字节保存两个FLOAT4_E2M1编码。连续输入x1、x2的逻辑shape分别为(k, m)、(k, n),对应的字节存储shape分别为(k, m/2)、(k, n/2)。66+- HIFLOAT8的T-T量化:
123- - 当m、n、k均大于0时,必须同时满足m × k > 2、n × k > 2。67+ - x1、x2均为HIFLOAT8,x1_scale、x2_scale均为FLOAT32且形状为`(1,)`。
124-- HIFLOAT8 T-T场景约束:68+ - group_size须为0。
125- - 输入和输出支持以下数据类型组合:x1、x2为HIFLOAT8,x1Scale、x2Scale为FLOAT32,yRef为FLOAT32。
126- - x1 shape为(k, m),x2 shape为(k, n),yRef shape为(m, n)。
127- - x1Scale shape为(1),x2Scale shape为(1),groupSize必须为0。
128- - 不支持bias、offset参数。
129 69 
130## 调用说明70## 调用说明
131 71 
132| 调用方式 | 样例代码 | 说明 |72| 调用方式 | 样例代码 | 说明 |
133-| ---------------- | --------------------------- | --------------------------------------------------- |73+| --- | --- | --- |
134-| aclnn接口 | [test_aclnn_quant_batch_matmul_inplace_add_mxfp8](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | 展示mxFP8量化场景的调用方式,其中`x1/x2`为FLOAT8、`x1Scale/x2Scale`为FLOAT8_E8M0,`groupSize=32`。 |74+| aclnn API | [MXFP8样例](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | 通过[aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd.md)调用,展示MXFP8量化。 |
135-| aclnn接口 | <a href="examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp">test_aclnn_quant_batch_matmul_inplace_add_mxfp4</a> | 展示MXFP4量化场景的调用方式,其中x1、x2为打包存储的FLOAT4_E2M1,x1Scale、x2Scale为FLOAT8_E8M0,yRef为FLOAT32,groupSize=32。 |75+| aclnn API | [MXFP4样例](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp) | 通过[aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd.md)调用,展示打包存储的FLOAT4_E2M1输入。 |
136-| aclnn接口 | [test_aclnn_quant_batch_matmul_inplace_add_TT](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | 展示当前支持组合`transposeX1=true`、`transposeX2=false`下HIFLOAT8 TT场景的调用方式。 |76+| aclnn API | [HIFLOAT8的T-T样例](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | 通过[aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd.md)调用,展示逐张量量化。 |
77+| GE图模式 | - | 通过[QuantBatchMatmulInplaceAdd算子IR](op_graph/quant_batch_matmul_inplace_add_proto.h)构图调用。 |
@@ -5,134 +5,75 @@ English | [简体中文](./README.md)
5## Applicable Products5## Applicable Products
6 6 
7| Product | Support |7| Product | Support |
8-| ---- | :----:|8+| --- | :---: |
9-| Ascend 950PR&950DT products | √ |9+| <term>Ascend 950PR&950DT products</term> | √ |
10-| Atlas A3 products | × |10+| <term>Atlas A3 products</term> | × |
11-| Atlas A2 products | × |11+| <term>Atlas A2 products</term> | × |
12-| Atlas 200I/500 A2 inference products | × |12+| <term>Atlas 200I/500 A2 inference products</term> | × |
13-| Atlas inference products | × |13+| <term>Atlas inference products</term> | × |
14-| Atlas training products | × |14+| <term>Atlas training products</term> | × |
15 15 
16## Function16## Function
17 17 
18-- Operator function: in the micro-batch training scenario, gradient accumulation of micro-batches is required, and a large number of fusion scenarios where QuantBatchMatmul is followed by InplaceAdd exist. The QuantBatchMatmulInplaceAdd operator fuses the preceding operators to improve network performance. It implements quantized matrix multiplication and addition, and its basic function is a combination of matrix multiplication and addition.18+- Operator function: Fuses quantized matrix multiplication and in-place addition, accumulating the matrix product into an existing y. This supports gradient accumulation during micro-batch training.
19 19 
20-- Formula:20+- Formulas: Supports MX (Microscaling) quantization and HIFLOAT8 T-T (per-tensor–per-tensor) quantization. See [Introduction to Quantization Modes](../../docs/en/context/quant_mode_introduction.md).
21 21 
22- - **MX quantization:**22+ In the following formulas, $X_1$ and $X_2$ are the matrices interpreted according to the transpose attributes, with shapes $(M,K)$ and $(K,N)$. M and N are the result matrix dimensions, and K is the reduction dimension. $Y_{\mathrm{in}}$ and $Y_{\mathrm{out}}$ denote y before and after in-place accumulation.
23+ 
24+ - MX quantization:
23 25 
24 $$26 $$
25- y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n]27+ Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+
28+ \sum_{j=0}^{L-1}\left[
29+ \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right)
30+ S_{1,m,j}S_{2,j,n}\right]
26 $$31 $$
27 32 
28- where $gsK$ denotes the quantization block size of the K axis, that is, 32; $x1Slice$ denotes the vector of length $gsK$ in row m of $x1$; $x2Slice$ denotes the vector of length $gsK$ in column n of $x2$. Slicing on the K axis starts from $j*gsK$, the value range of j is [0, kLoops), kLoops=ceil($K_i$ / $gsK$), and the case where the length of the last slice is less than $gsK$ is supported.33+ Here, $g_K=32$, $L=\lceil K/g_K\rceil$, and $g_j=\min(g_K,K-jg_K)$. $S_1$ and $S_2$ are the input scales for the corresponding K group. The last group may contain fewer than 32 elements.
29 34 
30- - **HIFLOAT8 T-T quantization:**35+ - HIFLOAT8 T-T quantization:
31 36 
32 $$37 $$
33- y[m,n] = (\sum_{k=0}^{K_i-1} (x1[m,k] * x2[k,n])) * (scale1 * scale2) + y[m,n]38+ Y_{\mathrm{out},m,n}=Y_{\mathrm{in},m,n}+
39+ \left(\sum_{k=0}^{K-1}X_{1,m,k}X_{2,k,n}\right)S_1S_2
34 $$40 $$
35 41 
36- where $scale1$ and $scale2$ correspond to $x1Scale$ and $x2Scale$ respectively, both being scalars (shape $(1)$). When `transposeX1`/`transposeX2` is set, the $x1$ and $x2$ in the formula participate in the computation with the transposed view.42+ Here, $S_1$ and $S_2$ correspond to x1_scale and x2_scale, respectively, and both have shape $(1,)$.
37 43 
38## Parameters44## Parameters
39 45 
40-<table class="tg"><thead>46+| Parameter | Input/Output/Attribute | Description | Data Type | Data Format |
41- <tr>47+| --- | --- | --- | --- | --- |
42- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Parameter name</span></th>48+| x1 | Input | Left matrix, corresponding to X₁ in the formulas. | FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1 | ND |
43- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Input/Output/Attribute</span></th>49+| x2 | Input | Right matrix, corresponding to X₂ in the formulas. | FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1 | ND |
44- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Description</span></th>50+| x2_scale | Input | Quantization scale for x2, corresponding to S₂ in the formulas. Uses FLOAT8_E8M0 for MX quantization and FLOAT32 for HIFLOAT8 T-T quantization. | FLOAT8_E8M0, FLOAT32 | ND |
45- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data type</span></th>51+| y | Input | Existing result used for in-place accumulation, corresponding to Y_in in the formulas. | FLOAT32 | ND |
46- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data format</span></th>52+| x1_scale | Optional input | Quantization scale for x1, corresponding to S₁ in the formulas. Required for both currently supported modes, MX and HIFLOAT8 T-T. Its data type must match x2_scale. | FLOAT8_E8M0, FLOAT32 | ND |
47- </tr></thead>53+| y | Output | Result after in-place accumulation, corresponding to Y_out in the formulas. Shares storage with input y. | FLOAT32 | ND |
48-<tbody>54+| transpose_x1 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x1: false disables transposition and true enables it.</li><li>Defaults to false; must be true for currently supported computations.</li></ul> | BOOL | - |
49- <tr>55+| transpose_x2 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x2: false disables transposition and true enables it.</li><li>Defaults to false; must remain false for currently supported computations.</li></ul> | BOOL | - |
50- <td class="tg-22a9">x1</td>56+| group_size | Optional attribute | <ul><li>Encodes quantization group sizes along M, N, and K, with 16 bits per component. For supported combinations, see <a href="#constraints">Constraints</a>.</li><li>Defaults to 0.</li></ul> | INT64 | - |
51- <td class="tg-22a9">Input</td>
52- <td class="tg-22a9">Left matrix in the matrix multiplication, corresponding to input x1 in the formula.</td>
53- <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td>
54- <td class="tg-22a9">ND</td>
55- </tr>
56- <tr>
57- <td class="tg-22a9">x2</td>
58- <td class="tg-22a9">Input</td>
59- <td class="tg-22a9">Right matrix in the matrix multiplication, corresponding to input x2 in the formula.</td>
60- <td class="tg-22a9">FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1</td>
61- <td class="tg-22a9">ND</td>
62- </tr>
63- <tr>
64- <td class="tg-22a9">x1Scale</td>
65- <td class="tg-22a9">Optional input</td>
66- <td class="tg-22a9">Scaling factor introduced by the quantization of x1 in the quantization parameters, corresponding to scale1 in the formula.</td>
67- <td class="tg-22a9">FLOAT8_E8M0 (MX), FLOAT32 (HIFLOAT8 TT)</td>
68- <td class="tg-22a9">ND</td>
69- </tr>
70- <tr>
71- <td class="tg-22a9">x2Scale</td>
72- <td class="tg-22a9">Input</td>
73- <td class="tg-22a9">Scaling factor introduced by the quantization of x2 in the quantization parameters, corresponding to scale2 in the formula.</td>
74- <td class="tg-22a9">FLOAT8_E8M0 (MX), FLOAT32 (HIFLOAT8 TT)</td>
75- <td class="tg-22a9">ND</td>
76- </tr>
77- <tr>
78- <td class="tg-22a9">yRef</td>
79- <td class="tg-22a9">Input and output</td>
80- <td class="tg-22a9">Result after the matrix multiplication and the addition accumulation, corresponding to input and output y in the formula.</td>
81- <td class="tg-22a9">FLOAT32</td>
82- <td class="tg-22a9">ND</td>
83- </tr>
84- <tr>
85- <td class="tg-22a9">transposeX1</td>
86- <td class="tg-22a9">Input</td>
87- <td class="tg-22a9">Indicates whether the input shape of x1 is transposed.</td>
88- <td class="tg-22a9">bool</td>
89- <td class="tg-22a9">-</td>
90- </tr>
91- <tr>
92- <td class="tg-22a9">transposeX2</td>
93- <td class="tg-22a9">Input</td>
94- <td class="tg-22a9">Indicates whether the input shape of x2 is transposed.</td>
95- <td class="tg-22a9">bool</td>
96- <td class="tg-22a9">-</td>
97- </tr>
98- <tr>
99- <td class="tg-22a9">groupSize</td>
100- <td class="tg-22a9">Input</td>
101- <td class="tg-22a9">Integer parameter used for the quantization group sizes along the m, n, and k directions of the input.</td>
102- <td class="tg-22a9">INT64</td>
103- <td class="tg-22a9">-</td>
104- </tr>
105-</tbody></table>
106- 
107-- <term>Ascend 950PR&950DT products</term>:
108- - x1 and x2 support the FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, and FLOAT4_E2M1 data types.
109- - x1Scale and x2Scale support the FLOAT8_E8M0 (MX) and FLOAT32 (HIFLOAT8 TT) data types.
110- - yRef supports only the FLOAT32 data type.
111- - Currently only transposeX1=true and transposeX2=false are supported.
112- - groupSize is composed by concatenating groupSizeM, groupSizeN, and groupSizeK: groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32.
113 57 
114## Constraints58## Constraints
115 59 
116-- Contiguous tensors are supported, and [non-contiguous tensors](../../docs/en/context/non_contiguous_tensor.md) support the transposition scenario.60+- Only transpose_x1=true and transpose_x2=false are currently supported. The logical input shapes of x1 and x2 are `(K, M)` and `(K, N)`, respectively, and y has shape `(M, N)`.
117-- groupSize-related constraints: the passed-in groupSize is internally decomposed according to the formula to obtain groupSizeM, groupSizeN, and groupSizeK. When one or more of them is 0, the values are inferred again according to the input shapes of x1/x2/x1Scale/x2Scale.61+- group_size is encoded as `group_size_k | (group_size_n << 16) | (group_size_m << 32)`. A zero component is inferred from the corresponding input and scale shapes. The inferred sizes must satisfy the selected quantization mode.
118-- Constraints of the dynamic quantization (MX quantization) scenario:62+- MX quantization:
119- - The inputs and outputs support the following data type combinations: x1 and x2 are FLOAT8_E5M2/FLOAT8_E4M3FN, x1Scale and x2Scale are FLOAT8_E8M0, and yRef is FLOAT32.63+ - x1 and x2 may each be FLOAT8_E4M3FN or FLOAT8_E5M2, or both may be FLOAT4_E2M1. Mixed FP4/FP8 inputs and FLOAT4_E1M2 are not supported.
120- - The inputs and outputs also support the following data type combination: x1 and x2 are both FLOAT4_E2M1, x1Scale and x2Scale are both FLOAT8_E8M0, and yRef is FLOAT32. FLOAT4_E1M2 inputs and mixed FP4 and FP8 inputs are not supported.64+ - x1_scale and x2_scale are FLOAT8_E8M0, with shapes `(ceil(K/64), M, 2)` and `(ceil(K/64), N, 2)`, respectively, where ceil rounds upward.
121- - The shape of x1 is (k, m), the shape of x2 is (k, n), the shape of x1Scale is (ceil(k/64), m, 2), the shape of x2Scale is (ceil(k/64), n, 2), the shape of yRef is (m, n), [gsM, gsN, gsK] is [1, 1, 32], and groupSize is 32.65+ - The resulting group sizes must be `[group_size_m, group_size_n, group_size_k]=[1, 1, 32]`. Set group_size=4295032864 explicitly, or set it to 32 to infer the M and N group sizes from the shapes.
122- - Additional constraints for MXFP4 quantization:66+ - For MXFP4, M and N must be nonnegative even numbers. x1 and x2 use packed 4-bit storage, with two FLOAT4_E2M1 elements per byte. For contiguous inputs, the byte-storage shapes are `(K, M/2)` and `(K, N/2)`, respectively.
123- - m and n must be non-negative even numbers.67+ - For MXFP4, when M, N, and K are all positive, both $M\times K>2$ and $N\times K>2$ must also hold.
124- - The preceding shapes are expressed in logical element counts. x1 and x2 use 4-bit packed storage, with each byte storing two FLOAT4_E2M1 codes. For contiguous inputs, the logical shapes of x1 and x2 are (k, m) and (k, n), respectively, and the corresponding byte storage shapes are (k, m/2) and (k, n/2).68+- HIFLOAT8 T-T quantization:
125- - When m, n, and k are all greater than 0, both m × k > 2 and n × k > 2 must hold.69+ - x1 and x2 are HIFLOAT8. x1_scale and x2_scale are FLOAT32 and both have shape `(1,)`.
126-- Constraints of the HIFLOAT8 T-T scenario:70+ - group_size must be 0.
127- - The inputs and outputs support the following data type combinations: x1 and x2 are HIFLOAT8, x1Scale and x2Scale are FLOAT32, and yRef is FLOAT32.
128- - The shape of x1 is (k, m), the shape of x2 is (k, n), and the shape of yRef is (m, n).
129- - The shape of x1Scale is (1), the shape of x2Scale is (1), and groupSize must be 0.
130- - The bias and offset parameters are not supported.
131 71 
132## Invocation Description72## Invocation Description
133 73 
134| Invocation method | Sample code | Description |74| Invocation method | Sample code | Description |
135-| ---------------- | --------------------------- | --------------------------------------------------- |75+| --- | --- | --- |
136-| aclnn interface | [test_aclnn_quant_batch_matmul_inplace_add_mxfp8](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | Shows the invocation method of the mxFP8 quantization scenario, where `x1/x2` are FLOAT8, `x1Scale/x2Scale` are FLOAT8_E8M0, and `groupSize=32`. |76+| aclnn API | [MXFP8 example](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp8.cpp) | Calls [aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd_en.md) to demonstrate MXFP8 quantization. |
137-| aclnn interface | <a href="examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp">test_aclnn_quant_batch_matmul_inplace_add_mxfp4</a> | Shows the invocation method of the MXFP4 quantization scenario, where x1 and x2 are packed FLOAT4_E2M1, x1Scale and x2Scale are FLOAT8_E8M0, yRef is FLOAT32, and groupSize=32. |77+| aclnn API | [MXFP4 example](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_mxfp4.cpp) | Calls [aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd_en.md) with packed FLOAT4_E2M1 inputs. |
138-| aclnn interface | [test_aclnn_quant_batch_matmul_inplace_add_TT](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | Shows the invocation method of the HIFLOAT8 TT scenario under the currently supported combination `transposeX1=true` and `transposeX2=false`. |78+| aclnn API | [HIFLOAT8 T-T example](examples/arch35/test_aclnn_quant_batch_matmul_inplace_add_TT.cpp) | Calls [aclnnQuantBatchMatmulInplaceAdd](docs/aclnnQuantBatchMatmulInplaceAdd_en.md) to demonstrate per-tensor quantization. |
79+| GE graph mode | - | Builds a graph using the [QuantBatchMatmulInplaceAdd operator IR](op_graph/quant_batch_matmul_inplace_add_proto.h). |
@@ -37,13 +37,13 @@
37 37 
38### 保留Bitcast的转置融合38### 保留Bitcast的转置融合
39 39 
40-对于“转置节点→`Bitcast`→目标算子”的结构,融合转置并保留`Bitcast`。`x1`和`x1_scale`可以分别包含一个`Bitcast`,不要求两路同时包含。`x2`和`x2_scale`支路保持不变。此模式的共享节点限制见使用约束。40+对于“转置节点→`Bitcast`→目标算子”的结构,融合转置并保留`Bitcast`。`x1`和`x1_scale`可以分别包含一个`Bitcast`,不要求两路同时包含。`x2`和`x2_scale`支路保持不变。此模式的共享节点限制见[使用约束](#使用约束)。
41 41 
42![保留Bitcast的转置融合](../../../docs/zh/figures/QuantBatchMatmulInplaceAddTransposeFusionPass_5.png)42![保留Bitcast的转置融合](../../../docs/zh/figures/QuantBatchMatmulInplaceAddTransposeFusionPass_5.png)
43 43 
44### 保留Bitcast的x1_scale Reshape融合44### 保留Bitcast的x1_scale Reshape融合
45 45 
46-`x1_scale`支路可以为“`Reshape`→`Bitcast`→目标算子”。当`Reshape`满足等价转置条件,且带`Bitcast`的模式被识别时,融合该`Reshape`并保留`Bitcast`。图中展示静态MX量化场景;动态shape组合的限制见使用约束。46+`x1_scale`支路可以为“`Reshape`→`Bitcast`→目标算子”。当`Reshape`满足等价转置条件,且带`Bitcast`的模式被识别时,融合该`Reshape`并保留`Bitcast`。图中展示静态MX量化场景;动态shape组合的限制见[使用约束](#使用约束)。
47 47 
48![保留Bitcast的x1_scale Reshape融合](../../../docs/zh/figures/QuantBatchMatmulInplaceAddTransposeFusionPass_6.png)48![保留Bitcast的x1_scale Reshape融合](../../../docs/zh/figures/QuantBatchMatmulInplaceAddTransposeFusionPass_6.png)
49 49 
@@ -79,10 +79,10 @@
79- 含`Bitcast`的模式仅用于待融合转换节点不被其他计算支路共享的图;共享支路的数值正确性不在该模式的保证范围内。79- 含`Bitcast`的模式仅用于待融合转换节点不被其他计算支路共享的图;共享支路的数值正确性不在该模式的保证范围内。
80- 带`Bitcast`的融合模式须通过目标平台的能力校验;不支持该模式的平台保持原图。此限制不影响不带`Bitcast`的模式。80- 带`Bitcast`的融合模式须通过目标平台的能力校验;不支持该模式的平台保持原图。此限制不影响不带`Bitcast`的模式。
81- 每个目标节点根据自身`x1`、`x2`的shape独立识别动态shape,不受此前处理的节点或图影响。动态shape的“`Reshape`→`Bitcast`”组合仍须满足平台能力及其他融合条件。此处动态shape指含未知维度,与MX动态量化不是同一概念。81- 每个目标节点根据自身`x1`、`x2`的shape独立识别动态shape,不受此前处理的节点或图影响。动态shape的“`Reshape`→`Bitcast`”组合仍须满足平台能力及其他融合条件。此处动态shape指含未知维度,与MX动态量化不是同一概念。
82-- 实际数据类型、量化模式、shape和转置属性须符合[QuantBatchMatmulInplaceAdd算子说明](../README.md)。该算子当前要求`transposeX1=true`、`transposeX2=false`;图示中的属性取反不扩展算子的支持范围。82+- 实际数据类型、量化模式、shape和转置属性须符合[QuantBatchMatmulInplaceAdd算子说明](../README.md)。该算子当前要求`transpose_x1=true`、`transpose_x2=false`;图示中的属性取反不扩展算子的支持范围。
83 83 
84## 支持的型号84## 支持的型号
85 85 
86<!-- npu="950" id1 -->86<!-- npu="950" id1 -->
87-Ascend 950PR&950DT系列产品87+- <term>Ascend 950PR&950DT系列产品</term>
88<!-- end id1 -->88<!-- end id1 -->
@@ -0,0 +1,308 @@
1+# QuantBatchMatmulInplaceAddTransposeFusionPass
2+ 
3+## Fusion Patterns
4+ 
5+This fusion rule folds a `Transpose` or `TransposeD` before the `x1` input of `QuantBatchMatmulInplaceAdd` into the operator and negates `transpose_x1`. When the conditions are met, a transpose or equivalent `Reshape` before `x1_scale` is also fused. An intermediate `Bitcast` is preserved. `x2`, `x2_scale`, and `transpose_x2` remain unchanged.
6+ 
7+Diagram conventions:
8+ 
9+- The inputs are `x1`, `x2`, `x1_scale`, `x2_scale`, and `yRef`. `yRef` is the accumulation input and corresponds to the operator input `y`. Output `y` preserves the in-place accumulation relationship.
10+- `t1` and `t2` denote `transpose_x1` and `transpose_x2` before fusion, respectively; `!t1` denotes logical negation. This rule applies only when `t1=false` and `t2=false`. After fusion, `transpose_x1=true` and `transpose_x2=false`.
11+- `M` and `N` are the numbers of rows and columns in the result matrix, and `K` is the reduction dimension of the matrix multiplication. Shapes are shown in square brackets. A singleton dimension has size 1.
12+- Inputs specifying the transpose permutation and the target Reshape shape are omitted. The diagrams illustrate the transformation; the actual inputs and fused result must satisfy the operator specifications.
13+ 
14+### Fusing Only the x1 Transpose
15+ 
16+The `Transpose` or `TransposeD` before `x1` is folded into the target operator, negating `transpose_x1`. A qualifying transpose before `x1_scale` is fused at the same time. The connections of `x2` and `x2_scale` are preserved.
17+ 
18+```mermaid
19+flowchart LR
20+ subgraph before["Before fusion"]
21+ direction TB
22+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
23+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
24+ before_x2["x2"] --> before_op
25+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Transpose or TransposeD"] --> before_op
26+ before_x2_scale["x2_scale"] --> before_op
27+ before_yRef["yRef"] --> before_op
28+ before_op --> before_y["y (in-place accumulation)"]
29+ end
30+ subgraph after["After fusion"]
31+ direction TB
32+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
33+ after_x1["x1"] --> after_op
34+ after_x2["x2"] --> after_op
35+ after_x1_scale["x1_scale"] --> after_op
36+ after_x2_scale["x2_scale"] --> after_op
37+ after_yRef["yRef"] --> after_op
38+ after_op --> after_y["y (in-place accumulation)"]
39+ end
40+```
41+ 
42+### No Fusion When Only x2 Has a Transpose
43+ 
44+Transformations only on the `x2` or `x2_scale` branch do not trigger this rule. The graph remains unchanged.
45+ 
46+```mermaid
47+flowchart LR
48+ subgraph before["Before fusion"]
49+ direction TB
50+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
51+ before_x1["x1"] --> before_op
52+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
53+ before_x1_scale["x1_scale"] --> before_op
54+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Transpose or TransposeD"] --> before_op
55+ before_yRef["yRef"] --> before_op
56+ before_op --> before_y["y (in-place accumulation)"]
57+ end
58+ subgraph after["Unchanged"]
59+ direction TB
60+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
61+ after_x1["x1"] --> after_op
62+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
63+ after_x1_scale["x1_scale"] --> after_op
64+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Transpose or TransposeD"] --> after_op
65+ after_yRef["yRef"] --> after_op
66+ after_op --> after_y["y (in-place accumulation)"]
67+ end
68+```
69+ 
70+### Fusing Only x1 When Both Sides Have Transposes
71+ 
72+When both data inputs have preceding transposes, only the transformations of `x1` and a qualifying `x1_scale` are fused. `Transpose`, `TransposeD`, `Reshape`, and `Bitcast` nodes before `x2` and `x2_scale`, and their connections, are preserved.
73+ 
74+```mermaid
75+flowchart LR
76+ subgraph before["Before fusion"]
77+ direction TB
78+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
79+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
80+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
81+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Transpose or TransposeD"] --> before_op
82+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Transpose or TransposeD"] --> before_op
83+ before_yRef["yRef"] --> before_op
84+ before_op --> before_y["y (in-place accumulation)"]
85+ end
86+ subgraph after["After fusion"]
87+ direction TB
88+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
89+ after_x1["x1"] --> after_op
90+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
91+ after_x1_scale["x1_scale"] --> after_op
92+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Transpose or TransposeD"] --> after_op
93+ after_yRef["yRef"] --> after_op
94+ after_op --> after_y["y (in-place accumulation)"]
95+ end
96+```
97+ 
98+### Fusing x1_scale Reshape in MX Quantization
99+ 
100+For static shapes, a `Reshape` of a three-dimensional FLOAT8_E8M0 `x1_scale` is treated as an equivalent transpose when it swaps the first two dimensions and at least one of those dimensions has size 1. For example, the shape changes from `[1, M, 2]` to `[M, 1, 2]`. This `Reshape` can be fused together with the `x1` transpose. A `Reshape` before `x2_scale` is preserved.
101+ 
102+```mermaid
103+flowchart LR
104+ subgraph before["Before fusion"]
105+ direction TB
106+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
107+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
108+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
109+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape"] --> before_op
110+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_op
111+ before_yRef["yRef"] --> before_op
112+ before_op --> before_y["y (in-place accumulation)"]
113+ end
114+ subgraph after["After fusion"]
115+ direction TB
116+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
117+ after_x1["x1"] --> after_op
118+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
119+ after_x1_scale["x1_scale"] --> after_op
120+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_op
121+ after_yRef["yRef"] --> after_op
122+ after_op --> after_y["y (in-place accumulation)"]
123+ end
124+```
125+ 
126+### Transpose Fusion Retaining Bitcast
127+ 
128+For a transpose → `Bitcast` → target operator sequence, the transpose is fused and `Bitcast` is preserved. `x1` and `x1_scale` may each have one `Bitcast`; both branches need not contain one. The `x2` and `x2_scale` branches remain unchanged. For restrictions on shared nodes, see [Usage Constraints](#constraints).
129+ 
130+```mermaid
131+flowchart LR
132+ subgraph before["Before fusion"]
133+ direction TB
134+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
135+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_x1_1["Bitcast"] --> before_op
136+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_x2_1["Bitcast"] --> before_op
137+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Transpose or TransposeD"] --> before_x1_scale_1["Bitcast"] --> before_op
138+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Transpose or TransposeD"] --> before_x2_scale_1["Bitcast"] --> before_op
139+ before_yRef["yRef"] --> before_op
140+ before_op --> before_y["y (in-place accumulation)"]
141+ end
142+ subgraph after["After fusion"]
143+ direction TB
144+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
145+ after_x1["x1"] --> after_x1_0["Bitcast"] --> after_op
146+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_x2_1["Bitcast"] --> after_op
147+ after_x1_scale["x1_scale"] --> after_x1_scale_0["Bitcast"] --> after_op
148+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Transpose or TransposeD"] --> after_x2_scale_1["Bitcast"] --> after_op
149+ after_yRef["yRef"] --> after_op
150+ after_op --> after_y["y (in-place accumulation)"]
151+ end
152+```
153+ 
154+### Fusing x1_scale Reshape While Preserving Bitcast
155+ 
156+The `x1_scale` branch may have a `Reshape` → `Bitcast` → target operator sequence. If `Reshape` is equivalent to a transpose and the pattern containing `Bitcast` is recognized, `Reshape` is fused and `Bitcast` is preserved. The diagram shows static-shape MX quantization. For restrictions on dynamic-shape combinations, see [Usage Constraints](#constraints).
157+ 
158+```mermaid
159+flowchart LR
160+ subgraph before["Before fusion"]
161+ direction TB
162+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
163+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_x1_1["Bitcast"] --> before_op
164+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_x2_1["Bitcast"] --> before_op
165+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape"] --> before_x1_scale_1["Bitcast"] --> before_op
166+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_x2_scale_1["Bitcast"] --> before_op
167+ before_yRef["yRef"] --> before_op
168+ before_op --> before_y["y (in-place accumulation)"]
169+ end
170+ subgraph after["After fusion"]
171+ direction TB
172+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
173+ after_x1["x1"] --> after_x1_0["Bitcast"] --> after_op
174+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_x2_1["Bitcast"] --> after_op
175+ after_x1_scale["x1_scale"] --> after_x1_scale_0["Bitcast"] --> after_op
176+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_x2_scale_1["Bitcast"] --> after_op
177+ after_yRef["yRef"] --> after_op
178+ after_op --> after_y["y (in-place accumulation)"]
179+ end
180+```
181+ 
182+### Preserving Connections for Single-Element Scales
183+ 
184+For HIFLOAT8 per-tensor quantization, for example, `x1_scale` and `x2_scale` are one-dimensional FLOAT32 tensors of shape `[1]`, each containing one quantization coefficient. If neither scale has a preceding transformation, their connections are preserved and only the `x1` transpose is fused. Here, `[1]` is distinct from the zero-dimensional tensor shape `[]`.
185+ 
186+```mermaid
187+flowchart LR
188+ subgraph before["Before fusion"]
189+ direction TB
190+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
191+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
192+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
193+ before_x1_scale["x1_scale<br/>FLOAT32, shape [1]"] --> before_op
194+ before_x2_scale["x2_scale<br/>FLOAT32, shape [1]"] --> before_op
195+ before_yRef["yRef"] --> before_op
196+ before_op --> before_y["y (in-place accumulation)"]
197+ end
198+ subgraph after["After fusion"]
199+ direction TB
200+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
201+ after_x1["x1"] --> after_op
202+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
203+ after_x1_scale["x1_scale<br/>FLOAT32, shape [1]"] --> after_op
204+ after_x2_scale["x2_scale<br/>FLOAT32, shape [1]"] --> after_op
205+ after_yRef["yRef"] --> after_op
206+ after_op --> after_y["y (in-place accumulation)"]
207+ end
208+```
209+ 
210+### Fusing x1_scale Reshape with Dynamic Shapes
211+ 
212+When a data input shape contains an unknown dimension, a `Reshape` before `x1_scale` can be recognized without the singleton-dimension condition required for static shapes. The input graph must ensure that this `Reshape` is equivalent to the required transpose. The diagram contains no `Bitcast`. This describes a fusion pattern and does not imply support for arbitrary unknown input dimensions.
213+ 
214+```mermaid
215+flowchart LR
216+ subgraph before["Before fusion"]
217+ direction TB
218+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
219+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
220+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
221+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape (dynamic)"] --> before_op
222+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape (dynamic)"] --> before_op
223+ before_yRef["yRef"] --> before_op
224+ before_op --> before_y["y (in-place accumulation)"]
225+ end
226+ subgraph after["After fusion"]
227+ direction TB
228+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
229+ after_x1["x1"] --> after_op
230+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
231+ after_x1_scale["x1_scale"] --> after_op
232+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape (dynamic)"] --> after_op
233+ after_yRef["yRef"] --> after_op
234+ after_op --> after_y["y (in-place accumulation)"]
235+ end
236+```
237+ 
238+### Other Reshape Patterns with Singleton Dimensions
239+ 
240+For static shapes, a preceding `Reshape` can also be recognized when `x1_scale` has at least two dimensions and at least one of its last two dimensions has size 1, for example, a shape change from `[8, 1]` to `[1, 8]`. The input graph must ensure equivalence to a transpose. The quantization mode and parameter combination must still satisfy the operator specifications.
241+ 
242+```mermaid
243+flowchart LR
244+ subgraph before["Before fusion"]
245+ direction TB
246+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
247+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
248+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
249+ before_x1_scale["x1_scale"] --> before_x1_scale_0["Reshape"] --> before_op
250+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_op
251+ before_yRef["yRef"] --> before_op
252+ before_op --> before_y["y (in-place accumulation)"]
253+ end
254+ subgraph after["After fusion"]
255+ direction TB
256+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
257+ after_x1["x1"] --> after_op
258+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
259+ after_x1_scale["x1_scale"] --> after_op
260+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_op
261+ after_yRef["yRef"] --> after_op
262+ after_op --> after_y["y (in-place accumulation)"]
263+ end
264+```
265+ 
266+### No Transformation Before x1_scale
267+ 
268+If `x1_scale` connects directly to the target operator, has no recognizable preceding transformation, or is an unconnected optional input, that branch remains unchanged and does not prevent fusion of the `x1` transpose. `x2_scale` must retain a valid input connection.
269+ 
270+```mermaid
271+flowchart LR
272+ subgraph before["Before fusion"]
273+ direction TB
274+ before_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = false<br/>transpose_x2 = false"]
275+ before_x1["x1"] --> before_x1_0["Transpose or TransposeD"] --> before_op
276+ before_x2["x2"] --> before_x2_0["Transpose or TransposeD"] --> before_op
277+ before_x1_scale["x1_scale"] --> before_op
278+ before_x2_scale["x2_scale"] --> before_x2_scale_0["Reshape"] --> before_op
279+ before_yRef["yRef"] --> before_op
280+ before_op --> before_y["y (in-place accumulation)"]
281+ end
282+ subgraph after["After fusion"]
283+ direction TB
284+ after_op["QuantBatchMatmulInplaceAdd<br/>transpose_x1 = true<br/>transpose_x2 = false"]
285+ after_x1["x1"] --> after_op
286+ after_x2["x2"] --> after_x2_0["Transpose or TransposeD"] --> after_op
287+ after_x1_scale["x1_scale"] --> after_op
288+ after_x2_scale["x2_scale"] --> after_x2_scale_0["Reshape"] --> after_op
289+ after_yRef["yRef"] --> after_op
290+ after_op --> after_y["y (in-place accumulation)"]
291+ end
292+```
293+ 
294+## Constraints
295+ 
296+- Before fusion, `transpose_x1=false` and `transpose_x2=false` are required so that the resulting attributes meet the backend requirements. Other attribute combinations leave the graph unchanged.
297+- A recognizable `Transpose` or `TransposeD` must precede `x1`, connecting directly to the target operator or through one `Bitcast`. A `Reshape` alone or a transpose only on the `x2` branch cannot trigger this rule.
298+- Data transposes must match matrix-multiplication transpose semantics, and scale transformations must preserve the correspondence between quantization coefficients and data. Arbitrary dimension permutations or Reshape operations are not equivalent transposes.
299+- Patterns containing `Bitcast` apply only when the transformation being fused is not shared with another computation branch. Numerical correctness on shared branches is not guaranteed by these patterns.
300+- Patterns containing `Bitcast` must pass the target platform's capability check. Unsupported platforms leave the graph unchanged. This restriction does not apply to patterns without `Bitcast`.
301+- Each target node determines whether shapes are dynamic from its own `x1` and `x2` shapes, independently of previously processed nodes or graphs. A dynamic-shape `Reshape` → `Bitcast` combination must still satisfy platform capability and other fusion requirements. Dynamic shape here means a shape with unknown dimensions and is distinct from MX dynamic quantization.
302+- Data types, quantization modes, shapes, and transpose attributes must satisfy the [QuantBatchMatmulInplaceAdd specifications](../README_en.md). The operator currently requires `transpose_x1=true` and `transpose_x2=false`. Negating the attribute in a diagram does not extend operator support.
303+ 
304+## Applicable Products
305+ 
306+<!-- npu="950" id1 -->
307+- <term>Ascend 950PR&950DT products</term>
308+<!-- end id1 -->
@@ -27,21 +27,25 @@
27 27 
28- 计算公式:28- 计算公式:
29 29 
30- - **MX量化:**30+ $X_1$、$X_2$表示应用转置属性后的矩阵,shape分别为$(M,K)$、$(K,N)$。$Y_{\mathrm{in}}$为调用前的`yRef`,$Y_{\mathrm{out}}$为原地更新后的结果;两者的shape均为$(M,N)$。
31+ 
32+ - MX(Microscaling,微缩放)量化:
31 33 
32 $$34 $$
33- y[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice)) * (scale1[m, j] * scale2[j, n])) + y[m,n]35+ Y_{\mathrm{out}}[m,n]=Y_{\mathrm{in}}[m,n]+\sum_{j=0}^{L-1}
36+ \left(\sum_{t=0}^{g_j-1}X_1[m,jg_K+t]X_2[jg_K+t,n]\right)S_1[m,j]S_2[j,n]
34 $$37 $$
35 38 
36- 其中,$gsK$ 代表K轴的量化的block size即32,$x1Slice$代表$x1$第m行长度为 $gsK$ 的向量,$x2Slice$代表$x2$第n列长度为 $gsK$ 的向量,K轴均从$j*gsK$起始切片,j的取值范围[0, kLoops), kLoops=ceil($K_i$ / $gsK$),支持最后的切片长度不足 $gsK$。39+ 其中,$g_K=32$,$L=\lceil K/g_K\rceil$,$g_j=\min(g_K,K-jg_K)$。$S_1$、$S_2$分别为`x1ScaleOptional`、`x2Scale`中对应分组的scale,存储shape见[约束说明](#约束说明)。
37 40 
38- - **HIFLOAT8 T-T量化:**41+ - HIFLOAT8 T-T量化:
39 42 
40 $$43 $$
41- y[m,n] = (\sum_{k=0}^{K_i-1} (x1[m,k] * x2[k,n])) * (scale1 * scale2) + y[m,n]44+ Y_{\mathrm{out}}[m,n]=Y_{\mathrm{in}}[m,n]+
45+ \left(\sum_{k=0}^{K-1}X_1[m,k]X_2[k,n]\right)s_1s_2
42 $$46 $$
43 47 
44- 其中,$scale1$和$scale2$分别对应$x1Scale$和$x2Scale$,均为标量(shape为 $(1)$);当设置transposeX1/transposeX2时,公式中的$x1$和$x2$按转置后的视图参与计算。48+ $s_1$、$s_2$分别对应`x1ScaleOptional`、`x2Scale`,shape均为$(1,)$。
45 49 
46## 函数原型50## 函数原型
47 51 
@@ -1,7 +1,5 @@
1# aclnnQuantBatchMatmulInplaceAdd1# aclnnQuantBatchMatmulInplaceAdd
2 2 
3-<!-- md-trans-meta sourceCommit=unknown translatedAt=2026-09-22T18:26:21.508Z -->
4- 
5## Applicable Products3## Applicable Products
6 4 
7<!-- npu="950" id1 -->5<!-- npu="950" id1 -->
@@ -3,182 +3,104 @@
3## 产品支持情况3## 产品支持情况
4 4 
5| 产品 | 是否支持 |5| 产品 | 是否支持 |
6-| ---- | :----:|6+| --- | :---: |
7-|Ascend 950PR&950DT系列产品|√|7+| <term>Ascend 950PR&950DT系列产品</term> | √ |
8-|Atlas A3系列产品|√|8+| <term>Atlas A3系列产品</term> | √ |
9-|Atlas A2系列产品|√|9+| <term>Atlas A2系列产品</term> | √ |
10-|Atlas 200I/500 A2推理产品|x|10+| <term>Atlas 200I/500 A2推理产品</term> | × |
11-|Atlas推理系列产品|√|11+| <term>Atlas推理系列产品</term> | √ |
12-|Atlas训练系列产品|x|12+| <term>Atlas训练系列产品</term> | × |
13-|Kirin X90处理器系列产品|√|13+| <term>KirinX90处理器系列产品</term> | √ |
14-|Kirin 9030处理器系列产品|√|14+| <term>Kirin9030处理器系列产品</term> | √ |
15 15 
16## 功能说明16## 功能说明
17 17 
18-- 算子功能:完成量化的矩阵乘计算,最小支持输入维度为2维,最大支持输入维度为6维。18+- 算子功能:完成量化矩阵乘计算,支持缩放、累加偏置和输出类型转换,输入的逻辑维数为2~6。支持的逐张量、逐token、逐通道及分组量化模式随产品和数据类型而异,模式定义见[量化模式介绍](../../docs/zh/context/quant_mode_introduction.md)。
19-- 计算公式:
20 19 
21- - <term>Atlas A2系列产品</term>、<term>Atlas A3系列产品</term>、<term>Ascend 950PR&950DT系列产品</term>:20+- 版本关系:QuantBatchMatmulV3是算子名称,与aclnn接口的版本号分别定义。[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4.md)相较[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3.md)增加了左矩阵的pertoken_scale输入;[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)提供更多量化参数,并按输入组合调用本算子或[QuantBatchMatmulV4](../quant_batch_matmul_v4/README.md)。
22- - 无pertoken无bias:
23 21 
24- $$22+- 计算公式:以下省略batch下标,$X_1$、$X_2$表示按transpose_x1、transpose_x2解释后的矩阵,形状分别为$(M,K)$、$(K,N)$。M、N为输出矩阵的行数、列数,K为归约维度;$S_1$、$S_2$分别对应pertoken_scale、scale。未提供pertoken_scale、bias、offset时,分别按1、0、0处理。
25- out = x1@x2 * scale + offset
26- $$
27 23 
28- - bias INT32:24+ - 非分组量化、无bias:
29 25 
30- $$26+ $$
31- out = (x1@x2 + bias) * scale + offset27+ y=(X_1@X_2)\odot S_2\odot S_1+offset
32- $$28+ $$
33 29 
34- - bias BFLOAT16/FLOAT32(此场景无offset):30+ - INT8或INT4输入且bias为INT32:
35 31 
36- $$32+ $$
37- out = x1@x2 * scale + bias33+ y=(X_1@X_2+bias)\odot S_2\odot S_1+offset
38- $$34+ $$
39 35 
40- - pertoken无bias:36+ - INT8或INT4输入且bias为BFLOAT16、FLOAT16或FLOAT32:
41 37 
42- $$38+ $$
43- out = x1@x2 * scale * pertokenScaleOptional39+ y=(X_1@X_2)\odot S_2\odot S_1+bias
44- $$40+ $$
45 41 
46- - pertoken, bias INT32(此场景无offset):42+ - FLOAT8_E4M3FN、FLOAT8_E5M2或HIFLOAT8的非分组量化,bias为FLOAT32:
47 43 
48- $$44+ $$
49- out = (x1@x2 + bias) * scale * pertokenScaleOptional45+ y=(X_1@X_2+bias)\odot S_2\odot S_1
50- $$46+ $$
51 47 
52- - pertoken, bias BFLOAT16/FLOAT16/FLOAT32(此场景无offset):48+ - INT32输出:直接输出整数累加结果,不执行缩放。
53 49 
54- $$50+ $$
55- out = x1@x2 * scale * pertokenScaleOptional + bias51+ y=X_1@X_2+bias
56- $$52+ $$
57 53 
58- - <term>Atlas推理系列产品</term>:54+ - MX(Microscaling,微缩放)、G-B和B-B分组量化:按K轴分组计算,再累加各组结果。
59- - 无pertoken、无bias:
60 55 
61- $$56+ $$
62- out = x1@x2 * scale + offset57+ y_{m,n}=\sum_{j=0}^{L-1}\left[
63- $$58+ \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right)
59+ S_{1,m,j}S_{2,j,n}\right]+bias_n
60+ $$
64 61 
65- - 无pertoken、bias INT32:62+ 其中,$g_K$为K方向分组大小,$L=\lceil K/g_K\rceil$,$g_j=\min(g_K,K-jg_K)$;$S_1$、$S_2$表示按对应量化分组广播后的scale。G-B和B-B模式不支持bias,公式中取$bias_n=0$。
66 63 
67- $$64+ 其中,$@$表示矩阵乘,$\odot$表示按量化参数形状广播的逐元素乘。offset仅用于支持的INT8输出组合,不能与pertoken_scale同时使用。上述公式表示量化计算关系,实际结果还需转换为dtype指定的数据类型。
68- out = (x1@x2 + bias) * scale + offset
69- $$
70- 
71- - 有pertoken、无bias
72- 
73- $$
74- out = x1@x2 * scale * pertokenScaleOptional
75- $$
76- 
77- - 有pertoken、bias INT32
78- 
79- $$
80- out = (x1@x2 + bias) * scale * pertokenScaleOptional
81- $$
82- 
83- - 有pertoken、bias FLOAT32
84- 
85- $$
86- out = x1@x2 * scale * pertokenScaleOptional + bias
87- $$
88 65 
89## 参数说明66## 参数说明
90 67 
91-<table class="tg"><thead>68+| 参数名 | 输入/输出/属性 | 描述 | 数据类型 | 数据格式 |
92- <tr>69+| --- | --- | --- | --- | --- |
93- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">参数名</span></th>70+| x1 | 输入 | 矩阵乘的左矩阵,对应公式中的X₁。 | INT8、INT4、HIFLOAT8、FLOAT8_E5M2、FLOAT8_E4M3FN、FLOAT4_E2M1 | ND、FRACTAL_NZ |
94- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">输入/输出/属性</span></th>71+| x2 | 输入 | 矩阵乘的右矩阵,对应公式中的X₂。 | INT8、INT4、HIFLOAT8、FLOAT8_E5M2、FLOAT8_E4M3FN、FLOAT4_E2M1 | ND、FRACTAL_NZ |
95- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">描述</span></th>72+| scale | 输入 | 右矩阵的量化缩放参数。UINT64、INT64类型存储编码后的量化参数,须按解码后的scale参与公式计算。 | UINT64、FLOAT32、INT64、BFLOAT16、FLOAT8_E8M0 | ND、FRACTAL_NZ |
96- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据类型</span></th>73+| offset | 可选输入 | 输出量化的偏置因子;未提供时按0处理,仅用于支持的INT8输出组合。 | FLOAT32 | ND |
97- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">数据格式</span></th>74+| bias | 可选输入 | 矩阵乘的累加偏置;未提供时不加偏置。累加位置由输入及bias的数据类型决定。 | INT32、BFLOAT16、FLOAT16、FLOAT32 | ND |
98- </tr></thead>75+| pertoken_scale | 可选输入 | 左矩阵的量化缩放参数;未提供时按1处理。逐token及分组量化场景须提供对应的缩放参数。 | FLOAT32、FLOAT8_E8M0 | ND |
99-<tbody>76+| y | 输出 | 矩阵乘及量化处理后的结果,数据类型由dtype指定。 | FLOAT16、INT8、BFLOAT16、INT32、FLOAT32 | ND、FRACTAL_NZ |
100- <tr>77+| dtype | 属性 | 输出y的数据类型枚举,支持0(FLOAT32)、1(FLOAT16)、2(INT8)、3(INT32)、27(BFLOAT16),具体取值受产品及输入组合限制。 | INT64 | - |
101- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">x1</span></td>78+| transpose_x1 | 可选属性 | <ul><li>是否转置x1的最后两维:false表示不转置,true表示转置。</li><li>默认值为false。</li></ul> | BOOL | - |
102- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输入</span></td>79+| transpose_x2 | 可选属性 | <ul><li>是否转置x2的最后两维:false表示不转置,true表示转置。</li><li>默认值为false。</li></ul> | BOOL | - |
103- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">矩阵乘运算中的左矩阵。</span></td>80+| group_size | 可选属性 | <ul><li>M、N、K方向量化分组大小的编码值,各分量占16位;支持的分组组合见<a href="#约束说明">约束说明</a>。</li><li>默认值为0。</li></ul> | INT64 | - |
104- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td>
105- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND, FRACTAL_NZ</span></td>
106- </tr>
107- <tr>
108- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">x2</span></td>
109- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">输入</span></td>
110- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">矩阵乘运算中的右矩阵。</span></td>
111- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td>
112- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND, FRACTAL_NZ</span></td>
113- </tr>
114- <tr>
115- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">scale</span></td>
116- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输入</span></td>
117- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">量化参数的缩放因子,对应公式的scale。</span></td>
118- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">UINT64, FLOAT32, INT64, BF16, FLOAT8_E8M0</span></td>
119- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td>
120- </tr>
121- <tr>
122- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">offset</span></td>
123- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">输入</span></td>
124- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">量化参数的偏置因子,对应公式的offset。</span></td>
125- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32</span></td>
126- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td>
127- </tr>
128- <tr>
129- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">bias</span></td>
130- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输入</span></td>
131- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">矩阵乘运算后累加的偏置,对应公式中的bias。</span></td>
132- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT32, BF16, FLOAT16, FLOAT32</span></td>
133- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td>
134- </tr>
135- <tr>
136- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">pertoken_scale</span></td>
137- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">输入</span></td>
138- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">量化参数的缩放因子,对应公式中的pertokenScaleOptional。</span></td>
139- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32, FLOAT8_E8M0</span></td>
140- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td>
141- </tr>
142- <tr>
143- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">y</span></td>
144- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">输出</span></td>
145- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">矩阵乘运算的计算结果。</span></td>
146- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">FLOAT16, INT8, BF16, INT32, FLOAT32</span></td>
147- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td>
148- </tr>
149-</tbody></table>
150 81 
151-- <term>Atlas A2系列产品</term>、<term>Atlas A3系列产品</term>:82+- <term>Atlas A3系列产品</term>、<term>Atlas A2系列产品</term>:x1、x2不支持HIFLOAT8、FLOAT8_E5M2、FLOAT8_E4M3FN、FLOAT4_E2M1;scale不支持FLOAT8_E8M0;pertoken_scale仅支持FLOAT32;y不支持FLOAT32。
152- - x1只支持INT8、INT4数据类型。83+- <term>Atlas推理系列产品</term>:x1、x2仅支持INT8;scale支持UINT64、INT64,逐token场景还支持FLOAT32;bias支持INT32,逐token场景还支持FLOAT32;pertoken_scale仅支持FLOAT32;y支持FLOAT16、INT8,逐token场景仅支持FLOAT16。逐token场景的y使用FRACTAL_NZ,其余场景使用ND。
153- - x2只支持INT8、INT4数据类型。84+- <term>KirinX90处理器系列产品</term>、<term>Kirin9030处理器系列产品</term>:x1、x2仅支持INT8,x1使用ND,x2使用FRACTAL_NZ;scale仅支持UINT64、INT64;bias仅支持INT32;pertoken_scale仅支持FLOAT32;y仅支持FLOAT16、INT8。
154- - scale只支持UINT64、FLOAT32、INT64、BF16数据类型。85+- <term>Ascend 950PR&950DT系列产品</term>:scale的FRACTAL_NZ格式仅用于MX全量化的ScaleNZ场景,要求x2也使用FRACTAL_NZ,且x1、x2类型相同,为FLOAT8_E4M3FN或FLOAT4_E2M1。布局和转换要求见[ScaleNZ约束](docs/aclnnQuantMatmulWeightNz.md#scalenz约束)。其他产品的scale使用ND。
155- - bias只支持INT32、BFLOAT16、FLOAT16、FLOAT32数据类型。
156- - offset只支持FLOAT32数据类型。
157- - pertoken_scale只支持FLOAT32数据类型。
158- - y只支持FLOAT16和BFLOAT16数据类型。
159-- <term>Atlas推理系列产品</term>:
160- - x1只支持INT8数据类型。
161- - x2只支持INT8数据类型。
162- - scale只支持UINT64、FLOAT32数据类型。
163- - bias只支持INT32数据类型,pertoken场景额外支持FLOAT32数据类型。
164- - pertoken_scale只支持FLOAT32数据类型。
165- - y只支持FLOAT16和INT32数据类型。
166-- Kirin X90/Kirin 9030处理器系列产品:
167- - x1、x2只支持INT8数据类型。
168- - scale只支持UINT64、INT64数据类型。
169- - offset只支持FLOAT32数据类型。
170- - bias只支持INT32数据类型。
171- - pertoken_scale只支持FLOAT32数据类型。
172- - y只支持FLOAT16和INT8数据类型。
173 86 
174## 约束说明87## 约束说明
175 88 
176-- 不支持空tensor。89+- 输入矩阵的K维必须一致,batch维须满足[广播关系](../../docs/zh/context/broadcast_relationship.md)。具体形状和产品限制见[算子IR说明](docs/ascendir_QuantBatchMatmulV3.md)。
177-- 支持连续tensor,[非连续tensor](../../docs/zh/context/non_contiguous_tensor.md)只支持转置场景。90+- INT4输入仅支持二维,transpose_x1须为false,输入的最内维须为偶数。
91+- group_size按`group_size_k | (group_size_n << 16) | (group_size_m << 32)`编码。分组分量为0时由输入形状推导,推导时对应维度须满足整除要求。MX、G-B、B-B模式的最终分组大小分别为`[1, 1, 32]`、`[1, 128, 128]`、`[128, 128, 128]`;其他量化模式的group_size为0。
92+- MX模式下,pertoken_scale、scale均须提供且类型为FLOAT8_E8M0。FLOAT4_E2M1输入的最内维须为偶数,且K须大于2。
93+- <term>Atlas A3系列产品</term>、<term>Atlas A2系列产品</term>:输入矩阵逻辑形状的最后一维不能超过65535。
94+- <term>Ascend 950PR&950DT系列产品</term>:x2使用ND时支持M=0或N=0并输出空Tensor;x2使用FRACTAL_NZ时支持M=0并输出空Tensor。其他场景不支持维度大小为0的输入。
178 95 
179## 调用说明96## 调用说明
180 97 
181-| 调用方式 | 样例代码 | 说明 |98+| 调用方式 | 样例代码 | 说明 |
182-| ---------------- | --------------------------- | --------------------------------------------------- |99+| --- | --- | --- |
183-| GEIR接口 | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | 通过[QuantBatchMatmulV3算子IR](docs/ascendir_QuantBatchMatmulV3.md)构图并调用算子。 |100+| GE图模式 | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | 通过[QuantBatchMatmulV3算子IR](docs/ascendir_QuantBatchMatmulV3.md)构图调用。 |
184-| aclnn接口 | [test_aclnn_quant_matmul_v3](examples/test_aclnn_quant_matmul_v3_at.cpp) | 通过<br>[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3.md)<br>[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4.md)<br>[aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz.md)<br>[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)<br>等方式调用QuantBatchMatmulV3算子。 |101+| aclnn API | [test_aclnn_quant_matmul_v3_at](examples/test_aclnn_quant_matmul_v3_at.cpp) | 通过[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3.md)调用。 |
102+| aclnn API | [test_aclnn_quant_matmul_v4_at2_at3](examples/test_aclnn_quant_matmul_v4_at2_at3.cpp) | 通过[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4.md)调用,支持左矩阵逐token缩放。 |
103+| aclnn API | [test_aclnn_quant_matmul_weight_nz](examples/test_aclnn_quant_matmul_weight_nz.cpp) | 通过[aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz.md)调用,使用FRACTAL_NZ格式的权重。 |
104+| aclnn API | [test_aclnn_quant_matmul_v5_hif8hif8](../quant_batch_matmul_v4/examples/arch35/test_aclnn_quant_matmul_v5_hif8hif8.cpp) | 通过[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)调用,样例展示由QuantBatchMatmulV3执行的HIFLOAT8量化场景。 |
105+ 
106+aclnnQuantMatmulV3、aclnnQuantMatmulV4已在接口声明中标记为待废弃,迁移说明见对应接口文档。
@@ -5,182 +5,104 @@ English | [简体中文](./README.md)
5## Applicable Products5## Applicable Products
6 6 
7| Product | Support |7| Product | Support |
8-| ---- | :----:|8+| --- | :---: |
9-|Ascend 950PR&950DT products|√|9+| <term>Ascend 950PR&950DT products</term> | √ |
10-|Atlas A3 products|√|10+| <term>Atlas A3 products</term> | √ |
11-|Atlas A2 products|√|11+| <term>Atlas A2 products</term> | √ |
12-|Atlas 200I/500 A2 inference products|x|12+| <term>Atlas 200I/500 A2 inference products</term> | × |
13-|Atlas inference products|√|13+| <term>Atlas inference products</term> | √ |
14-|Atlas training products|x|14+| <term>Atlas training products</term> | × |
15-|Kirin X90 processor series products|√|15+| <term>Kirin X90 processor products</term> | √ |
16-|Kirin 9030 processor series products|√|16+| <term>Kirin 9030 processor products</term> | √ |
17 17 
18## Function18## Function
19 19 
20-- Operator function: Performs quantized matrix multiplication. The input supports a minimum of 2 dimensions and a maximum of 6 dimensions.20+- Operator function: Performs quantized matrix multiplication with scaling, bias accumulation, and output type conversion. The logical input rank is 2–6. Supported per-tensor, per-token, per-channel, and grouped modes depend on the product and data type. See [Introduction to Quantization Modes](../../docs/en/context/quant_mode_introduction.md).
21-- Formula:
22 21 
23- - <term>Atlas A2 products</term>, <term>Atlas A3 products</term>, <term>Ascend 950PR&950DT products</term>:22+- Version relationship: QuantBatchMatmulV3 is the operator name; ACLNN interface versions are defined separately. [aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4_en.md) adds left-matrix pertoken_scale compared with [aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3_en.md). [aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5_en.md) provides additional quantization parameters and dispatches to this operator or [QuantBatchMatmulV4](../quant_batch_matmul_v4/README_en.md) according to the input combination.
24- - No pertoken and no bias:
25 23 
26- $$24+- Formulas: Batch indices are omitted below. $X_1$ and $X_2$ are the matrices interpreted according to transpose_x1 and transpose_x2, with shapes $(M,K)$ and $(K,N)$. M and N are the result matrix dimensions, and K is the reduction dimension. $S_1$ and $S_2$ correspond to pertoken_scale and scale. Omitted pertoken_scale, bias, and offset are treated as 1, 0, and 0, respectively.
27- out = x1@x2 * scale + offset
28- $$
29 25 
30- - bias INT32:26+ - Non-grouped quantization without bias:
31 27 
32- $$28+ $$
33- out = (x1@x2 + bias) * scale + offset29+ y=(X_1@X_2)\odot S_2\odot S_1+offset
34- $$30+ $$
35 31 
36- - bias BFLOAT16/FLOAT32 (no offset in this scenario):32+ - INT8 or INT4 inputs with INT32 bias:
37 33 
38- $$34+ $$
39- out = x1@x2 * scale + bias35+ y=(X_1@X_2+bias)\odot S_2\odot S_1+offset
40- $$36+ $$
41 37 
42- - pertoken and no bias:38+ - INT8 or INT4 inputs with BFLOAT16, FLOAT16, or FLOAT32 bias:
43 39 
44- $$40+ $$
45- out = x1@x2 * scale * pertokenScaleOptional41+ y=(X_1@X_2)\odot S_2\odot S_1+bias
46- $$42+ $$
47 43 
48- - pertoken and bias INT32 (no offset in this scenario):44+ - Non-grouped quantization with FLOAT8_E4M3FN, FLOAT8_E5M2, or HIFLOAT8 inputs and FLOAT32 bias:
49 45 
50- $$46+ $$
51- out = (x1@x2 + bias) * scale * pertokenScaleOptional47+ y=(X_1@X_2+bias)\odot S_2\odot S_1
52- $$48+ $$
53 49 
54- - pertoken and bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario):50+ - INT32 output: Returns the integer accumulation result directly, without scaling.
55 51 
56- $$52+ $$
57- out = x1@x2 * scale * pertokenScaleOptional + bias53+ y=X_1@X_2+bias
58- $$54+ $$
59 55 
60- - <term>Atlas inference products</term>:56+ - MX (Microscaling), G-B, and B-B grouped quantization: Compute by K groups and sum the group results.
61- - No pertoken and no bias:
62 57 
63- $$58+ $$
64- out = x1@x2 * scale + offset59+ y_{m,n}=\sum_{j=0}^{L-1}\left[
65- $$60+ \left(\sum_{t=0}^{g_j-1}X_{1,m,jg_K+t}X_{2,jg_K+t,n}\right)
61+ S_{1,m,j}S_{2,j,n}\right]+bias_n
62+ $$
66 63 
67- - No pertoken and bias INT32:64+ Here, $g_K$ is the group size along K, $L=\lceil K/g_K\rceil$, and $g_j=\min(g_K,K-jg_K)$. $S_1$ and $S_2$ are the scales broadcast according to the quantization groups. G-B and B-B do not support bias, so $bias_n=0$ in the formula.
68 65 
69- $$66+ Here, $@$ denotes matrix multiplication, and $\odot$ denotes elementwise multiplication with broadcasting according to the quantization-parameter shapes. offset is used only in supported INT8-output combinations and cannot be combined with pertoken_scale. These formulas describe the quantized computation; the result is also converted to the type specified by dtype.
70- out = (x1@x2 + bias) * scale + offset
71- $$
72- 
73- - pertoken and no bias
74- 
75- $$
76- out = x1@x2 * scale * pertokenScaleOptional
77- $$
78- 
79- - pertoken and bias INT32
80- 
81- $$
82- out = (x1@x2 + bias) * scale * pertokenScaleOptional
83- $$
84- 
85- - pertoken and bias FLOAT32
86- 
87- $$
88- out = x1@x2 * scale * pertokenScaleOptional + bias
89- $$
90 67 
91## Parameters68## Parameters
92 69 
93-<table class="tg"><thead>70+| Parameter | Input/Output/Attribute | Description | Data Type | Data Format |
94- <tr>71+| --- | --- | --- | --- | --- |
95- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Parameter name</span></th>72+| x1 | Input | Left matrix, corresponding to X₁ in the formulas. | INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1 | ND, FRACTAL_NZ |
96- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Input/Output/Attribute</span></th>73+| x2 | Input | Right matrix, corresponding to X₂ in the formulas. | INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1 | ND, FRACTAL_NZ |
97- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Description</span></th>74+| scale | Input | Right-matrix quantization scale. UINT64 and INT64 store encoded quantization parameters; use the decoded scale in the formulas. | UINT64, FLOAT32, INT64, BFLOAT16, FLOAT8_E8M0 | ND, FRACTAL_NZ |
98- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data type</span></th>75+| offset | Optional input | Output quantization offset. Treated as 0 when omitted; used only in supported INT8-output combinations. | FLOAT32 | ND |
99- <th class="tg-hlb2"><span style="font-weight:700;color:var(--theme-text);background-color:var(--theme-table-header-bg)">Data format</span></th>76+| bias | Optional input | Matrix-product accumulation bias. Omitted bias adds nothing. Its accumulation position depends on the input and bias data types. | INT32, BFLOAT16, FLOAT16, FLOAT32 | ND |
100- </tr></thead>77+| pertoken_scale | Optional input | Left-matrix quantization scale. Treated as 1 when omitted. The corresponding scales are required for per-token and grouped quantization. | FLOAT32, FLOAT8_E8M0 | ND |
101-<tbody>78+| y | Output | Result after matrix multiplication and quantization, with data type specified by dtype. | FLOAT16, INT8, BFLOAT16, INT32, FLOAT32 | ND, FRACTAL_NZ |
102- <tr>79+| dtype | Attribute | Output y data type: 0 (FLOAT32), 1 (FLOAT16), 2 (INT8), 3 (INT32), or 27 (BFLOAT16), subject to product and input-combination restrictions. | INT64 | - |
103- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">x1</span></td>80+| transpose_x1 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x1: false disables transposition and true enables it.</li><li>Defaults to false.</li></ul> | BOOL | - |
104- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Input</span></td>81+| transpose_x2 | Optional attribute | <ul><li>Whether to transpose the last two dimensions of x2: false disables transposition and true enables it.</li><li>Defaults to false.</li></ul> | BOOL | - |
105- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Left matrix in the matrix multiplication operation.</span></td>82+| group_size | Optional attribute | <ul><li>Encodes quantization group sizes along M, N, and K, with 16 bits per component. For supported combinations, see <a href="#constraints">Constraints</a>.</li><li>Defaults to 0.</li></ul> | INT64 | - |
106- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td>
107- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND, FRACTAL_NZ</span></td>
108- </tr>
109- <tr>
110- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">x2</span></td>
111- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Input</span></td>
112- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Right matrix in the matrix multiplication operation.</span></td>
113- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">INT8, INT4, HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1</span></td>
114- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND, FRACTAL_NZ</span></td>
115- </tr>
116- <tr>
117- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">scale</span></td>
118- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Input</span></td>
119- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Scale factor of the quantization parameter, corresponding to scale in the formula.</span></td>
120- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">UINT64, FLOAT32, INT64, BF16, FLOAT8_E8M0</span></td>
121- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td>
122- </tr>
123- <tr>
124- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">offset</span></td>
125- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Input</span></td>
126- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Offset factor of the quantization parameter, corresponding to offset in the formula.</span></td>
127- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32</span></td>
128- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td>
129- </tr>
130- <tr>
131- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">bias</span></td>
132- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Input</span></td>
133- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Bias accumulated after the matrix multiplication operation, corresponding to bias in the formula.</span></td>
134- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">INT32, BF16, FLOAT16, FLOAT32</span></td>
135- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td>
136- </tr>
137- <tr>
138- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">pertoken_scale</span></td>
139- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Input</span></td>
140- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">Scale factor of the quantization parameter, corresponding to pertokenScaleOptional in the formula.</span></td>
141- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">FLOAT32, FLOAT8_E8M0</span></td>
142- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--devui-base-bg, #ffffff)">ND</span></td>
143- </tr>
144- <tr>
145- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">y</span></td>
146- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Output</span></td>
147- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">Computation result of the matrix multiplication operation.</span></td>
148- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">FLOAT16, INT8, BF16, INT32, FLOAT32</span></td>
149- <td class="tg-22a9"><span style="color:var(--theme-aide-text);background-color:var(--theme-table-header-bg)">ND</span></td>
150- </tr>
151-</tbody></table>
152 83 
153-- <term>Atlas A2 products</term>, <term>Atlas A3 products</term>:84+- <term>Atlas A3 products</term>, <term>Atlas A2 products</term>: x1 and x2 do not support HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN, or FLOAT4_E2M1. scale does not support FLOAT8_E8M0; pertoken_scale supports only FLOAT32; y does not support FLOAT32.
154- - x1 supports only the INT8 and INT4 data types.85+- <term>Atlas inference products</term>: x1 and x2 support only INT8. scale supports UINT64 and INT64, and also FLOAT32 for per-token quantization. bias supports INT32, and also FLOAT32 for per-token quantization. pertoken_scale supports only FLOAT32. y supports FLOAT16 and INT8, but only FLOAT16 for per-token quantization. y uses FRACTAL_NZ for per-token quantization and ND otherwise.
155- - x2 supports only the INT8 and INT4 data types.86+- <term>Kirin X90 processor products</term>, <term>Kirin 9030 processor products</term>: x1 and x2 support only INT8; x1 uses ND and x2 uses FRACTAL_NZ. scale supports only UINT64 and INT64; bias only INT32; pertoken_scale only FLOAT32; y only FLOAT16 and INT8.
156- - scale supports only the UINT64, FLOAT32, INT64, and BF16 data types.87+- <term>Ascend 950PR&950DT products</term>: FRACTAL_NZ scale is used only for ScaleNZ in full MX quantization. x2 must also use FRACTAL_NZ, and x1 and x2 must share the same type, FLOAT8_E4M3FN or FLOAT4_E2M1. For layout and conversion requirements, see [ScaleNZ Constraints](docs/aclnnQuantMatmulWeightNz_en.md#scalenz-constraints). scale uses ND on other products.
157- - bias supports only the INT32, BFLOAT16, FLOAT16, and FLOAT32 data types.
158- - offset supports only the FLOAT32 data type.
159- - pertoken_scale supports only the FLOAT32 data type.
160- - y supports only the FLOAT16 and BFLOAT16 data types.
161-- <term>Atlas inference products</term>:
162- - x1 supports only the INT8 data type.
163- - x2 supports only the INT8 data type.
164- - scale supports only the UINT64 and FLOAT32 data types.
165- - bias supports only the INT32 data type, and the pertoken scenario additionally supports the FLOAT32 data type.
166- - pertoken_scale supports only the FLOAT32 data type.
167- - y supports only the FLOAT16 and INT32 data types.
168-- Kirin X90/Kirin 9030 processor series products:
169- - x1 and x2 support only the INT8 data type.
170- - scale supports only the UINT64 and INT64 data types.
171- - offset supports only the FLOAT32 data type.
172- - bias supports only the INT32 data type.
173- - pertoken_scale supports only the FLOAT32 data type.
174- - y supports only the FLOAT16 and INT8 data types.
175 88 
176## Constraints89## Constraints
177 90 
178-- Empty tensors are not supported.91+- The input K dimensions must match, and batch dimensions must satisfy the [broadcasting rules](../../docs/en/context/broadcast_relationship.md). For shapes and product-specific restrictions, see the [Operator IR Description](docs/ascendir_QuantBatchMatmulV3_en.md).
179-- Contiguous tensors are supported. [Non-contiguous tensors](../../docs/en/context/non_contiguous_tensor.md) support only the transpose scenario.92+- INT4 inputs support only two dimensions, require transpose_x1=false, and must have an even innermost dimension.
93+- group_size is encoded as `group_size_k | (group_size_n << 16) | (group_size_m << 32)`. Zero components are inferred from input shapes, with divisibility required for the corresponding dimensions. The resulting sizes for MX, G-B, and B-B must be `[1, 1, 32]`, `[1, 128, 128]`, and `[128, 128, 128]`, respectively. Other modes use group_size=0.
94+- MX requires both pertoken_scale and scale as FLOAT8_E8M0. FLOAT4_E2M1 inputs must have an even innermost dimension, and K must be greater than 2.
95+- <term>Atlas A3 products</term>, <term>Atlas A2 products</term>: The last logical dimension of each input matrix must not exceed 65535.
96+- <term>Ascend 950PR&950DT products</term>: ND x2 supports M=0 or N=0 and produces an empty tensor. FRACTAL_NZ x2 supports M=0 and produces an empty tensor. Other cases do not support inputs with zero-sized dimensions.
180 97 
181## Invocation Description98## Invocation Description
182 99 
183-| Invocation method | Sample code | Description |100+| Invocation method | Sample code | Description |
184-| ---------------- | --------------------------- | --------------------------------------------------- |101+| --- | --- | --- |
185-| aclnn interface | [test_aclnn_quant_matmul_v3](examples/test_aclnn_quant_matmul_v3_at.cpp) | Calls the QuantBatchMatmulV3 operator through <br>[aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3_en.md)<br>[aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4_en.md)<br>[aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz_en.md)<br>[aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5_en.md)<br>and other interfaces. |102+| GE graph mode | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | Builds a graph using the [QuantBatchMatmulV3 operator IR](docs/ascendir_QuantBatchMatmulV3_en.md). |
186-| GEIR interface | [test_geir_quant_batch_matmul_v3](examples/test_geir_quant_batch_matmul_v3.cpp) | Builds the graph and invokes the operator through the [QuantBatchMatmulV3 operator IR](docs/ascendir_QuantBatchMatmulV3.md). |103+| aclnn API | [test_aclnn_quant_matmul_v3_at](examples/test_aclnn_quant_matmul_v3_at.cpp) | Invokes [aclnnQuantMatmulV3](docs/aclnnQuantMatmulV3_en.md). |
104+| aclnn API | [test_aclnn_quant_matmul_v4_at2_at3](examples/test_aclnn_quant_matmul_v4_at2_at3.cpp) | Invokes [aclnnQuantMatmulV4](docs/aclnnQuantMatmulV4_en.md) with left-matrix per-token scaling. |
105+| aclnn API | [test_aclnn_quant_matmul_weight_nz](examples/test_aclnn_quant_matmul_weight_nz.cpp) | Invokes [aclnnQuantMatmulWeightNz](docs/aclnnQuantMatmulWeightNz_en.md) with FRACTAL_NZ weights. |
106+| aclnn API | [test_aclnn_quant_matmul_v5_hif8hif8](../quant_batch_matmul_v4/examples/arch35/test_aclnn_quant_matmul_v5_hif8hif8.cpp) | Invokes [aclnnQuantMatmulV5](../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5_en.md). The example demonstrates HIFLOAT8 quantization executed by QuantBatchMatmulV3. |
107+ 
108+aclnnQuantMatmulV3 and aclnnQuantMatmulV4 are marked for future deprecation in their interface declarations. See the corresponding interface documents for migration information.
@@ -64,7 +64,7 @@
64### 输入与转置要求64### 输入与转置要求
65 65 
66- `x1`和`x2`的原始`shape`维数均不得小于2,至少一路数据输入存在`Transpose`或`TransposeD`,否则不触发融合。数据输入前仅有`Reshape`不能触发融合。66- `x1`和`x2`的原始`shape`维数均不得小于2,至少一路数据输入存在`Transpose`或`TransposeD`,否则不触发融合。数据输入前仅有`Reshape`不能触发融合。
67-- `x1`、`x2`的数据类型均支持`INT8`、`INT4`、`FLOAT8_E4M3FN`、`FLOAT8_E5M2`、`HIFLOAT8`、`FLOAT4_E2M1`;输出数据类型支持`INT8`、`FLOAT16`、`BF16`、`INT32`、`FLOAT32`。实际组合还需满足目标产品的算子约束。67+- `x1`、`x2`的数据类型均支持`INT8`、`INT4`、`FLOAT8_E4M3FN`、`FLOAT8_E5M2`、`HIFLOAT8`、`FLOAT4_E2M1`;输出数据类型支持`INT8`、`FLOAT16`、`BFLOAT16`、`INT32`、`FLOAT32`。实际组合还需满足目标产品的算子约束。
68- 支持动态`shape`;`x1`或`x2`的`shape`存在未知维度时,按动态场景处理。68- 支持动态`shape`;`x1`或`x2`的`shape`存在未知维度时,按动态场景处理。
69- 数据输入的转置必须只交换最后两维,其他维度的顺序保持不变。例如,形状为`(B, M, K)`的输入转置为`(B, K, M)`。交换批次维度等其他转置方式不属于本规则的适用范围。对应的`scale`转换也须保持量化系数与数据的对应关系,具体要求见下一节。69- 数据输入的转置必须只交换最后两维,其他维度的顺序保持不变。例如,形状为`(B, M, K)`的输入转置为`(B, K, M)`。交换批次维度等其他转置方式不属于本规则的适用范围。对应的`scale`转换也须保持量化系数与数据的对应关系,具体要求见下一节。
70- `pertoken_scale`是可选输入,未连接时不处理该路`scale`。70- `pertoken_scale`是可选输入,未连接时不处理该路`scale`。
@@ -102,8 +102,8 @@
102 102 
103| 产品 | 普通规则 | Limit补充规则 |103| 产品 | 普通规则 | Limit补充规则 |
104| ---- | ---- | ---- |104| ---- | ---- | ---- |
105-| Ascend 950PR&950DT系列产品 | `x1`、`x2`均不受本节65535维度阈值限制,满足其他使用约束时执行转置融合 | 跳过本补充规则,转置融合由普通规则处理 |105+| <term>Ascend 950PR&950DT系列产品</term> | `x1`、`x2`均不受本节65535维度阈值限制,满足其他使用约束时执行转置融合 | 跳过本补充规则,转置融合由普通规则处理 |
106-| Atlas A2训练系列产品/Atlas A2推理系列产品、Atlas A3训练系列产品/Atlas A3推理系列产品 | `x1`要求`outer` ≤ 65535;`x2`要求`outer` ≤ 65535或格式为`FRACTAL_NZ` | 分别检查`x1`、`x2`;仅处理满足0 < `outer` ≤ 65535且`inner` > 65535的支路 |106+| <term>Atlas A2系列产品</term>、<term>Atlas A3系列产品</term> | `x1`要求`outer` ≤ 65535;`x2`要求`outer` ≤ 65535或格式为`FRACTAL_NZ` | 分别检查`x1`、`x2`;仅处理满足0 < `outer` ≤ 65535且`inner` > 65535的支路 |
107 107 
108Limit规则的维度条件同样适用于`FRACTAL_NZ`格式,不因格式而放宽。参与判断的`outer`或`inner`为未知值`-1`时,该支路不满足Limit规则的维度条件。108Limit规则的维度条件同样适用于`FRACTAL_NZ`格式,不因格式而放宽。参与判断的`outer`或`inner`为未知值`-1`时,该支路不满足Limit规则的维度条件。
109 109 
@@ -117,13 +117,13 @@ Limit规则的维度条件同样适用于`FRACTAL_NZ`格式,不因格式而放
117产品的数据类型与格式支持范围参见[QuantBatchMatmulV3算子说明](../README.md)。本规则按平台指令能力判断融合条件,算子支持某产品并不表示该产品支持本规则的全部融合模式。117产品的数据类型与格式支持范围参见[QuantBatchMatmulV3算子说明](../README.md)。本规则按平台指令能力判断融合条件,算子支持某产品并不表示该产品支持本规则的全部融合模式。
118 118 
119<!-- npu="910b" id1 -->119<!-- npu="910b" id1 -->
120-Atlas A2训练系列产品/Atlas A2推理系列产品120+- <term>Atlas A2系列产品</term>
121<!-- end id1 -->121<!-- end id1 -->
122 122 
123<!-- npu="A3" id2 -->123<!-- npu="A3" id2 -->
124-Atlas A3训练系列产品/Atlas A3推理系列产品124+- <term>Atlas A3系列产品</term>
125<!-- end id2 -->125<!-- end id2 -->
126 126 
127<!-- npu="950" id3 -->127<!-- npu="950" id3 -->
128-Ascend 950PR&950DT系列产品128+- <term>Ascend 950PR&950DT系列产品</term>
129<!-- end id3 -->129<!-- end id3 -->
@@ -1,81 +1,79 @@
1# QuantBatchMatmulV3TransposeFusionPass1# QuantBatchMatmulV3TransposeFusionPass
2 2 
3-<!-- md-trans-meta sourceCommit=unknown translatedAt=2026-10-12T17:10:14.145Z pushedAt=2026-10-13T01:47:29.772Z -->
4- 
5## Fusion Patterns3## Fusion Patterns
6 4 
7-This fusion rule folds the explicit transpose in the `x1` or `x2` input branch of `QuantBatchMatmulV3` into the matmul computation, and can also process both branches at the same time. The recognized transpose node types are `Transpose` and `TransposeD`. When fusing the `x1` branch, invert the `transpose_x1` attribute of the `QuantBatchMatmulV3` node; when fusing the `x2` branch, invert the `transpose_x2` attribute of the node. Inverting means changing `false` to `true`, or `true` to `false`.5+This fusion rule folds explicit transposes on the `x1` or `x2` input branch of `QuantBatchMatmulV3` into matrix multiplication, and can process both branches together. It recognizes `Transpose` and `TransposeD` nodes. Fusing the `x1` branch inverts the node's `transpose_x1` attribute; fusing the `x2` branch inverts `transpose_x2`. Inversion changes `false` to `true` or `true` to `false`.
8 6 
9-In this document, "bypass" refers to the following connection adjustment: connect the input of a conversion node directly to the successor node of that conversion node, so that the current branch no longer passes through the conversion node. After the bypass, keep the conversion node if it still has other data consumers; otherwise, delete it. The successor node can be `QuantBatchMatmulV3` or the retained `Bitcast`.7+In this document, "bypass" means connecting a conversion node's input directly to its successor so that the current branch no longer passes through that conversion node. Retain the bypassed node if other data consumers still use it; otherwise, remove it. The successor can be `QuantBatchMatmulV3` or a retained `Bitcast`.
10 8 
11-The following figures assume that the [Input and Transpose Requirements](#input-and-transpose-requirements) and [Dimension and Platform Restrictions](#dimension-and-platform-constraints) are met. The variables in the figures are described as follows:9+The diagrams below assume that the [Input and Transpose Requirements](#input-and-transpose-requirements) and [Dimension and Platform Constraints](#dimension-and-platform-constraints) are satisfied. Variables used in the diagrams are defined below:
12 10 
13-- **Input Mapping**: Shows `x1`, `x2`, and the two `scale` inputs respectively. `x1_scale` denotes the input `pertoken_scale` (IR index 5), and `x2_scale` denotes the input `scale` (IR index 2); these two names are aliases used in the figures, not new operator parameters or operator attributes.11+- **Input mapping**: The diagrams show `x1`, `x2`, and their two `scale` branches. `x1_scale` denotes input `pertoken_scale` (IR index 5), and `x2_scale` denotes input `scale` (IR index 2). These names are diagram aliases, not additional operator parameters or attributes.
14-- **Symbol Definition**: `t1` and `t2` denote the values of the `transpose_x1` and `transpose_x2` attributes of `QuantBatchMatmulV3` before fusion, respectively; `!` denotes logical inversion.12+- **Symbols**: `t1` and `t2` denote the `transpose_x1` and `transpose_x2` attributes of `QuantBatchMatmulV3` before fusion. `!` denotes logical negation.
15-- **Source Tensor**: A source tensor with the same name before and after fusion refers to the same tensor.13+- **Source tensors**: A source tensor with the same name before and after fusion denotes the same tensor.
16-- **Omitted Content**: The figures omit `bias`, `offset`, the input specifying the transpose dimension order, and the input specifying the shape after reshape, and show only the data connections relevant to fusion.14+- **Omitted items**: The diagrams omit `bias`, `offset`, transpose-permutation inputs, and target-shape inputs. Only connections relevant to fusion are shown.
17 15 
18-### Fusion-Only x1 Side Transpose16+### Fusing Only the x1 Transpose
19 17 
20-When there exists one transpose node in each of the `x1` branch and the `x1_scale` branch, bypass respectively the two transpose nodes, and connect their inputs directly to the `x1` and `pertoken_scale` input ports of `QuantBatchMatmulV3`. Meanwhile, invert the `transpose_x1` attribute of the `QuantBatchMatmulV3` node. The connections of the `x2` and `scale` inputs and the `transpose_x2` attribute of the node remain unchanged.18+When the `x1` and `x1_scale` branches each contain a transpose node, bypass both transpose nodes and connect their inputs directly to the `x1` and `pertoken_scale` ports of `QuantBatchMatmulV3`, respectively. Invert the node's `transpose_x1` attribute. Keep the `x2` and `scale` connections and the `transpose_x2` attribute unchanged.
21 19 
22-![Fusion-only x1 and x1_scale transposes, four-way input independently connected](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_1.png)20+![Fusing the x1 and x1_scale transposes with four independent input branches](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_1.png)
23 21 
24-### Fusion-only x2 Side Transpose22+### Fusing Only the x2 Transpose
25 23 
26-When there exists one transpose node each in the `x2` branch and the `x2_scale` branch, bypass the two transpose nodes respectively, and connect their inputs directly to the `x2` and `scale` input ports of `QuantBatchMatmulV3`. Also, invert the `transpose_x2` attribute of the `QuantBatchMatmulV3` node. Keep the connections of the `x1` and `pertoken_scale` inputs and the `transpose_x1` attribute of the node unchanged.24+When the `x2` and `x2_scale` branches each contain a transpose node, bypass both transpose nodes and connect their inputs directly to the `x2` and `scale` ports of `QuantBatchMatmulV3`, respectively. Invert the node's `transpose_x2` attribute. Keep the `x1` and `pertoken_scale` connections and the `transpose_x1` attribute unchanged.
27 25 
28-![Fusion-only x2 and x2_scale transposes, four-way input connected independently](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_2.png)26+![Fusing the x2 and x2_scale transposes with four independent input branches](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_2.png)
29 27 
30-### Simultaneously Fusing Transposes on Both Sides28+### Fusing Transposes on Both Sides
31 29 
32-When there exists one transpose node in each of the four branches of `x1`, `x2`, `x1_scale`, and `x2_scale`, and both the `x1` and `x2` sides meet the usage constraints listed at the beginning of this section, the four transpose nodes are bypassed respectively. The inputs of the four transpose nodes connect to the `x1`, `x2`, `pertoken_scale`, and `scale` input ports of `QuantBatchMatmulV3`, respectively; the `transpose_x1` and `transpose_x2` attributes of the `QuantBatchMatmulV3` node are both inverted. Matching only the structure in the figure is insufficient to trigger fusion. Constraints such as data type, dimension, and platform capability must also be satisfied.30+When each of the `x1`, `x2`, `x1_scale`, and `x2_scale` branches contains a transpose node, and both data branches meet the constraints listed at the start of this section, bypass all four transpose nodes. Connect their inputs to the `x1`, `x2`, `pertoken_scale`, and `scale` ports of `QuantBatchMatmulV3`, respectively, and invert both `transpose_x1` and `transpose_x2`. Matching the illustrated structure alone is insufficient: the data type, dimension, and platform capability constraints must also be satisfied.
33 31 
34-![Four transposes of two data inputs and two scale inputs fused respectively](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_3.png)32+![Fusing the four transposes on the two data and two scale branches](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_3.png)
35 33 
36-### Reshape Fusion for Normal Scale34+### Reshape Fusion for Normal scale
37 35 
38-When there exists one transpose node in each of the `x1` and `x2` input branches, and one `Reshape` equivalent to the required transpose in each of the two `scale` branches, bypass the two transpose nodes and the two `Reshape` nodes respectively. The four branches connect to their corresponding `QuantBatchMatmulV3` input ports, and the `transpose_x1` and `transpose_x2` attributes of the `QuantBatchMatmulV3` node are both inverted. For identification requirements of static normal `scale`, see [Reshape Requirements for scale](#reshape-requirements-for-scale). The figure shows the case where both sides can be fused. You can also fuse only the side that satisfies the constraints.36+When each data branch contains a transpose node and each `scale` branch contains a `Reshape` equivalent to the required transpose, bypass both transpose nodes and both `Reshape` nodes. Connect the four branches to their corresponding `QuantBatchMatmulV3` input ports and invert both `transpose_x1` and `transpose_x2`. For static normal `scale` requirements, see [Reshape Requirements for scale](#reshape-requirements-for-scale). The figure shows fusion on both sides; only the side meeting the constraints can also be fused.
39 37 
40-![Fusion of the Reshapes of two normal scale branches with their corresponding data transposes](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_4.png)38+![Fusing each normal scale Reshape with its corresponding data transpose](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_4.png)
41 39 
42-### Reshape Fusion of MX Scale40+### Reshape Fusion for MX scale
43 41 
44-Taking the `x2` side as an example: the `x2` input is preceded by `Transpose` or `TransposeD`, `x2_scale` is of the `FLOAT8_E8M0` type, and its `Reshape` changes (1, N, 2) to (N, 1, 2). During fusion, the transpose node in the `x2` branch and the `Reshape` in the `x2_scale` branch are bypassed, and their inputs are connected to the `x2` and `scale` input ports of `QuantBatchMatmulV3` respectively; the `transpose_x2` attribute of the `QuantBatchMatmulV3` node is inverted. The connections of `x1` and `x1_scale` remain unchanged. The same processing logic applies when the `x1` side meets the corresponding conditions.42+For example, on the `x2` side, the data input is preceded by `Transpose` or `TransposeD`, `x2_scale` has type `FLOAT8_E8M0`, and its `Reshape` changes (1, N, 2) to (N, 1, 2). Bypass the data transpose and scale `Reshape`, connect their inputs to the `x2` and `scale` ports of `QuantBatchMatmulV3`, respectively, and invert `transpose_x2`. Keep the `x1` and `x1_scale` connections unchanged. Apply the same logic on the `x1` side when its corresponding conditions are met.
45 43 
46-![MX scenario showing x1, x2, the two scales, and the 3D Reshape respectively](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_5.png)44+![MX inputs showing x1, x2, both scales, and a three-dimensional Reshape](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_5.png)
47 45 
48### Transpose Fusion Retaining Bitcast46### Transpose Fusion Retaining Bitcast
49 47 
50-`Bitcast` reinterprets the bit representation of data as the target data type. When each of the four-way inputs is connected as "transpose node → `Bitcast` → `QuantBatchMatmulV3`", the fusion directly connects the inputs of the four transpose nodes to the subsequent `Bitcast` respectively, retains the four `Bitcast` nodes and their connections to `QuantBatchMatmulV3`, and performs attribute inversion on the `transpose_x1` and `transpose_x2` attributes of the `QuantBatchMatmulV3` node. After fusion, the data type of the `Bitcast` output remains unchanged.48+`Bitcast` reinterprets the data bits as the target data type. When each of the four branches follows "transpose node → `Bitcast` → `QuantBatchMatmulV3`", connect each transpose node's input directly to its following `Bitcast`. Retain all four `Bitcast` nodes and their connections to `QuantBatchMatmulV3`, and invert both `transpose_x1` and `transpose_x2`. The output data type of each `Bitcast` remains unchanged.
51 49 
52-![Fusion of transposes on four-way inputs with Bitcast retained respectively](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_6.png)50+![Fusing the four input transposes while retaining their Bitcast nodes](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_6.png)
53 51 
54-The input branches are not required to contain `Bitcast` at the same time. A transpose node can connect to the matmul directly or through a `Bitcast`; fusion across multiple consecutive `Bitcast` nodes is not supported. For the `scale` branch, the node before the `Bitcast` can also be a `Reshape` that meets the requirements described below. For platform restrictions and the applicability boundary of dynamic `Reshape`, see the usage constraints.52+The input branches do not all have to contain `Bitcast`. A transpose can connect to matrix multiplication directly or through one `Bitcast`; fusion across multiple consecutive `Bitcast` nodes is unsupported. On a `scale` branch, the node preceding `Bitcast` can also be a `Reshape` meeting the requirements below. For platform restrictions and the scope of dynamic `Reshape` support, see Constraints.
55 53 
56-### Reshape Fusion for Dynamic Scale54+### Reshape Fusion for Dynamic scale
57 55 
58-If `x1` or `x2` contains unknown dimensions, handle it as a dynamic scenario. Taking the `x2` side as an example, the fusion bypasses the transpose node on the `x2` branch and the `Reshape` on the `x2_scale` branch, connects their inputs to the `x2` and `scale` input ports of `QuantBatchMatmulV3` respectively, and will invert the `transpose_x2` attribute of the `QuantBatchMatmulV3` node. `x1` and `x1_scale` keep their original connections. The dynamic scenario does not perform the dimension condition check for a static `Reshape`, but the input graph must still ensure transpose equivalence between the `Reshape` and the required `scale`.56+Treat the scenario as dynamic if `x1` or `x2` contains an unknown dimension. For example, on the `x2` side, bypass the data transpose and `x2_scale` `Reshape`, connect their inputs to the `x2` and `scale` ports of `QuantBatchMatmulV3`, respectively, and invert `transpose_x2`. Keep `x1` and `x1_scale` unchanged. Static `Reshape` dimension checks are not performed in dynamic scenarios, but the input graph must still ensure that the `Reshape` is equivalent to the required scale transpose.
59 57 
60-![Four-way inputs shown separately in the dynamic scenario, with only the x2 side transpose and Reshape processed](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_7.png)58+![Four independent inputs in a dynamic scenario, with only the x2 transpose and Reshape fused](../../../docs/en/figures/QuantBatchMatmulV3TransposeFusionPass_7.png)
61 59 
62-The `scale` conversion nodes in each figure are the specific structures of the illustrated scenario. A conversion node is not required on every `scale` branch. A `scale` branch is processed only when the `Transpose` or `TransposeD` of the corresponding data input is fused; when `pertoken_scale` is not connected, that input is not processed. A bypassed node is retained if it still has other data consumers; otherwise, it is removed from the graph.60+The scale conversion nodes shown in each figure are specific to that example. A conversion node is not required on every `scale` branch. Process a `scale` branch only when the corresponding data input's `Transpose` or `TransposeD` is fused; do not process `pertoken_scale` if it is unconnected. Retain a bypassed node if other data consumers still use it; otherwise, remove it from the graph.
63 61 
64## Constraints62## Constraints
65 63 
66### Input and Transpose Requirements64### Input and Transpose Requirements
67 65 
68-- The original `shape` of both `x1` and `x2` must have at least 2 dimensions. At least one data input must connect to `Transpose` or `TransposeD`; otherwise, fusion is not triggered. A data input preceded only by `Reshape` does not trigger fusion.66+- The original shapes of `x1` and `x2` must each have at least two dimensions. At least one data input must have a preceding `Transpose` or `TransposeD`; otherwise, fusion is not triggered. A `Reshape` alone before a data input cannot trigger fusion.
69-- The data types of both `x1` and `x2` support `INT8`, `INT4`, `FLOAT8_E4M3FN`, `FLOAT8_E5M2`, `HIFLOAT8`, and `FLOAT4_E2M1`. The output data types support `INT8`, `FLOAT16`, `BF16`, `INT32`, and `FLOAT32`. The actual combinations must also meet the operator constraints of the target product.67+- `x1` and `x2` each support `INT8`, `INT4`, `FLOAT8_E4M3FN`, `FLOAT8_E5M2`, `HIFLOAT8`, and `FLOAT4_E2M1`. The output supports `INT8`, `FLOAT16`, `BFLOAT16`, `INT32`, and `FLOAT32`. The actual combination must also satisfy the target product's operator constraints.
70-- Dynamic `shape` is supported. If the `shape` of `x1` or `x2` contains unknown dimensions, it is processed as a dynamic scenario.68+- Dynamic shapes are supported. If the shape of `x1` or `x2` contains an unknown dimension, the case is treated as dynamic.
71-- The transpose of a data input must only swap the last two dimensions, with the order of other dimensions unchanged. For example, an input of shape `(B, M, K)` is transposed to `(B, K, M)`. Other transposes, such as swapping the batch dimension, are out of the scope of this rule. The corresponding `scale` conversion must also maintain the correspondence between quantization coefficients and data. For details, see the next section.69+- A data transpose must swap only the last two dimensions and preserve the order of all other dimensions, for example, `(B, M, K)` → `(B, K, M)`. Other permutations, such as swapping batch dimensions, are outside this rule's scope. The corresponding scale transformation must also preserve the correspondence between quantization coefficients and data. See the next section.
72-- `pertoken_scale` is an optional input. If it is not connected, the corresponding `scale` is not processed.70+- `pertoken_scale` is optional. Its scale branch is not processed when unconnected.
73 71 
74-### Restrictions in the Bitcast Scenario72+### Restrictions for Bitcast Patterns
75 73 
76-On some platforms, the matrix multiplication instruction supports mixed INT8 and INT4 inputs. On such platforms, if one of the four-way inputs matches the structure in the following table, this rule retains the entire `QuantBatchMatmulV3` node and all of its input branches, and does not perform this transpose fusion. This restriction applies even if the current node does not use mixed INT8 and INT4 inputs.74+Some platforms support mixed INT8 and INT4 matrix multiplication inputs. On those platforms, if any of the four input branches matches a pattern in the table below, retain the entire `QuantBatchMatmulV3` node and all its input branches without performing this transpose fusion. This restriction applies even when the current node does not use mixed INT8 and INT4 inputs.
77 75 
78-| Input Branch | Connection Structure That Triggers the Preceding Constraint |76+| Input branch | Connection pattern that triggers the restrictions above |
79| ---- | ---- |77| ---- | ---- |
80| `x1`, `x2`, or either of the two `scale` branches | `Transpose` or `TransposeD` → `Bitcast` → `QuantBatchMatmulV3` |78| `x1`, `x2`, or either of the two `scale` branches | `Transpose` or `TransposeD` → `Bitcast` → `QuantBatchMatmulV3` |
81| Either of the two `scale` branches | `Reshape` that meets the fusion requirements in the next section → `Bitcast` → `QuantBatchMatmulV3` |79| Either of the two `scale` branches | `Reshape` that meets the fusion requirements in the next section → `Bitcast` → `QuantBatchMatmulV3` |
@@ -104,8 +102,8 @@ Let `outer` denote the penultimate dimension of the original `shape` of the data
104 102 
105| Product | General Rule | Limit Supplementary Rule |103| Product | General Rule | Limit Supplementary Rule |
106| ---- | ---- | ---- |104| ---- | ---- | ---- |
107-| <term>Ascend 950PR&950DT products</term> | `x1` and `x2` are not subject to the 65535 dimension threshold in this section. Transpose fusion is performed when other usage constraints are met. | This supplementary rule is skipped, and transpose fusion is handled by the general rule. |105+| <term>Ascend 950PR&950DT products</term> | Neither `x1` nor `x2` is subject to this section's 65535-dimension threshold. Transpose fusion runs when the other constraints are met. | This supplementary rule is skipped; transpose fusion is handled by the regular rule. |
108-| <term>Atlas A2 products</term> and <term>Atlas A3 products</term> | `x1` requires `outer` ≤ 65535; `x2` requires `outer` ≤ 65535 or the `FRACTAL_NZ` format. | Checks `x1` and `x2` respectively; processes only branches that meet 0 < `outer` ≤ 65535 and `inner` > 65535. |106+| <term>Atlas A2 products</term>, <term>Atlas A3 products</term> | `x1` requires `outer` ≤ 65535. `x2` requires `outer` ≤ 65535 or `FRACTAL_NZ` format. | Checks `x1` and `x2` independently and processes only branches with 0 < `outer` ≤ 65535 and `inner` > 65535. |
109 107 
110The dimension conditions of the Limit rule also apply to the `FRACTAL_NZ` format; they are not relaxed due to the format. If the `outer` or `inner` value involved in the check is the unknown value `-1`, the branch does not satisfy the dimension conditions of the Limit rule.108The dimension conditions of the Limit rule also apply to the `FRACTAL_NZ` format; they are not relaxed due to the format. If the `outer` or `inner` value involved in the check is the unknown value `-1`, the branch does not satisfy the dimension conditions of the Limit rule.
111 109 
@@ -116,16 +114,16 @@ The dimension conditions of the Limit rule also apply to the `FRACTAL_NZ` format
116 114 
117## Applicable Products115## Applicable Products
118 116 
119-See [QuantBatchMatmulV3 Operator Description](../README.md) for the data types and formats supported by the product. This rule determines fusion conditions based on platform instruction capabilities. Operator support for a product does not mean that the product supports all fusion modes of this rule.117+See [QuantBatchMatmulV3 Operator Description](../README_en.md) for the data types and formats supported by the product. This rule determines fusion conditions based on platform instruction capabilities. Operator support for a product does not mean that the product supports all fusion modes of this rule.
120 118 
121<!-- npu="910b" id1 -->119<!-- npu="910b" id1 -->
122-<term>Atlas A2 products</term>120+- <term>Atlas A2 products</term>
123<!-- end id1 -->121<!-- end id1 -->
124 122 
125<!-- npu="A3" id2 -->123<!-- npu="A3" id2 -->
126-<term>Atlas A3 products</term>124+- <term>Atlas A3 products</term>
127<!-- end id2 -->125<!-- end id2 -->
128 126 
129<!-- npu="950" id3 -->127<!-- npu="950" id3 -->
130-<term>Ascend 950PR&950DT products</term>128+- <term>Ascend 950PR&950DT products</term>
131<!-- end id3 -->129<!-- end id3 -->
@@ -2,13 +2,13 @@
2 2 
3## 融合模式3## 融合模式
4 4 
5-该融合将MatMul/MatMulV2/MatMulV3、BatchMatMul/BatchMatMulV2/BatchMatMulV3算子转换为QuantBatchMatMulV3算子,并插入scale为1.0的常量作为scale输入,bias输入、转置属性及输出y的数据类型保持不变。5+该融合将MatMul/MatMulV2/MatMulV3、BatchMatMul/BatchMatMulV2/BatchMatMulV3算子转换为QuantBatchMatmulV3算子,并插入scale为1.0的常量作为scale输入,bias输入、转置属性及输出y的数据类型保持不变。
6 6 
7-![](../../../docs/zh/figures/zz_matmul_to_qbmmv3_fusion_pass_1.png)7+![MatMul转换为QuantBatchMatmulV3的融合前后示意图](../../../docs/zh/figures/zz_matmul_to_qbmmv3_fusion_pass_1.png)
8 8 
9## 使用约束9## 使用约束
10 10 
11-- MatMul类型包括BatchMatMul/BatchMatMulV2/MatMul/MatMulV2。11+- MatMul类型包括MatMul、MatMulV2、MatMulV3、BatchMatMul、BatchMatMulV2、BatchMatMulV3。
12- 输入的数据类型支持HIFLOAT8。12- 输入的数据类型支持HIFLOAT8。
13<!-- npu="950" id1 -->13<!-- npu="950" id1 -->
14- <term>Ascend 950PR&950DT系列产品</term>:支持输入为INT8且输出为INT32的组合。14- <term>Ascend 950PR&950DT系列产品</term>:支持输入为INT8且输出为INT32的组合。
@@ -17,5 +17,5 @@
17## 支持的型号17## 支持的型号
18 18 
19<!-- npu="950" id2 -->19<!-- npu="950" id2 -->
20-Ascend 950PR&950DT系列产品20+- <term>Ascend 950PR&950DT系列产品</term>
21<!-- end id2 -->21<!-- end id2 -->
@@ -1,17 +1,15 @@
1# ZZMatMulTOQBMMV3FusionPass1# ZZMatMulTOQBMMV3FusionPass
2 2 
3-<!-- md-trans-meta sourceCommit=unknown translatedAt=2026-10-12T17:10:34.905Z pushedAt=2026-10-13T01:47:29.775Z -->3+## Fusion Patterns
4 4 
5-## Fusion Pattern5+This fusion converts MatMul/MatMulV2/MatMulV3 and BatchMatMul/BatchMatMulV2/BatchMatMulV3 into QuantBatchMatmulV3, inserting a constant scale of 1.0. The bias input, transpose attributes, and output y data type are preserved.
6 6 
7-This fusion converts the MatMul/MatMulV2/MatMulV3 and BatchMatMul/BatchMatMulV2/BatchMatMulV3 operators into the QuantBatchMatMulV3 operator, and inserts a constant with a scale of 1.0 as the scale input. The bias input, transpose attribute, and data type of output y remain unchanged.7+![Before and after converting MatMul to QuantBatchMatmulV3](../../../docs/en/figures/ZZMatMulTOQBMMV3FusionPass_1.png)
8- 
9-![](../../../docs/en/figures/ZZMatMulTOQBMMV3FusionPass_1.png)
10 8 
11## Constraints9## Constraints
12 10 
13-- The MatMul type includes BatchMatMul/BatchMatMulV2/MatMul/MatMulV2.11+- MatMul types include MatMul, MatMulV2, MatMulV3, BatchMatMul, BatchMatMulV2, and BatchMatMulV3.
14-- The input supports the HIFLOAT8 data type.12+- The supported input data type is HIFLOAT8.
15<!-- npu="950" id1 -->13<!-- npu="950" id1 -->
16- <term>Ascend 950PR&950DT products</term>: Support the combination of INT8 input and INT32 output.14- <term>Ascend 950PR&950DT products</term>: Support the combination of INT8 input and INT32 output.
17<!-- end id1 -->15<!-- end id1 -->
@@ -19,5 +17,5 @@ This fusion converts the MatMul/MatMulV2/MatMulV3 and BatchMatMul/BatchMatMulV2/
19## Applicable Products17## Applicable Products
20 18 
21<!-- npu="950" id2 -->19<!-- npu="950" id2 -->
22-<term>Ascend 950PR&950DT products</term>20+- <term>Ascend 950PR&950DT products</term>
23<!-- end id2 -->21<!-- end id2 -->
@@ -1,6 +1,6 @@
1# aclnnQuantMatmulV31# aclnnQuantMatmulV3
2 2 
3-**须知:该接口后续版本会废弃,请使用最新aclnnQuantMatmulV5接口。**3+**须知:该接口后续版本会废弃,请使用[aclnnQuantMatmulV5](../../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)接口。**
4 4 
5## 产品支持情况5## 产品支持情况
6 6 
@@ -1,6 +1,6 @@
1# aclnnQuantMatmulV41# aclnnQuantMatmulV4
2 2 
3-**须知:该接口后续版本会废弃,请使用最新aclnnQuantMatmulV5接口。**3+**须知:该接口后续版本会废弃,请使用[aclnnQuantMatmulV5](../../quant_batch_matmul_v4/docs/aclnnQuantMatmulV5.md)接口。**
4 4 
5## 产品支持情况5## 产品支持情况
6 6 
Mmatmul/quant_matmul_activation_quant/tests/README_en.md+40-30File content is under review, please try again later