| Files | Last commit | Last update |
|---|---|---|
| 6 days ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 4 days ago | ||
| 4 days ago | ||
| 25 days ago | ||
| 1 month ago | ||
| 4 days ago | ||
| 6 days ago | ||
| 6 days ago |
SwigluGroupQuant
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | × |
| Atlas A2 products | × |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
Function
Description:
The SwigluGroupQuant operator implements the fused computation of the SwiGLU activation function and group quantization. Five quantization modes are supported:
- quant_mode=0: Block Quant (FP8 block quantization, fixed 128-element groups)
- quant_mode=1: MX Quant (FP8 MX quantization, fixed 32-element groups)
- quant_mode=2: HiFp8 Static Quant (HiFp8 static quantization)
- quant_mode=3: HiFp8 Dynamic Quant (HiFp8 dynamic quantization)
- quant_mode=5: MX Quant V2 (variant Clipped SwiGLU + cuBLAS MX FP8 quantization, fixed 32-element groups)
Formula:
Basic computation flow
Step 1: GroupIndex processing (optional) → Compute real_bs
Step 2: Input splitting (processes only the first real_bs rows)
Step 3: Clamp processing (optional, processes only the first real_bs rows)
Step 4: SwiGLU activation / variant SwiGLU activation (processes only the first real_bs rows)
Step 5: Weight weighting (optional, processes only the first real_bs rows)
Step 6: Quantization (processes only the first real_bs rows)
Step 1: GroupIndex processing (optional)
When group_index is provided, it dynamically computes the number of tokens to actually process:
group_sum=∑g=0G−1group_index[g]\text{group\_sum} = \sum_{g=0}^{G-1} \text{group\_index}[g]
real_bs=min(group_sum,N)\text{real\_bs} = \min(\text{group\_sum}, N)
where:
- GG is the number of MoE expert groups
- NN is the first dimension of the input tensor (preset batch size)
- All subsequent steps process only the first real_bs\text{real\_bs} rows of data
MoE scenario: In MoE inference, different experts may process different numbers of tokens. group_index allows dynamic adjustment of the processing range to avoid processing empty data.
Step 2: Input splitting
The input tensor x∈RN×D\mathbf{x} \in \mathbb{R}^{N \times D} is split along the last dimension into two parts:
x0[n,d]=x[n,d],d∈[0,D/2)\mathbf{x}_0[n, d] = \mathbf{x}[n, d], \quad d \in [0, D/2)
x1[n,d]=x[n,d+D/2],d∈[0,D/2)\mathbf{x}_1[n, d] = \mathbf{x}[n, d + D/2], \quad d \in [0, D/2)
Step 3: Clamp processing (optional)
When clamp_limit > 0, the inputs are clamped:
x0′[n,d]=min(x0[n,d],c)\mathbf{x}_0'[n, d] = \min(\mathbf{x}_0[n, d], c)
x1′[n,d]=min(max(x1[n,d],−c),c)\mathbf{x}_1'[n, d] = \min(\max(\mathbf{x}_1[n, d], -c), c)
where cc is clamp_limit.
Purpose of Clamp:
- x0\mathbf{x}_0 (gating branch) is limited to the positive range [0,c][0, c] to prevent gradient vanishing of sigmoid
- x1\mathbf{x}_1 (linear branch) is limited to the symmetric range [−c,c][-c, c] to prevent numeric overflow
Step 4: SwiGLU activation / variant SwiGLU activation
1. Standard SwiGLU (quant_mode=0/1/2/3)
Element-wise computation:
yorigin[t,h]=SiLU(x0′[t,h])⋅x1′[t,h]\mathbf{y}_{\text{origin}}[t,h] =\operatorname{SiLU}(\mathbf{x}_0'[t,h])\cdot\mathbf{x}_1'[t,h]
where:
SiLU(z)=z⋅σ(z)=z1+e−z\operatorname{SiLU}(z)=z\cdot\sigma(z)=\frac{z}{1+e^{-z}}
Therefore:
yorigin[t,h]=x0′[t,h]1+e−x0′[t,h]⋅x1′[t,h]\mathbf{y}_{\text{origin}}[t,h] =\frac{\mathbf{x}_0'[t,h]}{1+e^{-\mathbf{x}_0'[t,h]}}\cdot\mathbf{x}_1'[t,h]
2. Variant Clipped SwiGLU (quant_mode=5)
quant_mode=5 supports alpha and bias. The activation formula is:
yorigin[t,h]=x0′[t,h]⋅σ (αx0′[t,h])⋅(x1′[t,h]+β)\mathbf{y}_{\text{origin}}[t,h] =\mathbf{x}_0'[t,h]\cdot \sigma\!\left(\alpha\mathbf{x}_0'[t,h]\right)\cdot \left(\mathbf{x}_1'[t,h]+\beta\right)
where:
- α\alpha corresponds to the attribute
alpha, with a default value of 1.01.0; - β\beta corresponds to the attribute
bias, with a default value of 0.00.0; - Clamp is performed before
alphaandbiasparticipate in the activation computation.
The equivalent step-by-step computation is:
t1[t,h]=−αx0′[t,h]t2[t,h]=et1[t,h]t3[t,h]=1+t2[t,h]t4[t,h]=x0′[t,h]t3[t,h]t5[t,h]=x1′[t,h]+βyorigin[t,h]=t4[t,h]⋅t5[t,h]\begin{aligned} t_1[t,h]&=-\alpha\mathbf{x}_0'[t,h] \\ t_2[t,h]&=e^{t_1[t,h]} \\ t_3[t,h]&=1+t_2[t,h] \\ t_4[t,h]&=\frac{\mathbf{x}_0'[t,h]}{t_3[t,h]} \\ t_5[t,h]&=\mathbf{x}_1'[t,h]+\beta \\ \mathbf{y}_{\text{origin}}[t,h]&=t_4[t,h]\cdot t_5[t,h] \end{aligned}
When α=1.0\alpha=1.0 and β=0.0\beta=0.0, the variant formula degenerates to standard SwiGLU.
yOrigin output: When output_origin=true, yOrigin stores the preceding activation result, that is, the SwiGLU result before Weight weighting; for quant_mode=5, it already includes the effects of Clamp, alpha, and bias.
Step 5: Weight weighting (optional)
When weight is provided, the SwiGLU output is weighted:
yweighted[n,d]=yswiglu[n,d]⋅w[n]\mathbf{y}_{\text{weighted}}[n, d] = \mathbf{y}_{\text{swiglu}}[n, d] \cdot w[n]
where w[n]w[n] is the weight value of the n-th token.
MoE scenario: weight comes from the softmax output of the expert router and represents the weight of this token for the current expert.
Step 6: Quantization
quant_mode=0 (Block Quant)
Group division: The output is divided along the last dimension into groups of 128 elements:
y=[g0,g1,…,gK],K=⌈D/2/128⌉\mathbf{y} = [\mathbf{g}_0, \mathbf{g}_1, \ldots, \mathbf{g}_K], \quad K = \lceil D/2 / 128 \rceil
Each group is gi∈RN×128\mathbf{g}_i \in \mathbb{R}^{N \times 128}.
Non-finite value masking and absolute value computation:
z[n,j]=yweighted[n,j]⋅0(generate a zero tensor)mfinite[n,j]=(z[n,j]=z[n,j])yabs[n,j]={∣yweighted[n,j]∣,mfinite[n,j]0,otherwise\begin{aligned} \mathbf{z}[n, j] &= \mathbf{y}_{\text{weighted}}[n, j] \cdot 0 \quad \text{(generate a zero tensor)} \\ \mathbf{m}_{\text{finite}}[n, j] &= (\mathbf{z}[n, j] = \mathbf{z}[n, j]) \\ \mathbf{y}_{\text{abs}}[n, j] &= \begin{cases} |\mathbf{y}_{\text{weighted}}[n, j]|, & \mathbf{m}_{\text{finite}}[n, j] \\ 0, & \text{otherwise} \end{cases} \end{aligned}
Masking principle: NaN has the property NaN != NaN; meanwhile Inf * 0 also yields NaN, so this step masks NaN and Inf when computing amax.
Scale computation:
For the i-th group (containing 128 consecutive elements):
ai=maxj=0127yabs[j]a_i = \max_{j=0}^{127} \mathbf{y}_{\text{abs}}[j]
a^i=max(ai,10−4)\hat{a}_i = \max(a_i, 10^{-4})
siraw=a^iMfp8s_i^{\text{raw}} = \frac{\hat{a}_i}{M_{\text{fp8}}}
where Mfp8M_{\text{fp8}} takes the following values:
- FP8 E4M3FN: Mfp8=448.0M_{\text{fp8}} = 448.0
- FP8 E5M2: Mfp8=57344.0M_{\text{fp8}} = 57344.0
Scale output and InvScale computation:
When round_scale=false:
si=siraw,InvScalei=Mfp8a^i=1sis_i = s_i^{\text{raw}}, \quad \text{InvScale}_i = \frac{M_{\text{fp8}}}{\hat{a}_i} = \frac{1}{s_i}
When round_scale=true, the scale is rounded up to a power of 2:
ei=⌈log2(siraw)⌉e_i = \lceil \log_2(s_i^{\text{raw}}) \rceil
si=2ei,InvScalei=2−eis_i = 2^{e_i}, \quad \text{InvScale}_i = 2^{-e_i}
where sis_i is written to the FLOAT32 scale output.
Quantization computation:
yscaled[n,j]=yweighted[n,j]⋅InvScalei,j∈group i\mathbf{y}_{\text{scaled}}[n, j] = \mathbf{y}_{\text{weighted}}[n, j] \cdot \text{InvScale}_i, \quad j \in \text{group } i
If yscaled[n,j]\mathbf{y}_{\text{scaled}}[n,j] is NaN or Inf, the implementation uses the original yweighted[n,j]\mathbf{y}_{\text{weighted}}[n,j] as the FP8 cast input:
ycast_in[n,j]={yscaled[n,j],yscaled[n,j] is finiteyweighted[n,j],otherwise\mathbf{y}_{\text{cast\_in}}[n, j] = \begin{cases} \mathbf{y}_{\text{scaled}}[n, j], & \mathbf{y}_{\text{scaled}}[n, j] \text{ is finite} \\ \mathbf{y}_{\text{weighted}}[n, j], & \text{otherwise} \end{cases}
yquant[n,j]=cast_fp8_rint(ycast_in[n,j])\mathbf{y}_{\text{quant}}[n, j] = \text{cast\_fp8\_rint}(\mathbf{y}_{\text{cast\_in}}[n, j])
where cast_fp8_rint is the type conversion from FP32 to FP8, using the RINT (round to nearest) mode.
quant_mode=1 (MX Quant)
MX quantization principle: adopts the combination of E8M0 Scale + FP8 Data.
Grouping: every 32 elements form one group:
y=[g0,g1,…,gK],gi∈R32\mathbf{y} = [\mathbf{g}_0, \mathbf{g}_1, \ldots, \mathbf{g}_K], \quad \mathbf{g}_i \in \mathbb{R}^{32}
Amax computation:
ai=maxj=031∣gi[j]∣a_i = \max_{j=0}^{31} |\mathbf{g}_i[j]|
a^i=max(ai,10−4)\hat{a}_i = \max(a_i, 10^{-4})
Raw scale computation:
siraw=a^iMfp8s_i^{\text{raw}} = \frac{\hat{a}_i}{M_{\text{fp8}}}
where Mfp8M_{\text{fp8}} takes the following values:
- FP8 E4M3FN: Mfp8=448.0M_{\text{fp8}} = 448.0
- FP8 E5M2: Mfp8=57344.0M_{\text{fp8}} = 57344.0
quant_mode=1 supports only round_scale=true, which rounds the raw scale up to a power of 2:
ei=⌈log2(siraw)⌉e_i = \lceil \log_2(s_i^{\text{raw}}) \rceil
The equivalent computation based on the FP32 bit pattern is:
ei=E(siraw)−127+1mantissa(siraw)≠0e_i = E(s_i^{\text{raw}}) - 127 + \mathbf{1}_{\text{mantissa}(s_i^{\text{raw}}) \ne 0}
E8M0 scale encoding:
sie8m0=ei+127s_i^{\text{e8m0}} = e_i + 127
where sie8m0s_i^{\text{e8m0}} is written to the FLOAT8_E8M0 scale output, and the actual scale value represented is 2ei2^{e_i}.
InvScale computation:
InvScalei=2−ei\text{InvScale}_i = 2^{-e_i}
Quantization computation:
yquant[j]=cast_fp8_rint(yweighted[j]⋅InvScalei),j∈group i\mathbf{y}_{\text{quant}}[j] = \text{cast\_fp8\_rint}\left(\mathbf{y}_{\text{weighted}}[j] \cdot \text{InvScale}_i\right), \quad j \in \text{group } i
quant_mode=2 (HiFp8 Static Quant)
Static quantization description: uses the pre-provided invScale to perform scaled quantization on the weighted SwiGLU output.
Case 1: no GroupIndex (groupIndex is empty):
yquant[n,d]=hif8_cast(yweighted[n,d]⋅invScale[0]),n∈[0,N),d∈[0,D/2)\mathbf{y}_{\text{quant}}[n, d] = \text{hif8\_cast}\left(\mathbf{y}_{\text{weighted}}[n, d] \cdot \text{invScale}[0]\right), \quad n \in [0, N), \quad d \in [0, D/2)
where hif8_cast is the HiFloat8 type conversion function.
Case 2: with GroupIndex (groupIndex is not empty):
Let GG be the number of MoE expert groups, and groupIndex[g]\text{groupIndex}[g] denote the number of tokens processed by the g-th expert.
Compute the start and end indices of each group:
start(0)=0,end(g)=∑k=0ggroupIndex[k],start(g)=end(g−1)\text{start}^{(0)} = 0, \quad \text{end}^{(g)} = \sum_{k=0}^{g} \text{groupIndex}[k], \quad \text{start}^{(g)} = \text{end}^{(g-1)}
For the g-th group, quantization uses the corresponding scaling factor invScale[g]\text{invScale}[g]:
yquant[n,d]=hif8_cast(yweighted[n,d]⋅invScale[g]),n∈[start(g),end(g)),d∈[0,D/2)\mathbf{y}_{\text{quant}}[n, d] = \text{hif8\_cast}\left(\mathbf{y}_{\text{weighted}}[n, d] \cdot \text{invScale}[g]\right), \quad n \in [\text{start}^{(g)}, \text{end}^{(g)}), \quad d \in [0, D/2)
MoE scenario: In MoE inference, different experts process different numbers of tokens. groupIndex identifies the token range processed by each expert, and invScale is the static scaling factor precomputed for each expert.
quant_mode=3 (HiFp8 Dynamic Quant)
Dynamic quantization description: computes the scaling factor dynamically from the weighted SwiGLU output for quantization.
Case 1: no GroupIndex (groupIndex is empty):
Compute the global maximum absolute value:
amax=max(maxn∈[0,N),d∈[0,D/2)∣yweighted[n,d]∣,ϵ)a_{\max} = \max\left(\max_{n \in [0, N), d \in [0, D/2)} |\mathbf{y}_{\text{weighted}}[n, d]|, \epsilon\right)
where ϵ\epsilon is a constant for numerical stability.
Compute the scaling factor:
s=amaxMhif8s = \frac{a_{\max}}{M_{\text{hif8}}}
where Mhif8M_{\text{hif8}} is dstTypeMax, representing the maximum finite value of the HiFloat8 type.
Quantization computation:
yquant[n,d]=hif8_cast(yweighted[n,d]s),n∈[0,N),d∈[0,D/2)\mathbf{y}_{\text{quant}}[n, d] = \text{hif8\_cast}\left(\frac{\mathbf{y}_{\text{weighted}}[n, d]}{s}\right), \quad n \in [0, N), \quad d \in [0, D/2)
where hif8_cast is the HiFloat8 type conversion function.
Case 2: with GroupIndex (groupIndex is not empty):
Let GG be the number of MoE expert groups, and groupIndex[g]\text{groupIndex}[g] denote the number of tokens processed by the g-th expert.
Compute the start and end indices of each group:
start(0)=0,end(g)=∑k=0ggroupIndex[k],start(g)=end(g−1)\text{start}^{(0)} = 0, \quad \text{end}^{(g)} = \sum_{k=0}^{g} \text{groupIndex}[k], \quad \text{start}^{(g)} = \text{end}^{(g-1)}
For the g-th group, extract the corresponding data:
y(g)=yweighted[start(g):end(g),:]\mathbf{y}^{(g)} = \mathbf{y}_{\text{weighted}}[\text{start}^{(g)}:\text{end}^{(g)}, :]
Compute the maximum absolute value of this group:
amax(g)=max(maxn∈[start(g),end(g)),d∈[0,D/2)∣y(g)[n,d]∣,ϵ)a_{\max}^{(g)} = \max\left(\max_{n \in [\text{start}^{(g)}, \text{end}^{(g)}), d \in [0, D/2)} |\mathbf{y}^{(g)}[n, d]|, \epsilon\right)
Compute the scaling factor of this group:
s(g)=amax(g)Mhif8s^{(g)} = \frac{a_{\max}^{(g)}}{M_{\text{hif8}}}
Quantize this group:
yquant[n,d]=hif8_cast(yweighted[n,d]s(g)),n∈[start(g),end(g)),d∈[0,D/2)\mathbf{y}_{\text{quant}}[n, d] = \text{hif8\_cast}\left(\frac{\mathbf{y}_{\text{weighted}}[n, d]}{s^{(g)}}\right), \quad n \in [\text{start}^{(g)}, \text{end}^{(g)}), \quad d \in [0, D/2)
MoE scenario: In MoE inference, different experts process different numbers of tokens. groupIndex identifies the token range processed by each expert, and each group computes its scaling factor independently to adapt to different data distributions.
quant_mode=5 (MxQuant cuBLAS)
MX quantization principle: adopts the combination of E8M0 Scale + FP8 Data.
Grouping: every 32 elements form one group; fewer than 32 elements are padded with 0 to participate in the computation:
y=[g0,g1,…,gK],gi∈R32\mathbf{y} = [\mathbf{g}_0, \mathbf{g}_1, \ldots, \mathbf{g}_K], \quad \mathbf{g}_i \in \mathbb{R}^{32}
Amax computation (cuBLAS has no lower bound):
ai=maxj=031∣gi[j]∣a_i = \max_{j=0}^{31} |\mathbf{g}_i[j]|
Raw scale computation:
siraw=aiMfp8s_i^{\text{raw}} = \frac{a_i}{M_{\text{fp8}}}
where Mfp8M_{\text{fp8}} takes the following values:
- FP8 E4M3FN: Mfp8=448.0M_{\text{fp8}} = 448.0
- FP8 E5M2: Mfp8=57344.0M_{\text{fp8}} = 57344.0
Exponent rounding up (cuBLAS): extract the unbiased exponent EiE_i and mantissa MiM_i from siraws_i^{\text{raw}}. To ensure that quantization does not overflow, the exponent is rounded up:
Ei∗={Ei+1,siraw is a normal number, Ei<254 and Mi>0Ei+1,siraw is a subnormal number and Mi>0.5Ei,otherwiseE_i^{*} = \begin{cases} E_i + 1, & s_i^{\text{raw}} \text{ is a normal number, } E_i < 254 \text{ and } M_i > 0 \\ E_i + 1, & s_i^{\text{raw}} \text{ is a subnormal number and } M_i > 0.5 \\ E_i, & \text{otherwise} \end{cases}
si=2Ei∗s_i = 2^{E_i^{*}}
E8M0 scale encoding: sis_i is stored into FLOAT8_E8M0 as an unbiased exponent:
sie8m0=Ei∗+127s_i^{\text{e8m0}} = E_i^{*} + 127
All-zero block handling: when ai=0a_i = 0, sie8m0=0s_i^{\text{e8m0}} = 0 (no amax lower bound, consistent with the dual-axis operator).
Quantization computation:
yquant[t,j]=cast_fp8_rint (yweighted[t,j]⋅1si),j∈group i\mathbf{y}_{\text{quant}}[t, j] = \text{cast\_fp8\_rint}\!\left(\mathbf{y}_{\text{weighted}}[t, j] \cdot \frac{1}{s_i}\right), \quad j \in \text{group } i
where cast_fp8_rint is the type conversion from FP32 to FP8, using the RINT (round to nearest) mode.
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| x | Input | SwiGLU input. When quantMode is 5, the shape supports only the 2D [T,D]; in other modes, the shape is [...,D] with 2 to 8 dimensions (2 to 7 dimensions when quantMode is 1). When quantMode is 5, D must be greater than or equal to 64 and divisible by 64; in other modes, D of a non-empty input must be greater than or equal to 256 and divisible by 256. When quantMode is 0 or 1, an empty tensor is supported; when quantMode is 2 or 3, an empty tensor is not supported. When quantMode is 0, 1, or 5, only FLOAT16 and BFLOAT16 are supported; when quantMode is 2 or 3, FLOAT, FLOAT16, and BFLOAT16 are supported. | FLOAT, FLOAT16, BFLOAT16 | ND |
| weight | Input (optional) | MOE weight tensor, used for the weighting computation of the SwiGLU output. When quantMode is 0 or 1, an empty tensor is supported; when quantMode is 2 or 3, an empty tensor is not supported. When it is not empty, quantMode=5 supports FLOAT16, BFLOAT16, and FLOAT32, and other modes support only FLOAT32; the dimensions range from 1 to 8, and the number of elements must equal the product of the element counts of x over all dimensions except the last one. | quantMode=5: FLOAT16, BFLOAT16, FLOAT32; other modes: FLOAT32 | ND |
| group_index | Input (optional) | Number of group tokens in count mode. When quantMode is 0 or 1, it does not support being an empty tensor alone; when quantMode is 2 or 3, an empty tensor is not supported. When it is not empty, the data type is INT64 and the shape is [G]. | INT64 | ND |
| scale | Input (optional) | invScale tensor for static quantization input when quantMode=2. It is used only when quant_mode is 2. If groupIndex exists, the shape is [G]; otherwise, the shape is [1]. | FLOAT32 | ND |
| dst_type | Attribute | Target quantization type. It takes effect when quantMode is 0, 1, or 5. The supported values are 35 and 36, which represent FLOAT8_E5M2 and FLOAT8_E4M3FN respectively. | INT64 | - |
| quant_mode | Attribute | Quantization mode. The supported values are 0, 1, 2, 3, and 5. 0 indicates the Block FP8 mode. 1 indicates the original MX mode. 2 indicates the HIFP8 static quantization mode. 3 indicates the HIFP8 dynamic quantization mode. 5 indicates the MX V2 mode. | INT64 | - |
| block_size | Attribute | Quantization block size. 0 indicates using the default block size of the current quantization mode. When quantMode is 0, 0 or 128 is supported; when quantMode is 1 or 5, 0 or 32 is supported; when quantMode is 2 or 3, this parameter does not take effect. | INT64 | - |
| round_scale | Attribute | Whether to round the scale to a power of 2. When quantMode is 1 or 5, it must be true. When quantMode is 2 or 3, this parameter does not take effect. | BOOL | - |
| clamp_limit | Attribute | Clamp threshold before the SwiGLU computation. -1.0 indicates that clamp is disabled. When clamp is enabled, clampLimit must be greater than 0. | FLOAT | - |
| dst_type_max | Attribute | Maximum finite value of the target quantization type. This parameter takes effect only when quantMode is 3. The default value is 15.0. | FLOAT | - |
| output_origin | Attribute | Whether to output the SwiGLU result before Weight weighting. true indicates a valid yOrigin output; when it is false, the yOrigin output is invalid. | BOOL | - |
| y | Output | Quantization output. When quantMode is 0, 1, or 5, the data type is the same as dstType; when quantMode is 2 or 3, the data type is HIFLOAT8. The shape is [...,D/2] in all cases. When quantMode is 0 or 1, an empty tensor is supported; when quantMode is 2 or 3, an empty tensor is not supported. | HIFLOAT8, FLOAT8_E5M2, FLOAT8_E4M3FN | ND |
| y_scale | Output | Quantization scale output. When quantMode is 0, the shape is [...,ceil((D/2)/128)] and the data type is FLOAT32. When quantMode is 1 or 5, the shape is [...,ceil(ceil((D/2)/32)/2),2] and the data type is FLOAT8_E8M0. When quantMode is 2 or 3, the shape is [1] without groupIndex and [G] with groupIndex, and the data type is FLOAT32. When quantMode is 0 or 1, an empty tensor is supported; when quantMode is 2 or 3, an empty tensor is not supported. | FLOAT32, FLOAT8_E8M0 | ND |
| y_origin | Output | SwiGLU result before Weight weighting. The shape is [...,D/2]. The data type must be consistent with x. A null pointer is not supported. When quantMode is 0 or 1, an empty tensor is supported; when quantMode is 2 or 3, an empty tensor is not supported. | FLOAT, FLOAT16, BFLOAT16 | ND |
Constraints
-
Deterministic computation: aclnnSwigluGroupQuant is deterministic by default.
-
When quantMode is 0, only FP8 output is supported, and blockSize supports 0 or 128.
-
When quantMode is 1, FP8 output is supported, blockSize supports 0 or 32, and roundScale must be true.
-
When quantMode is 5, x supports only the 2D [T,D]; FP8 output is supported, blockSize supports 0 or 32, roundScale must be true, and groupIndex and scale are not supported. clampLimit must be a finite value; -1.0 indicates that Clamp is disabled, otherwise it must be greater than 0.
-
When quantMode is 2 or 3, HIFP8 quantization output is supported, and dstType, blockSize, and roundScale do not take effect. The dimensions of input x are [T, D] or [B, S, D], which must meet the following specification constraints:
Specification item Specification Description B 1~31 - S 0~128K - D/2 512, 768, 1024, 1536, 1792, 2048, 2560, 4096 - dstTypeMax 15, 56, 224, 32768 - -
The data type of yScale must match quantMode: FLOAT32 for Block FP8, FLOAT8_E8M0 for MX, and FLOAT32 for HIFP8.
-
Element values in groupIndex must be greater than or equal to 0.
Invocation Description
| Invocation method | Invocation sample | Description |
|---|---|---|
| aclnn invocation | test_aclnn_swiglu_group_quant | Calls the SwigluGroupQuant operator through the aclnnSwigluGroupQuant interface. |
| aclnn V2 invocation | test_aclnn_swiglu_group_quant_v2 | Calls quant_mode=5 of the SwigluGroupQuant operator through the aclnnSwigluGroupQuantV2 interface. |
| Graph-mode invocation | - | Calls the SwigluGroupQuant operator by constructing a graph through the operator IR. |