| Files | Last commit | Last update |
|---|---|---|
| 4 days ago | ||
| 2 months ago | ||
| 4 months ago | ||
| 4 days ago | ||
| 4 months ago | ||
| 9 days ago | ||
| 11 months ago | ||
| 1 month ago | ||
| 17 days ago |
SwiGluQuant
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
| kirin X90 processor series products | √ |
| kirin 9030 processor series products | √ |
Function
-
Description: Adds a quant operation after the SwiGlu activation function to implement the SwiGluQuant computation of the input x. int8 or int4 quantized output is supported.
-
Operator support scope: currently, SwiGluQuant supports the MoE scenario (with groupIndexOptional passed in) and the non-MoE scenario (with groupIndexOptional left empty). The input x and group_index of SwiGluQuant come from the outputs of the GroupedMatMul operator and MoeInitRouting. Through the group_index input parameter, it implements MoE grouped dynamic quantization, static per_tensor quantization, and static per_channel quantization.
-
Formula for dynamic quantization in the MoE scenario:
Act=SwiGlu(x)=Swish(A)∗BYtmp0=Act[0 :g[0], :]∗smooth_scales[0 :g[0], :],i=0Ytmpi=Act[g[i] :g[i+1], :]∗smooth_scales[g[i] :g[i+1], :],i∈(0,G)∩Zscale=dstTypeScale/row_max(abs(Ytmp)) Act = SwiGlu(x) = Swish(A)*B \\ Y_{tmp}^0 = Act[0\colon g[0],\colon] * smooth\_scales[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))
Y=Cast(Mul(Ytmp,Scale)) Y = Cast(Mul(Y_{tmp}, Scale))
Here, A denotes the first half of the input x, B denotes the second half of the input x, g denotes group_index, and G is the number of groups of group_index. For int8 quantization, dstTypeScale=127dstTypeScale = 127 (127 is the maximum value of int8); for int4 quantization, dstTypeScale=7dstTypeScale = 7 (7 is the maximum value of int4).
-
Formula for static quantization in the MoE scenario:
Act=SwiGLU(x)=Swish(A)∗BYtmp0=Act(0 :g[0], :)∗smooth_scales[0 :g[0], :]+offsets[0 :g[0], :],i=0Ytmpi=Act[g[i] :g[i+1], :]∗smooth_scales[g[i] :g[i+1], :]+offsets[g[i] :g[i+1], :],i∈(0,G)∩Z Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp}^0 = Act(0\colon g[0],\colon) * smooth\_scales[0\colon g[0],\colon] + offsets[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon] + offsets[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\
Y=Cast(Ytmp) Y = Cast(Y_{tmp})
Here, A denotes the first half of the input x, B denotes the second half of the input x, g denotes group_index, and G is the number of groups of group_index.
-
Formula for dynamic quantization in the non-MoE scenario (groupIndexOptional left empty):
Act=SwiGLU(x)=Swish(A)∗BYtmp=Act∗smooth_scales(0, :)scale=dstTypeScale/row_max(abs(Ytmp)) Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act* smooth\_scales(0,\colon)\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))
Y=Cast(Mul(Ytmp,Scale)) Y = Cast(Mul(Y_{tmp}, Scale))
Here, A denotes the first half of the input x, B denotes the second half of the input x. For int8 quantization, dstTypeScale=127dstTypeScale = 127 (127 is the maximum value of int8); for int4 quantization, dstTypeScale=7dstTypeScale = 7 (7 is the maximum value of int4).
-
Formula for static quantization in the non-MoE scenario (groupIndexOptional left empty):
Act=SwiGLU(x)=Swish(A)∗BYtmp=Act∗smooth_scales(0, :)+offsets(0, :) Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act * smooth\_scales(0,\colon) + offsets(0,\colon) \\
Y=Cast(Ytmp) Y = Cast(Y_{tmp})
Here, A denotes the first half of the input x, B denotes the second half of the input x.
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| x | Input | Input x in the formula. | FLOAT16, BFLOAT16, FLOAT | ND |
| smoothScalesOptional | Input | The smooth_scales variable in the formula. | FLOAT | ND |
| offsetsOptional | Input | Input offsets in the formula. | FLOAT | ND |
| groupIndexOptional | Input | The group_index variable in the formula. | INT32 | ND |
| activateLeft | Attribute |
|
BOOL | - |
| quantModeOptional | Attribute |
|
STRING | - |
| groupListType | Attribute |
|
INT64_T | - |
| dstType | Attribute |
|
INT64_T | - |
| yOut | Output | y in the formula. | INT8, INT4 | ND |
| scaleOut | Output | scale in the formula. | FLOAT | ND |
- Kirin X90/kirin 9030 processor series products: the
xdata type does not support BFLOAT16; theydata type does not support INT4.
Constraints
None.
Invocation Description
| Invocation method | Invocation sample | Description |
|---|---|---|
| aclnn invocation | test_aclnn_swi_glu_quant | Invokes the SwiGluQuant operator through the aclnnSwiGluQuant interface. |
| aclnn invocation | test_aclnn_swi_glu_quant_v2 | Invokes the SwiGluQuant operator through the aclnnSwiGluQuantV2 interface. |
| Graph-mode invocation | - | Invokes the SwiGluQuant operator by building a graph through the operator IR. |