FilesLast commitLast update
4 days ago
2 months ago
4 months ago
4 days ago
4 months ago
9 days ago
11 months ago
1 month ago
17 days ago
README

SwiGluQuant

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×
kirin X90 processor series products √
kirin 9030 processor series products √

Function

  • Description: Adds a quant operation after the SwiGlu activation function to implement the SwiGluQuant computation of the input x. int8 or int4 quantized output is supported.

  • Operator support scope: currently, SwiGluQuant supports the MoE scenario (with groupIndexOptional passed in) and the non-MoE scenario (with groupIndexOptional left empty). The input x and group_index of SwiGluQuant come from the outputs of the GroupedMatMul operator and MoeInitRouting. Through the group_index input parameter, it implements MoE grouped dynamic quantization, static per_tensor quantization, and static per_channel quantization.

  • Formula for dynamic quantization in the MoE scenario:

    Act=SwiGlu(x)=Swish(A)∗BYtmp0=Act[0 ⁣:g[0], ⁣:]∗smooth_scales[0 ⁣:g[0], ⁣:],i=0Ytmpi=Act[g[i] ⁣:g[i+1], ⁣:]∗smooth_scales[g[i] ⁣:g[i+1], ⁣:],i∈(0,G)∩Zscale=dstTypeScale/row_max(abs(Ytmp)) Act = SwiGlu(x) = Swish(A)*B \\ Y_{tmp}^0 = Act[0\colon g[0],\colon] * smooth\_scales[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))

    Y=Cast(Mul(Ytmp,Scale)) Y = Cast(Mul(Y_{tmp}, Scale))

    Here, A denotes the first half of the input x, B denotes the second half of the input x, g denotes group_index, and G is the number of groups of group_index. For int8 quantization, dstTypeScale=127dstTypeScale = 127 (127 is the maximum value of int8); for int4 quantization, dstTypeScale=7dstTypeScale = 7 (7 is the maximum value of int4).

  • Formula for static quantization in the MoE scenario:

    Act=SwiGLU(x)=Swish(A)∗BYtmp0=Act(0 ⁣:g[0], ⁣:)∗smooth_scales[0 ⁣:g[0], ⁣:]+offsets[0 ⁣:g[0], ⁣:],i=0Ytmpi=Act[g[i] ⁣:g[i+1], ⁣:]∗smooth_scales[g[i] ⁣:g[i+1], ⁣:]+offsets[g[i] ⁣:g[i+1], ⁣:],i∈(0,G)∩Z Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp}^0 = Act(0\colon g[0],\colon) * smooth\_scales[0\colon g[0],\colon] + offsets[0\colon g[0],\colon], i=0 \\ Y_{tmp}^i = Act[g[i]\colon g[i+1], \colon] * smooth\_scales[g[i]\colon g[i+1], \colon] + offsets[g[i]\colon g[i+1], \colon], i \in (0, G) \cap \mathbb{Z}\\

    Y=Cast(Ytmp) Y = Cast(Y_{tmp})

    Here, A denotes the first half of the input x, B denotes the second half of the input x, g denotes group_index, and G is the number of groups of group_index.

  • Formula for dynamic quantization in the non-MoE scenario (groupIndexOptional left empty):

    Act=SwiGLU(x)=Swish(A)∗BYtmp=Act∗smooth_scales(0, ⁣:)scale=dstTypeScale/row_max(abs(Ytmp)) Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act* smooth\_scales(0,\colon)\\ scale=dstTypeScale/row\_max(abs(Y_{tmp}))

    Y=Cast(Mul(Ytmp,Scale)) Y = Cast(Mul(Y_{tmp}, Scale))

    Here, A denotes the first half of the input x, B denotes the second half of the input x. For int8 quantization, dstTypeScale=127dstTypeScale = 127 (127 is the maximum value of int8); for int4 quantization, dstTypeScale=7dstTypeScale = 7 (7 is the maximum value of int4).

  • Formula for static quantization in the non-MoE scenario (groupIndexOptional left empty):

    Act=SwiGLU(x)=Swish(A)∗BYtmp=Act∗smooth_scales(0, ⁣:)+offsets(0, ⁣:) Act = SwiGLU(x) = Swish(A)*B \\ Y_{tmp} = Act * smooth\_scales(0,\colon) + offsets(0,\colon) \\

    Y=Cast(Ytmp) Y = Cast(Y_{tmp})

    Here, A denotes the first half of the input x, B denotes the second half of the input x.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
x Input Input x in the formula. FLOAT16, BFLOAT16, FLOAT ND
smoothScalesOptional Input The smooth_scales variable in the formula. FLOAT ND
offsetsOptional Input Input offsets in the formula. FLOAT ND
groupIndexOptional Input The group_index variable in the formula. INT32 ND
activateLeft Attribute
  • Denotes whether the left matrix participates in the computation.
  • Users must pass this parameter.
BOOL -
quantModeOptional Attribute
  • Users must pass this parameter.
  • "static" indicates static quantization, "dynamic" indicates dynamic quantization, and "dynamic_msd" indicates dynamic MSD quantization. Currently, only "dynamic" dynamic quantization and "static" static quantization are supported. Static quantization supports only per_tensor quantization and per_channel quantization.
STRING -
groupListType Attribute
  • Users must pass this parameter.
  • 0 indicates the cumsum mode, and 1 indicates the count mode. Currently, only 0 (cumsum mode) and 1 (count mode) are supported.
INT64_T -
dstType Attribute
  • Users must pass this parameter.
  • 2 indicates that yOut is the int8 quantized output, and 29 indicates that yOut is the int4 quantized output. Currently, only 2 and 29 are supported as inputs, and the default value is 2.
INT64_T -
yOut Output y in the formula. INT8, INT4 ND
scaleOut Output scale in the formula. FLOAT ND
  • Kirin X90/kirin 9030 processor series products: the x data type does not support BFLOAT16; the y data type does not support INT4.

Constraints

None.

Invocation Description

Invocation method Invocation sample Description
aclnn invocation test_aclnn_swi_glu_quant Invokes the SwiGluQuant operator through the aclnnSwiGluQuant interface.
aclnn invocation test_aclnn_swi_glu_quant_v2 Invokes the SwiGluQuant operator through the aclnnSwiGluQuantV2 interface.
Graph-mode invocation - Invokes the SwiGluQuant operator by building a graph through the operator IR.