FilesLast commitLast update
9 days ago
2 months ago
1 month ago
9 days ago
9 days ago
9 days ago
11 months ago
1 month ago
17 days ago
README

DequantSwigluQuant

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×
kirin X90 processor series products √
kirin 9030 processor series products √

Function

  • Operator function: Adds dequant and quant operations before and after the Swish gated linear unit activation function to implement the DequantSwigluQuant computation of x.

  • Formula when swiglu_mode is 0:

    dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)

    swigluOuti=Swiglu(dequantOuti)=Swish(Ai)∗BiswigluOut_i = Swiglu(dequantOut_i)=Swish(A_i)*B_i

    outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)

    Here, Ai represents the first half of dequantOuti, and Bi represents the second half of dequantOuti.

  • Formula when swiglu_mode is 1:

    dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)

    x_glu=x_glu.clamp(min=None,max=clamp_limit)x\_glu = x\_glu.clamp(min=None, max=clamp\_limit)

    x_linear=x_linear.clamp(min=−clamp_limit,max=clamp_limit)x\_linear = x\_linear.clamp(min=-clamp\_limit, max=clamp\_limit)

    out_glu=x_glu∗sigmoid(glu_alpha∗x_glu)out\_glu = x\_glu * sigmoid(glu\_alpha * x\_glu)

    swigluOuti=out_glu∗(x_linear+glu_bias)swigluOut_i = out\_glu * (x\_linear + glu\_bias)

    outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)

    Here, x_glu represents the even-index part of dequantOuti, and x_linear represents the odd-index part of dequantOuti.

  • Formula when swiglu_mode is 2:

    The computation logic is the same as when swiglu_mode is 1 (both are the GPT-OSS variant SwiGLU, using clamp_limit, glu_alpha, and glu_bias):

    dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)

    x_glu=x_glu.clamp(min=None,max=clamp_limit)x\_glu = x\_glu.clamp(min=None, max=clamp\_limit)

    x_linear=x_linear.clamp(min=−clamp_limit,max=clamp_limit)x\_linear = x\_linear.clamp(min=-clamp\_limit, max=clamp\_limit)

    out_glu=x_glu∗sigmoid(glu_alpha∗x_glu)out\_glu = x\_glu * sigmoid(glu\_alpha * x\_glu)

    swigluOuti=out_glu∗(x_linear+glu_bias)swigluOut_i = out\_glu * (x\_linear + glu\_bias)

    outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)

    The difference from swiglu_mode 1 lies in how x_glu and x_linear are split: when swiglu_mode is 2, x_glu represents the first half of dequantOuti, and x_linear represents the second half of dequantOuti (the same split method as when swiglu_mode is 0); when swiglu_mode is 1, the split interleaves even and odd indices.

  • Formula when swiglu_mode is 3:

    dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)

    gate=dequantOuti[:,:H/2]gate = dequantOut_i[:, :H/2]

    up=dequantOuti[:,H/2:]up = dequantOut_i[:, H/2:]

    gate=SiLU(gate)=gate∗sigmoid(gate)gate = SiLU(gate) = gate * sigmoid(gate)

    gate=gate.clamp(max=clamp_limit)gate = gate.clamp(max=clamp\_limit)

    up=up.clamp(min=−clamp_limit,max=clamp_limit)up = up.clamp(min=-clamp\_limit, max=clamp\_limit)

    swigluOuti=gate∗upswigluOut_i = gate * up

    outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)

    Here, gate represents the first half of dequantOuti, and up represents the second half of dequantOuti (the same split method as when swiglu_mode is 0). swiglu_mode 3 uses the SiLU activation function (consistent with swiglu_mode 0) and the clamp limiting mechanism (using clamp_limit), but does not use the glu_alpha and glu_bias parameters.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
x Input Input data to be processed, x in the formula. The input cannot contain ±inf or nan. FLOAT16, BFLOAT16, INT32 ND
weight_scale Input The input cannot contain ±inf or nan. FLOAT ND
activation_scale Input Dequantization scale of the activation function. The input cannot contain ±inf or nan. FLOAT ND
bias Input The bias of Matmul, bias in the formula. The input cannot contain ±inf or nan. FLOAT, FLOAT16, BFLOAT16, INT32 ND
quant_scale Input Quantization scale, quant_scale in the formula. The input cannot contain ±inf or nan. FLOAT ND
quant_offset Input Quantization offset. The input cannot contain ±inf or nan. FLOAT ND
group_index Input The group_index required for MoE grouping. The input cannot contain ±inf or nan. INT64 ND
activate_left Attribute Specifies whether to perform the swiglu activation on the left half of the input. BOOL -
quant_mode Attribute Indicates that dynamic quantization is used. STRING -
dst_type Attribute Specifies the data type of output y. INT64 -
round_mode Attribute Specifies the rounding mode for the output y result. STRING -
activate_dim Attribute Specifies the selected split axis for the swish computation. INT64 -
swiglu_mode Attribute Specifies the swiglu computation mode. Values are 0/1/2/3: 0 is traditional swiglu; 1 is variant swiglu (even-odd chunking); 2 is variant swiglu (contiguous first-half/second-half chunking); 3 is variant swiglu (contiguous first-half/second-half chunking, with clamp after silu) INT64 -
clamp_limit Attribute Specifies the threshold value used by the variant swiglu. FLOAT -
glu_alpha Attribute Specifies the parameter used by the variant swiglu. FLOAT -
glu_bias Attribute Specifies the bias parameter used by the variant swiglu. FLOAT -
y Output - INT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1, FLOAT4_E1M2 ND
scale Output - FLOAT ND
  • Kirin X90/kirin 9030 processor series products:
    • Input x: The data type does not support BFLOAT16.
    • Input bias: The data type does not support BFLOAT16.
    • Input quant_scale: The data type does not support FLOAT16.
    • Output y: The data type does not support FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1, or FLOAT4_E1M2.

Constraints

  • Ascend 950PR&950DT products:

    • The dimension of input x corresponding to activate_dim must be a multiple of 2, and the number of dimensions of x must be greater than 1.
    • When the data type of input x is INT32, weight_scale cannot be empty; when the data type of input x is not INT32, weight_scale cannot be provided as an input and a null pointer is passed in.
    • When the data type of input x is not INT32, activation_scale cannot be provided as an input and the parameter is set to a null pointer.
    • When the data type of input x is not INT32, bias cannot be provided as an input and the parameter is set to a null pointer.
    • When the data type of output y is FLOAT4_E2M1 or FLOAT4_E1M2, the last dimension of y must be a multiple of 2.
    • The tail axis of output y does not exceed 5120.
  • Atlas A2 products, Atlas A3 products:

    • The four parameters swiglu_mode, clamp_limit, glu_alpha, and glu_bias are used for the GPT-OSS variant SwiGLU.
    • The last dimension of x must be a multiple of 2, and the number of dimensions of x must be greater than 1.
    • When quant_mode is static, quant_scale and quant_offset are 1-dimensional with a value of 1; when quant_mode is dynamic, quant_scale and quant_offset
    • The memory size of input tensors supported by the operator has an upper limit. Validation formula: weight_scale tensor memory size + bias tensor memory size + quant_scale tensor memory size + quant_offset tensor memory size + (activation_scale tensor memory size + scale tensor memory size) / 40 + x tensor last-dimension H memory size * 10 < 192KB.

Invocation Description

Invocation method Invocation sample Description
aclnn invocation test_aclnn_dequant_swiglu_quant Calls the DequantSwigluQuant operator through the aclnnDequantSwigluQuant interface.
aclnn invocation test_aclnn_dequant_swiglu_quant_v2 Calls the DequantSwigluQuant operator through the aclnnDequantSwigluQuantV2 interface.
Graph-mode invocation - Calls the DequantSwigluQuant operator by building a graph through the operator IR.