| Files | Last commit | Last update |
|---|---|---|
| 9 days ago | ||
| 2 months ago | ||
| 1 month ago | ||
| 9 days ago | ||
| 9 days ago | ||
| 9 days ago | ||
| 11 months ago | ||
| 1 month ago | ||
| 17 days ago |
DequantSwigluQuant
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
| kirin X90 processor series products | √ |
| kirin 9030 processor series products | √ |
Function
-
Operator function: Adds dequant and quant operations before and after the Swish gated linear unit activation function to implement the DequantSwigluQuant computation of x.
-
Formula when swiglu_mode is 0:
dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)
swigluOuti=Swiglu(dequantOuti)=Swish(Ai)∗BiswigluOut_i = Swiglu(dequantOut_i)=Swish(A_i)*B_i
outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)
Here, Ai represents the first half of dequantOuti, and Bi represents the second half of dequantOuti.
-
Formula when swiglu_mode is 1:
dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)
x_glu=x_glu.clamp(min=None,max=clamp_limit)x\_glu = x\_glu.clamp(min=None, max=clamp\_limit)
x_linear=x_linear.clamp(min=−clamp_limit,max=clamp_limit)x\_linear = x\_linear.clamp(min=-clamp\_limit, max=clamp\_limit)
out_glu=x_glu∗sigmoid(glu_alpha∗x_glu)out\_glu = x\_glu * sigmoid(glu\_alpha * x\_glu)
swigluOuti=out_glu∗(x_linear+glu_bias)swigluOut_i = out\_glu * (x\_linear + glu\_bias)
outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)
Here, x_glu represents the even-index part of dequantOuti, and x_linear represents the odd-index part of dequantOuti.
-
Formula when swiglu_mode is 2:
The computation logic is the same as when swiglu_mode is 1 (both are the GPT-OSS variant SwiGLU, using clamp_limit, glu_alpha, and glu_bias):
dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)
x_glu=x_glu.clamp(min=None,max=clamp_limit)x\_glu = x\_glu.clamp(min=None, max=clamp\_limit)
x_linear=x_linear.clamp(min=−clamp_limit,max=clamp_limit)x\_linear = x\_linear.clamp(min=-clamp\_limit, max=clamp\_limit)
out_glu=x_glu∗sigmoid(glu_alpha∗x_glu)out\_glu = x\_glu * sigmoid(glu\_alpha * x\_glu)
swigluOuti=out_glu∗(x_linear+glu_bias)swigluOut_i = out\_glu * (x\_linear + glu\_bias)
outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)
The difference from swiglu_mode 1 lies in how x_glu and x_linear are split: when swiglu_mode is 2, x_glu represents the first half of dequantOuti, and x_linear represents the second half of dequantOuti (the same split method as when swiglu_mode is 0); when swiglu_mode is 1, the split interleaves even and odd indices.
-
Formula when swiglu_mode is 3:
dequantOuti=Dequant(xi)dequantOut_i = Dequant(x_i)
gate=dequantOuti[:,:H/2]gate = dequantOut_i[:, :H/2]
up=dequantOuti[:,H/2:]up = dequantOut_i[:, H/2:]
gate=SiLU(gate)=gate∗sigmoid(gate)gate = SiLU(gate) = gate * sigmoid(gate)
gate=gate.clamp(max=clamp_limit)gate = gate.clamp(max=clamp\_limit)
up=up.clamp(min=−clamp_limit,max=clamp_limit)up = up.clamp(min=-clamp\_limit, max=clamp\_limit)
swigluOuti=gate∗upswigluOut_i = gate * up
outi=Quant(swigluOuti)out_i = Quant(swigluOut_i)
Here, gate represents the first half of dequantOuti, and up represents the second half of dequantOuti (the same split method as when swiglu_mode is 0). swiglu_mode 3 uses the SiLU activation function (consistent with swiglu_mode 0) and the clamp limiting mechanism (using clamp_limit), but does not use the glu_alpha and glu_bias parameters.
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| x | Input | Input data to be processed, x in the formula. The input cannot contain ±inf or nan. | FLOAT16, BFLOAT16, INT32 | ND |
| weight_scale | Input | The input cannot contain ±inf or nan. | FLOAT | ND |
| activation_scale | Input | Dequantization scale of the activation function. The input cannot contain ±inf or nan. | FLOAT | ND |
| bias | Input | The bias of Matmul, bias in the formula. The input cannot contain ±inf or nan. | FLOAT, FLOAT16, BFLOAT16, INT32 | ND |
| quant_scale | Input | Quantization scale, quant_scale in the formula. The input cannot contain ±inf or nan. | FLOAT | ND |
| quant_offset | Input | Quantization offset. The input cannot contain ±inf or nan. | FLOAT | ND |
| group_index | Input | The group_index required for MoE grouping. The input cannot contain ±inf or nan. | INT64 | ND |
| activate_left | Attribute | Specifies whether to perform the swiglu activation on the left half of the input. | BOOL | - |
| quant_mode | Attribute | Indicates that dynamic quantization is used. | STRING | - |
| dst_type | Attribute | Specifies the data type of output y. | INT64 | - |
| round_mode | Attribute | Specifies the rounding mode for the output y result. | STRING | - |
| activate_dim | Attribute | Specifies the selected split axis for the swish computation. | INT64 | - |
| swiglu_mode | Attribute | Specifies the swiglu computation mode. Values are 0/1/2/3: 0 is traditional swiglu; 1 is variant swiglu (even-odd chunking); 2 is variant swiglu (contiguous first-half/second-half chunking); 3 is variant swiglu (contiguous first-half/second-half chunking, with clamp after silu) | INT64 | - |
| clamp_limit | Attribute | Specifies the threshold value used by the variant swiglu. | FLOAT | - |
| glu_alpha | Attribute | Specifies the parameter used by the variant swiglu. | FLOAT | - |
| glu_bias | Attribute | Specifies the bias parameter used by the variant swiglu. | FLOAT | - |
| y | Output | - | INT8, FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1, FLOAT4_E1M2 | ND |
| scale | Output | - | FLOAT | ND |
- Kirin X90/kirin 9030 processor series products:
- Input
x: The data type does not support BFLOAT16. - Input
bias: The data type does not support BFLOAT16. - Input
quant_scale: The data type does not support FLOAT16. - Output
y: The data type does not support FLOAT8_E5M2, FLOAT8_E4M3FN, FLOAT4_E2M1, or FLOAT4_E1M2.
- Input
Constraints
-
Ascend 950PR&950DT products:
- The dimension of input x corresponding to activate_dim must be a multiple of 2, and the number of dimensions of x must be greater than 1.
- When the data type of input x is INT32, weight_scale cannot be empty; when the data type of input x is not INT32, weight_scale cannot be provided as an input and a null pointer is passed in.
- When the data type of input x is not INT32, activation_scale cannot be provided as an input and the parameter is set to a null pointer.
- When the data type of input x is not INT32, bias cannot be provided as an input and the parameter is set to a null pointer.
- When the data type of output y is FLOAT4_E2M1 or FLOAT4_E1M2, the last dimension of y must be a multiple of 2.
- The tail axis of output y does not exceed 5120.
-
Atlas A2 products, Atlas A3 products:
- The four parameters swiglu_mode, clamp_limit, glu_alpha, and glu_bias are used for the GPT-OSS variant SwiGLU.
- The last dimension of x must be a multiple of 2, and the number of dimensions of x must be greater than 1.
- When quant_mode is static, quant_scale and quant_offset are 1-dimensional with a value of 1; when quant_mode is dynamic, quant_scale and quant_offset
- The memory size of input tensors supported by the operator has an upper limit. Validation formula: weight_scale tensor memory size + bias tensor memory size + quant_scale tensor memory size + quant_offset tensor memory size + (activation_scale tensor memory size + scale tensor memory size) / 40 + x tensor last-dimension H memory size * 10 < 192KB.
Invocation Description
| Invocation method | Invocation sample | Description |
|---|---|---|
| aclnn invocation | test_aclnn_dequant_swiglu_quant | Calls the DequantSwigluQuant operator through the aclnnDequantSwigluQuant interface. |
| aclnn invocation | test_aclnn_dequant_swiglu_quant_v2 | Calls the DequantSwigluQuant operator through the aclnnDequantSwigluQuantV2 interface. |
| Graph-mode invocation | - | Calls the DequantSwigluQuant operator by building a graph through the operator IR. |