| Files | Last commit | Last update |
|---|---|---|
| 13 days ago | ||
| 2 months ago | ||
| 1 month ago | ||
| 9 days ago | ||
| 9 days ago | ||
| 17 days ago | ||
| 7 months ago | ||
| 23 days ago | ||
| 17 days ago |
SwigluMxQuant
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | × |
| Atlas A2 products | × |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
| Kirin X90 processor series products | × |
| Kirin 9030 processor series products | × |
Function
-
Operator function: This is a fused operator that implements the combined computation of the SwiGLU activation function and dynamic block quantization. It first computes the SwiGLU activation function on the input, then performs block-based dynamic quantization on the result, and outputs a low-precision FP4/FP8 tensor and the corresponding scale factors.
-
Formula:
Stage 1: SwiGLU activation function
gate, hidden = split(x, dim=dim)
swish = sigmoid(gate) * gate
act = swish * hidden
Stage 2: Dynamic block quantization
Dynamic quantization is performed block by block along the axis dimension with blocksize=32. When scale_alg=0 (OCP), the elements {V_i} in each block are quantized by the following formulas:
shared_exp = floor(log2(max_i(|V_i|))) - emax
mxscale = 2^shared_exp
y_i = cast_to_dst_type(V_i / mxscale, round_mode)
Where emax is the exponent of the largest regular number of the target data type. The values are as follows:
Data type emax FLOAT4_E2M1 2 FLOAT4_E1M2 0 FLOAT8_E4M3FN 8 FLOAT8_E5M2 15 When scale_alg=1 (cuBLAS, only FP8 types), block scale factor derivation is used. For the formulas, refer to aclnnSwigluMxQuant.
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| x | Input | Input tensor. Its size along the dimension specified by activate_dim must be a multiple of 2. | FLOAT16, BFLOAT16 | ND |
| group_index | Optional input | The shape must be 1-dimensional, and shape[0] must be greater than 0 and less than or equal to 256; each value of data must be an integer greater than or equal to 0, the sum of all values must be greater than 0, and the sum must be less than or equal to the total number of rows of x after the SwiGLU operation (that is, the total number of rows of the SwiGLU output y: when activate_dim is the last axis, it equals the product of the remaining axes of x excluding the last axis; when activate_dim is not the last axis, it equals half of the total number of rows of x). | INT32, INT64 | ND |
| y | Output | Output tensor after quantization. Its shape is the same as x, with the activate_dim dimension being half of that of x. | FLOAT4_E2M1, FLOAT4_E1M2, FLOAT8_E4M3FN, FLOAT8_E5M2 | ND |
| mxscale | Output | Scale factor of each quantization block (32 elements per group). The shape along the axis dimension is the value of the corresponding axis of y divided by 32, rounded up, and aligned to an even number; the last dimension is fixed to 2 and stores the scales of two adjacent quantization blocks. When axis=-2 and group_index exists, the shape along the axis dimension is the value of the corresponding axis of y integer-divided by 64 plus group_num. | FLOAT8_E8M0 | ND |
| activate_dim | Attribute | Split dimension of SwiGLU. The value range is [-1, -2, xDim-2, xDim-1] (xDim is the number of dimensions of input x, that is, the last two dimensions). When activate_dim is not the last dimension, swiglu_mode cannot be 1. | INT64 | - |
| activate_left | Attribute | Indicates whether the SwiGLU activation is applied to the first half or the second half of the input. When it is false, the second half is activated. This parameter does not take effect when swiglu_mode=1. | BOOL | - |
| swiglu_mode | Attribute | SwiGLU computation mode: 0=traditional SwiGLU, 1=odd-even interleaved variant, 2=front-back split clamp variant, 3=front-back split sigmoid-clamp variant. The value range is [0, 3]. When activate_dim is not the last dimension, swiglu_mode cannot be 1. | INT64 | - |
| clamp_limit | Attribute | Clamp threshold of the SwiGLU variants (swiglu_mode=1/2/3). It must be greater than 0. | FLOAT | - |
| glu_alpha | Attribute | Sigmoid scale coefficient of the SwiGLU variants (swiglu_mode=1/2). | FLOAT | - |
| glu_bias | Attribute | Bias of the linear part of the SwiGLU variants (swiglu_mode=1/2). | FLOAT | - |
| group_mode | Attribute | Takes effect when group_index exists. 0=count mode, 1=cumsum mode. Currently, only 0 is supported. | INT64 | - |
| axis | Attribute | Quantization axis. Block-based quantization is performed along this dimension. The value range is [-1, -2, xDim-2, xDim-1] (xDim is the number of dimensions of input x, that is, the last two dimensions). | INT64 | - |
| dst_type | Attribute | Target quantization type: 40=FP4_E2M1, 41=FP4_E1M2, 36=FP8_E4M3FN, 35=FP8_E5M2. | INT64 | - |
| round_mode | Attribute | Rounding mode, used for type conversion during quantization. The value is "rint", "floor", or "round". | STRING | - |
| scale_alg | Attribute | Scale algorithm: 0=OCP, 1=cuBLAS, 2=RNE (reserved). Currently, only 0 and 1 are supported. | INT64 | - |
| max_dtype_value | Attribute | Reserved parameter. It does not take effect in the current version. | FLOAT | - |
Constraints
- Input x supports 2-7 dimensional tensors. Its size along the dimension specified by activate_dim must be divisible by 2.
- activate_dim and axis must take the last two dimensions of input x. The value range is [-1, -2, xDim-2, xDim-1] (xDim is the number of dimensions of input x).
- When activate_dim is not the last dimension, swiglu_mode cannot be 1.
- When swiglu_mode is 2 or 3, axis must be -1.
- When activate_dim = -2 or axis = -2 and group_index exists, input x must be 2-dimensional.
- When dst_type is an FP4 type, the last dimension of the output shape must be divisible by 2.
- When dst_type is an FP4 type, scale_alg must be 0.
- When group_index exists, it must be 1-dimensional, and shape[0] must be greater than 0 and less than or equal to 256; each value of data must be an integer greater than or equal to 0, the sum of all values must be greater than 0, and the sum must be less than or equal to the total number of rows of x after the SwiGLU operation (that is, the total number of rows of the SwiGLU output y: when activate_dim is the last axis, it equals the product of the remaining axes of x excluding the last axis; when activate_dim is not the last axis, it equals half of the total number of rows of x).
- FP8 output types support only the "rint" rounding mode.
- Empty tensors are not supported: when any dimension in the shape of input x, optional input group_index, or outputs y and mxscale is 0 (the number of elements is 0), operator validation fails and an error is reported.
- Input x must be 2-7 dimensional. rank=0 scalar tensors are not supported.
- The caller allocates memory for outputs y/mxscale according to the shape inferred by the interface, and the operator writes the quantization result completely back to the output tensors. The parts of the output beyond the range covered by the sum of all elements of group_index are not cleared and contain garbage data.
- The memory spaces of inputs and outputs must not overlap. In-place modification (inplace) scenarios are not supported.
- The operator does not modify any input data during execution.
Invocation Description
| Invocation method | Invocation sample | Description |
|---|---|---|
| aclnn invocation | test_aclnn_swiglu_mx_quant | Invokes the SwigluMxQuant operator through the aclnnSwigluMxQuant interface. |