FilesLast commitLast update
17 days ago
2 months ago
1 month ago
8 days ago
8 days ago
5 days ago
11 months ago
1 month ago
17 days ago
README

DynamicQuantV2

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products √
Atlas training products √
Kirin X90 processor series products √
Kirin 9030 processor series products √

Function

  • Operator function: performs pertoken/pertensor/perchannel symmetric/asymmetric dynamic quantization for the input tensor. In the MOE scenario, the smooth_scales of each expert are different, which are distinguished by the input group_index.

  • Formula:

    • Symmetric quantization:

      • If smooth_scales is not provided:

        scale=row_max(abs(x))/dtypeMax scale=row\_max(abs(x))/dtypeMax

        y=round(x/scale) y=round(x/scale)

      • If smooth_scales is provided:

        input=x⋅smoothscales input = x\cdot smooth_scales

        scale=row_max(abs(input))/dtypeMax scale=row\_max(abs(input))/dtypeMax

        y=round(input/scale) y=round(input/scale)

    • Asymmetric quantization:

      • If smooth_scales is not provided:

        scale=(row_max(x)−row_min(x))/(dtypeMax−dtypeMin) scale=(row\_max(x) - row\_min(x))/(dtypeMax - dtypeMin)

        offset=dtypeMax−row_max(x)/scale offset=dtypeMax-row\_max(x)/scale

        y=round(x/scale+offset) y=round(x/scale+offset)

      • If smooth_scales is provided:

        input=x⋅smoothscales input = x\cdot smooth_scales

        scale=(row_max(input)−row_min(input))/(dtypeMax−dtypeMin) scale=(row\_max(input) - row\_min(input))/(dtypeMax - dtypeMin)

        offset=dtypeMax−row_max(input)/scale offset=dtypeMax-row\_max(input)/scale

        y=round(input/scale+offset) y=round(input/scale+offset)

    Where:

    • row_max represents the computation of the maximum value of each row.
    • row_min represents the computation of the minimum value of each row.
    • dtypeMax is the maximum value of the output data type.
    • dtypeMin is the minimum value of the output data type.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
x Input
  • The input tensor of the operator, corresponding to `x` in the formula.
  • The number of dimensions of the shape must be greater than 1.
FLOAT16, BFLOAT16 ND
smooth_scales Optional input
  • The input tensor of the operator, corresponding to `smooth_scales` in the formula.
  • The data type is the same as the data type of x. When group_index is empty, the shape is 1D and Dim[0] is the last dimension of x. When group_index is not empty, the shape is 2D. Dim[0] is the number of experts (E). E must not be greater than 1024, and it must not be less than 1 when x is not empty. Dim[1] is the last dimension of x.
FLOAT16, BFLOAT16 ND
group_index Optional input
  • Specifies the index of groups.
  • 1D shape is supported, which is [E, ]. The first dimension of the shape is the same as the first dimension of the shape of smooth_scales.
INT32 ND
dst_type Optional attribute
  • Specifies the data type of the output y. DT_INT4(29), DT_INT8(2), DT_FLOAT8_E5M2(35), DT_FLOAT8_E4M3FN(36), and DT_HIFLOAT8(34) are supported.
  • The default value is DT_INT8(2).
INT -
is_symmetrical Optional attribute
  • Specifies whether to use symmetric quantization.
  • The default value is false.
BOOL -
quant_mode Optional attribute
  • Specifies the quantization mode. "pertoken", "pertensor", and "perchannel" are supported.
  • The default value is "pertoken".
STRING -
dst_type_max Optional attribute
  • Specifies the range of the quantization output. It takes effect only in the Hifloat8 scenario. The supported value range is 0~32768.
  • The default value is 0.
FLOAT -
y Output
  • The output tensor after quantization, corresponding to `y` in the formula.
  • The shape is the same as the input x. When the data type of y is INT4, the last dimension of x must be divisible by 2.
INT8, INT4, FLOAT8_E5M2, FLOAT8_E4M3FN, HIFLOAT8 ND
scale Output
  • The scaling coefficient after quantization, corresponding to `scale` in the formula.
  • When quant_mode is "pertoken", the shape is the same as the shape of x with the last dimension removed. When quant_mode is "pertensor", the shape is (1,).
FLOAT32 ND
offset Output
  • The offset used for asymmetric quantization, corresponding to `offset` in the formula.
  • The shape is the same as the shape of scale.
FLOAT32 ND
  • Atlas inference products and Atlas training products:

    • Input x: only the FLOAT16 data type is supported.
    • Output y: only the INT8 data type is supported.
    • The inputs smooth_scales and group_index are reserved parameters and do not participate in the computation in the current version.
    • Input dst_type: only value 2 can be configured.
  • Atlas A3 products and Atlas A2 products: only the INT8 and INT4 data types are supported for the output y.

  • Kirin X90/Kirin 9030 processor series products:

    • Input x: the BFLOAT16 data type is not supported.
    • Optional input smooth_scales: the BFLOAT16 data type is not supported.
    • Output y: the data types INT4, FLOAT8_E5M2, FLOAT8_E4M3FN, and HIFLOAT8 are not supported.

Constraints

  • E should not be greater than the product of the dimensions of x after the last dimension is removed (S).
  • The values of group_index should increase, in the range from 0 to S. The last value should be S. Otherwise, the result is meaningless.

Invocation Description

Invocation method Sample code Description
aclnn invocation test_aclnn_dynamic_quant_v2 Invoke the DynamicQuantV2 operator through the aclnnDynamicQuantV2 interface.
Graph mode - Invoke the DynamicQuantV2 operator by constructing a graph through the operator IR.