FilesLast commitLast update
12 days ago
2 months ago
2 months ago
9 days ago
23 days ago
19 days ago
9 days ago
1 month ago
17 days ago
README

DynamicMxQuantWithDualAxis

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products ×
Atlas A2 products ×
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function

  • Operator function: Performs MX quantization with the FLOAT4-class or FLOAT8-class destination data type on the -1 axis and the -2 axis at the same time. On the given -1 axis and -2 axis, for every 32 values, the operator computes the quantization scales mxscale1 and mxscale2 corresponding to the two groups of values, divides all elements of each group by the corresponding mxscale1 or mxscale2, converts them to the corresponding dst_type according to round_mode, and obtains the quantization results y1 and y2. When dst_type is FLOAT8_E4M3FN or FLOAT8_E5M2, the value of scale_alg specifies different algorithms for computing mxscale.

  • Axis merging description: During implementation, the operator merges all axes before the -2 axis (exclusive). For a tensor whose input shape is (d0,d1,...,dn−3,dn−2,dn−1)(d_0, d_1, ..., d_{n-3}, d_{n-2}, d_{n-1}), the dimensions before the -2 axis, (d0,d1,...,dn−3)(d_0, d_1, ..., d_{n-3}), are merged into one dimension; this is equivalent to reshaping the input to (d0×d1×...×dn−3,dn−2,dn−1)(d_0 \times d_1 \times ... \times d_{n-3}, d_{n-2}, d_{n-1}) and then performing the quantization computation.

  • Formula:

    • Scenario 1: when scale_alg is 0, the implementation follows the OCP Microscaling Formats (Mx) Specification:

    • Groups the input x into groups of 32 values along the -1 axis; a group of 32 values {{Vi}i=132}\{\{V_i\}_{i=1}^{32}\} is quantized to {mxscale1,{Pi}i=132}\{mxscale1, \{P_i\}_{i=1}^{32}\}

      shared_exp=floor(log2(maxi(∣Vi∣)))−emaxshared\_exp = floor(log_2(max_i(|V_i|))) - emax

      mxscale1=2shared_expmxscale1 = 2^{shared\_exp}

      Pi=cast_to_dst_type(Vi/mxscale1,round_mode), i from 1 to 32P_i = cast\_to\_dst\_type(V_i/mxscale1, round\_mode), \space i\space from\space 1\space to\space 32

    • At the same time, groups the input x into groups of 32 values along the -2 axis; a group of 32 values {{Vj}j=132}\{\{V_j\}_{j=1}^{32}\} is quantized to {mxscale2,{Pj}j=132}\{mxscale2, \{P_j\}_{j=1}^{32}\}

      shared_exp=floor(log2(maxj(∣Vj∣)))−emaxshared\_exp = floor(log_2(max_j(|V_j|))) - emax

      mxscale2=2shared_expmxscale2 = 2^{shared\_exp}

      Pj=cast_to_dst_type(Vj/mxscale2,round_mode), j from 1 to 32P_j = cast\_to\_dst\_type(V_j/mxscale2, round\_mode), \space j\space from\space 1\space to\space 32

    • The PiP_{i} values quantized along the -1 axis form the output y1 at the positions of the corresponding ViV_{i}, and mxscale1 forms the output mxscale1 according to the groups on the -1 axis dimension. The PjP_{j} values quantized along the -2 axis form the output y2 at the positions of the corresponding VjV_{j}, and mxscale2 forms the output mxscale2 according to the groups on the -2 axis dimension.

    • emax: the exponent of the largest normal number of the corresponding data type.

      DataType emax
      FLOAT4_E2M1 2
      FLOAT4_E1M2 0
      FLOAT8_E4M3FN 8
      FLOAT8_E5M2 15
    • Scenario 2: when scale_alg is 1, only FP8 types are involved (round-up algorithm):

      • Groups the input x into groups of 32 values along the -1 axis (or the -2 axis), computes the block scaling factor Sfp32bS_{fp32}^b for each group separately, and then maps all elements in the group to the target low-precision type FP8.
      • Finds the maximum absolute value of the values in the group: Amax(Dfp32b)=max({∣di∣}i=132)Amax(D_{fp32}^b) = max(\{|d_i|\}_{i=1}^{32})
      • Maps FP32 into the range representable by the target data type FP8: Sfp32b=Amax(Dfp32b)Amax(DType)S_{fp32}^b = \frac{Amax(D_{fp32}^b)}{Amax(DType)}
      • Extracts the unbiased exponent EintbE_{int}^b and the mantissa MfixpbM_{fixp}^b from the block scaling factor and performs conditional rounding up.
      • Computes the block scaling factor: Sue8m0b=2EintbS_{ue8m0}^b=2^{E_{int}^b}; computes the block conversion factor: Rfp32b=1fp32(Sue8m0b)R_{fp32}^b=\frac{1}{fp32(S_{ue8m0}^b)}
      • Applies quantization to each element in the group: di=DType(dfp32i⋅Rfp32b)d^i = DType(d_{fp32}^i \cdot R_{fp32}^b)
    • Scenario 3: when scale_alg is 2, only the FP4_E2M1 type is involved (DynamicDtypeRange algorithm, supported only by the V2 interface):

      • When dstTypeMax = 0.0/6.0/7.0, the operator computes scale using the exponent-field addValueBit carry method.
      • When dstTypeMax is another custom value, the operator computes scale using the FP32-precision invDstTypeMax multiplication method.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
x Input Data to be quantized, corresponding to Vi and di in the formulas.
When the destination type is FLOAT4_E2M1 or FLOAT4_E1M2, the last dimension of x must be an even number.
FLOAT16, BFLOAT16 ND
round_mode Optional Attribute Data conversion mode.
When dst_type is 40/41(FLOAT4_E2M1/FLOAT4_E1M2), {"rint", "floor", "round"} are supported;
when dst_type is 35/36(FLOAT8_E5M2/FLOAT8_E4M3FN), only {"rint"} is supported;
when a null pointer is passed in, the "rint" mode is used.
STRING -
dst_type Optional Attribute Specifies the types of y1 and y2 after data conversion.
The input range is {35, 36, 40, 41}, corresponding to {35:FLOAT8_E5M2, 36:FLOAT8_E4M3FN, 40:FLOAT4_E2M1, 41:FLOAT4_E1M2} respectively.
INT64 -
scale_alg Optional Attribute Computation methods of mxscale1 and mxscale2.
Supported values are 0, 1, and 2; 0 represents the OCP implementation (scenario 1), 1 represents the round-up implementation (scenario 2), and 2 represents the DynamicDtypeRange implementation (scenario 3).
When dst_type is FLOAT4_E1M2, only 0 is supported.
When dst_type is FLOAT4_E2M1, only 0 and 2 are supported.
When dst_type is FLOAT8, only 0 and 1 are supported.
INT64 -
dst_type_max Optional Attribute Value of maxType, corresponding to Amax(DType) in the formulas.
Supported values are 0.0 and 6.0-12.0; 0.0 means that Amax(DType) is the maximum value of the quantization result data type, and 6.0-12.0 means that Amax(DType) is the passed-in value.
This value can be set only when FP4_E2M1 is used and scale_alg is 2.
DOUBLE -
y1 Output Result corresponding to the input x after quantization along the -1 axis, corresponding to Pi and di in the formulas.
The shape is the same as that of the input x.
FLOAT4_E2M1, FLOAT4_E1M2, FLOAT8_E4M3FN, FLOAT8_E5M2 ND
mxscale1 Output Quantization scale corresponding to each group on the -1 axis, corresponding to mxscale1 and Sb in the formulas.
The shape is the -1 axis value of x divided by 32 and rounded up, with even padding and a pad fill value of 0.
FLOAT8_E8M0 ND
y2 Output Result corresponding to the input x after quantization along the -2 axis, corresponding to Pj and dj in the formulas.
The shape is the same as that of the input x.
FLOAT4_E2M1, FLOAT4_E1M2, FLOAT8_E4M3FN, FLOAT8_E5M2 ND
mxscale2 Output Quantization scale corresponding to each group on the -2 axis, corresponding to mxscale2 and Sb in the formulas.
The shape is the -2 axis value of x divided by 32 and rounded up, with even padding and a pad fill value of 0.
The mxscale2 output requires interleaving processing for every two rows of data.
FLOAT8_E8M0 ND

Constraints

  • The shape constraints on x, mxscale1, and mxscale2 are as follows:
    • The number of dimensions of x should be greater than or equal to 2.
    • rank(mxscale1) = rank(x) + 1.
    • rank(mxscale2) = rank(x) + 1.
    • mxscale1.shape[-2] = (ceil(x.shape[-1] / 32) + 2 - 1) / 2.
    • mxscale2.shape[-3] = (ceil(x.shape[-2] / 32) + 2 - 1) / 2.
    • mxscale1.shape[-1] = 2.
    • mxscale2.shape[-1] = 2.
    • Other dimensions are the same as those of the input x.
    • Example: when the shape of the input x is [B, M, N] and the destination data type is FP8-class, the corresponding y1 and y2 have the shape [B, M, N], mxscale1 has the shape [B, M, (ceil(N/32)+2-1)/2, 2], and mxscale2 has the shape [B, (ceil(M/32)+2-1)/2, N, 2].

Invocation Description

Invocation method Sample code Description
aclnn invocation (V1) test_aclnn_dynamic_mx_quant_with_dual_axis Invokes the operator through the aclnnDynamicMxQuantWithDualAxis interface; supports scale_alg=0/1.
aclnn invocation (V2) test_aclnn_dynamic_mx_quant_with_dual_axis_v2 Invokes the operator through the aclnnDynamicMxQuantWithDualAxisV2 interface; supports scale_alg=0/1/2.
Graph-mode invocation - Builds the graph and invokes the DynamicMxQuantWithDualAxis operator through the operator IR.