FilesLast commitLast update
7 days ago
2 months ago
3 months ago
15 days ago
15 days ago
9 days ago
2 months ago
3 months ago
1 month ago
17 days ago
README

SituMxQuant

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products ×
Atlas A2 products ×
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function

  • Operator function: fuses the Situ activation function and dynamic MX (Microscaling) quantization into one operator.

  • Formula:

    1. Situ activation:

      situa=β×tanh⁡(gate/β)×sigmoid(gate)situ_a = \beta \times \tanh(gate / \beta) \times sigmoid(gate)

      When linear_beta > 0:

      up=linear_beta×tanh⁡(up/linear_beta)up = linear\_beta \times \tanh(up / linear\_beta)

      situOut=situa×upsituOut = situ_a \times up

      Here, when activate_left is true, gate takes the first half of x and up takes the second half; when activate_left is false, gate takes the second half of x and up takes the first half.

    2. MX quantization (OCP algorithm):

      shared_exp=floor(log2(max(∣situOuti∣)))−emaxshared\_exp = floor(log2(max(|situOut_i|))) - emax

      y_scale=2shared_exp(E8M0)y\_scale = 2^{shared\_exp} (E8M0)

      y=cast_to_fp8(situOut/y_scale)y = cast\_to\_fp8(situOut / y\_scale)

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
x Input Data to be processed, corresponding to x in the formula. The last dimension must be a multiple of 2. The input does not support containing ±inf or nan. FLOAT16, BFLOAT16 ND
beta Attribute beta parameter of the Situ activation, corresponding to β in the formula. It must be greater than 0. The default value is 1.0. FLOAT -
linear_beta Attribute linear_beta parameter of the Situ activation, corresponding to linear_beta in the formula. It is not enabled when the value is ≤ 0. The default value is 0.0. FLOAT -
activate_left Attribute Indicates whether gate takes the first half or the second half of x, corresponding to activate_left in the formula. The default value is false. BOOL -
axis Attribute Denotes the quantization axis, the axis for computing max and scale in the formula. Currently only -1 is supported. The default value is -1. INT -
dst_type Attribute Denotes the data type of output y, corresponding to the target type of cast_to_fp8 in the formula: 36=FLOAT8_E4M3FN and 35=FLOAT8_E5M2. The default value is 36. INT -
round_mode Attribute Denotes the rounding mode of quantization, the rounding method of cast in the formula. "rint", "round", and "floor" are supported. FP8 outputs support only "rint". The default value is "rint". STRING -
y Output Quantized output, corresponding to y in the formula. FLOAT8_E4M3FN, FLOAT8_E5M2 ND
y_scale Output Scale of MX quantization (E8M0 format), corresponding to y_scale in the formula. FLOAT8_E8M0 ND

Constraints

  • The last dimension of x must be a multiple of 2.
  • x must have at least 1 dimension.
  • axis currently supports only -1 (tail-axis quantization).
  • dst_type supports 36 (FLOAT8_E4M3FN) or 35 (FLOAT8_E5M2).
  • round_mode must be "rint".
  • The data type of y must match dst_type, and the data type of y_scale must be FLOAT8_E8M0.
  • The shapes of y and y_scale must be consistent with the derived results (see the Parameters section).
  • The shape constraints of y_scale are as follows:
    • H = x.shape[-1] / 2
    • scaleNum = ceil(H / 64)
    • y_scale.shape = x.shape[:-1] + [scaleNum, 2]

Invocation Description

Invocation method Invocation sample Description
aclnn invocation test_aclnn_situ_mx_quant Invokes the SituMxQuant operator through the aclnnSituMxQuant interface.
Graph-mode invocation - Invokes the SituMxQuant operator through graph construction with the operator IR.