DynamicQuantV2
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | √ |
| Atlas training products | √ |
| Kirin X90 processor series products | √ |
| Kirin 9030 processor series products | √ |
Function
-
Operator function: performs pertoken/pertensor/perchannel symmetric/asymmetric dynamic quantization for the input tensor. In the MOE scenario, the smooth_scales of each expert are different, which are distinguished by the input group_index.
-
Formula:
-
Symmetric quantization:
-
If smooth_scales is not provided:
scale=row_max(abs(x))/dtypeMax scale=row\_max(abs(x))/dtypeMax
y=round(x/scale) y=round(x/scale)
-
If smooth_scales is provided:
input=x⋅smoothscales input = x\cdot smooth_scales
scale=row_max(abs(input))/dtypeMax scale=row\_max(abs(input))/dtypeMax
y=round(input/scale) y=round(input/scale)
-
-
Asymmetric quantization:
-
If smooth_scales is not provided:
scale=(row_max(x)−row_min(x))/(dtypeMax−dtypeMin) scale=(row\_max(x) - row\_min(x))/(dtypeMax - dtypeMin)
offset=dtypeMax−row_max(x)/scale offset=dtypeMax-row\_max(x)/scale
y=round(x/scale+offset) y=round(x/scale+offset)
-
If smooth_scales is provided:
input=x⋅smoothscales input = x\cdot smooth_scales
scale=(row_max(input)−row_min(input))/(dtypeMax−dtypeMin) scale=(row\_max(input) - row\_min(input))/(dtypeMax - dtypeMin)
offset=dtypeMax−row_max(input)/scale offset=dtypeMax-row\_max(input)/scale
y=round(input/scale+offset) y=round(input/scale+offset)
-
Where:
- row_max represents the computation of the maximum value of each row.
- row_min represents the computation of the minimum value of each row.
- dtypeMax is the maximum value of the output data type.
- dtypeMin is the minimum value of the output data type.
-
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| x | Input |
|
FLOAT16, BFLOAT16 | ND |
| smooth_scales | Optional input |
|
FLOAT16, BFLOAT16 | ND |
| group_index | Optional input |
|
INT32 | ND |
| dst_type | Optional attribute |
|
INT | - |
| is_symmetrical | Optional attribute |
|
BOOL | - |
| quant_mode | Optional attribute |
|
STRING | - |
| dst_type_max | Optional attribute |
|
FLOAT | - |
| y | Output |
|
INT8, INT4, FLOAT8_E5M2, FLOAT8_E4M3FN, HIFLOAT8 | ND |
| scale | Output |
|
FLOAT32 | ND |
| offset | Output |
|
FLOAT32 | ND |
-
Atlas inference products and Atlas training products:
- Input
x: only the FLOAT16 data type is supported. - Output
y: only the INT8 data type is supported. - The inputs
smooth_scalesandgroup_indexare reserved parameters and do not participate in the computation in the current version. - Input
dst_type: only value 2 can be configured.
- Input
-
Atlas A3 products and Atlas A2 products: only the INT8 and INT4 data types are supported for the output
y. -
Kirin X90/Kirin 9030 processor series products:
- Input
x: the BFLOAT16 data type is not supported. - Optional input
smooth_scales: the BFLOAT16 data type is not supported. - Output
y: the data types INT4, FLOAT8_E5M2, FLOAT8_E4M3FN, and HIFLOAT8 are not supported.
- Input
Constraints
- E should not be greater than the product of the dimensions of x after the last dimension is removed (S).
- The values of
group_indexshould increase, in the range from 0 to S. The last value should be S. Otherwise, the result is meaningless.
Invocation Description
| Invocation method | Sample code | Description |
|---|---|---|
| aclnn invocation | test_aclnn_dynamic_quant_v2 | Invoke the DynamicQuantV2 operator through the aclnnDynamicQuantV2 interface. |
| Graph mode | - | Invoke the DynamicQuantV2 operator by constructing a graph through the operator IR. |