FilesLast commitLast update
12 days ago
2 months ago
18 hours ago
3 months ago
4 days ago
3 months ago
12 days ago
12 days ago
README

AmpUpdateScale

English | 简体中文

Product Support Status

Product Supported
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function Description

  • Operator function: implements the dynamic Scale update logic in AMP (Automatic Mixed Precision) training. It dynamically adjusts the loss scale based on the current scale value, the growth tracker counter, and whether Inf/NaN is found.

  • Computation formula:

    updated_scale={current_scale×backoff_factorif found_inf≠0current_scale×growth_factorif growth_tracker + 1 = growth_interval and new_scale is finitecurrent_scaleotherwise\text{updated\_scale} = \begin{cases} \text{current\_scale} \times \text{backoff\_factor} & \text{if found\_inf} \neq 0 \\ \text{current\_scale} \times \text{growth\_factor} & \text{if growth\_tracker + 1 = growth\_interval and new\_scale is finite} \\ \text{current\_scale} & \text{otherwise} \end{cases}

    updated_growth_tracker={0if found_inf≠0 or growth triggeredgrowth_tracker+1otherwise\text{updated\_growth\_tracker} = \begin{cases} 0 & \text{if found\_inf} \neq 0 \text{ or growth triggered} \\ \text{growth\_tracker} + 1 & \text{otherwise} \end{cases}

    where:

    • current_scale\text{current\_scale}: the current loss scale value
    • found_inf\text{found\_inf}: a flag that indicates whether Inf/NaN is detected (0 indicates normal, and non-0 indicates that Inf/NaN is found)
    • growth_tracker\text{growth\_tracker}: a counter of the number of steps in which Inf/NaN does not occur consecutively
    • growth_factor\text{growth\_factor}: the scale growth factor (usually 2.0)
    • backoff_factor\text{backoff\_factor}: the scale backoff factor (usually 0.5)
    • growth_interval\text{growth\_interval}: the interval in steps for triggering scale growth
  • Usage scenario: dynamic loss scaling in AMP training, which is used to prevent gradient underflow in FP16/BF16 mixed-precision training.

Parameter Description

Parameter Name Input/Output/Attribute Description Data Type Data Format
current_scale Input The current loss scale value. FLOAT, FLOAT16, BFLOAT16 ND
growth_tracker Input A counter of the number of steps in which Inf/NaN does not occur consecutively. INT32 ND
found_inf Input A flag that indicates whether Inf/NaN is detected. 0 indicates normal, and non-0 indicates that Inf/NaN is found. FLOAT, FLOAT16, BFLOAT16 ND
growth_factor Attribute The scale growth factor, usually set to 2.0. FLOAT -
backoff_factor Attribute The scale backoff factor, usually set to 0.5. FLOAT -
growth_interval Attribute The interval in steps for triggering scale growth. INT -
updated_scale Output The updated loss scale value. FLOAT, FLOAT16, BFLOAT16 ND
updated_growth_tracker Output The updated growth tracker counter. INT32 ND
  • Shape constraint: All input and output tensors are scalars, and the shape is [1].
  • Data type constraint: The data type of current_scale must be the same as those of found_inf and updated_scale.

Constraint Description

None

Invocation Description

Invocation Method Invocation Sample Description
aclnn invocation test_aclnn_amp_update_scale Invokes the AmpUpdateScale operator through the aclnnAmpUpdateScale API.