FilesLast commitLast update
13 days ago
26 days ago
21 days ago
2 months ago
21 days ago
3 months ago
26 days ago
18 days ago
README

LarsV2Update

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products √
Atlas inference products √
Atlas training products √

Function

  • Operator function: performs the LARS-V2 optimizer gradient update, which adaptively adjusts the trust coefficient of the gradient according to the L2 norm of the weight, and is used in large-batch training scenarios.

  • Formula:

    w_norm=w_square_sumw\_norm = \sqrt{w\_square\_sum}

    g_norm=g_square_sumg\_norm = \sqrt{g\_square\_sum}

    coeff=hyperpara×w_normweight_decay×w_norm+g_norm+epsiloncoeff = \frac{hyperpara \times w\_norm}{weight\_decay \times w\_norm + g\_norm + epsilon}

    • If use_clip = True:

      coeff=max⁡(0, min⁡(coefflearning_rate, 1))coeff = \max\left(0,\ \min\left(\frac{coeff}{learning\_rate},\ 1\right)\right)

    grad_weight=w×weight_decay+ggrad\_weight = w \times weight\_decay + g

    g_new=grad_weight×coeffg\_new = grad\_weight \times coeff

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
w Input Weight tensor. FLOAT ND
g Input Gradient tensor, with the same data type and shape as w. FLOAT ND
w_square_sum Input Sum of squares of the weights (precomputed by SquareSumAll), a scalar. FLOAT ND
g_square_sum Input Sum of squares of the gradients, a scalar. FLOAT ND
weight_decay Input Weight decay coefficient, a scalar. FLOAT ND
learning_rate Input Learning rate, a scalar. It participates in the computation only when use_clip=True. FLOAT ND
hyperpara Attribute LARS trust coefficient (eta), with the default value 0.001. Float -
epsilon Attribute Small constant to prevent division by zero, with the default value 0.00001. Float -
use_clip Attribute Whether to clip coeff to [0,1], with the default value False. Bool -
g_new Output Updated gradient, with the same data type and shape as w. FLOAT ND

Constraints

  • Inputs w and g must have the same shape and data type (the Tiling stage checks that the dtype is DT_FLOAT and that the shapes are exactly identical; see op_host/arch35/lars_v2_update_tiling_arch35.cpp).
  • Output g_new has the same type, shape, and dtype as w (InferShape directly copies the shape of w without broadcasting; the Tiling stage performs a fallback check that g_new and w have the same shape).
  • w_square_sum, g_square_sum, weight_decay, and learning_rate are always FLOAT-type scalars (the number of elements is 1, and they are broadcast to the shape of w for the computation; the Tiling stage checks the dtype and the number of elements).
  • Supports 0 to 8 dimensions (a 0-dimensional scalar is processed as 1 dimension, and the Tiling stage checks that the number of dimensions does not exceed 8).
  • Supports empty tensors (the number of elements is 0; for example, when the shape contains a 0 dimension, the output is an empty tensor without an error; on the kernel side, totalTiles==0 short-circuits the entire execution, and each tile is skipped when its count <= 0).
  • NaN in the input data propagates according to IEEE 754 semantics (the clipping semantics for NaN in hardware vector instructions may differ from the framework reference implementation; the upper-layer framework must guarantee NaN input scenarios, and the operator does not intercept NaN).
  • When inf/nan is passed in for attributes hyperpara/epsilon, no check or interception is performed, and they directly participate in the computation according to IEEE 754 floating-point semantics (hyperpara=inf→g_new is all ±Inf; hyperpara/epsilon=nan→g_new is all NaN; epsilon=inf→coeff=0→g_new is all zeros), consistent with the NaN propagation policy for inputs.
  • The total number of elements has no independent software limit: the single-axis length and the total scale are constrained only by the device HBM capacity (the three full-size fp32 buffers of w/g/g_new occupy 12×numel bytes in total); GM offsets and the entire shape/stride chain use int64, and extra-large shapes are handled through Tiling chunking.
  • When use_clip=True, learning_rate must be non-zero (it participates in the coeff/learning_rate clipping, and a value of 0 causes division by zero); epsilon must not be configured with a value that makes the denominator weight_decay×w_norm+g_norm+epsilon equal to 0 (for example, all-zero norms and epsilon=0).

Invocation Description

Invocation method Sample code Description
GE graph mode test_geir_lars_v2_update Invokes the LarsV2Update operator through graph construction with the operator IR; the sample includes CPU golden numerical verification (dtype/shape/element-wise).