FilesLast commitLast update
22 days ago
3 months ago
22 days ago
22 days ago
22 days ago
9 days ago
1 month ago
22 days ago
18 days ago
README

SparseApplyFtrlV2

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function

  • Operator function: performs the Ftrl-proximal V2 optimization algorithm update on sparse gradients at specified indices, including the l2_shrinkage regularization term. It targets large-scale sparse feature training scenarios.

  • Formula:

For each index i in indices:

grad_shrink={grad[i]l2_shrinkage=0grad[i]+2×l2_shrinkage×var[i]otherwiseaccumnew[i]=accum[i]+grad[i]2accum_pow[i]=accum[i]−lr_poweraccum_new_pow[i]=accumnew[i]−lr_powerdelta=(accum_pow[i]−accum_new_pow[i])/lrlinearnew[i]=linear[i]+grad_shrink+delta×var[i]l1_adjust=clamp(linearnew[i],−l1,l1)x=l1_adjust−linearnew[i]y=accum_new_pow[i]/lr+2×l2varnew[i]=x/y\begin{aligned} \text{grad\_shrink} &= \begin{cases} \text{grad}[i] & l2\_shrinkage = 0 \\ \text{grad}[i] + 2 \times l2\_shrinkage \times \text{var}[i] & \text{otherwise} \end{cases} \\ \text{accum}_{\text{new}}[i] &= \text{accum}[i] + \text{grad}[i]^2 \\ \text{accum\_pow}[i] &= \text{accum}[i]^{-lr\_power} \\ \text{accum\_new\_pow}[i] &= \text{accum}_{\text{new}}[i]^{-lr\_power} \\ \text{delta} &= (\text{accum\_pow}[i] - \text{accum\_new\_pow}[i]) / lr \\ \text{linear}_{\text{new}}[i] &= \text{linear}[i] + \text{grad\_shrink} + \text{delta} \times \text{var}[i] \\ \text{l1\_adjust} &= \text{clamp}(\text{linear}_{\text{new}}[i], -l1, l1) \\ x &= \text{l1\_adjust} - \text{linear}_{\text{new}}[i] \\ y &= \text{accum\_new\_pow}[i] / lr + 2 \times l2 \\ \text{var}_{\text{new}}[i] &= x / y \end{aligned}

Usually lr_power = -0.5. Only the rows specified in indices are updated, and the other rows stay unchanged. If indices[i] is out of bounds (<0 or >=N), that row is skipped.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
var Input The parameter to be optimized, shape [N, D1, D2, ...] float32, float16, bfloat16 ND
accum Input The accumulation of squared gradients, with the same shape as var Same as var ND
linear Input The accumulation of linear terms, with the same shape as var Same as var ND
grad Input The sparse gradient, shape [M, D1, D2, ...] Same as var ND
indices Input The sparse indices, a 1D vector, shape [M] int32, int64 ND
lr Input The learning rate, a scalar Same as var ND
l1 Input The L1 regularization coefficient, a scalar Same as var ND
l2 Input The L2 regularization coefficient, a scalar Same as var ND
l2_shrinkage Input The L2 shrinkage coefficient, a scalar Same as var ND
lr_power Input The learning rate power, a scalar Same as var ND
var Output The updated parameter to be optimized, with the same shape/dtype as input var Same as var ND
accum Output The updated accumulation of squared gradients, with the same shape/dtype as input accum Same as var ND
linear Output The updated accumulation of linear terms, with the same shape/dtype as input linear Same as var ND
use_locking Attribute Whether to use a lock mechanism (multi-thread safe). Default value: false. The current implementation does not read this attribute; it is kept only for interface compatibility bool -

Constraints

  • The shapes of var, accum, and linear must be exactly the same.
  • var is at least 1D (N >= 1). In the ND format it has at most 8 dimensions (1 <= rank <= 8).
  • indices must be a 1D vector.
  • grad.dim(0) must equal indices.dim(0) (that is, M = len(indices)).
  • var.shape[1:] must equal grad.shape[1:], that is, the trailing dimensions match.
  • lr, l1, l2, l2_shrinkage, and lr_power must be scalars (with shape [] or [1]).
  • The values in indices must be within the range [0, var.shape[0]) and must be unique. Because indices must be unique, M = len(indices) must not exceed var.shape[0] (M <= N). If indices contains duplicate values, the results are unpredictable (consistent with the TensorFlow behavior).
  • The dtypes of all data inputs (var/accum/linear/grad/lr/l1/l2/l2_shrinkage/lr_power) must be the same.
  • When indices is empty (len(indices) == 0), the computation is skipped, and var/accum/linear stay unchanged.
  • lr should be greater than 0; otherwise the denominator is 0 and Inf is produced.
  • lr_power should not be greater than 0 (usually -0.5). accum should be non-negative (the implementation clamps negative accum to 0 before the power operation, to avoid NaN).
  • The values of all inputs (including scalar hyperparameters) should be finite and should not contain NaN/Inf or extreme values beyond the representation range of the dtype. Otherwise the intermediate quantities (grad², accum^(-lr_power)) overflow, and the results are not guaranteed to match the reference.
  • To obtain finite and numerically stable results, the recommended value ranges are: lr∈(0,1], l1/l2/l2_shrinkage∈[0,1], lr_power∈[-1,0), and accum∈[1e-3,100]. For var/linear/grad, the recommended magnitude is no more than 10 for fp16/bf16 and no more than 1e3 for fp32 (the combination with accum and lr_power must still not overflow).
  • fp16/bf16 is internally computed as fp32 and then cast back to the original type.

Invocation Description

Invocation method Sample code Description
GE graph mode test_geir_sparse_apply_ftrl_v2 Calls the SparseApplyFtrlV2 operator by building the graph through the operator IR.