| Files | Last commit | Last update |
|---|---|---|
| 22 days ago | ||
| 3 months ago | ||
| 22 days ago | ||
| 22 days ago | ||
| 22 days ago | ||
| 9 days ago | ||
| 1 month ago | ||
| 22 days ago | ||
| 18 days ago |
SparseApplyFtrlV2
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
Function
-
Operator function: performs the Ftrl-proximal V2 optimization algorithm update on sparse gradients at specified indices, including the
l2_shrinkageregularization term. It targets large-scale sparse feature training scenarios. -
Formula:
For each index i in indices:
grad_shrink={grad[i]l2_shrinkage=0grad[i]+2×l2_shrinkage×var[i]otherwiseaccumnew[i]=accum[i]+grad[i]2accum_pow[i]=accum[i]−lr_poweraccum_new_pow[i]=accumnew[i]−lr_powerdelta=(accum_pow[i]−accum_new_pow[i])/lrlinearnew[i]=linear[i]+grad_shrink+delta×var[i]l1_adjust=clamp(linearnew[i],−l1,l1)x=l1_adjust−linearnew[i]y=accum_new_pow[i]/lr+2×l2varnew[i]=x/y\begin{aligned} \text{grad\_shrink} &= \begin{cases} \text{grad}[i] & l2\_shrinkage = 0 \\ \text{grad}[i] + 2 \times l2\_shrinkage \times \text{var}[i] & \text{otherwise} \end{cases} \\ \text{accum}_{\text{new}}[i] &= \text{accum}[i] + \text{grad}[i]^2 \\ \text{accum\_pow}[i] &= \text{accum}[i]^{-lr\_power} \\ \text{accum\_new\_pow}[i] &= \text{accum}_{\text{new}}[i]^{-lr\_power} \\ \text{delta} &= (\text{accum\_pow}[i] - \text{accum\_new\_pow}[i]) / lr \\ \text{linear}_{\text{new}}[i] &= \text{linear}[i] + \text{grad\_shrink} + \text{delta} \times \text{var}[i] \\ \text{l1\_adjust} &= \text{clamp}(\text{linear}_{\text{new}}[i], -l1, l1) \\ x &= \text{l1\_adjust} - \text{linear}_{\text{new}}[i] \\ y &= \text{accum\_new\_pow}[i] / lr + 2 \times l2 \\ \text{var}_{\text{new}}[i] &= x / y \end{aligned}
Usually
lr_power = -0.5. Only the rows specified inindicesare updated, and the other rows stay unchanged. Ifindices[i]is out of bounds (<0 or >=N), that row is skipped.
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| var | Input | The parameter to be optimized, shape [N, D1, D2, ...] | float32, float16, bfloat16 | ND |
| accum | Input | The accumulation of squared gradients, with the same shape as var | Same as var | ND |
| linear | Input | The accumulation of linear terms, with the same shape as var | Same as var | ND |
| grad | Input | The sparse gradient, shape [M, D1, D2, ...] | Same as var | ND |
| indices | Input | The sparse indices, a 1D vector, shape [M] | int32, int64 | ND |
| lr | Input | The learning rate, a scalar | Same as var | ND |
| l1 | Input | The L1 regularization coefficient, a scalar | Same as var | ND |
| l2 | Input | The L2 regularization coefficient, a scalar | Same as var | ND |
| l2_shrinkage | Input | The L2 shrinkage coefficient, a scalar | Same as var | ND |
| lr_power | Input | The learning rate power, a scalar | Same as var | ND |
| var | Output | The updated parameter to be optimized, with the same shape/dtype as input var | Same as var | ND |
| accum | Output | The updated accumulation of squared gradients, with the same shape/dtype as input accum | Same as var | ND |
| linear | Output | The updated accumulation of linear terms, with the same shape/dtype as input linear | Same as var | ND |
| use_locking | Attribute | Whether to use a lock mechanism (multi-thread safe). Default value: false. The current implementation does not read this attribute; it is kept only for interface compatibility | bool | - |
Constraints
- The shapes of var, accum, and linear must be exactly the same.
- var is at least 1D (N >= 1). In the ND format it has at most 8 dimensions (1 <= rank <= 8).
- indices must be a 1D vector.
- grad.dim(0) must equal indices.dim(0) (that is, M = len(indices)).
- var.shape[1:] must equal grad.shape[1:], that is, the trailing dimensions match.
- lr, l1, l2, l2_shrinkage, and lr_power must be scalars (with shape [] or [1]).
- The values in indices must be within the range [0, var.shape[0]) and must be unique. Because indices must be unique, M = len(indices) must not exceed var.shape[0] (M <= N). If indices contains duplicate values, the results are unpredictable (consistent with the TensorFlow behavior).
- The dtypes of all data inputs (var/accum/linear/grad/lr/l1/l2/l2_shrinkage/lr_power) must be the same.
- When indices is empty (len(indices) == 0), the computation is skipped, and var/accum/linear stay unchanged.
- lr should be greater than 0; otherwise the denominator is 0 and Inf is produced.
- lr_power should not be greater than 0 (usually -0.5). accum should be non-negative (the implementation clamps negative accum to 0 before the power operation, to avoid NaN).
- The values of all inputs (including scalar hyperparameters) should be finite and should not contain NaN/Inf or extreme values beyond the representation range of the dtype. Otherwise the intermediate quantities (grad², accum^(-lr_power)) overflow, and the results are not guaranteed to match the reference.
- To obtain finite and numerically stable results, the recommended value ranges are:
lr∈(0,1],l1/l2/l2_shrinkage∈[0,1],lr_power∈[-1,0), andaccum∈[1e-3,100]. Forvar/linear/grad, the recommended magnitude is no more than 10 for fp16/bf16 and no more than 1e3 for fp32 (the combination withaccumandlr_powermust still not overflow). - fp16/bf16 is internally computed as fp32 and then cast back to the original type.
Invocation Description
| Invocation method | Sample code | Description |
|---|---|---|
| GE graph mode | test_geir_sparse_apply_ftrl_v2 | Calls the SparseApplyFtrlV2 operator by building the graph through the operator IR. |