FilesLast commitLast update
10 days ago
10 days ago
3 months ago
10 days ago
10 days ago
10 days ago
1 month ago
10 days ago
10 days ago
README

SparseApplyRMSProp

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function

  • Operator function: Performs an update of the RMSProp optimization algorithm on the rows of the specified sparse indexes. According to the indices vector, the operator gathers the corresponding rows from var/ms/mom, applies the RMSProp update formulas, and scatter-writes the updated rows back. Missed rows remain unchanged.

  • Formula:

For each index i in indices:

msnew[i]=ρ⋅msold[i]+(1−ρ)⋅grad[i]2momnew[i]=momentum⋅momold[i]+lr⋅grad[i]/msnew[i]+ϵvarnew[i]=varold[i]−momnew[i]\begin{aligned} \text{ms}_{\text{new}}[i] &= \rho \cdot \text{ms}_{\text{old}}[i] + (1 - \rho) \cdot \text{grad}[i]^2 \\ \text{mom}_{\text{new}}[i] &= \text{momentum} \cdot \text{mom}_{\text{old}}[i] + \text{lr} \cdot \text{grad}[i] / \sqrt{\text{ms}_{\text{new}}[i] + \epsilon} \\ \text{var}_{\text{new}}[i] &= \text{var}_{\text{old}}[i] - \text{mom}_{\text{new}}[i] \end{aligned}

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
var Input Parameter matrix to be optimized, with the shape [N, D1, D2, ...]. FLOAT16, BFLOAT16, FLOAT32 ND
ms Input Exponential moving average of squared gradients, with the shape the same as var. FLOAT16, BFLOAT16, FLOAT32 ND
mom Input Momentum accumulation, with the shape the same as var. FLOAT16, BFLOAT16, FLOAT32 ND
lr Input Learning rate scalar, with the shape [1]. FLOAT16, BFLOAT16, FLOAT32 ND
rho Input Decay rate scalar (typical value 0.9), with the shape [1]. FLOAT16, BFLOAT16, FLOAT32 ND
momentum Input Momentum coefficient scalar (typical value 0.0 to 0.9), with the shape [1]. FLOAT16, BFLOAT16, FLOAT32 ND
epsilon Input Smoothing term scalar (typical value 1e-8), with the shape [1]. FLOAT16, BFLOAT16, FLOAT32 ND
grad Input Sparse gradient, with the shape [M, D1, D2, ...], where M = len(indices). FLOAT16, BFLOAT16, FLOAT32 ND
indices Input Sparse indexes, with the shape [M]. The values should be in the range [0, N) and should be unique. INT32, INT64 ND
var Output Updated parameter matrix, with the shape the same as the input var. FLOAT16, BFLOAT16, FLOAT32 ND
ms Output Updated EMA of squared gradients, with the shape the same as the input ms. FLOAT16, BFLOAT16, FLOAT32 ND
mom Output Updated momentum accumulation, with the shape the same as the input mom. FLOAT16, BFLOAT16, FLOAT32 ND
use_locking Attribute Whether to use a mutex lock to protect the update. The default value is false. BOOL -

Constraints

  • All data tensors (var, ms, mom, lr, rho, momentum, epsilon, grad) must use the same dtype (float16/bfloat16/float32).
  • The data type of indices is int32 or int64.
  • var.shape == ms.shape == mom.shape.
  • grad.shape[0] == indices.shape[0].
  • var.shape[1:] == grad.shape[1:], that is, the trailing dimensions of var and grad must match.
  • The rank of var is >= 1.
  • The rank of indices is 1.
  • The values in indices should be in the range [0, var.shape[0]) and should be unique. If indices contains duplicate values, the result is unpredictable (consistent with the TensorFlow behavior).
  • When indices is empty (len(indices) == 0), the computation is skipped, and var/ms/mom remain unchanged.
  • For fp16/bf16, the internal computation is performed in fp32 and then cast back to the original type.

Invocation Description

Invocation method Sample code Description
GE graph mode test_geir_sparse_apply_rms_prop Invokes the SparseApplyRMSProp operator by constructing a graph with the SparseApplyRMSProp operator IR. A TensorFlow plugin is also provided to map the TensorFlow operators SparseApplyRMSProp and ResourceSparseApplyRMSProp to this operator.