| Files | Last commit | Last update |
|---|---|---|
| 10 days ago | ||
| 10 days ago | ||
| 3 months ago | ||
| 10 days ago | ||
| 10 days ago | ||
| 10 days ago | ||
| 1 month ago | ||
| 10 days ago | ||
| 10 days ago |
SparseApplyRMSProp
English | 简体中文
Applicable Products
| Product | Support |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
Function
-
Operator function: Performs an update of the RMSProp optimization algorithm on the rows of the specified sparse indexes. According to the indices vector, the operator gathers the corresponding rows from var/ms/mom, applies the RMSProp update formulas, and scatter-writes the updated rows back. Missed rows remain unchanged.
-
Formula:
For each index i in indices:
msnew[i]=ρ⋅msold[i]+(1−ρ)⋅grad[i]2momnew[i]=momentum⋅momold[i]+lr⋅grad[i]/msnew[i]+ϵvarnew[i]=varold[i]−momnew[i]\begin{aligned} \text{ms}_{\text{new}}[i] &= \rho \cdot \text{ms}_{\text{old}}[i] + (1 - \rho) \cdot \text{grad}[i]^2 \\ \text{mom}_{\text{new}}[i] &= \text{momentum} \cdot \text{mom}_{\text{old}}[i] + \text{lr} \cdot \text{grad}[i] / \sqrt{\text{ms}_{\text{new}}[i] + \epsilon} \\ \text{var}_{\text{new}}[i] &= \text{var}_{\text{old}}[i] - \text{mom}_{\text{new}}[i] \end{aligned}
Parameters
| Parameter name | Input/Output/Attribute | Description | Data type | Data format |
|---|---|---|---|---|
| var | Input | Parameter matrix to be optimized, with the shape [N, D1, D2, ...]. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| ms | Input | Exponential moving average of squared gradients, with the shape the same as var. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| mom | Input | Momentum accumulation, with the shape the same as var. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| lr | Input | Learning rate scalar, with the shape [1]. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| rho | Input | Decay rate scalar (typical value 0.9), with the shape [1]. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| momentum | Input | Momentum coefficient scalar (typical value 0.0 to 0.9), with the shape [1]. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| epsilon | Input | Smoothing term scalar (typical value 1e-8), with the shape [1]. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| grad | Input | Sparse gradient, with the shape [M, D1, D2, ...], where M = len(indices). | FLOAT16, BFLOAT16, FLOAT32 | ND |
| indices | Input | Sparse indexes, with the shape [M]. The values should be in the range [0, N) and should be unique. | INT32, INT64 | ND |
| var | Output | Updated parameter matrix, with the shape the same as the input var. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| ms | Output | Updated EMA of squared gradients, with the shape the same as the input ms. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| mom | Output | Updated momentum accumulation, with the shape the same as the input mom. | FLOAT16, BFLOAT16, FLOAT32 | ND |
| use_locking | Attribute | Whether to use a mutex lock to protect the update. The default value is false. | BOOL | - |
Constraints
- All data tensors (var, ms, mom, lr, rho, momentum, epsilon, grad) must use the same dtype (float16/bfloat16/float32).
- The data type of indices is int32 or int64.
- var.shape == ms.shape == mom.shape.
- grad.shape[0] == indices.shape[0].
- var.shape[1:] == grad.shape[1:], that is, the trailing dimensions of var and grad must match.
- The rank of var is >= 1.
- The rank of indices is 1.
- The values in indices should be in the range [0, var.shape[0]) and should be unique. If indices contains duplicate values, the result is unpredictable (consistent with the TensorFlow behavior).
- When indices is empty (len(indices) == 0), the computation is skipped, and var/ms/mom remain unchanged.
- For fp16/bf16, the internal computation is performed in fp32 and then cast back to the original type.
Invocation Description
| Invocation method | Sample code | Description |
|---|---|---|
| GE graph mode | test_geir_sparse_apply_rms_prop | Invokes the SparseApplyRMSProp operator by constructing a graph with the SparseApplyRMSProp operator IR. A TensorFlow plugin is also provided to map the TensorFlow operators SparseApplyRMSProp and ResourceSparseApplyRMSProp to this operator. |