FilesLast commitLast update
1 month ago
1 month ago
1 month ago
16 days ago
1 month ago
1 month ago
16 days ago
1 month ago
18 days ago
README

English | 简体中文

SparseApplyAdagrad

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function

  • Operator function: SparseApplyAdagrad is the sparse update operator of the Adagrad optimization algorithm. It selects the corresponding rows in var and accum according to indices, and updates the parameter variable and the gradient square accumulator in place with the sparse gradient grad. Rows not hit by indices remain unchanged. This operator is commonly used in large-scale sparse parameter training scenarios such as Embedding.

  • Formula:

For each index index = indices[i] in indices:

accumnew[index]={accumold[index]+grad[i]2,update_slots=Trueaccumold[index],update_slots=False\text{accum}_{\text{new}}[\text{index}] = \begin{cases} \text{accum}_{\text{old}}[\text{index}] + \text{grad}[i]^2, & \text{update\_slots} = \text{True} \\ \text{accum}_{\text{old}}[\text{index}], & \text{update\_slots} = \text{False} \end{cases}

varnew[index]=varold[index]−lr×grad[i]accumnew[index]+10−8\text{var}_{\text{new}}[\text{index}] = \text{var}_{\text{old}}[\text{index}] - \text{lr} \times \frac{\text{grad}[i]}{\sqrt{\text{accum}_{\text{new}}[\text{index}]} + 10^{-8}}

Here, 1e-8 is a fixed float32 numerical stability constant inside the kernel.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
var Input Parameter tensor to be updated, with shape [N, D1, D2, ...]. The first dimension is the sparse index dimension. FLOAT ND
accum Input Gradient square accumulator, with the same shape and dtype as var. FLOAT ND
lr Input Learning rate scalar, with shape [] or [1]. FLOAT ND
grad Input Sparse gradient tensor, with shape [M, D1, D2, ...], where M equals the length of indices. FLOAT ND
indices Input Sparse index vector, with shape [M], which specifies the rows to be updated in var/accum. INT32 ND
var Output Updated parameter tensor, with the same shape and dtype as the input var. FLOAT ND
accum Output Updated gradient square accumulator, with the same shape and dtype as the input accum. FLOAT ND
use_locking Attribute Whether to use a lock to protect variable updates. Default value: false. BOOL -
update_slots Attribute Whether to update accum. Default value: true. When the value is true, accum += grad * grad is performed. BOOL -

Constraints

  • var and accum must have the same shape, and the rank is at least 2.
  • lr must be a scalar, with shape [] or [1].
  • indices must be a 1D vector.
  • The rank of grad must be the same as that of var. grad.shape[0] must equal indices.shape[0], and grad.shape[1:] must equal var.shape[1:].
  • The data types of var, accum, lr, and grad must be consistent. Currently, only float32 is supported.
  • The data type of indices currently supports only int32.
  • The values in indices must satisfy 0 <= indices[i] < var.shape[0] and should be unique. If indices contains duplicate values, the result is unpredictable.
  • epsilon is fixed to float32 1e-8 and is not an input parameter.
  • use_locking is integrated as a compatibility attribute. Currently, no additional concurrency semantics are guaranteed.

Invocation Description

Invocation method Invocation sample Description
Graph-mode invocation test_geir_sparse_apply_adagrad.cpp Calls the SparseApplyAdagrad operator through graph construction with the operator IR.
TensorFlow graph-mode invocation - Maps the TensorFlow operator SparseApplyAdagrad to this operator through the TensorFlow plugin.

For the graph-mode invocation description in Kernel invocation, install the operator package and run:

bash build.sh --run_example sparse_apply_adagrad graph --soc=ascend950