FilesLast commitLast update
23 days ago
2 months ago
3 days ago
18 days ago
3 days ago
2 months ago
5 days ago
3 days ago
1 month ago
17 days ago
README

SwishGrad

English | 简体中文

Applicable Products

Product Support
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products √
Atlas inference products √
Atlas training products √

Function

  • Operator function: the backpropagation of the Swish activation function, used to compute the gradient of the Swish activation function.

  • Formula:

    • Formula of the Swish function

    s(x)=x∗σ(βx)s(x) = x*\sigma(\beta x)

    • Derivative implementation of the Swish function formula

    s′(x)=βs(x)+σ(βx)(1−βs(x))=σ(βx)∗(1+βx(1−σ(βx))) s^\prime(x)= \beta s(x)+\sigma(\beta x)(1-\beta s(x))= \sigma(\beta x)*(1+\beta x(1-\sigma(\beta x)))

    gradInput=gradOutput∗s′(x)gradInput = gradOutput * s^\prime(x)

    σ(x)=11+e−x\sigma(x) = {\frac{1} {1+{e}^{-x}}}

    Here σ(x)\sigma(x) is the Sigmoid function, s(x)s(x) is the Swish function, and s′(x)s^\prime(x) is the derivative of the Swish function.

Parameters

Parameter name Input/Output/Attribute Description Data type Data format
grad Input Denotes the gradient of the forward output of the Swish activation function, gradOutput in the formula. BFLOAT16, FLOAT16, FLOAT ND
x Input Denotes the tensor used to compute the activation function, x in the formula. BFLOAT16, FLOAT16, FLOAT ND
y Input Denotes the forward output result of the Swish activation function, σ(x) in the formula. BFLOAT16, FLOAT16, FLOAT ND
grad_x Output Denotes the gradient value of the Swish forward input, that is, the result after differentiating the input. BFLOAT16, FLOAT16, FLOAT ND
scale Optional attribute Denotes an adjustable parameter, a scalar used to control the shape and slope of the Swish function, β in the formula. FLOAT32 -

Constraints

None

Invocation Description

Invocation method Sample code Description
aclnn interface test_aclnn_swish_grad Calls the SwishGrad operator through the aclnnSwishBackward interface.
Graph mode - Calls the SwishGrad operator by building a graph through operator IR.