FilesLast commitLast update
10 days ago
2 months ago
10 months ago
1 month ago
10 months ago
1 month ago
10 months ago
15 days ago
15 days ago
README

TransformBiasRescaleQkv

English | 简体中文

Product Support Status

Product Supported
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×
Kirin X90 processor series products √
Kirin 9030 processor series products √

Function Description

  • Operator function: The TransformBiasRescaleQkv operator is an API for processing the query (Query), key (Key), and value (Value) vectors in the multi-head attention mechanism. It adjusts the bias (Bias) and rescale (Rescale) factors of these vectors to optimize the attention computation process.

  • Computation formula: The element-wise computation process is as follows:

    qo=(qi+qbias)/dim_per_head q_o=(q_i+q_{bias})/\sqrt{dim\_per\_head}\\

    ko=ki+kbias k_o=k_i+k_{bias}\\

    vo=vi+vbias v_o=v_i+v_{bias}

    In the formula:

    • dim_per_head is the dimension of each attention head.
    • qo, ko, and vo are the output elements of the query (Query), key (Key), and value (Value) vectors respectively.
    • qi, ki, and vi are the input elements of the query (Query), key (Key), and value (Value) vectors respectively.
    • qbias, kbias, and vbias are the offsets of the input elements of the query (Query), key (Key), and value (Value) vectors respectively.

Parameter Description

Parameter Name Input/Output/Attribute Description Data Type Data Format
qkv Input Inputs qi, ki, and vi in the formula. BFLOAT16, FLOAT32, FLOAT16 ND
qkvBias Input Inputs qbias, kbias, and vbias in the formula. BFLOAT16, FLOAT32, FLOAT16 ND
numHeads Attribute
  • Number of input heads.
  • The value must be greater than 0.
INT64 -
qOut Output qo in the formula. BFLOAT16, FLOAT32, FLOAT16 ND
kOut Output ko in the formula. BFLOAT16, FLOAT32, FLOAT16 ND
vOut Output vo in the formula. BFLOAT16, FLOAT32, FLOAT16 ND
  • Kirin X90/Kirin 9030 processor series products: BFLOAT16 is not supported.

Constraint Description

  • The data types of the inputs qkv and qkvBias and the outputs qOut, kOut, and vOut must be the same.
  • If the input value is NaN, the output is also NaN. If the input is Inf, the output is also Inf.
  • If the input is -Inf, the output is also -Inf.

Invocation Description

Invocation Method Sample Code Description
aclnn invocation test_aclnn_transform_bias_rescale_qkv Invokes the TransformBiasRescaleQkv operator through the aclnnTransformBiasRescaleQkv API.
Graph-mode invocation - Invokes the TransformBiasRescaleQkv operator by building a graph through the operator IR.