FilesLast commitLast update
16 days ago
1 month ago
18 days ago
1 month ago
1 month ago
1 month ago
10 days ago
9 days ago
README

NormRopeConcat

English | 简体中文

Product Support

Product Supported
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Function Description

  • Operator function: In the attention mechanism of a (multimodal) transformer, performs normalization (Norm), rotary position encoding (Rope), and feature concatenation (Concat) on query, key, and Value:

    • Normalization (Norm) currently supports LayerNorm, AFFINE LayerNorm, RmsNorm, and AFFINE RmsNorm types.
    • Rotary position encoding (Rope) supports the Interleave and Half types.
    • Feature concatenation (Concat) supports concatenation along the sequence dimension, and the concatenation order matters.
  • Computation formula (taking Query (video) and EncoderQuery (text) as an example):

    $$
    

    hiddenState_q = \text{Norm}(query, normQueryWeight, normQueryBias, eps) \ hiddenState_{eq} = \text{Norm}(encoderQuery, normEncoderQueryWeight, normEncoderQueryBias, eps) \ concatenatedHiddenState = \text{Concat}(hiddenState_q, hiddenState_{eq}) \ transposedHiddenState = \text{Transpose}(concatenatedHiddenState, (0, 2, 1, 3)) \ hiddenState = \text{RoPE}(concatenatedHiddenState, ropeSin, ropeCos)

    1. The input and output layouts are as follows: the shape of the input query is (B, S, N, D), and the shape of the output hiddenState is (B, N, S, D), where B is batch, S is sequenceLen, N is headNum, and D is headDim.

    2. Norm has five modes (normType): NONE(0), LAYER_NORM(1), LAYER_NORM_AFFINE(2), RMS_NORM(3), RMS_NORM_AFFINE(4), where: When normType = NONE:

      hiddenStateq=queryhiddenState_q = query

      When normType = LAYER_NORM

      queryMeanb,s,n=1D∑i=0D−1queryb,s,nqueryVarb,s,n=1D∑i=0D−1(query−queryMeanb,s,n)2queryRstdb,s,n=1queryVarb,s,n+ϵhiddenStateq=(query−queryMean)∗queryRstdqueryMean_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D-1}query_{b,s,n} \\ queryVar_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D-1}(query-queryMean_{b,s,n})^2 \\ queryRstd_{b,s,n}= \frac{1}{\sqrt{queryVar_{b,s,n}+\epsilon}} \\ hiddenState_q = (query-queryMean)*queryRstd

      When normType = LAYER_NORM_AFFINE, based on the preceding formulas:

      hiddenStateq=normQueryWeight∗hiddenStateq+normQueryBiashiddenState_q = normQueryWeight*hiddenState_q + normQueryBias

      When normType = RMS_NORM:

      queryMs=1D∑i=0D−1(queryb,s,n)2queryRms=1queryMs+ϵhiddenStateq=query∗queryRmsqueryMs = \frac{1}{D}\sum_{i=0}^{D-1}(query_{b,s,n})^2 \\ queryRms = \frac{1}{\sqrt{queryMs+\epsilon}} \\ hiddenState_q = query * queryRms

      When normType = RMS_NORM_AFFINE, based on the preceding formulas:

      hiddenStateq=normQueryWeight∗hiddenStateqhiddenState_q = normQueryWeight*hiddenState_q

    3. Concat means concatenation along the sequence dimension, and the concatenation order matters (concatOrder). When concatOrder=0, hiddenStateqhiddenState_q comes before hiddenStateeqhiddenState_{eq}; when concatOrder=1, hiddenStateqhiddenState_q comes after hiddenStateeqhiddenState_{eq}.

    4. RoPE has three modes (ropeType): NONE(0), INTERLEAVE(1), HALF(2). When ropeType=NONE, the output is returned directly without transformation. For other cases, refer to the following:

        def image_rotary_emb(hidden_states, rope_sin, rope_cos, mode=1):
            out = torch.empty_like(hidden_states)
            if mode == 1: # interleave
                x = hidden_states.view(*hidden_states.shape[:-1], -1, 2)
                x1, x2 = x[..., 0], x[..., 1]
                rotated_x = torch.stack([-x2, x1],dim=-1).flatten(3)
                out = hidden_states.float() * rope_cos + rotated_x.float()*rope_sin
                return out.type_as(hidden_states)
            else: # half
                x1, x2 = hidden_states.reshape(*hidden_states.shape[:-1], 2, -1).unbind(-2)
                rotated_x = torch.cat([-x2, x1],dim=-1)
                out = hidden_states.float() * rope_cos + rotated_x.float()*rope_sin
                return out.type_as(hidden_states)
      
    5. The shape of the RoPE input ropeSin is (seqRope, D), where

    seqRope<=min(seqQuery+seqEncoderQuery,seqKey+seqEncoderKey)seqRope <= min(seqQuery+seqEncoderQuery, seqKey+seqEncoderKey)

    1. In the training scenario, queryMean, queryRstd, encoderQueryMean, encoderQueryRstd are output for subsequent backward computation.

Parameter Description

Parameter Name Input/Output/Attribute Description Data Type Data Format
query Input Represents the Query in the attention mechanism FLOAT16, BFLOAT16, FLOAT ND
key Input Represents the Key in the attention mechanism FLOAT16, BFLOAT16, FLOAT ND
value Input Represents the Value in the attention mechanism FLOAT16, BFLOAT16, FLOAT ND
encoderQuery Input Represents the Query in the attention mechanism, from EncoderHiddenState, and can be a null pointer FLOAT16, BFLOAT16, FLOAT ND
encoderKey Input Represents the Key in the attention mechanism, from EncoderHiddenState, and can be a null pointer FLOAT16, BFLOAT16, FLOAT ND
encoderValue Input Represents the Value in the attention mechanism, from EncoderHiddenState, and can be a null pointer FLOAT16, BFLOAT16, FLOAT ND
normQueryWeight Input Represents the affine transformation parameters of LayerNorm, applied to Query FLOAT16, BFLOAT16, FLOAT ND
normQueryBias Input Represents the affine transformation parameters of LayerNorm, applied to Query FLOAT16, BFLOAT16, FLOAT ND
normKeyWeight Input Represents the affine transformation parameters of LayerNorm, applied to Key FLOAT16, BFLOAT16, FLOAT ND
normKeyBias Input Represents the affine transformation parameters of LayerNorm, applied to Key FLOAT16, BFLOAT16, FLOAT ND
normAddedQueryWeight Input Represents the affine transformation parameters of LayerNorm, applied to encoderQuery FLOAT16, BFLOAT16, FLOAT ND
normAddedQueryBias Input Represents the affine transformation parameters of LayerNorm, applied to encoderQuery FLOAT16, BFLOAT16, FLOAT ND
normAddedKeyWeight Input Represents the affine transformation parameters of LayerNorm, applied to encoderKey FLOAT16, BFLOAT16, FLOAT ND
normAddedKeyBias Input Represents the affine transformation parameters of LayerNorm, applied to encoderKey FLOAT16, BFLOAT16, FLOAT ND
ropeSin Input Represents the sine encoding of RoPE FLOAT16, BFLOAT16, FLOAT ND
ropeCos Input Represents the cosine encoding of RoPE FLOAT16, BFLOAT16, FLOAT ND
normType Attribute Represents the normalization type applied to q and k. 0: no normalization, 1: LayerNorm, 2: LayerNormAffine, 3: RmsNorm, 4: RmsNormAffine int64 ND
normAddedType Attribute Represents the normalization type applied to encoderQuery and encoderKey. 0: no normalization, 1: LayerNorm, 2: LayerNormAffine, 3: RmsNorm, 4: RmsNormAffine int64 ND
ropeType Attribute Represents the RoPE mode, of the int64 type. 0: no RoPE, 1: Interleave, 2: Half int64 ND
concatOrder Attribute Represents the concatenation order, of the int64 type. 0: query first, 1: query last int64 ND
eps Attribute Represents the epsilon value in normalization float32 ND
isTraining Attribute Indicates whether it is the training phase, and determines whether the values used for backward computation are output bool ND
queryOutput Output Represents the query output in the attention mechanism FLOAT16, BFLOAT16, FLOAT ND
keyOutput Output Represents the key output in the attention mechanism FLOAT16, BFLOAT16, FLOAT ND
valueOutput Output Represents the value output in the attention mechanism FLOAT16, BFLOAT16, FLOAT ND
normQueryMean Output Query mean output in LayerNorm, used for backward computation FLOAT ND
normQueryRstd Output Query standard deviation output in LayerNorm, used for backward computation FLOAT ND
normKeyMean Output Key mean output in LayerNorm, used for backward computation FLOAT ND
normKeyRstd Output Key standard deviation output in LayerNorm, used for backward computation FLOAT ND
normEncoderQueryMean Output encoderQuery mean output in LayerNorm, used for backward computation FLOAT ND
normEncoderQueryRstd Output encoderQuery standard deviation output in LayerNorm, used for backward computation FLOAT ND
normEncoderKeyMean Output encoderKey mean output in LayerNorm, used for backward computation FLOAT ND
normEncoderKeyRstd Output encoderKey standard deviation output in LayerNorm, used for backward computation FLOAT ND

Constraints

  • Deterministic computation:
    • aclnnNormRopeConcat uses a deterministic implementation by default.
  • The data types of query, key, value, encoderQuery, encoderKey, and encoderValue must be the same.
  • The headDim length is in the range [1~1024] and must be an even number.
  • The seqRope length is in the range [1~Min(seqQuery+seqEncoderQuery, seqKey+seqEncoderKey)].

Calling Instructions

Calling Method Calling Sample Description
aclnn call test_aclnn_norm_rope_concat.cpp Calls the NormRopeConcat operator through the aclnnNormRopeConcat API.