| Files | Last commit | Last update |
|---|---|---|
| 16 days ago | ||
| 1 month ago | ||
| 18 days ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 10 days ago | ||
| 9 days ago |
NormRopeConcat
English | 简体中文
Product Support
| Product | Supported |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
Function Description
-
Operator function: In the attention mechanism of a (multimodal) transformer, performs normalization (Norm), rotary position encoding (Rope), and feature concatenation (Concat) on query, key, and Value:
- Normalization (Norm) currently supports LayerNorm, AFFINE LayerNorm, RmsNorm, and AFFINE RmsNorm types.
- Rotary position encoding (Rope) supports the Interleave and Half types.
- Feature concatenation (Concat) supports concatenation along the sequence dimension, and the concatenation order matters.
-
Computation formula (taking Query (video) and EncoderQuery (text) as an example):
$$hiddenState_q = \text{Norm}(query, normQueryWeight, normQueryBias, eps) \ hiddenState_{eq} = \text{Norm}(encoderQuery, normEncoderQueryWeight, normEncoderQueryBias, eps) \ concatenatedHiddenState = \text{Concat}(hiddenState_q, hiddenState_{eq}) \ transposedHiddenState = \text{Transpose}(concatenatedHiddenState, (0, 2, 1, 3)) \ hiddenState = \text{RoPE}(concatenatedHiddenState, ropeSin, ropeCos)
-
The input and output layouts are as follows: the shape of the input
queryis(B, S, N, D), and the shape of the outputhiddenStateis(B, N, S, D), where B is batch, S is sequenceLen, N is headNum, and D is headDim. -
Norm has five modes (
normType):NONE(0), LAYER_NORM(1), LAYER_NORM_AFFINE(2), RMS_NORM(3), RMS_NORM_AFFINE(4), where: WhennormType = NONE:hiddenStateq=queryhiddenState_q = query
When
normType = LAYER_NORMqueryMeanb,s,n=1D∑i=0D−1queryb,s,nqueryVarb,s,n=1D∑i=0D−1(query−queryMeanb,s,n)2queryRstdb,s,n=1queryVarb,s,n+ϵhiddenStateq=(query−queryMean)∗queryRstdqueryMean_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D-1}query_{b,s,n} \\ queryVar_{b,s,n} = \frac{1}{D}\sum_{i=0}^{D-1}(query-queryMean_{b,s,n})^2 \\ queryRstd_{b,s,n}= \frac{1}{\sqrt{queryVar_{b,s,n}+\epsilon}} \\ hiddenState_q = (query-queryMean)*queryRstd
When
normType = LAYER_NORM_AFFINE, based on the preceding formulas:hiddenStateq=normQueryWeight∗hiddenStateq+normQueryBiashiddenState_q = normQueryWeight*hiddenState_q + normQueryBias
When
normType = RMS_NORM:queryMs=1D∑i=0D−1(queryb,s,n)2queryRms=1queryMs+ϵhiddenStateq=query∗queryRmsqueryMs = \frac{1}{D}\sum_{i=0}^{D-1}(query_{b,s,n})^2 \\ queryRms = \frac{1}{\sqrt{queryMs+\epsilon}} \\ hiddenState_q = query * queryRms
When
normType = RMS_NORM_AFFINE, based on the preceding formulas:hiddenStateq=normQueryWeight∗hiddenStateqhiddenState_q = normQueryWeight*hiddenState_q
-
Concat means concatenation along the sequence dimension, and the concatenation order matters (
concatOrder). WhenconcatOrder=0, hiddenStateqhiddenState_q comes before hiddenStateeqhiddenState_{eq}; whenconcatOrder=1, hiddenStateqhiddenState_q comes after hiddenStateeqhiddenState_{eq}. -
RoPE has three modes (
ropeType):NONE(0), INTERLEAVE(1), HALF(2). WhenropeType=NONE, the output is returned directly without transformation. For other cases, refer to the following:def image_rotary_emb(hidden_states, rope_sin, rope_cos, mode=1): out = torch.empty_like(hidden_states) if mode == 1: # interleave x = hidden_states.view(*hidden_states.shape[:-1], -1, 2) x1, x2 = x[..., 0], x[..., 1] rotated_x = torch.stack([-x2, x1],dim=-1).flatten(3) out = hidden_states.float() * rope_cos + rotated_x.float()*rope_sin return out.type_as(hidden_states) else: # half x1, x2 = hidden_states.reshape(*hidden_states.shape[:-1], 2, -1).unbind(-2) rotated_x = torch.cat([-x2, x1],dim=-1) out = hidden_states.float() * rope_cos + rotated_x.float()*rope_sin return out.type_as(hidden_states) -
The shape of the RoPE input
ropeSinis(seqRope, D), where
seqRope<=min(seqQuery+seqEncoderQuery,seqKey+seqEncoderKey)seqRope <= min(seqQuery+seqEncoderQuery, seqKey+seqEncoderKey)
- In the training scenario,
queryMean, queryRstd, encoderQueryMean, encoderQueryRstdare output for subsequent backward computation.
-
Parameter Description
| Parameter Name | Input/Output/Attribute | Description | Data Type | Data Format |
|---|---|---|---|---|
| query | Input | Represents the Query in the attention mechanism | FLOAT16, BFLOAT16, FLOAT | ND |
| key | Input | Represents the Key in the attention mechanism | FLOAT16, BFLOAT16, FLOAT | ND |
| value | Input | Represents the Value in the attention mechanism | FLOAT16, BFLOAT16, FLOAT | ND |
| encoderQuery | Input | Represents the Query in the attention mechanism, from EncoderHiddenState, and can be a null pointer | FLOAT16, BFLOAT16, FLOAT | ND |
| encoderKey | Input | Represents the Key in the attention mechanism, from EncoderHiddenState, and can be a null pointer | FLOAT16, BFLOAT16, FLOAT | ND |
| encoderValue | Input | Represents the Value in the attention mechanism, from EncoderHiddenState, and can be a null pointer | FLOAT16, BFLOAT16, FLOAT | ND |
| normQueryWeight | Input | Represents the affine transformation parameters of LayerNorm, applied to Query | FLOAT16, BFLOAT16, FLOAT | ND |
| normQueryBias | Input | Represents the affine transformation parameters of LayerNorm, applied to Query | FLOAT16, BFLOAT16, FLOAT | ND |
| normKeyWeight | Input | Represents the affine transformation parameters of LayerNorm, applied to Key | FLOAT16, BFLOAT16, FLOAT | ND |
| normKeyBias | Input | Represents the affine transformation parameters of LayerNorm, applied to Key | FLOAT16, BFLOAT16, FLOAT | ND |
| normAddedQueryWeight | Input | Represents the affine transformation parameters of LayerNorm, applied to encoderQuery | FLOAT16, BFLOAT16, FLOAT | ND |
| normAddedQueryBias | Input | Represents the affine transformation parameters of LayerNorm, applied to encoderQuery | FLOAT16, BFLOAT16, FLOAT | ND |
| normAddedKeyWeight | Input | Represents the affine transformation parameters of LayerNorm, applied to encoderKey | FLOAT16, BFLOAT16, FLOAT | ND |
| normAddedKeyBias | Input | Represents the affine transformation parameters of LayerNorm, applied to encoderKey | FLOAT16, BFLOAT16, FLOAT | ND |
| ropeSin | Input | Represents the sine encoding of RoPE | FLOAT16, BFLOAT16, FLOAT | ND |
| ropeCos | Input | Represents the cosine encoding of RoPE | FLOAT16, BFLOAT16, FLOAT | ND |
| normType | Attribute | Represents the normalization type applied to q and k. 0: no normalization, 1: LayerNorm, 2: LayerNormAffine, 3: RmsNorm, 4: RmsNormAffine | int64 | ND |
| normAddedType | Attribute | Represents the normalization type applied to encoderQuery and encoderKey. 0: no normalization, 1: LayerNorm, 2: LayerNormAffine, 3: RmsNorm, 4: RmsNormAffine | int64 | ND |
| ropeType | Attribute | Represents the RoPE mode, of the int64 type. 0: no RoPE, 1: Interleave, 2: Half | int64 | ND |
| concatOrder | Attribute | Represents the concatenation order, of the int64 type. 0: query first, 1: query last | int64 | ND |
| eps | Attribute | Represents the epsilon value in normalization | float32 | ND |
| isTraining | Attribute | Indicates whether it is the training phase, and determines whether the values used for backward computation are output | bool | ND |
| queryOutput | Output | Represents the query output in the attention mechanism | FLOAT16, BFLOAT16, FLOAT | ND |
| keyOutput | Output | Represents the key output in the attention mechanism | FLOAT16, BFLOAT16, FLOAT | ND |
| valueOutput | Output | Represents the value output in the attention mechanism | FLOAT16, BFLOAT16, FLOAT | ND |
| normQueryMean | Output | Query mean output in LayerNorm, used for backward computation | FLOAT | ND |
| normQueryRstd | Output | Query standard deviation output in LayerNorm, used for backward computation | FLOAT | ND |
| normKeyMean | Output | Key mean output in LayerNorm, used for backward computation | FLOAT | ND |
| normKeyRstd | Output | Key standard deviation output in LayerNorm, used for backward computation | FLOAT | ND |
| normEncoderQueryMean | Output | encoderQuery mean output in LayerNorm, used for backward computation | FLOAT | ND |
| normEncoderQueryRstd | Output | encoderQuery standard deviation output in LayerNorm, used for backward computation | FLOAT | ND |
| normEncoderKeyMean | Output | encoderKey mean output in LayerNorm, used for backward computation | FLOAT | ND |
| normEncoderKeyRstd | Output | encoderKey standard deviation output in LayerNorm, used for backward computation | FLOAT | ND |
Constraints
- Deterministic computation:
- aclnnNormRopeConcat uses a deterministic implementation by default.
- The data types of query, key, value, encoderQuery, encoderKey, and encoderValue must be the same.
- The headDim length is in the range [1~1024] and must be an even number.
- The seqRope length is in the range [1~Min(seqQuery+seqEncoderQuery, seqKey+seqEncoderKey)].
Calling Instructions
| Calling Method | Calling Sample | Description |
|---|---|---|
| aclnn call | test_aclnn_norm_rope_concat.cpp | Calls the NormRopeConcat operator through the aclnnNormRopeConcat API. |