| Files | Last commit | Last update |
|---|---|---|
| 16 days ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 13 days ago | ||
| 10 days ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 9 days ago |
RopeWithSinCosCache
English | 简体中文
Product Support Status
| Product | Supported |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | √ |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
| Kirin X90 processor products | √ |
| Kirin 9030 processor products | √ |
Description
-
Operator function: To improve performance, the inference network passes the sin and cos inputs through a cache, and the operator performs the rotary positional encoding computation.
-
Computation formulas:
- mrope mode: The shape of positions is [m, numTokens], where m is the element count of mropeSection, and 3 or 4 is supported:
cosSin[i]=cosSinCache[positions[i]]cosSin[i] = cosSinCache[positions[i]]
cos,sin=cosSin.chunk(2,dim=−1)cos, sin = cosSin.chunk(2, dim=-1)
(1) When cacheMode is 0:
-
The element count of mropeSection is 3:
cos0=cos[0,:,:mropeSection[0]]cos0 = cos[0, :, :mropeSection[0]]
cos1=cos[1,:,mropeSection[0]:(mropeSection[0]+mropeSection[1])]cos1 = cos[1, :, mropeSection[0]:(mropeSection[0] + mropeSection[1])]
cos2=cos[2,:,(mropeSection[0]+mropeSection[1]):(mropeSection[0]+mropeSection[1]+mropeSection[2])]cos2 = cos[2, :, (mropeSection[0] + mropeSection[1]):(mropeSection[0] + mropeSection[1] + mropeSection[2])]
cos=torch.cat((cos0,cos1,cos2),dim=−1)cos = torch.cat((cos0, cos1, cos2), dim=-1)
sin0=sin[0,:,:mropeSection[0]]sin0 = sin[0, :, :mropeSection[0]]
sin1=sin[1,:,mropeSection[0]:(mropeSection[0]+mropeSection[1])]sin1 = sin[1, :, mropeSection[0]:(mropeSection[0] + mropeSection[1])]
sin2=sin[2,:,(mropeSection[0]+mropeSection[1]):(mropeSection[0]+mropeSection[1]+mropeSection[2])]sin2 = sin[2, :, (mropeSection[0] + mropeSection[1]):(mropeSection[0] + mropeSection[1] + mropeSection[2])]
sin=torch.cat((sin0,sin1,sin2),dim=−1)sin= torch.cat((sin0, sin1, sin2), dim=-1)
queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]
queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]
-
The element count of mropeSection is 4:
cos=torch.cat([m[i] for i,m in enumerate(cos.split(mropeSection,dim=−1))],dim=−1)cos = torch.cat([m[i]\ for\ i, m\ in\ enumerate(cos.split(mropeSection, dim=-1))], dim=-1)
sin=torch.cat([m[i] for i,m in enumerate(sin.split(mropeSection,dim=−1))],dim=−1)sin = torch.cat([m[i]\ for\ i, m\ in\ enumerate(sin.split(mropeSection, dim=-1))], dim=-1)
queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]
queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]
(2) When cacheMode is 1:
cosTmp=coscosTmp = cos
cos[...,1:mropeSection[1]∗3:3]=cosTmp[1,...,1:mropeSection[1]∗3:3]cos [..., 1:mropeSection[1] * 3:3] = cosTmp[1, ..., 1:mropeSection[1] * 3:3]
cos[...,2:mropeSection[1]∗3:3]=cosTmp[2,...,2:mropeSection[1]∗3:3]cos[..., 2:mropeSection[1] * 3:3] = cosTmp[2, ..., 2:mropeSection[1] * 3:3]
sinTmp=sinsinTmp = sin
sin[...,1:mropeSection[1]∗3:3]=sinTmp[1,...,1:mropeSection[1]∗3:3]sin[..., 1:mropeSection[1] * 3:3] = sinTmp [1, ..., 1:mropeSection[1] * 3:3]
sin[...,2:mropeSection[1]∗3:3]=sinTmp[2,...,2:mropeSection[1]∗3:3]sin[..., 2:mropeSection[1] * 3:3] = sinTmp [2, ..., 2:mropeSection[1] * 3:3]
queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]
queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]
(1) rotate_half (GPT-NeoX style) computation mode:
x1,x2=torch.chunk(queryRot,2,dim=−1)x1, x2 = torch.chunk(queryRot, 2, dim=-1)
o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]
o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]
queryRot=torch.cat((o1,o2),dim=−1)queryRot = torch.cat((o1, o2), dim=-1)
query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)
(2) rotate_interleaved (GPT-J style) computation mode:
x1=queryRot[...,::2]x1 = queryRot[..., ::2]
x2=queryRot[...,1::2]x2 = queryRot[..., 1::2]
o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]
o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]
queryRot=torch.stack((o1,o2),dim=−1)queryRot = torch.stack((o1, o2), dim=-1)
query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)
- rope mode: The shape of positions is [numTokens]:
cosSin[i]=cosSinCache[positions[i]]cosSin[i] = cosSinCache[positions[i]]
cos,sin=cosSin.chunk(2,dim=−1)cos, sin = cosSin.chunk(2, dim=-1)
queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]
queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]
(1) rotate_half (GPT-NeoX style) computation mode:
x1,x2=torch.chunk(queryRot,2,dim=−1)x1, x2 = torch.chunk(queryRot, 2, dim=-1)
o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]
o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]
queryRot=torch.cat((o1,o2),dim=−1)queryRot = torch.cat((o1, o2), dim=-1)
query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)
(2) rotate_interleaved (GPT-J style) computation mode:
x1=queryRot[...,::2]x1 = queryRot[..., ::2]
x2=queryRot[...,1::2]x2 = queryRot[..., 1::2]
o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]
o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]
queryRot=torch.stack((o1,o2),dim=−1)queryRot = torch.stack((o1, o2), dim=-1)
query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)
Parameter Description
| Parameter Name | Input/Output/Attribute | Description | Data Type | Data Format |
|---|---|---|---|---|
| positions | Input | aclTensor on the device, the input index. | INT64 | ND |
| queryIn | Input | aclTensor on the device, the first tensor on which rotary positional encoding is performed, `query` in the formulas. | BFLOAT16, FLOAT16, FLOAT32 | ND |
| keyIn | Input | aclTensor on the device, the second tensor on which rotary positional encoding is performed. | BFLOAT16, FLOAT16, FLOAT32 | ND |
| cosSinCache | Input | aclTensor on the device, the positional encoding tensor that participates in the computation. | BFLOAT16, FLOAT16, FLOAT32 | ND |
| mropeSection | Attribute | Under the mrope mode, used to integrate the positional encoding tensor information of the inputs, `mropeSection` in the formulas. Passing nullptr indicates the rope mode. | aclIntArray | - |
| headSize | Attribute | Dimension size of each attention head. | INT64 | - |
| isNeoxStyle | Attribute | true indicates the rotate\_half (GPT-NeoX style) computation mode, and false indicates the rotate\_interleaved (GPT-J style) computation mode. | BOOL | - |
| cacheMode | Attribute | Concatenation mode of cos and sin. 0 indicates the segmented mode, and 1 indicates the interleaved mode. Provided only by the V2 API. | INT64 | - |
| queryOut | Output | Output result of query after rotary positional encoding. | FLOAT, FLOAT16, BFLOAT16 | ND |
| keyOut | Output | Output result of key after rotary positional encoding. | FLOAT, FLOAT16, BFLOAT16 | ND |
- Kirin X90 processor products and Kirin 9030 processor products: BFLOAT16 is not supported.
Constraints
- queryIn, keyIn, and cosSinCache support only 2-dimensional shape inputs.
- The size of dimension 1 of queryIn and keyIn must be a multiple of headSize.
- The data types of the queryIn, keyIn, and cosSinCache inputs must be identical.
- headSize: when the data type is BFLOAT16 or FLOAT16, it is a multiple of 32; when the data type is FLOAT32, it is a multiple of 16.
- rotaryDim: always less than or equal to headSize; when the data type is BFLOAT16 or FLOAT16, it is a multiple of 32; when the data type is FLOAT32, it is a multiple of 16. In the mrope mode, the sum of all elements of mropeSection should equal half of rotaryDim.
- The values of the input tensor positions should be less than the dimension-0 maxSeqLen of cosSinCache.
- aclnnRopeWithSinCosCache is deterministic by default.
- When invoked through the aclnn APIs, mropeSection and cacheMode are subject to the following constraints:
- The aclnnRopeWithSinCosCache API supports only mropeSection values of [16, 24, 24], [24, 20, 20], [8, 12, 12], and [16, 16, 16, 16], and cacheMode is fixed to 0. Atlas A5 products support only [16, 24, 24] and [24, 20, 20].
- The aclnnRopeWithSinCosCacheV2 API supports mropeSection values of [11, 11, 10], [16, 24, 24], [24, 20, 20], [8, 12, 12], and [16, 16, 16, 16], and cacheMode supports only 0 and 1. When mropeSection is [16, 16, 16, 16], cacheMode supports only 0. This API does not support Atlas A5 products.
Calling Description
| Calling Method | Sample Code | Description |
|---|---|---|
| aclnn API | test_aclnn_rope_with_sin_cos_cache | Invokes the RopeWithSinCosCache operator through the aclnnRopeWithSinCosCache API. |