FilesLast commitLast update
16 days ago
1 month ago
1 month ago
13 days ago
10 days ago
1 month ago
1 month ago
1 month ago
9 days ago
README

RopeWithSinCosCache

English | 简体中文

Product Support Status

Product Supported
Ascend 950PR&950DT products √
Atlas A3 products √
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×
Kirin X90 processor products √
Kirin 9030 processor products √

Description

  • Operator function: To improve performance, the inference network passes the sin and cos inputs through a cache, and the operator performs the rotary positional encoding computation.

  • Computation formulas:

    1. mrope mode: The shape of positions is [m, numTokens], where m is the element count of mropeSection, and 3 or 4 is supported:

    cosSin[i]=cosSinCache[positions[i]]cosSin[i] = cosSinCache[positions[i]]

    cos,sin=cosSin.chunk(2,dim=−1)cos, sin = cosSin.chunk(2, dim=-1)

    (1) When cacheMode is 0:

    • The element count of mropeSection is 3:

      cos0=cos[0,:,:mropeSection[0]]cos0 = cos[0, :, :mropeSection[0]]

      cos1=cos[1,:,mropeSection[0]:(mropeSection[0]+mropeSection[1])]cos1 = cos[1, :, mropeSection[0]:(mropeSection[0] + mropeSection[1])]

      cos2=cos[2,:,(mropeSection[0]+mropeSection[1]):(mropeSection[0]+mropeSection[1]+mropeSection[2])]cos2 = cos[2, :, (mropeSection[0] + mropeSection[1]):(mropeSection[0] + mropeSection[1] + mropeSection[2])]

      cos=torch.cat((cos0,cos1,cos2),dim=−1)cos = torch.cat((cos0, cos1, cos2), dim=-1)

      sin0=sin[0,:,:mropeSection[0]]sin0 = sin[0, :, :mropeSection[0]]

      sin1=sin[1,:,mropeSection[0]:(mropeSection[0]+mropeSection[1])]sin1 = sin[1, :, mropeSection[0]:(mropeSection[0] + mropeSection[1])]

      sin2=sin[2,:,(mropeSection[0]+mropeSection[1]):(mropeSection[0]+mropeSection[1]+mropeSection[2])]sin2 = sin[2, :, (mropeSection[0] + mropeSection[1]):(mropeSection[0] + mropeSection[1] + mropeSection[2])]

      sin=torch.cat((sin0,sin1,sin2),dim=−1)sin= torch.cat((sin0, sin1, sin2), dim=-1)

      queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]

      queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]

    • The element count of mropeSection is 4:

      cos=torch.cat([m[i] for i,m in enumerate(cos.split(mropeSection,dim=−1))],dim=−1)cos = torch.cat([m[i]\ for\ i, m\ in\ enumerate(cos.split(mropeSection, dim=-1))], dim=-1)

      sin=torch.cat([m[i] for i,m in enumerate(sin.split(mropeSection,dim=−1))],dim=−1)sin = torch.cat([m[i]\ for\ i, m\ in\ enumerate(sin.split(mropeSection, dim=-1))], dim=-1)

      queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]

      queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]

    (2) When cacheMode is 1:

    cosTmp=coscosTmp = cos

    cos[...,1:mropeSection[1]∗3:3]=cosTmp[1,...,1:mropeSection[1]∗3:3]cos [..., 1:mropeSection[1] * 3:3] = cosTmp[1, ..., 1:mropeSection[1] * 3:3]

    cos[...,2:mropeSection[1]∗3:3]=cosTmp[2,...,2:mropeSection[1]∗3:3]cos[..., 2:mropeSection[1] * 3:3] = cosTmp[2, ..., 2:mropeSection[1] * 3:3]

    sinTmp=sinsinTmp = sin

    sin[...,1:mropeSection[1]∗3:3]=sinTmp[1,...,1:mropeSection[1]∗3:3]sin[..., 1:mropeSection[1] * 3:3] = sinTmp [1, ..., 1:mropeSection[1] * 3:3]

    sin[...,2:mropeSection[1]∗3:3]=sinTmp[2,...,2:mropeSection[1]∗3:3]sin[..., 2:mropeSection[1] * 3:3] = sinTmp [2, ..., 2:mropeSection[1] * 3:3]

    queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]

    queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]

    (1) rotate_half (GPT-NeoX style) computation mode:

    x1,x2=torch.chunk(queryRot,2,dim=−1)x1, x2 = torch.chunk(queryRot, 2, dim=-1)

    o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]

    o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]

    queryRot=torch.cat((o1,o2),dim=−1)queryRot = torch.cat((o1, o2), dim=-1)

    query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)

    (2) rotate_interleaved (GPT-J style) computation mode:

    x1=queryRot[...,::2]x1 = queryRot[..., ::2]

    x2=queryRot[...,1::2]x2 = queryRot[..., 1::2]

    o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]

    o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]

    queryRot=torch.stack((o1,o2),dim=−1)queryRot = torch.stack((o1, o2), dim=-1)

    query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)

    1. rope mode: The shape of positions is [numTokens]:

    cosSin[i]=cosSinCache[positions[i]]cosSin[i] = cosSinCache[positions[i]]

    cos,sin=cosSin.chunk(2,dim=−1)cos, sin = cosSin.chunk(2, dim=-1)

    queryRot=query[...,:rotaryDim]queryRot = query[..., :rotaryDim]

    queryPass=query[...,rotaryDim:]queryPass = query[..., rotaryDim:]

    (1) rotate_half (GPT-NeoX style) computation mode:

    x1,x2=torch.chunk(queryRot,2,dim=−1)x1, x2 = torch.chunk(queryRot, 2, dim=-1)

    o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]

    o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]

    queryRot=torch.cat((o1,o2),dim=−1)queryRot = torch.cat((o1, o2), dim=-1)

    query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)

    (2) rotate_interleaved (GPT-J style) computation mode:

    x1=queryRot[...,::2]x1 = queryRot[..., ::2]

    x2=queryRot[...,1::2]x2 = queryRot[..., 1::2]

    o1[i]=x1[i]∗cos[i]−x2[i]∗sin[i]o1[i] = x1[i] * cos[i] - x2[i] * sin[i]

    o2[i]=x2[i]∗cos[i]+x1[i]∗sin[i]o2[i] = x2[i] * cos[i] + x1[i] * sin[i]

    queryRot=torch.stack((o1,o2),dim=−1)queryRot = torch.stack((o1, o2), dim=-1)

    query=torch.cat((queryRot,queryPass),dim=−1)query = torch.cat((queryRot, queryPass), dim=-1)

Parameter Description

Parameter Name Input/Output/Attribute Description Data Type Data Format
positions Input aclTensor on the device, the input index. INT64 ND
queryIn Input aclTensor on the device, the first tensor on which rotary positional encoding is performed, `query` in the formulas. BFLOAT16, FLOAT16, FLOAT32 ND
keyIn Input aclTensor on the device, the second tensor on which rotary positional encoding is performed. BFLOAT16, FLOAT16, FLOAT32 ND
cosSinCache Input aclTensor on the device, the positional encoding tensor that participates in the computation. BFLOAT16, FLOAT16, FLOAT32 ND
mropeSection Attribute Under the mrope mode, used to integrate the positional encoding tensor information of the inputs, `mropeSection` in the formulas. Passing nullptr indicates the rope mode. aclIntArray -
headSize Attribute Dimension size of each attention head. INT64 -
isNeoxStyle Attribute true indicates the rotate\_half (GPT-NeoX style) computation mode, and false indicates the rotate\_interleaved (GPT-J style) computation mode. BOOL -
cacheMode Attribute Concatenation mode of cos and sin. 0 indicates the segmented mode, and 1 indicates the interleaved mode. Provided only by the V2 API. INT64 -
queryOut Output Output result of query after rotary positional encoding. FLOAT, FLOAT16, BFLOAT16 ND
keyOut Output Output result of key after rotary positional encoding. FLOAT, FLOAT16, BFLOAT16 ND
  • Kirin X90 processor products and Kirin 9030 processor products: BFLOAT16 is not supported.

Constraints

  • queryIn, keyIn, and cosSinCache support only 2-dimensional shape inputs.
  • The size of dimension 1 of queryIn and keyIn must be a multiple of headSize.
  • The data types of the queryIn, keyIn, and cosSinCache inputs must be identical.
  • headSize: when the data type is BFLOAT16 or FLOAT16, it is a multiple of 32; when the data type is FLOAT32, it is a multiple of 16.
  • rotaryDim: always less than or equal to headSize; when the data type is BFLOAT16 or FLOAT16, it is a multiple of 32; when the data type is FLOAT32, it is a multiple of 16. In the mrope mode, the sum of all elements of mropeSection should equal half of rotaryDim.
  • The values of the input tensor positions should be less than the dimension-0 maxSeqLen of cosSinCache.
  • aclnnRopeWithSinCosCache is deterministic by default.
  • When invoked through the aclnn APIs, mropeSection and cacheMode are subject to the following constraints:
    • The aclnnRopeWithSinCosCache API supports only mropeSection values of [16, 24, 24], [24, 20, 20], [8, 12, 12], and [16, 16, 16, 16], and cacheMode is fixed to 0. Atlas A5 products support only [16, 24, 24] and [24, 20, 20].
    • The aclnnRopeWithSinCosCacheV2 API supports mropeSection values of [11, 11, 10], [16, 24, 24], [24, 20, 20], [8, 12, 12], and [16, 16, 16, 16], and cacheMode supports only 0 and 1. When mropeSection is [16, 16, 16, 16], cacheMode supports only 0. This API does not support Atlas A5 products.

Calling Description

Calling Method Sample Code Description
aclnn API test_aclnn_rope_with_sin_cos_cache Invokes the RopeWithSinCosCache operator through the aclnnRopeWithSinCosCache API.