FilesLast commitLast update
1 month ago
10 days ago
10 days ago
1 month ago
10 days ago
1 month ago
1 month ago
10 days ago
README

RopeQuantKvcache

English | 简体中文

Product Support Status

Product Supported
Ascend 950PR&950DT products √
Atlas A3 products ×
Atlas A2 products √
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×
Kirin X90 processor products √
Kirin 9030 processor products √

Description

  • Operator function: Splits the trailing axis of the input tensor into q, k, and v, and applies rotary positional encoding to q and k. The current kernel writes out q and updates the quantized k and v to kCacheRef and vCacheRef according to indice. The reserved k and v outputs are currently not written.

Parameter Description

Parameter Name Input/Output/Attribute Description Data Type Data Format
qkv Input aclTensor on the device, the tensor to be split. FLOAT16 ND
cos Input aclTensor on the device, the tensor used for rotary positional encoding. FLOAT16 ND
sin Input aclTensor on the device, the tensor used for rotary positional encoding. FLOAT16 ND
quant_scale Input aclTensor on the device, the tensor of quantization scale parameters. FLOAT32 ND
quant_offset Input aclTensor on the device, the tensor of quantization offsets. INT32 ND
k_cache Input Input used for in-place update. INT8 ND
v_cache Input Input used for in-place update. INT8 ND
indice Input Index used to update the quantized result INT32 ND
size_splits Attribute Used to split qkv. INT64 -
layout Attribute Data layout mode of qkv. String -
kv_output Attribute Controls whether to output the original k and v. BOOL -
q Output Result of the split q after rotary positional encoding. FLOAT16 ND
k Output Result of the split k after rotary positional encoding. FLOAT16 ND
v Output The split v. FLOAT16 ND
k_cache Output Result of the split k after rotary positional encoding and quantization. INT8 ND
v_cache Output Result of the split v after quantization. INT8 ND

Constraints

  • qkv is 3-dimensional [B, S, H], where H=(Nq+Nkv+Nkv)*D. cos and sin are 4-dimensional [B, S, 1, D], whose B, S, and D must be identical to those of the split k. Broadcast is not supported.
  • Both quant_scale and quant_offset are 1-dimensional [D]. kCache and vCache are both 4-dimensional [B, S_cache, Nkv, D] with identical shapes, and S_cache must be able to hold the update positions. The element count of indice is B.
  • size_splits contains 3 items [Nq*D, Nkv*D, Nkv*D], and the sum of the three items must equal the trailing axis H of qkv. Neither layout nor kv_output is currently consumed by the implementation.
  • Dynamic shape (-1) and dynamic rank (-2) are not supported.

Calling Description

Calling Method Calling Sample Description
Graph mode invocation test_geir_rope_quant_kvcache Invokes the operator in graph mode.