| Files | Last commit | Last update |
|---|---|---|
| 1 month ago | ||
| 10 days ago | ||
| 10 days ago | ||
| 1 month ago | ||
| 10 days ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 10 days ago |
RopeQuantKvcache
English | 简体中文
Product Support Status
| Product | Supported |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | × |
| Atlas A2 products | √ |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
| Kirin X90 processor products | √ |
| Kirin 9030 processor products | √ |
Description
- Operator function: Splits the trailing axis of the input tensor into q, k, and v, and applies rotary positional encoding to q and k. The current kernel writes out q and updates the quantized k and v to kCacheRef and vCacheRef according to indice. The reserved k and v outputs are currently not written.
Parameter Description
| Parameter Name | Input/Output/Attribute | Description | Data Type | Data Format |
|---|---|---|---|---|
| qkv | Input | aclTensor on the device, the tensor to be split. | FLOAT16 | ND |
| cos | Input | aclTensor on the device, the tensor used for rotary positional encoding. | FLOAT16 | ND |
| sin | Input | aclTensor on the device, the tensor used for rotary positional encoding. | FLOAT16 | ND |
| quant_scale | Input | aclTensor on the device, the tensor of quantization scale parameters. | FLOAT32 | ND |
| quant_offset | Input | aclTensor on the device, the tensor of quantization offsets. | INT32 | ND |
| k_cache | Input | Input used for in-place update. | INT8 | ND |
| v_cache | Input | Input used for in-place update. | INT8 | ND |
| indice | Input | Index used to update the quantized result | INT32 | ND |
| size_splits | Attribute | Used to split qkv. | INT64 | - |
| layout | Attribute | Data layout mode of qkv. | String | - |
| kv_output | Attribute | Controls whether to output the original k and v. | BOOL | - |
| q | Output | Result of the split q after rotary positional encoding. | FLOAT16 | ND |
| k | Output | Result of the split k after rotary positional encoding. | FLOAT16 | ND |
| v | Output | The split v. | FLOAT16 | ND |
| k_cache | Output | Result of the split k after rotary positional encoding and quantization. | INT8 | ND |
| v_cache | Output | Result of the split v after quantization. | INT8 | ND |
Constraints
qkvis 3-dimensional[B, S, H], whereH=(Nq+Nkv+Nkv)*D.cosandsinare 4-dimensional[B, S, 1, D], whose B, S, and D must be identical to those of the split k. Broadcast is not supported.- Both
quant_scaleandquant_offsetare 1-dimensional[D]. kCache and vCache are both 4-dimensional[B, S_cache, Nkv, D]with identical shapes, andS_cachemust be able to hold the update positions. The element count ofindiceis B. size_splitscontains 3 items[Nq*D, Nkv*D, Nkv*D], and the sum of the three items must equal the trailing axis H ofqkv. Neitherlayoutnorkv_outputis currently consumed by the implementation.- Dynamic shape (-1) and dynamic rank (-2) are not supported.
Calling Description
| Calling Method | Calling Sample | Description |
|---|---|---|
| Graph mode invocation | test_geir_rope_quant_kvcache | Invokes the operator in graph mode. |