| Files | Last commit | Last update |
|---|---|---|
| 18 days ago | ||
| 1 month ago | ||
| 29 days ago | ||
| 29 days ago | ||
| 29 days ago | ||
| 29 days ago | ||
| 1 month ago | ||
| 1 month ago | ||
| 26 days ago | ||
| 11 days ago |
IndexerQuantCache
English | 简体中文
Product Support
| Product | Supported |
|---|---|
| Ascend 950PR&950DT products | √ |
| Atlas A3 products | × |
| Atlas A2 products | × |
| Atlas 200I/500 A2 inference products | × |
| Atlas inference products | × |
| Atlas training products | × |
Description
-
Operator function: Compresses and updates the KV Cache in place during the Epilog phase of the Indexer attention mechanism. The operator compresses the
float16/bfloat16activation valuexinto FP8 (E4M3/E5M2), INT8 (uint8), or MX-FP4 format based on the quantization mode, and scatters the quantization results tocachebased onslot_mapping. Except in HiFloat8 mode, the corresponding scale is also written tocache_scale. Tokens with a value of -1 inslot_mappingare skipped. Four quantization modes are supported: MX-FP8, Normal, HiFloat8, and MX-FP4. -
Computation formula:
Quantization is performed on the last dimension of
x(the d dimension). In MX-FP8 and MX-FP4 modes, elements are grouped every 32 elements. In Normal mode, each row (one token) is a group and the entire row shares one scale. The quantization group is denoted as xgx_g:scaleg=max(∣xg∣)Q_MAX,qi=round(xiscaleg)scale_g = \frac{\max(|x_g|)}{Q\_MAX}, \quad q_i = \mathrm{round}\left(\frac{x_i}{scale_g}\right)
quant_mode=0(MX-FP8): The scale is stored asfloat8_e8m0, and the scale is rounded whenround_scale=true.quant_mode=1(Normal): Dynamic quantization is performed row by row. One scale is computed for each row and stored asfloat32.quant_mode=2(HiFloat8): x×x_scalex \times x\_scale is converted to hifloat8, andcache_scaleis neither computed nor written back.quant_mode=3(MX-FP4): The values are quantized to FP4 in standard MX blocks (every 32 elements) (uint8 packs 2 fp4 values per byte), and the scale is stored asfloat8_e8m0.
Parameter Description
| Parameter | Input/Output/Attribute | Description | Data Type | Data Format |
|---|---|---|---|---|
| cache | Input/Output | KV Cache vector cache of the current layer, updated in place, that is, the quantization target in the formula. In MX-FP4 mode, it is uint8, with 2 fp4 values packed per byte. | FLOAT8_E4M3FN, FLOAT8_E5M2, UINT8, FLOAT4_E2M1, FLOAT4_E1M2 | ND |
| cache_scale | Input/Output | Quantization scale tensor. In HiFloat8 mode, this input remains unchanged and is not written back. | FLOAT, FLOAT8_E8M0 | ND |
| x | Input | Activation input to be quantized, that is, x in the formula. The shape is [bs, d]. |
FLOAT16, BFLOAT16 | ND |
| slot_mapping | Input | Index mapping from token to cache slot. A value of -1 indicates that the token is skipped. The shape is [bs]. | INT32 | ND |
| cache | Output | KV Cache updated in place, which is the same tensor as the input cache. |
FLOAT8_E4M3FN, FLOAT8_E5M2, UINT8, FLOAT4_E2M1, FLOAT4_E1M2 | ND |
| cache_scale | Output | Scale written back in place in MX-FP8, Normal, and MX-FP4 modes. In HiFloat8 mode, the input value remains unchanged. | FLOAT, FLOAT8_E8M0 | ND |
| quant_mode | Optional attribute | Quantization mode: 0=MX-FP8, 1=Normal, 2=HiFloat8, 3=MX-FP4. Default value: 1. | INT | - |
| round_scale | Optional attribute | Whether to round the scale in MX-FP8 mode (quant_mode=0). Default value: true. | BOOL | - |
| x_scale | Optional attribute | Global scale multiplier in HiFloat8 mode (quant_mode=2). Default value: 1.0. | FLOAT | - |
Constraints
- The last dimension
dofxmust be divisible by 32 andd ≤ 8192. - The dimensions of
slot_mappingmust equal the dimensions ofxminus 1, that is,slot_mappingis the flattening of all dimensions ofxexcept the last dimension. - The data type combination must match the quantization mode: In Normal/HiFloat8 mode, cache is FP8/UINT8 and cache_scale is FLOAT. In MX-FP8 mode, cache is FP8 and cache_scale is FLOAT8_E8M0. In MX-FP4 mode, cache is FP4 and cache_scale is FLOAT8_E8M0.
- Tokens with a value of -1 in
slot_mappingare skipped. The value range of other valid elements is [0, num_slots - 1], and element values must not repeat. Correctness of results is not guaranteed when duplicates exist.
Calling Instructions
| Calling Method | Sample Code | Description |
|---|---|---|
| aclnn API | test_aclnn_indexer_quant_cache | Calls the IndexerQuantCache operator through aclnnIndexerQuantCache. |
| Graph mode | - | Calls the IndexerQuantCache operator by connecting to the GE graph mode through the operator IR. |
| PyTorch API | - | Calls the indexer_quant_cache operator through the indexer_quant_cache API. |