FilesLast commitLast update
7 days ago
1 month ago
26 days ago
7 days ago
9 days ago
16 days ago
3 months ago
11 days ago
9 days ago
README

QuantCompressor

English | 简体中文

Product Support

Product Supported
Ascend 950PR&950DT products √
Atlas A3 products ×
Atlas A2 products ×
Atlas 200I/500 A2 inference products ×
Atlas inference products ×
Atlas training products ×

Description

  • Operator function: QuantCompressor is a preprocessing operator of SMLA and QLI in the inference scenario. It is the quantized version of Compressor. It compresses every 4 or 128 tokens of the KV cache into one, and then each token performs DSA computation with these compressed KV caches. In the case of long sequences, QuantCompressor can effectively reduce the computation overhead. The difference from Compressor is that the inputs xx, WKVW^{KV}, and WGateW^{Gate} are HIFLOAT8 quantized data and directly participate in Matmul computation as HIFLOAT8 (natively supported by hardware). Then the FLOAT32 Matmul output is multiplied by the merged scale factor for dequantization, thereby reducing memory usage and movement overhead. The main computation process is:

    1. Matmul and dequantization: Perform Matmul on the HIFLOAT8 quantized input XX and WKVW^{KV} to obtain a FLOAT32 result, then multiply it by the merged scale factor x_descale⋅wkv_descalex\_descale \cdot wkv\_descale to complete dequantization and obtain kv_statekv\_state. Perform Matmul on XX and WGateW^{Gate} to obtain a FLOAT32 result, then multiply it by the merged scale factor x_descale⋅wgate_descalex\_descale \cdot wgate\_descale to complete dequantization and obtain score_statescore\_state. x_descale is a per-tensor scale (a single scalar), and wkv_descale and wgate_descale are per-channel scales (the number of channels is coff*D). After merging, the scale is still per-channel. kv_statekv\_state and score_statescore\_state are updated according to the input start_pos and cu_seqlens.
    2. When coff is 2, rearrange the data of kv_statekv\_state and score_statescore\_state.
    3. Group score_statescore\_state by the compression ratio and add it to ApeApe, then perform the softmax computation. Compute Mul on the softmax result and kv_statekv\_state, and then perform the ReduceSum computation.
  • Computation formula:

    1. Compute matrix multiplication and dequantization (quant_mode=1, A8W8_A_HIFP8_PER_TENSOR_W_HIFP8_PER_CHANNEL):

    Perform Matmul with HIFLOAT8 input. After the FLOAT32 result is output, multiply it by the merged scale factor (the product of x_descale and wkv/wgate_descale) to complete dequantization:

    C4A:[kv_statea,score_statea]=(Xhif8@[Whif8aKV,Whif8aGate])⋅(x_descale⋅[wkv_descalea,wgate_descalea]),[kv_stateb,score_stateb]=(Xhif8@[Whif8bKV,Whif8bGate])⋅(x_descale⋅[wkv_descaleb,wgate_descaleb]);C4A:\left[kv\_state^a, score\_state^a\right] = (X_{hif8} @ \left[W^{aKV}_{hif8}, W^{aGate}_{hif8}\right]) \cdot (x\_descale \cdot [wkv\_descale^a, wgate\_descale^a]), \left[kv\_state^b, score\_state^b\right] = (X_{hif8} @ \left[W^{bKV}_{hif8}, W^{bGate}_{hif8}\right]) \cdot (x\_descale \cdot [wkv\_descale^b, wgate\_descale^b]);

    C128A:[kv_state,score_state]=(Xhif8@[Whif8KV,Whif8Gate])⋅(x_descale⋅[wkv_descale,wgate_descale])C128A:\left[kv\_state, score\_state\right] = (X_{hif8} @ \left[W^{KV}_{hif8}, W^{Gate}_{hif8}\right]) \cdot (x\_descale \cdot [wkv\_descale, wgate\_descale])

    x_descale is a per-tensor scalar (shape [1,]), and wkv_descale and wgate_descale are per-channel vectors (shape [coff*D,], scaled along the weight output channel dimension). After the two are multiplied, they merge into a per-channel scale factor.

    1. Compute grouped addition:

    C4A:score_statei′=[score_state[4(i−1)+1:4i,:]a;score_state[4i+1:4(i+1),:]b]+Ape, i=1,2,⋯ ,s4;C4A:score\_state_i^\prime = \left[score\_state_{\left[4(i-1)+1:4i,:\right]}^a; score\_state_{\left[4i+1:4(i+1),:\right]}^b\right] + Ape,~i=1,2,\cdots, \frac{s}{4};

    C128A:score_statei′=score_state[128(i−1)+1:128i,:]+Ape, i=1,2,⋯ ,s128;C128A:score\_state_i^\prime = score\_state_{\left[128(i-1)+1:128i,:\right]} + Ape,~i=1,2,\cdots, \frac{s}{128};

    1. Compute grouped Softmax:

    C4A:Si′=softmax(score_statei′), i=1,2,⋯ ,s4;C4A:S_i^\prime = softmax(score\_state_i^\prime),~i=1,2,\cdots, \frac{s}{4};

    C128A:Si′=softmax(score_statei′), i=1,2,⋯ ,s128;C128A:S_i^\prime = softmax(score\_state_i^\prime),~i=1,2,\cdots, \frac{s}{128};

    1. Compute the Hadamard product:

    C4A:(SH)i=Si′⊙[kv_state[4(i−1)+1:4i,:]a;kv_state[4i+1:4(i+1),:]b], i=1,2,⋯ ,s4;C4A:(S_H)_i = S_i^\prime \odot \left[kv\_state^a_{\left[4(i-1)+1:4i,:\right]} ;kv\_state^b_{\left[4i+1:4(i+1),:\right]}\right],~i=1,2,\cdots, \frac{s}{4};

    C128A:SH=Si′⊙kv_state;C128A:S_H = S_i^\prime \odot kv\_state;

    1. Compute grouped summation along the compression axis:

    C4A:CiComp=[1]1×8@(SH)i, i=1,2,⋯ ,s4;C4A:C_{i}^{\text{Comp}} = \left[1\right]_{1\times8} @ (S_H)_i, ~i=1,2,\cdots, \frac{s}{4};

    C128A:CiComp=[1]1×128@(SH)i, i=1,2,⋯ ,s128;C128A:C_{i}^{\text{Comp}} = \left[1\right]_{1\times128} @ (S_H)_i, ~i=1,2,\cdots, \frac{s}{128};

Parameters

Parameter Input/Output/Attribute Description Data Type Data Format
x Input X in the formula, indicating the original data without compression. It is the HIFLOAT8 quantized input. HIFLOAT8 ND
wkv Input WKV in the formula, indicating the KV compression weight. It is the HIFLOAT8 quantized input. HIFLOAT8 ND
wgate Input WGate in the formula, indicating the gate compression weight. It is the HIFLOAT8 quantized input. HIFLOAT8 ND
state_cache Input/Output [kv_state, score_state] in the formula, indicating the historical data of kv_state and score_state. FLOAT ND
ape Input Ape in the formula, indicating positional biases. FLOAT ND
x_descale Optional input Dequantization scale factor of x, with per-tensor scale. Required when quant_mode=1. FLOAT ND
wkv_descale Optional input Dequantization scale factor of wkv, with per-channel scale and coff*D channels. Required when quant_mode=1. FLOAT ND
wgate_descale Optional input Dequantization scale factor of wgate, with per-channel scale and coff*D channels. Required when quant_mode=1. FLOAT ND
state_block_table Optional input Indicates the block mapping table used for state_cache storage. In the cachemode=1/2 scenarios supported by the current operator, this parameter is mandatory. INT32 ND
cu_seqlens Optional input Indicates the number of valid tokens in each batch. INT32 ND
seqused Optional input Indicates the number of tokens that actually participate in compression in each batch.
If None is specified, it indicates the same as the Sequence Length of each batch.
INT32 ND
start_pos Optional input Indicates the start position of computation. INT32 ND
quant_mode Attribute Quantization mode. The value range is [1]. 1 indicates A8W8_A_HIFP8_PER_TENSOR_W_HIFP8_PER_CHANNEL (HIFLOAT8 input; x is dequantized with per-tensor scale, and wkv/wgate with per-channel scale). INT32 -
cmp_ratio Attribute Used for sparse computation. It indicates the data compression ratio, which is an integer in the range [2, 128]. INT32 -
coff Optional attribute Indicates whether to perform overlap data rearrangement.
coff=1: no overlap data rearrangement is required; coff=2: overlap data rearrangement is required.
The default value is 1.
INT32 -
cache_mode Optional attribute Indicates the storage mode of state_cache.
cache_mode=1: continuous buffer; cache_mode=2: circular buffer.
The default value is 1.
INT32 -
state_cache_stride_dim0 Optional attribute Indicates the stride of axis 0 of state_cache. The default value is 0. INT32 -
cmp_kv Output Indicates the compressed data. The content of the pad rows at the tail that do not form a complete compression group is undefined (the operator does not write them, and the memory content before the call is retained). The caller should consume only the valid compressed rows. BFLOAT16 ND

Constraints

  • Dimension meanings of the x parameter: B (Batch Size) indicates the batch size of the input sample, S (Sequence Length) indicates the sequence length of the input sample, H (Head Size) indicates the size of the hidden layer, D (Head Dim) indicates the smallest unit size of the hidden layer, and T indicates the cumulative sum of the sequence lengths of input samples across all batches.
  • Input shape restrictions:
    • state_cache supports the input shape [block_num,block_size,2*coff*D], with block_num>0 required. When cache_mode=2, block_size >= coff * cmp_ratio + S - 1 must be satisfied.
    • If the dimensions of x adopt BS axis merging, that is, the input shape of x is [T,H]:
      • The input shape of cu_seqlens must be [B+1,]. Each element in this parameter indicates the total number of tokens in the current batch and all previous batches, that is, the prefix sum. Therefore, the value of each element must be greater than or equal to that of the previous element, the first element must be 0, and the last element must be T.
      • seqused supports the input shape [B,]. The number of valid tokens in each batch must be less than or equal to the corresponding Sequence Length, that is, seqused[n] <= cu_seqlens[n+1] - cu_seqlens[n], and not less than 0.
      • When cache_mode=1, state_block_table supports the input shape [B,ceil(Smax/block_size)]. Smax is the largest Sequence Length in each batch, that is, Smax=max(start_pos)+max(cu_seqlens[n+1] - cu_seqlens[n]). When cache_mode=2, state_block_table supports the input shape [B].
      • cmp_kv has the output shape [min(T,T//cmp_ratio+B),D]: compressed_tokens + compressed_tokens + ... + compressed_tokens + pad.
    • If the dimensions of x do not adopt BS axis merging, that is, the input shape of x is [B,S,H]:
      • cu_seqlens must be empty.
      • seqused supports the input shape [B,]. The number of valid tokens in each batch must be less than or equal to the corresponding Sequence Length, that is, seqused[n] <= S, and not less than 0.
      • When cache_mode=1, state_block_table supports the input shape [B,ceil(Smax/block_size)]. Smax is the largest Sequence Length in each batch, that is, Smax=max(start_pos)+S. When cache_mode=2, state_block_table supports the input shape [B].
      • cmp_kv has the output shape [B,ceil(S/cmp_ratio),D]: (compressed_tokens+pad0) + (compressed_tokens+pad1) + ... + (compressed_tokens+padN).
  • Input value range restrictions:
    • This interface supports generalization of B and S, with the following scenario restrictions:
      • In some long-sequence scenarios, an excessively large computation amount may cause an error that exceeds NPU memory. Note: the computation amount is affected by the input shape of x; the larger the values, the larger the computation amount. Typical long-sequence scenarios (that is, a large product of B and S, or a large T) include but are not limited to the following:
      B S H
      100 65525 4096
      25 261120 4096
      100 131072 4096
      100 261120 4096
  • This interface supports B, S, and T being 0. That is, input parameters whose shapes are related to the B, S, and T values allow empty tensors, and the other input parameters do not support empty tensors. In this scenario, state_cache is not updated, and the output cmp_kv is an empty tensor.
  • The value of each element in state_block_table ranges from [0, block_num), where block_num is the size of dimension 0 of state_cache. Element values are directly used as the block index of state_cache, and out-of-range values cause illegal memory access. When an element value is 0: under cache_mode=1 (continuous buffer), the write operation on state_cache skips this position; under cache_mode=2 (circular buffer), neither the read nor the write operation skips it. The operator does not check for duplicates, and the caller must ensure the uniqueness of element values: under cache_mode=1, the element value 0 is an "unallocated" sentinel value and may appear repeatedly, while non-zero values must be globally unique; under cache_mode=2, 0 is a valid physical block number, and all element values must be globally unique. Duplicates cause multiple logical blocks/batches to write to the same physical block area, resulting in data overwrite of state_cache.
  • Input attribute restrictions:
    • quant_mode is 1 (A8W8_A_HIFP8_PER_TENSOR_W_HIFP8_PER_CHANNEL). In this case, x_descale, wkv_descale, and wgate_descale are required inputs.
    • D of 128/512 is supported.
    • H from 1K to 10K, aligned to 512, is supported.
    • block_size from 1 to 1024 is supported.
    • The following three typical combination scenarios are supported:
      • C4A: D=512, coff=2, cmp_ratio=4;
      • C4Li: D=128, coff=2, cmp_ratio=4;
      • C128A: D=512, coff=1, cmp_ratio=128.
  • Deterministic computation and batch consistency:
    • Deterministic implementation by default. Multiple calls with the same input produce the same result.
    • Set ACL_OPT_DETERMINISTIC to 3 through aclrtSetSysParamOpt() to enable batch consistency. After it is enabled, the computation result is independent of the batch size and position.

Calling Instructions

Calling Method Sample Code Description
aclnn API test_aclnn_quant_compressor Calls the QuantCompressor operator through the aclnnQuantCompressor interface.
PyTorch API - Calls the QuantCompressor operator through the quant_compressor interface.