MACAQ: better W4A4 accuracy with a fused inference path

Project repository: huawei-csl/MACaQ.

Add dense and grouped MACAQ W4A4 matmul for Ascend 950 using PTO-ISA through the ascend_ops_nn PyTorch extension.
MACAQ (Model Agnostic Calibrated Quantization) applies channel scaling,
block-32 BF16 Hadamard rotation, MXFP4 activation quantization and matrix
multiplication in one mixed Vector/Cube kernel, producing BF16 output.
Calibration, weight transformation (W / D) @ H and Flip-Refinement are offline.

The key result is better quantization accuracy with competitive small-batch
kernel latency. The online accuracy transform is included in every MACAQ
timing below. Native CANN quantization and matmul are unchanged independent
baselines. Large-prefill and grouped losses remain visible.

Accuracy results

Model perplexity: 23.7% lower than ordinary W4A4

Qwen3-0.6B on held-out WikiText-2; lower is better:

This is a CPU model-reference pilot, using 16,384 held-out tokens
(16,368 scored predictions, context 1,024). All 196 decoder linear projections
are quantized; embeddings and the LM head remain BF16. Calibration uses
1,024 training tokens, with no held-out hyperparameter tuning. It is not
full-dataset perplexity or full-model NPU execution. Algorithm 2 model
perplexity has not been measured.

Actual NPU outputs: lower error on all nine fixed projections

MACAQ lowers normalized mean squared error against both native FP4 scaling
presets on q/up/down projections at layers 0, 14 and 27. Native error is
1.58–4.61× MACAQ error for algorithm 0 and
1.68–4.14× for algorithm 2.
The measured ops-nn, TileLang regular and TileLang prepared outputs agree
exactly with the MACAQ CPU reference on all nine fixtures. Evaluation uses
1,024 separate held-out test tokens. Unsupported ops-nn dimensions are
zero-padded for accuracy coverage only, then sliced; those calls make no latency claim.

Accuracy protocol, all projection rows, grouped arithmetic checks and raw data.
Runtime and accuracy are separate experiments; this is not a joint
model-level speed-and-accuracy result.

Latency against existing native CANN operators

Baseline: npu_dynamic_mx_quant (block32 E2M1, round-to-even, scale_alg=0)
followed by npu_quant_matmul, or npu_grouped_matmul for grouped workloads.
MACAQ additionally performs scaling and Hadamard rotation. Each numerical
function is checked against its own CPU reference. Our split MACAQ paths
are ablations, not existing external baselines.

Native latency / MACAQ latency; above 1 favors MACAQ. Geometric means
give every configuration equal weight: all 24 (K,N) pairs at dense
M=16/128/4096 plus four-expert grouped workloads, for 96 cases in total.
Win counts compare medians and do not establish statistical significance.

Both small-batch groups are faster on the geometric mean, while the grouped mean is near parity. One call per configuration has total native/MACAQ time ratio 0.930×; MACAQ is slower in that mixture because larger cases contribute more time.

The accuracy gain comes without a latency penalty on the faster measured
workloads. It is not free for every shape: large prefill and many grouped
cases still lose. No end-to-end NPU throughput gain is claimed.

Complete protocol, all 96 rows, heatmap and raw samples.

Related implementation: TileLang PR #44.

AI/Agent disclosure

• AI-assisted implementation