Research map / Architectures and efficiency / Quantization
Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AESPA, a learning-based weight-only PTQ method for LLMs that quantizes layer-wise but optimizes attention-wise reconstruction objectives with pre-computed statistics, achieving near block-wise accuracy at much lower cost.
Problem
Post-training quantization of hyper-scale Transformers is costly for block-wise reconstruction methods (e.g., BRECQ), while learning-free methods ignore cross-layer dependency inside attention and rely on nearest rounding, hurting low-bit accuracy.
Approach
Quantize each projection (Q, K, V) separately in a layer-wise manner, but use reconstruction objectives of the attention output (derived via first-order Taylor approximation and an upper bound) that are computed from pre-computed expectations such as E[X X^T], E[K^T K], E[Q^T Q], and E[X A^T A X^T], enabling efficient weight-rounding optimization.
Key results
- At INT3 on WikiText-2, AESPA reaches 6.579 PPL on LLaMA-7B versus 6.716 for OmniQuant, and BRECQ runs out of memory on LLaMA models.
- At INT2 on WikiText-2 for OPT-6.7B, AESPA achieves 15.71 PPL where OmniQuant gives 4.9e3 and BRECQ runs out of memory.
- On INT2 zero-shot, AESPA averages 42.61 on LLaMA-7B versus 39.97 for Z-Fold and 37.70 for OmniQuant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention-output reconstruction rather than layer-output reconstruction (ablation) | OPT-125M quantization with layer-wise AdaRound · OPT-125M, W2A16 | perplexity: 160.7 -> 69.23 | Helped |
| Use attention-output reconstruction with separate layer-wise quantization rather than joint block-wise quantization (ablation) | OPT-125M quantization · OPT-125M, W2A16 | perplexity: 60.38 -> 69.23 | Hurt |
| Use attention-output reconstruction rather than layer-output reconstruction (ablation) | OPT-125M quantization with layer-wise AdaRound · OPT-125M, W3A16 | perplexity: 35.44 -> 32.89 | Helped |
| Use Hessian E[X A^T A X^T] instead of E[X X^T] (ablation) | Z-Fold · WikiText-2, OPT-125M, INT2 | perplexity: 190.1 -> 146.4 | Helped |
| Use Hessian E[X A^T A X^T] instead of E[X X^T] (ablation) | Z-Fold · PTB, OPT-125M, INT2 | perplexity: 331.6 -> 212.8 | Helped |
| Use Hessian E[X A^T A X^T] instead of E[X X^T] (ablation) | Z-Fold · C4, OPT-125M, INT2 | perplexity: 125.3 -> 112.6 | Helped |
| Optimize a weight-rounding policy after determining quantization parameters, rather than use nearest rounding | BLOOM models with INT2 quantization · WikiText-2, BLOOM-560M | perplexity: 65.45 -> 44.91 | Helped |
| Use attention-wise reconstruction and layer-wise quantization instead of layer-wise reconstruction (ablation) | OPT-125M quantization · OPT-125M, W4A16 | perplexity: 29.51 -> 28.98 | Mixed |
| Quantize query and key projections simultaneously instead of separately (ablation) | aespa on OPT-125M · OPT-125M quantization | quantization time: 3.5 hours; 44 times longer than the proposed aespa | Hurt |
| Use attention-wise reconstruction with layer-wise quantization instead of conventional layer-wise methods | aespa · LLaMA-7B, INT2 zero-shot tasks | average accuracy: 39.97 -> 42.61 | Helped |
| Use attention-wise reconstruction with layer-wise quantization instead of block-wise Brecq | aespa · WikiText-2, OPT-125M, INT2 | perplexity: 60.38 -> 71.18 | Hurt |
Surprising findings
- BRECQ performs best on 2-bit OPT-125M but degrades sharply with model size, which the authors attribute to hyperparameters tuned for ImageNet.
- Layer-wise quantization targeting attention-wise reconstruction performs close to joint block-wise quantization (BRECQ) on OPT-125M.
Limitations
- Only weight-only quantization is studied; activation quantization is left for future work.
- Focuses on attention output and not the full Transformer block, leaving nonlinear and normalization effects unmodeled.
- Results for BRECQ on larger models are affected by OOM and hyperparameter choices.
Experimental setup
Datasets: C4
Benchmarks: WikiText-2 C4 PTB ARC HellaSwag MMLU
Metrics: perplexity zero-shot accuracy processing time FLOPs
Baselines: BRECQ OmniQuant AffineQuant OPTQ Z-Fold RTN AdaRound
Builds on: AdaRound BRECQ Z-Fold OPTQ
Closest papers by meaning
- PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.90)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.89)
- ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.89)
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.88)
- Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers (CVPR 2025 · similarity 0.87)
- ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.