Research map / Architectures and efficiency / Quantization
BoA: Attention-aware Post-training Quantization without Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BoA, a backpropagation-free PTQ method that derives attention-aware Hessians to account for inter-layer dependencies within attention when optimizing quantized weights, combined with QuaRot/SpinQuant transformations.
Problem
Post-training quantization of LLMs with billions of parameters: gradient-based weight rounding is impractical, while existing backpropagation-free Hessian methods (GPTQ) assume layer independence and ignore inter-layer dependencies inside attention.
Approach
Replace the layer-wise reconstruction-error Hessian with attention-reconstruction-error Hessians for W_Q, W_K, W_V and W_out (via first-order Taylor expansion of softmax), then make them tractable with Hessian relaxation, Kronecker-factored inverse/Cholesky, and head-wise simultaneous quantization.
Key results
- On LLaMA3.2-3B at INT3 without transformation, BoA reaches 59.38% average zero-shot accuracy vs. 39.19% for GPTQ.
- On transformed LLaMA2-13B at INT2, BoA reaches 8.237 Wiki2 PPL vs. 9.593 for QuaRot-GPTQ and 12.92 for OmniQuant (average accuracy 56.92%).
- Under W2A4KV4 with SpinQuant, BoA reaches 47.14% average accuracy on LLaMA2-7B, versus 37.54% for SpinQuant-GPTQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention-aware Hessians to model inter-layer dependencies instead of GPTQ's layer-wise Hessian | BoA vs GPTQ on LLaMA2-7B INT2 · LLaMA2-7B INT2, WikiText-2 | Wiki2 PPL: 30.85 -> 12.76 | Helped |
| Use attention-aware Hessians to model inter-layer dependencies instead of GPTQ's layer-wise Hessian | BoA vs GPTQ on LLaMA2-7B INT2 · LLaMA2-7B INT2, zero-shot tasks | Average accuracy: 42.50 -> 45.81 | Helped |
| Use attention-aware Hessians to model inter-layer dependencies instead of GPTQ's layer-wise Hessian | BoA vs GPTQ on LLaMA3.2-3B INT3 · LLaMA3.2-3B INT3, zero-shot tasks | Average accuracy: 39.19 -> 59.38 | Helped |
| Apply QuaRot transformation before BoA | BoA on LLaMA2-7B INT2 · LLaMA2-7B INT2, WikiText-2 | Wiki2 PPL: 12.76 -> 10.42 | Helped |
| Apply QuaRot transformation before BoA | BoA on LLaMA2-7B INT2 · LLaMA2-7B INT2, zero-shot tasks | Average accuracy: 42.50 -> 51.00 | Helped |
| Apply SpinQuant transformation before BoA | BoA on LLaMA2-7B W2A4KV4 · LLaMA2-7B W2A4KV4, zero-shot tasks | Average accuracy: 37.54 -> 47.14 | Helped |
| Replace BoA's full attention-aware value-projection Hessian with the standard Hessian (ablation) | BoA on LLaMA-30B · LLaMA-30B, WikiText-2 | Wiki2 PPL: 6.669 -> 7.007 | Hurt |
| Enable simultaneous quantization across attention heads (ablation) | BoA on LLaMA-30B · LLaMA-30B, quantization processing | Processing time (hour): 135.4 -> 3.295 | Helped |
Surprising findings
- Quantizing the value projection with the full attention-aware Hessian requires large memory, so a relaxed variant that uses the standard Hessian for values is used in practice.
Limitations
- Processing time is longer than GPTQ because rows within each head are quantized sequentially.
- Attention-aware Hessians for value projection require large memory (about 32.8 GB for 30B), requiring a relaxation.
- Mixed-precision and vector-quantization methods are excluded from comparison.
Experimental setup
Datasets: WikiText-2
Benchmarks: WikiText-2 perplexity ARC-challenge ARC-easy BoolQ HellaSwag LAMBADA OpenbookQA PIQA WinoGrande
Metrics: perplexity zero-shot accuracy processing time memory cost
Baselines: RTN GPTQ OmniQuant AffineQuant DuQuant QuaRot-GPTQ SpinQuant-GPTQ
Builds on: GPTQ AdaRound BRECQ QuaRot SpinQuant
Closest papers by meaning
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.91)
- Evaluating Quantized Large Language Models (ICML 2024 · similarity 0.90)
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- AffineQuant: Affine Transformation Quantization for Large Language Models (ICLR 2024 · similarity 0.88)
- ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.88)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.