Research map / Architectures and efficiency / Quantization

ERQ: Error Reduction for Post-Training Quantization of Vision Transformers

ICML 2024 · Architectures and efficiency › Quantization · approach: Post-training quantization of ViTs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ERQ, a two-step post-training quantization method for ViTs that first reduces activation quantization error via ridge-regression weight updates (Aqer) and then reduces weight quantization error with an iterative rounding refinement plus ridge regression (Wqer).

Problem

Post-training quantization of vision transformers suffers large output error because weight and activation quantization errors are entangled, which degrades accuracy especially at low bit-widths (W3A4, W4A4).

Approach

Decouple the joint weight-activation MSE into two sequential subproblems: a closed-form ridge-regression weight adjustment that absorbs activation quantization error (Aqer), and an iterative quantize-and-correct procedure (Wqer) that refines rounding directions using a Gaussian-based proxy of output error and then compensates with ridge regression on the remaining full-precision weights.

Key results

• ERQ improves W3A4 ViT-S top-1 accuracy by 22.36% over GPTQ (45.68% vs 23.32%).
• On W4A4 DeiT-S, the full ERQ reaches 72.56% vs 68.41% baseline; Aqer alone gives +3.04 points.
• ERQ gives +0.5 box AP and +0.3 mask AP on W4A4 Mask R-CNN with Swin-T on COCO.

Measured effects reported in the paper

Surprising findings

• Rounding Refinement cuts MIPQ time from about 10 hours to 4 minutes at a small accuracy cost (69.55% to 69.24% on W4A4 DeiT-S).

Limitations

• Only layers with weights are handled; quantization error of self-attention is not considered.
• Not evaluated on large language models due to compute limits.
• Rounding refinement is restricted to flipping rounding directions, with limited flexibility.

Experimental setup

Datasets: ImageNet COCO

Benchmarks: ImageNet COCO

Metrics: top-1 accuracy box AP mask AP

Baselines: FQ-ViT PTQ4ViT APQ-ViT GPTQ RepQ-ViT AdaRound BRECQ QDrop PD-Quant

Builds on: GPTQ RepQ-ViT BRECQ AdaRound QDrop

Closest papers by meaning

• APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.92)
• AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.92)
• GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.91)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.91)
• VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.90)
• Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization (CVPR 2026 · similarity 0.89)