Research map / Architectures and efficiency / Quantization
APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APHQ-ViT, a reconstruction-based post-training quantization method for ViTs using an average perturbation Hessian loss and a ReLU-replacement MLP reconstruction, achieving strong 3/4-bit results with uniform quantizers.
Problem
Reconstruction-based PTQ methods for CNNs fail on Vision Transformers at ultra-low bit-widths, due to inaccurate output-importance estimation via Fisher-approximated Hessian losses and large accuracy loss when quantizing post-GELU activations.
Approach
Replaces the Fisher-based Hessian approximation with an average perturbation Hessian computed by finite differences of gradients over perturbed outputs (averaged across samples), and replaces GELU with ReLU in MLPs during a reconstruction step with a clamp loss to make post-GELU activations quantization-friendly.
Key results
- At W3/A3 on ImageNet, APHQ-ViT beats the second-best method DopQ-ViT by 10.71% on DeiT-T and averages a 7.21% improvement.
- At W4/A4 on COCO with Mask R-CNN Swin-T, APHQ-ViT reaches 38.9 box AP vs 37.5 for the best prior method (I&S-ViT and DopQ-ViT).
- Ablation on W3/A3 ViT-S: QDrop 38.31% rises to 63.17% with APH loss and MLP Reconstruction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APHQ-ViT instead of the strongest listed baseline at 3-bit quantization | ViT models · ImageNet | top-1 accuracy: APHQ-ViT exceeds the second-best method by an average of 7.21% | Helped |
| Use APHQ-ViT instead of DopQ-ViT | DeiT-T · ImageNet, 3-bit | top-1 accuracy: 44.71 -> 55.42 | Helped |
| Use APHQ-ViT instead of QDrop (ablation) | ViT-S · ImageNet, W3/A3 | top-1 accuracy: 38.31 -> 59.11 | Helped |
| Use APHQ-ViT with APH loss and MLP Reconstruction instead of APH loss alone (ablation) | ViT-S · ImageNet, W3/A3 | top-1 accuracy: 59.11 -> 63.17 | Helped |
| Replace MSE loss with APH loss (ablation) | ViT-S quantization reconstruction · ImageNet, W3/A3 | top-1 accuracy: 38.31 -> 59.11 | Helped |
| Replace BRECQ-based Hessian loss with APH loss (ablation) | ViT-S quantization reconstruction · ImageNet, W3/A3 | top-1 accuracy: 54.33 -> 59.11 | Helped |
| Replace Perturbation Hessian loss with averaged Perturbation Hessian loss (ablation) | ViT-S quantization reconstruction · ImageNet, W3/A3 | top-1 accuracy: 55.14 -> 59.11 | Helped |
| Replace GELU with ReLU and reconstruct the MLP (ablation) | ViT-B · ImageNet | top-1 accuracy: 84.54 -> 84.84 | Helped |
| Replace GELU with ReLU and reconstruct the MLP (ablation) | ViT-S · ImageNet | top-1 accuracy: 81.39 -> 80.90 | Hurt |
| Replace GELU with ReLU and reconstruct the MLP (ablation) | DeiT-T · ImageNet | top-1 accuracy: 72.21 -> 71.07 | Hurt |
| Replace GELU with ReLU and reconstruct the MLP (ablation) | DeiT-S · ImageNet | top-1 accuracy: 79.85 -> 79.38 | Hurt |
| Replace GELU with ReLU and reconstruct the MLP (ablation) | Swin-S · ImageNet | top-1 accuracy: 83.23 -> 83.12 | Hurt |
Surprising findings
- Replacing GELU with ReLU in MLP via MLP Reconstruction keeps accuracy near full precision (e.g., ViT-B 84.84 vs 84.54) and even improves ViT-B.
Limitations
- MLP Reconstruction alone drops accuracy by over 1% on DeiT-T.
- Results rely on uniform quantizers and hyperparameters inherited from QDrop; the method is evaluated only on specific ViT backbones.
Experimental setup
Datasets: ImageNet COCO
Benchmarks: ImageNet COCO
Metrics: top-1 accuracy box AP mask AP latency throughput
Baselines: PTQ4ViT APQ-ViT RepQ-ViT ERQ IGQ-ViT AdaLog I&S-ViT DopQ-ViT QDrop OASQ LSQ
Builds on: BRECQ QDrop AdaRound RepQ-ViT
Closest papers by meaning
- ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.92)
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.92)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.92)
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.90)
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.