Research map / Architectures and efficiency / Quantization
FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIMA-Q, a block-wise post-training quantization method for ViTs that replaces the BRECQ diagonal Hessian/FIM loss with a diagonal-plus-low-rank (DPLR) Fisher approximation derived from KL divergence, using a standard uniform quantizer.
Problem
Post-training quantization of Vision Transformers suffers large accuracy drops, especially at low bit-widths, and the Hessian/Fisher-based reconstruction loss used by prior methods (BRECQ) is approximated poorly for ViTs.
Approach
Shows that the FIM diagonal is linearly related to the KL-divergence gradient (not its square as in BRECQ), derives rank-one and low-rank FIM approximations that keep off-diagonal inter-token correlations, and combines them with the diagonal term into a DPLR-FIM reconstruction loss built on QDrop.
Key results
- At 3/3-bit on ImageNet, FIMA-Q averages about 5.31% above the second-best method across ViT/DeiT/Swin models.
- At 3-bit DPLR-FIM lifts MSE accuracy by 23.04% on ViT-S and 18.18% on DeiT-S.
- At W4/A4 on COCO, FIMA-Q reaches the best APb/APm in most Mask R-CNN and Cascade Mask R-CNN settings with a uniform quantizer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FIMA-Q with uniform quantization at W3/A3 instead of the strongest reported baseline | ViT-based classifiers · ImageNet, W3/A3 | top-1 accuracy: 64.09 vs 54.72; 77.63 vs 74.75; 55.55 vs 46.88; 69.13 vs 50.95; 76.54 vs 72.97; 77.26 vs 74.67; 78.82 vs 69.75 | Helped |
| Use FIMA-Q with uniform quantization at W4/A4 instead of the strongest reported baseline | ViT-based classifiers · ImageNet, W4/A4 | top-1 accuracy: 76.68 vs 75.69; 83.04 vs 82.63; 66.84 vs 66.31; 76.87 vs 75.84; 80.33 vs 80.13; 81.82 vs 80.98; 83.60 vs 83.34 | Helped |
| Use FIMA-Q rather than the strongest listed baseline | Mask R-CNN and Cascade Mask R-CNN with Swin backbones · COCO, W4/A4, APb | APb: 38.7 vs 37.5; 44.2 vs 44.7; 48.7 vs 48.2; 50.4 vs 50.3 | Mixed |
| Add diagonal and low-rank FIM components together (DPLR-FIM) (ablation) | FIMA-Q reconstruction loss · ImageNet, W3/A3 across ViT architectures | top-1 accuracy: 64.09 vs 60.02; 77.63 vs 76.29; 55.55 vs 55.54; 69.13 vs 68.68; 76.54 vs 76.32; 77.26 vs 75.08; 78.82 vs 77.87 | Helped |
| Add the low-rank FIM component to diagonal FIM (ablation) | FIMA-Q reconstruction loss · ImageNet, ViT-S, W3/A3 | top-1 accuracy: 60.02 -> 64.09 | Helped |
| Replace DPLR-FIM loss with MSE loss (ablation) | FIMA-Q · ImageNet, ViT-S, W3/A3 | top-1 accuracy: 64.09 -> 41.05 | Hurt |
| Replace DPLR-FIM loss with BRECQ-FIM loss (ablation) | FIMA-Q · ImageNet, ViT-S, W3/A3 | top-1 accuracy: 64.09 -> 14.65 | Hurt |
| Increase DPLR-FIM rank k (ablation) | FIMA-Q · ImageNet, W3/A3 | top-1 accuracy | Helped |
| Increase DPLR-FIM rank k (ablation) | FIMA-Q · ImageNet, W4/A4 | top-1 accuracy | No clear effect |
Surprising findings
- The BRECQ diagonal FIM loss performs worse than a simple MSE loss on ViTs at 3-bit (e.g., BRECQ-FIM 11.61 vs MSE 74.75 on ViT-B).
- PTQ4ViT and RepQ-ViT collapse to near-random accuracy (about 0.1%) on several models at 3-bit.
Limitations
- Low-rank term requires the perturbation matrix to have full rank k, handled via a progressive rank schedule.
- Detection/segmentation adaptation uses RPN classification outputs as the task output rather than a true task loss.
- Training time grows substantially with rank for large models such as Swin-B.
Experimental setup
Datasets: ImageNet COCO
Benchmarks: ImageNet COCO
Metrics: top-1 accuracy APb APm
Baselines: PTQ4ViT RepQ-ViT APQ-ViT AdaLog I&S-ViT DopQ-ViT QDrop ERQ IGQ-ViT OASQ NoisyQuant BRECQ-FIM
Builds on: BRECQ QDrop AdaRound
Closest papers by meaning
- APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.92)
- ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.91)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.91)
- PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.90)
- Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression (NeurIPS 2025 · similarity 0.90)
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.