Research map / Architectures and efficiency / Quantization
MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MimiQ, a data-free quantization method for ViTs that aligns inter-head attention in synthetic data generation (SSIM-based loss) and distills head-wise attention outputs from a full-precision teacher during quantized fine-tuning.
Problem
Data-free quantization of vision transformers fails in low-bit settings (e.g., W4/A4) because synthetic data produce misaligned inter-head attention maps and BN-based CNN DFQ methods do not transfer to ViTs.
Approach
Align head-wise attention maps for the same query patch during synthetic data generation using absolute SSIM as the distance, and distill head-wise attention outputs (DSSIM) from the full-precision teacher into the quantized student, in addition to output KL matching.
Key results
- On ImageNet W4/A4, MimiQ reaches 62.91% on ViT-B and 74.10% on DeiT-B (min-max), versus 2.83% and 45.81% for the best prior ViT-DFQ baseline PSAQ-ViT V2 on ViT-B and DeiT-B respectively.
- Maximum gain over baselines in the low-bit setting is +51.18 points (ViT-B, W4/A4).
- On COCO W4/A4 with Swin-T, MimiQ achieves 26.41 AP_box versus 0.06 for PSAQ V1 and 4.52 for PSAQ V2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MimiQ instead of PSAQ-ViT V2 | ViT-B quantized to W4/A4 · ImageNet | accuracy: 02.83 -> 62.91 | Helped |
| Use MimiQ instead of PSAQ-ViT V2 | ViT-B quantized to W4/A8 · ImageNet | accuracy: 84.02 -> 84.20 | Helped |
| Use MimiQ instead of PSAQ V2 | Swin-S quantized to W4/A4 · COCO | AP_box: 12.12 -> 34.97 | Helped |
| Use MimiQ instead of PSAQ V2 | Swin-S quantized to W4/A4 · ADE20K | mIoU: 12.13 -> 38.29 | Helped |
| Add inter-head coherency loss to synthesis (ablation) | MimiQ on ViT-T, W4/A4 · ImageNet | accuracy: 13.28 -> 42.99 | Helped |
| Add class loss to synthesis (ablation) | MimiQ on ViT-T, W4/A4 · ImageNet | accuracy: 12.31 -> 42.99 | Helped |
| Remove total variation loss from synthesis (ablation) | MimiQ on ViT-T, W4/A4 · ImageNet | accuracy: 38.38 -> 42.99 | No clear effect |
| Add head-wise attention distillation (ablation) | MimiQ on ViT-T, W4/A4 · ImageNet | accuracy: 39.61 -> 42.99 | Helped |
| Add head-wise attention distillation (ablation) | MimiQ on ViT-B, W4/A4 · ImageNet | accuracy: 39.67 -> 62.91 | Helped |
Surprising findings
- MimiQ outperforms Real-Data FT on DeiT-S, DeiT-B and Swin-B in some settings, attributed to head-wise attention distillation.
- Synthetic-calibrated RepQ-ViT drops sharply at W4/A4 (19.96 on ViT-B vs 68.48 with real data).
Limitations
- Synthetic-data generation cost grows with dataset size (10k samples ~19.8h on ViT-B)
- Synthetic calibration can hurt some real-data quantizers (RepQ-ViT W4/A4)
- Some settings underperform baselines slightly (e.g., Swin-B/Swin-S at W8/A8)
Experimental setup
Datasets: ImageNet synthetic data generated from full-precision ViT/DeiT/Swin
Benchmarks: ImageNet COCO ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU SSIM
Baselines: Real-Data FT GDFQ Qimera AdaDFQ PSAQ-ViT V1 PSAQ-ViT V2 CLAMP-ViT PTQ4ViT FQ-ViT RepQ-ViT
Builds on: GDFQ PSAQ-ViT LSQ Qimera SSIM
Closest papers by meaning
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.89)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.89)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.89)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.88)
- Instance-Aware Group Quantization for Vision Transformers (CVPR 2024 · similarity 0.88)
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.