Research map / Architectures and efficiency / Quantization
Instance-Aware Group Quantization for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IGQ-ViT, a post-training quantization method for vision transformers that dynamically groups activation channels and softmax attention rows per input instance, with a BOP-constrained ILP for per-layer group counts.
Problem
Post-training quantization of vision transformers at low bit-widths (e.g., 4-bit) degrades badly because per-channel activation ranges and softmax attention distributions vary strongly across channels, tokens and input instances, which fixed layer-wise or fixed-group quantizers cannot handle.
Approach
Split activation channels (inputs of FC layers) and softmax attention rows into groups dynamically at runtime for each input instance, assigning each to the quantizer with minimum distance via EM-style alternating optimization, and allocate per-layer group counts by minimizing KL-divergence perturbation under a bit-operation constraint.
Key results
- Under 6/6-bit, IGQ-ViT with 12 groups is within 0.9% of the per-channel upper bound on ImageNet across ViT, DeiT, and Swin models.
- Under 4/4-bit on ImageNet, IGQ-ViT (12 groups) reaches 79.32% for ViT-B and 79.23% for DeiT-B, consistently outperforming RepQ-ViT (68.48% and 75.61%).
- On COCO with Swin-T Mask R-CNN at 4/4-bit, IGQ-ViT (12 groups) achieves 41.0 box AP and 38.8 mask AP versus 36.1 and 36.0 for RepQ-ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply instance-aware grouping with 12 groups instead of RepQ-ViT | ViT-S · ImageNet validation, 4/4-bit | top-1 accuracy: 65.05 -> 73.61 | Helped |
| Apply instance-aware grouping with 12 groups instead of RepQ-ViT | DeiT-B · ImageNet validation, 4/4-bit | top-1 accuracy: 75.61 -> 79.23 | Helped |
| Apply instance-aware grouping with 12 groups instead of RepQ-ViT | Mask R-CNN with Swin-T · COCO, 4/4-bit | APbox: 36.1 -> 41.0 | Helped |
| Apply instance-aware grouping with 12 groups instead of RepQ-ViT | Cascade Mask R-CNN with Swin-T · COCO, 4/4-bit | APmask: 41.4 -> 42.4 | Helped |
| Increase group count from 8 to 12 (ablation) | IGQ-ViT on ViT-S · ImageNet validation, 4/4-bit | top-1 accuracy: 72.99 -> 73.61 | Helped |
| Increase group count from 8 to 12 (ablation) | IGQ-ViT on DeiT-B · ImageNet validation, 4/4-bit | top-1 accuracy: 78.85 -> 79.23 | Helped |
| Increase group count from 8 to 12 (ablation) | IGQ-ViT on Swin-T · ImageNet validation, 4/4-bit | top-1 accuracy: 76.91 -> 77.77 | Helped |
| Add group-size allocation rather than using a fixed group size (ablation) | IGQ-ViT on ViT-S · ImageNet, 4/4-bit, 6 groups | top-1 accuracy: 71.44 -> 71.80 | Helped |
| Add group-size allocation rather than using a fixed group size (ablation) | IGQ-ViT on DeiT-B · ImageNet, 4/4-bit, 8 groups | top-1 accuracy: 78.85 -> 79.04 | Helped |
| Add IGQ to linear operations while retaining layer-wise attention quantization (ablation) | IGQ-ViT on ViT-S · ImageNet, 4/4-bit | top-1 accuracy: 42.82 -> 57.32 | Helped |
| Add IGQ to softmax attentions while retaining layer-wise quantization for linear operations (ablation) | IGQ-ViT on ViT-S · ImageNet, 4/4-bit | top-1 accuracy: 42.82 -> 53.69 | Helped |
| Use instance-aware grouping for both linear operations and softmax attention instead of layer-wise quantization (ablation) | IGQ-ViT on ViT-S · ImageNet, 4/4-bit | top-1 accuracy: 42.82 -> 72.99 | Helped |
Surprising findings
- Sorting channels by dynamic range before grouping (PEG) does not improve over plain grouping for DeiT-B and Swin-T, suggesting per-channel ranges vary strongly across input instances.
Limitations
- Runtime grouping adds min/max computation, assignment, and floating-point group summation overhead compared with layer-wise quantization.
- Group size allocation relies on KL-divergence perturbation measured per layer and an ILP solver, adding calibration complexity.
Experimental setup
Datasets: ImageNet COCO
Benchmarks: ImageNet COCO
Metrics: top-1 accuracy box AP mask AP BOPs
Baselines: PTQ4ViT APQ-ViT RepQ-ViT Layer-wise quantization Grouping consecutive channels Sorting before grouping channels (PEG)
Builds on: PTQ4ViT RepQ-ViT APQ-ViT Q-bert group quantization VS-quant
Closest papers by meaning
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.91)
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.90)
- APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.89)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.88)
- MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.88)
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.