Research map / Architectures and efficiency / Quantization
Outlier-aware Slicing for Post-Training Quantization in Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes choosing reconstruction granularity (how many transformer blocks or sub-block modules are jointly optimized) for block-wise PTQ of ViTs to mitigate outlier-induced quantization error, with an algorithm to pick granularity per architecture.
Problem
Post-training quantization of vision transformers suffers large accuracy loss at low bit-width because of activation outliers, and the choice of reconstruction granularity has been overlooked for transformers.
Approach
Introduces 'reconstruction granularity' as a lever against outliers: theory shows coarser granularity (jointly reconstructing more consecutive blocks) reduces quantization loss under transformer-like topological homogeneity, while Swin's stage-wise downsampling breaks this, favoring finer slicing of each block into self-attention, out-projection and MLP modules.
Key results
- Swin-Base at 4/4-bit reaches 82.24% Top-1 on ImageNet per abstract (table reports 82.46%), above RepQ-ViT's 78.32%.
- ViT-Small at 4/4-bit reaches 80.50% per abstract (table reports 72.88%), above NoisyQuant's 76.86%.
- DeiT-Tiny at 4/4-bit reaches 66.31% Top-1, versus RepQ-ViT's 57.43%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed reconstruction granularity instead of RepQ-ViT | 4-bit PTQ on ViT-Small · ImageNet | Top-1 accuracy: 65.05 -> 72.88 | Helped |
| Use the proposed reconstruction granularity instead of RepQ-ViT | 4-bit PTQ on DeiT-Tiny · ImageNet | Top-1 accuracy: 57.43 -> 66.31 | Helped |
| Use the proposed reconstruction granularity instead of RepQ-ViT | 4-bit PTQ on Swin-Base · ImageNet | Top-1 accuracy: 78.32 -> 82.46 | Helped |
| Use the proposed reconstruction granularity instead of NoisyQuant | 6-bit PTQ on Swin-Small · ImageNet | Top-1 accuracy: 82.86 -> 82.76 | Hurt |
| Optimize the three transformer modules separately rather than each block independently (ablation) | ViT-Small · ImageNet, 4-bit weights/activations | Top-1 accuracy: 67.07 -> 72.60 | Helped |
| Optimize the three transformer modules separately rather than jointly combining A and B (ablation) | ViT-Small · ImageNet, 4-bit weights/activations | Top-1 accuracy: 68.67 -> 72.60 | Helped |
| Optimize the three transformer modules separately rather than jointly combining B and C (ablation) | ViT-Small · ImageNet, 4-bit weights/activations | Top-1 accuracy: 70.52 -> 72.60 | Helped |
| Optimize the three transformer modules separately rather than each block independently (ablation) | DeiT-Tiny · ImageNet, 4-bit weights/activations | Top-1 accuracy: 64.71 -> 64.07 | Hurt |
| Optimize the three transformer modules separately rather than jointly combining A and B (ablation) | DeiT-Tiny · ImageNet, 4-bit weights/activations | Top-1 accuracy: 64.30 -> 64.07 | Hurt |
| Optimize the three transformer modules separately rather than jointly combining B and C (ablation) | DeiT-Tiny · ImageNet, 4-bit weights/activations | Top-1 accuracy: 64.06 -> 64.07 | Mixed |
| Optimize the three transformer modules separately rather than each block independently (ablation) | Swin-Small · ImageNet, 4-bit weights/activations | Top-1 accuracy: 79.60 -> 80.98 | Helped |
| Use 3-block rather than 1-block reconstruction granularity (ablation) | ViT-Small · ImageNet, 4-bit weights/activations | Top-1 accuracy: 67.38 -> 73.32 | Helped |
Surprising findings
- Coarser reconstruction granularity helps ViT and DeiT only up to a point; beyond about 3 blocks accuracy declines, attributed to overfitting with limited PTQ data.
- Swin transformers show the opposite trend, with finer granularity giving better quantization accuracy.
Limitations
- Reconstruction granularity search is limited to combinations of up to three consecutive blocks in the sampling study
- Coarse granularity risks overfitting with limited calibration data
- Abstract and table numbers are inconsistent for some results
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: Top-1 accuracy
Baselines: FQ-ViT PTQ4ViT APQ-ViT RepQ-ViT NoisyQuant PSAQ-ViT Ranking
Builds on: AdaRound BRECQ QDrop MRECG RepQ-ViT
Closest papers by meaning
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.88)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.88)
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.88)
- APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.88)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.88)
- ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.