Research map / Architectures and efficiency / Quantization
GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GradQ-ViT, an INT8 gradient quantization framework for ViT training combining IQR-based quantization points, a Cross-Huber blend loss, gradient scaling, and adaptive learning rate allocation.
Problem
Gradient quantization for Vision Transformer training is unstable because gradients have narrow ranges with outliers and ViT loss landscapes are complex, and prior CNN-oriented methods (e.g., stochastic rounding) are ill-suited to ViTs.
Approach
Assigns quantization points adaptively based on interquartile range (log quantization inside IQR, uniform-like allocation to outlier tails), uses a blended cross-entropy plus Huber loss to limit outliers, rescales dequantized gradients to match the original magnitude and direction, and sets per-layer learning rates from quantization error and gradient cosine similarity.
Key results
- With W/A/G at INT8, DeiT-Base improves 0.52% over its FP32 baseline (82.32% vs 81.8%) and Swin-Base improves 0.21% (84.71% vs 84.5%) on ImageNet.
- On MobileViT-S, INT8 W/A/G training drops accuracy only 0.09% (78.96% vs 79.05% baseline) while the naive SR baseline drops 13.03%.
- A custom CUDA 11.8 kernel on RTX 3090 gives a 2.06× iteration speedup for MobileViT with INT8 forward and backward versus FP32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GradQ-ViT instead of Q-ViT | DeiT-S · ImageNet, DeiT-S | accuracy: 80.11 -> 80.26 | Helped |
| Use GradQ-ViT instead of the strongest listed baseline | DeiT-B · ImageNet, DeiT-B | accuracy: 79.7 -> 82.32 | Helped |
| Use GradQ-ViT instead of the strongest listed baseline | Swin-T · ImageNet, Swin-T | accuracy: 78.3 -> 81.15 | Helped |
| Use GradQ-ViT instead of Q-ViT | Swin-S · ImageNet, Swin-S | accuracy: 80.59 -> 83.37 | Helped |
| Use GradQ-ViT instead of the FP32 baseline | DeiT-B · ImageNet, DeiT-B | accuracy: +0.52% | Helped |
| Use GradQ-ViT instead of the FP32 baseline | Swin-B · ImageNet, Swin-B | accuracy: +0.21% | Helped |
| Quantize MobileViT-S weights, activations, and gradients to INT8 with GradQ-ViT | MobileViT-S · ImageNet, MobileViT-S | accuracy: 0.09% accuracy drop | Hurt |
| Quantize MobileViT forward and backward operations to INT8 | MobileViT on RTX 3090 · MobileViT, RTX 3090 | iteration speedup: 2.06× | Helped |
| Add IQS to ALA (ablation) | DeiT-Tiny · ImageNet, DeiT-Tiny | accuracy: 65.62 -> 69.48 | Helped |
| Add CH-Loss to ALA + IQS (ablation) | DeiT-Tiny · ImageNet, DeiT-Tiny | accuracy: 69.48 -> 71.27 | Helped |
| Add GS to ALA + IQS + CH-Loss (ablation) | DeiT-Tiny · ImageNet, DeiT-Tiny | accuracy: 71.27 -> 72.21 | Helped |
Surprising findings
- Quantizing gradients to INT8 with deterministic rounding reportedly slightly exceeds the FP32 baseline on DeiT-Base and Swin-Base, despite the paper's own argument that nearest rounding is biased.
Limitations
- MobileViT baselines against prior gradient quantization methods were not presented because none exist for it, so comparisons are limited to SR.
- Swin-Tiny showed a slight accuracy degradation (0.05% drop claim aside, Table 2 shows improvement only in some rows) and the paper acknowledges mixed results.
- Hyperparameters alpha and beta of the learning rate rule are set heuristically (1 and 0 early, swapped later) and the Cross-Huber delta and beta values are not analyzed.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: top-1 accuracy accuracy drop speedup
Baselines: SR Quantformer Q-ViT
Builds on: Zhu et al. 2020 INT8 backpropagation quantization Chmiel et al. 2021 logarithmic quantization Huber loss stochastic rounding
Closest papers by meaning
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.91)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.91)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.91)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.90)
- APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.89)
- MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.