Research map / Architectures and efficiency / Attention and transformers
ViT-Calibrator: Decision Stream Calibration for Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViT-Calibrator, a two-stage fine-tuning method (token feedback calibration plus LRP-based class-token dimension distillation) that improves accuracy of pretrained ViT variants on CIFAR-100, ImageNet-1K and ImageNet-Real.
Problem
Vision Transformers are opaque, and improving them usually requires architecture trial and error; the paper asks whether misdirected information flow in the decision stream (token and class-token dimension level) can be diagnosed and calibrated to boost classification.
Approach
Uses attention rollout and LRP relevance to find that foreground tokens and sparse class-token dimensions drive decisions, then adds a feedback module from deep to shallow layers (token stage) and a mask-based self-distillation that suppresses class-irrelevant dimensions (dimension stage), without retraining the backbone.
Key results
- On ImageNet-1K, the combined calibration raises DeiT-T from 72.17% to 74.19% top-1 (+2.02) and ViT-T from 75.45% to 77.19%.
- On CIFAR-100, ViT-T improves from 87.44% to 88.20% and DeiT-T from 84.32% to 85.24% with combined calibration.
- The paper reports that gains shrink for large-parameter models and as the number of attention heads grows, and that the dimension stage yields only small gains on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token and dimension calibration | Eva-L · CIFAR-100 | accuracy: +0.07 | Helped |
| add token and dimension calibration | Eva-L · ImageNet-1K | accuracy: +0.19 | Helped |
| add token and dimension calibration | Eva-L · ImageNet-Real | accuracy: +0.13 | Helped |
| add token calibration alone (ablation) | Eva-L · CIFAR-100 | accuracy: +0.05 | Helped |
| add dimension calibration alone (ablation) | Eva-L · CIFAR-100 | accuracy: +0.11 | Helped |
| add token calibration alone (ablation) | Eva-L · ImageNet-1K | accuracy: +0.24 | Helped |
| add dimension calibration alone (ablation) | Eva-L · ImageNet-1K | accuracy: +0.05 | Helped |
| add token calibration alone (ablation) | Eva-L · ImageNet-Real | accuracy: +0.16 | Helped |
| add dimension calibration alone (ablation) | Eva-L · ImageNet-Real | accuracy: +0.12 | Helped |
| increase feedback output layer count from 1 to 3 (ablation) | ViT-Calibrator on DeiT-T · ImageNet-1K | Top-1 accuracy: 73.4 -> 73.8 | Helped |
| increase feedback output layer count from 1 to 3 (ablation) | ViT-Calibrator on DeiT-T · ImageNet-1K | Top-5 accuracy: 91.3 -> 91.9 | Helped |
| set feedback target layer to 3 instead of 1 (ablation) | ViT-Calibrator on DeiT-T · ImageNet-1K | Top-1 accuracy: 72.1 -> 73.1 | Helped |
Surprising findings
- Calibrated models often perform lower in practical application, according to the authors, despite dimension calibration yielding large gains in their analysis.
- Setting the feedback target layer to the starting shallow layer lowers accuracy, while deeper targets help.
Limitations
- Gains are limited for large-parameter models and with more attention heads.
- Feedback module adds training and inference overhead.
- Calibrated models sometimes perform lower in practice; category-aware dimension selection needs refinement.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 ImageNet-Real
Benchmarks: CIFAR-100 ImageNet-1K ImageNet-Real
Metrics: top-1 accuracy top-5 accuracy
Baselines: ViT DeiT FlexiViT EVA
Builds on: ViT DeiT attention rollout layer-wise relevance propagation Swin Transformer
Closest papers by meaning
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.86)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.85)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.85)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.85)
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.85)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.