Research map / Architectures and efficiency / Attention and transformers
BVT-IMA: Binary Vision Transformer with Information-Modified Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BVT-IMA, which corrects the attention maps of binarized vision transformers with a small learnable information table indexed by Hamming distance, recovering information lost from constant token moduli.
Problem
Binarized vision transformers suffer large accuracy drops, partly because binary self-attention shifts highlighted attention regions away from the correct tokens.
Approach
Binary query/key tokens have constant moduli, so attention scores lose information quantity; the paper models attention scores as Bernoulli trials and replaces the lost information with learnable per-head modification factors indexed by the Hamming distance (d+1 factors), applied via lookup at inference with the same float operations.
Key results
- On CIFAR-100 with DeiT-Small, BVT-IMA reaches 75.77% top-1 vs 71.10% for GSB and 66.41% for BiT at 1-1 bit widths.
- On ImageNet-1k with DeiT-Base, BVT-IMA reaches 62.65% top-1 vs 38.19% for BiT (1-1) and 74.06% vs 69.60% for BiViT (1-1/32).
- On TinyImageNet with Swin-Tiny, BVT-IMA reaches 60.85% vs 58.66% for BiViT; with NesT-Tiny it reaches 64.23% vs 52.21% for BiViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add information-modified attention | binary DeiT-Tiny · CIFAR-100 | Top1 accuracy: 43.59% -> 62.46% | Helped |
| add information-modified attention | binary DeiT-Small · CIFAR-100 | Top1 accuracy: 66.41% -> 75.77% | Helped |
| add information-modified attention | binary DeiT-Tiny · TinyImageNet | Top1 accuracy: 24.32% -> 39.67% | Helped |
| add information-modified attention | binary DeiT-Small · TinyImageNet | Top1 accuracy: 38.77% -> 43.42% | Helped |
| add information-modified attention | binary DeiT-Tiny · ImageNet-1k | Top1 accuracy: 21.68% -> 30.03% | Helped |
| add information-modified attention | binary DeiT-Small · ImageNet-1k | Top1 accuracy: 30.73% -> 47.98% | Helped |
| add information-modified attention | binary DeiT-Base · ImageNet-1k | Top1 accuracy: 38.19% -> 62.65% | Helped |
| add information-modified attention | binary Swin-Tiny · TinyImageNet | Top1 accuracy: 58.66% -> 60.85% | Helped |
| add information-modified attention | binary NesT-Tiny · TinyImageNet | Top1 accuracy: 52.21% -> 64.23% | Helped |
| add information-modified attention | binary Swin-Tiny, activations 1-1/32 · ImageNet-1k | Top1 accuracy: 70.80% -> 72.03% | Helped |
| add information-modified attention | binary NesT-Tiny, activations 1-1/32 · ImageNet-1k | Top1 accuracy: 68.70% -> 71.00% | Helped |
| initialize the information table with the Bernoulli prior rather than all factors set to 1 (ablation) | DeiT-Tiny with information-modified attention · CIFAR-100 | Top1 accuracy: 46.04% -> 62.46% | Helped |
Surprising findings
- Naively migrating NLP binarization methods (BiBERT, BiT) to vision transformers can drop accuracy severely, and BiBERT fails to converge on NesT-Tiny for ImageNet-1k.
Limitations
- Binarized attention still underperforms real-valued ViTs by large margins (e.g., 43.99% vs 74.48% on ImageNet-1k DeiT-Tiny)
- Information table initialization uses a heuristic hyper-parameter m and an approximation that sets an uncertain term to 1
- Evaluation covers only image classification on three benchmarks
Experimental setup
Datasets: CIFAR-100 TinyImageNet ImageNet-1k
Benchmarks: CIFAR-100 TinyImageNet ImageNet-1k
Metrics: Top-1 accuracy BOPs FLOPs OPs consistent relative magnitude ratio
Baselines: BiT GSB Q-ViT BiBERT BiViT
Builds on: BiT BiBERT LSQ DeiT
Closest papers by meaning
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.88)
- MLP Can Be A Good Transformer Learner (CVPR 2024 · similarity 0.85)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.85)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.84)
- MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.84)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.