Research map / Architectures and efficiency / Attention and transformers

Bi-ViT: Pushing the Limit of Vision Transformer Quantization

AAAI 2024 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Bi-ViT, a fully-binarized vision transformer that adds learnable head-wise scaling factors in self-attention and a ranking-aware distillation loss, reaching large ImageNet gains over Bi-Real Net baselines on DeiT and Swin.

Problem

Fully binarizing vision transformers (weights and activations to 1-bit) causes severe accuracy collapse, traced to attention distortion in binarized self-attention from gradient vanishing and ranking disorder of attention scores.

Approach

Reactivate vanished gradients in binarized self-attention with learnable head-wise scaling factors that set the STE clip range, and rectify attention ranking disorder by distilling the relative order of the real-valued teacher's attention maps into the binarized student.

Key results

• Bi-ViT with DeiT-Tiny reaches 28.7% Top-1 on ImageNet (1-bit), a 22.1-point gain over the Bi-Real Net baseline at 6.6%.
• With Swin-Tiny, Bi-ViT outperforms the Bi-Real Net baseline by 21.4 points (Top-1 79.4 vs. 55.5 reported in table as 1-bit result of 55.5+21.4, i.e., baseline 34.1).
• Ablation on DeiT-Tiny: learnable scaling factor alone gives 24.4% and ranking-aware distillation alone gives 12.5%, with both combined giving 28.7%.

Measured effects reported in the paper

Surprising findings

• Binarizing only the MHSA module (w1a1) collapses accuracy to 4.4% even with MLP kept at full precision, while binarizing the MLP alone keeps 26.3% accuracy.

Limitations

• Accuracy of fully-binarized ViTs remains far below real-valued counterparts (e.g., 28.7% vs 72.2% for DeiT-Tiny).
• Speedups are theoretical OPs counts rather than measured on hardware.
• No publicly available fully-binarized ViT codebase existed, so baselines were re-implemented by the authors.

Experimental setup

Datasets: ImageNet ILSVRC12

Benchmarks: ImageNet ILSVRC12 COCO object detection with Mask R-CNN and Cascade Mask R-CNN (Swin-Tiny backbone)

Metrics: Top-1 accuracy Top-5 accuracy OPs model size

Baselines: Bi-Real Net RBONN BiBERT Q-ViT

Builds on: Bi-Real Net ReActNet DeiT Swin Transformer BiBERT

Closest papers by meaning

• GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.91)
• MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.89)
• AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.89)
• RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.89)
• Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.89)
• GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.89)