Research map / Architectures and efficiency / Attention and transformers

BHViT: Binarized Hybrid Vision Transformer

CVPR 2025 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 32 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BHViT, a hybrid CNN-ViT architecture with full binarization (multi-scale grouped dilated conv, binary window/global attention, shift-based binary MLP, quantization-decomposed attention binarization, and an L1 regularization loss for weight oscillation) that reaches SOTA among binary ViTs on CIFAR-10, ImageNet-1K, and segmentation.

Problem

Binarizing Vision Transformers causes severe accuracy drops compared with binarized CNNs, due to gradient vanishing through binarized attention, poor token discrimination in binary attention matrices, and weight oscillation under Adam in binary layers.

Approach

Design a binarization-friendly hybrid architecture (grouped dilated convolutions in early stages, windowed multi-head attention with global pooled tokens in late stages), binarize attention via quantization decomposition into multiple binary attention matrices, add shift operations to the binary MLP and full-precision residual shortcuts around Q/K/V, and add an L1 regularization loss on latent wei

Key results

• On ImageNet-1K, BHViT-Small† (full-precision downsampling) reaches 70.1% top-1, versus 49.5% for ReActNet applied to DeiT-Small and 67.7% for BinaryViT.
• On CIFAR-10, BHViT-Small achieves 95.0% top-1 versus 92.8% for ReCU (ResNet-18 1-1 binary).
• On ADE20K, BHViT reaches 65.63% pixAcc and 14.87% mIoU, above ReActNet (62.77%, 9.22%).

Measured effects reported in the paper

Surprising findings

• Applying ReActNet's binarization directly to DeiT-Small yields only about 49.5% ImageNet top-1, far below the same method on ResNet-18 (about 65.5%).

Limitations

• Pure ViT variant is 1.3% worse than the hybrid, and pure CNN performs worst, so gains depend on the hybrid design.
• Binary ViTs underperform binary CNNs on small datasets such as CIFAR-10 due to weaker inductive bias; the paper notes ViTs need large training data.

Experimental setup

Datasets: ImageNet-1K CIFAR-10 ADE20K RS-LVF

Benchmarks: ImageNet-1K CIFAR-10 ADE20K RS-LVF

Metrics: Top-1 accuracy mIoU pixel accuracy mAcc OPs model size

Baselines: ReActNet ReCU IR-Net RBNN ProxConnect++ GSB BiT Bi-ViT BiBERT BVT-IMA BinaryViT BiViT

Builds on: BiReal-Net ReActNet BinaryViT Bi-ViT DeiT GSB

Closest papers by meaning

• Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.83)
• BVT-IMA: Binary Vision Transformer with Information-Modified Attention (AAAI 2024 · similarity 0.83)
• Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing (CVPR 2025 · similarity 0.82)
• ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration (NeurIPS 2024 · similarity 0.82)
• PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.81)
• RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.81)