Research map / Architectures and efficiency / Attention and transformers

RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers

ICML 2025 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RePaViT, which keeps most FFN channels idle (bypassing GELU) so the two FFN projections can be structurally reparameterized into smaller linear layers at inference, yielding large speedups on ViTs with accuracy gains on large models.

Problem

ViT inference latency is dominated increasingly by FFN layers as model size grows, and existing efficient-ViT methods (token pruning/merging, hybrids, pruning) do not reparameterize FFN layers, so FFN latency remains a bottleneck for large ViTs.

Approach

A channel idle mechanism keeps a large fraction of FFN hidden channels bypassing the nonlinearity during training; BatchNorm replaces LayerNorm so normalizations and the linear shortcut can be folded into weights post-training, turning the two FFN projections plus shortcut into three smaller linear transforms at test time.

Key results

• RePa-ViT-Huge/0.75 reduces parameters by 41.5% and raises throughput by 68.7% while top-1 accuracy goes from 80.3% to 81.4%.
• RePa-ViT-Large/0.75 gives +66.8% speed and +1.7% top-1 (80.3% to 82.0%) versus the vanilla ViT-Large.
• RePa-DeiT-Base/0.75 gives +57.8% speed with a 0.5% accuracy drop (81.8% to 81.3%).

Measured effects reported in the paper

Surprising findings

• Large RePaViT variants (ViT-Large, ViT-Huge) are more accurate than their vanilla backbones despite roughly 41% fewer parameters.
• Training directly with the reparameterized (reduced) structure causes training collapse for several large models, so train-time overparameterization is needed.

Limitations

• Accuracy drops noticeably for tiny models (e.g., RePa-DeiT-Tiny loses 2.7% at theta 0.5 and 7.9% at theta 0.75), attributed to reduced nonlinearity.
• Efficiency gains are reported on specific GPUs (A6000) and ViT-Large/Huge results rely on custom training with 4096 batch size on H100s.
• Compared with pruning on DeiT-Small, LPViT achieves higher accuracy and lower GMACs in some settings.

Experimental setup

Datasets: ImageNet-1k MSCOCO ADE20K LAION-400M

Benchmarks: ImageNet-1k MSCOCO ADE20K

Metrics: top-1 accuracy throughput (images/second) GMACs parameters AP mIoU latency

Baselines: DeiT Swin Transformer LV-ViT WDPruning X-Pruner DC-ViT LPViT SLAB

Builds on: RepVGG SLAB ShuffleNetv2 DeiT Swin Transformer

Closest papers by meaning

• Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.88)
• Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
• Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.87)
• Make RepVGG Greater Again: A Quantization-Aware Approach (AAAI 2024 · similarity 0.87)
• RepViT: Revisiting Mobile CNN From ViT Perspective (CVPR 2024 · similarity 0.86)
• RMT: Retentive Networks Meet Vision Transformers (CVPR 2024 · similarity 0.86)