Research map / Architectures and efficiency / Attention and transformers
SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRepBN, which progressively replaces LayerNorm with re-parameterized BatchNorm during training, plus a simplified linear attention (ReLU kernel + depth-wise conv) to build efficient vision and language transformers.
Problem
Transformers are costly to deploy because LayerNorm needs online statistics at inference and softmax attention is quadratic in token count; naively swapping LayerNorm for BatchNorm causes inferior accuracy or training collapse.
Approach
Progressively transition from LayerNorm to a re-parameterized BatchNorm (RepBN = BN(X) + ηX, mergeable into BN at inference) via a linearly decaying mixing coefficient, and replace softmax attention with a simplified ReLU-kernel linear attention plus a depth-wise convolution branch.
Key results
- SLAB-Swin-S reaches 83.6% top-1 on ImageNet-1K at 16.2 ms latency, 2.4 ms faster than Flatten-Swin-S at 0.1% higher accuracy (per paper: Flatten-Swin-S 83.5%, 18.6 ms).
- PRepBN on DeiT-T improves accuracy from 72.2% (LN) to 73.6% with throughput 3432 to 4194 images/s.
- On Wikitext-103 with Adaptive Inputs, PRepBN lowers inference time from 13.9 to 12.9 ms/token with similar perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LayerNorm with PRepBN | DeiT-S · ImageNet-1K | top-1 accuracy: 78.8 -> 80.2 | Helped |
| Replace LayerNorm with PRepBN | Swin-T · ImageNet-1K | top-1 accuracy: 81.3 -> 81.4 | Helped |
| Replace Flatten-Swin-S with SLAB-Swin-S | Swin-S · ImageNet-1K | latency: 18.6 -> 16.2 | Helped |
| Replace Flatten-Swin-S with SLAB-Swin-S | Swin-S · ImageNet-1K | top-1 accuracy: 83.5% -> 83.6% | Helped |
| Add simplified linear attention to Flatten-DeiT-T (ablation) | Flatten-DeiT-T · ImageNet-1K | latency: 15.2 -> 10.2 | Helped |
| Add simplified linear attention to Flatten-DeiT-T (ablation) | Flatten-DeiT-T · ImageNet-1K | accuracy: 74.1 -> 73.0 | Hurt |
| Add PRepBN after simplified linear attention (ablation) | Flatten-DeiT-T + SLA · ImageNet-1K | accuracy: 73.0 -> 74.3 | Helped |
| Add PRepBN after simplified linear attention (ablation) | Flatten-DeiT-T + SLA · ImageNet-1K | latency: 10.2 -> 9.6 | Helped |
| Add simplified linear attention to Flatten-PVT-T (ablation) | Flatten-PVT-T · ImageNet-1K | accuracy: 77.8 -> 75.2 | Hurt |
| Add PRepBN after simplified linear attention (ablation) | Flatten-PVT-T + SLA · ImageNet-1K | accuracy: 75.2 -> 76.5 | Helped |
| Add progressive LayerNorm-to-BatchNorm strategy (ablation) | DeiT-T-BN · ImageNet-1K | accuracy: 71.9 -> 73.1 | Helped |
| Add RepBN to the progressive BatchNorm model (ablation) | DeiT-T-BN + Progressive Strategy · ImageNet-1K | accuracy: 73.1 -> 73.6 | Helped |
Surprising findings
- Directly training BatchNorm-based transformers causes inferior performance or training collapse (e.g., DeiT-S and Flatten-Swin-T).
Limitations
- Latency measured only on a single V100 GPU, so gains may not transfer to other hardware.
- Some SLAB variants trail Flatten baselines slightly (e.g., SLAB-DeiT-S 80.0% vs Flatten-DeiT-S 80.4%).
Experimental setup
Datasets: ImageNet-1K COCO Wikitext-103
Benchmarks: ImageNet-1K COCO Wikitext-103
Metrics: top-1 accuracy throughput latency (ms) FLOPs AP perplexity average accuracy over LM benchmarks
Baselines: LayerNorm transformers BN+FFNBN Flatten Transformer Swin Transformer PVT Adaptive Inputs
Builds on: RepVGG Flatten Transformer Swin Transformer DeiT
Closest papers by meaning
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.89)
- Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers (NeurIPS 2023 · similarity 0.88)
- Improving Transformer Optimization Through Better Initialization (ICML 2020 · similarity 0.88)
- The Need for Speed: Pruning Transformers with One Recipe (ICLR 2024 · similarity 0.87)
- Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.86)
- Impact of Layer Norm on Memorization and Generalization in Transformers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.