Research map / Architectures and efficiency / Attention and transformers
Memory Efficient Transformer Adapter for Dense Predictions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes META, a memory-efficient ViT adapter that shares layer normalization between attention and FFN branches, uses cross-shaped self-attention to reduce reshaping, and adds a lightweight conv branch and cascaded heads for dense prediction.
Problem
ViT adapters achieve good accuracy on dense prediction but are slowed at inference by memory-access operations such as standard normalization and frequent reshaping.
Approach
An adapter block that shares one layer normalization between parallel attention and FFN branches, uses cross-shaped (horizontal/vertical stripe) self-attention to avoid frequent reshapes, adds a parallel depthwise-conv branch for local inductive bias, and cascades heads across injectors/extractors.
Key results
- META-B with Mask R-CNN 3x+MS on MS-COCO reaches 44.3 AP^m and 51.2 AP^b with 115.3M params, versus ViT-Adapter-B at 43.6 AP^m with 120.2M params.
- META reports 8.1 GB memory consumption versus 15.2 GB for ViT-Adapter and 13.0 GB for LoSA in the detection setting.
- On ADE20K with UperNet 160k and ViT-B (IN-1k), META-B reaches 49.4 mIoU versus 48.8 for ViT-Adapter-B (single scale).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use only the Attn branch as the adapter (ablation) | ViT-B adapter · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 41.3 -> 32.1 | Hurt |
| Add the FFN branch to the Attn branch (ablation) | ViT-B adapter · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 41.3 -> 43.4 | Helped |
| Use only the FFN branch as the adapter (ablation) | ViT-B adapter · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 41.3 -> 30.8 | Hurt |
| Add the Conv branch to Attn + FFN (ablation) | ViT-B adapter · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 43.4 -> 44.1 | Helped |
| Add the cascaded mechanism to Attn + FFN (ablation) | ViT-B adapter · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 43.4 -> 43.6 | Helped |
| Add the Conv branch and cascaded mechanism (ablation) | ViT-B adapter · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 43.4 -> 44.3 | Helped |
| Share layer normalization between Attn and FFN (ablation) | ViT-B adapter with Attn + FFN · MS-COCO val, Mask R-CNN 3× +MS | APᵐ: 43.2 -> 43.4 | No clear effect |
| Use only the Attn branch as the adapter (ablation) | ViT-B adapter · ADE20K val, UperNet 160k | mIoU: 46.1 -> 33.7 | Hurt |
| Add the FFN branch to the Attn branch (ablation) | ViT-B adapter · ADE20K val, UperNet 160k | mIoU: 46.1 -> 46.8 | Helped |
| Use only the FFN branch as the adapter (ablation) | ViT-B adapter · ADE20K val, UperNet 160k | mIoU: 46.1 -> 31.1 | Hurt |
| Add the Conv branch to Attn + FFN (ablation) | ViT-B adapter · ADE20K val, UperNet 160k | mIoU: 46.8 -> 47.9 | Helped |
| Add the cascaded mechanism to Attn + FFN (ablation) | ViT-B adapter · ADE20K val, UperNet 160k | mIoU: 46.8 -> 48.0 | Helped |
Surprising findings
- Using the attention branch alone as the adapter drops AP^m from 41.3 to 32.1, and the FFN branch alone also degrades accuracy.
- Sharing layer normalization barely changes AP (43.4 vs 43.2 non-shared) but improves FPS and memory.
Limitations
- Theoretical analysis (information entropy and MMD argument) is informal and relies on loosely justified case analysis.
- Cascaded mechanism increases FLOPs and memory, per the authors' ablation.
- Evaluated only on plain ViT backbones and two datasets (COCO, ADE20K).
Experimental setup
Datasets: MS-COCO ADE20K ImageNet-1k ImageNet-22k
Benchmarks: MS-COCO ADE20K
Metrics: AP^b AP^m mIoU FLOPs #Params memory consumption FPS
Baselines: ViT-Adapter LoSA AdaptFormer FacT-TK ViTDet Swin Transformer UperNet Semantic FPN
Builds on: ViT-Adapter Cross-Shaped Window Attention ViT Mask R-CNN
Closest papers by meaning
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.88)
- ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions (CVPR 2024 · similarity 0.87)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.87)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.86)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.86)
- Your ViT is Secretly an Image Segmentation Model (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.