Research map / Architectures and efficiency / Pruning and efficient tuning
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CoDA with k-to-all attention and reduction r=3 instead of dense Parallel Adapter | T5 v1.1 Base, Large, and XL · MNLI, RTE, BoolQ | accuracy: 86.6 vs 87.1; 72.6 vs 71.5; 76.6 vs 77.9 (Base); +0.2 Δ Avg | Mixed |
| Increase reduction factor from r=3 to r=5 (ablation) | CoDA k-to-all · MNLI, RTE, BoolQ across T5 v1.1 sizes | average accuracy: +0.2 -> −0.6 Δ Avg | Hurt |
| Use k-to-all attention rather than k-to-k attention (ablation) | CoDA · MNLI, RTE, BoolQ across T5 v1.1 sizes | average accuracy: −0.3 -> +0.2 Δ Avg at r=3; −2.1 -> −0.6 Δ Avg at r=5 | Improved |
| Use soft top-k routing instead of independent sigmoid gates (ablation) | CoDA · MNLI, RTE, BoolQ across T5 v1.1 sizes | average accuracy: −2.0 at r=3; −1.6 at r=5 | Improved |
| Use learned soft top-k routing instead of selecting the first k tokens (ablation) | CoDA · MNLI, RTE, BoolQ across T5 v1.1 sizes | average accuracy: −4.4 at r=3; −12.9 at r=5 | Improved |
| Use CoDA with reduction r=3 rather than dense Parallel Adapter | T5 v1.1 models · MNLI, RTE, BoolQ | Transformer computation: 3-5x computation reduction | Improved |
| Pretrain CoDA for more steps (ablation) | CoDA initialized from pretrained dense models · Downstream classification tasks | accuracy | Improved |
| Apply conditional token computation | CoDA versus Parallel Adapter · NLP tasks | inference speed: 2x to 8x | Improved |
| Combine conditional computation with LoRA at reduction r=3 | LoRA · Six language tasks | average score: 84.0 vs 84.0 | No clear effect |
| Combine conditional computation with LoRA at reduction r=5 | LoRA · Six language tasks | average score: 84.0 vs 83.7 | Hurt |
Closest papers by meaning
- Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference (AAAI 2025 · similarity 0.86)
- Efficient Test-Time Adaptation of Vision-Language Models (CVPR 2024 · similarity 0.85)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.85)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.85)
- Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation (NeurIPS 2026 · similarity 0.84)
- READ-PVLA: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
