Research map / Architectures and efficiency / Pruning and efficient tuning
ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADePT, a parameter-efficient prompt tuning method that pairs a short soft prompt with a shallow token-shared feed-forward network producing input-dependent token embedding offsets, outperforming PT, DePT and other PEFT methods on 23 NLP tasks across T5-base, T5-3B, CodeGen-350M and Llama3-8B.
Problem
Position-based token embedding offsets in DePT make the same token embed differently depending on position and keep offsets tiny, limiting generalization and optimization of decomposed prompt tuning for PLM adaptation.
Approach
Replace DePT's position-indexed low-rank offset matrix with a shallow two-layer MLP shared across all tokens that maps each token embedding to its own offset, so offsets are input-adaptive and position-independent, while keeping a short soft prompt for fast inference and a parameter count matched to PT.
Key results
- On T5-base, ADePT reaches 86.6 GLUE mean and 77.4 SuperGLUE mean with 76.1K parameters, versus 84.8 and 60.0 for PT and 85.9 and 76.5 for DePT.
- On T5-3B over six large tasks, ADePT averages 79.2 F1/accuracy versus 79.0 for PT and 78.4 for DePT, below LoRA's 80.7.
- On MBPP with Llama3-8B, ADePT gets 43.22 pass@1 versus 42.50 for DePT and 18.27 for PT, but LoRA reaches 49.08.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace position-based DePT offsets with token-adaptive ADePT offsets | T5-base · GLUE mean | score: 85.9 -> 86.6 | Helped |
| replace position-based DePT offsets with token-adaptive ADePT offsets | T5-base · SuperGLUE mean | score: 76.5 -> 77.4 | Helped |
| replace position-based DePT offsets with token-adaptive ADePT offsets | T5-base · MRQA mean | F1: $73.0$ -> ${73.6} | Helped |
| replace position-based DePT offsets with token-adaptive ADePT offsets | T5-base · Other datasets mean | accuracy: $86.3$ -> ${86.4} | No clear effect |
| add token offsets to the soft-prompt-only ADePT variant (ablation) | ADePT · RTE | score: 58.3 -> 82.0 | Helped |
| add token offsets to the soft-prompt-only DePT variant (ablation) | DePT · RTE | score: 78.4 -> 79.1 | Helped |
| set ADePT bottleneck size to 5 instead of 10 or 20 (ablation) | ADePT · RTE | score: 82.0 -> 78.4 | Hurt |
| set ADePT bottleneck size to 30 instead of 10 or 20 (ablation) | ADePT · RTE | score: 82.0 -> 79.9 | Hurt |
| vary ADePT soft prompt length from 20 to 80 (best at 60) (ablation) | ADePT · RTE | score: 72.7; 77.7; 79.1; 82.0; 80.6; 79.1 | Mixed |
| remove the soft prompt and use only the feed-forward network (ablation) | ADePT · RTE | score: 82.0 -> 73.4 | Hurt |
| fine-tune the prompt and embedding matrix instead of ADePT (ablation) | T5-base · RTE | score: 82.0 -> 76.3 | Hurt |
| cyclically left-shift DePT offsets by different amounts (ablation) | DePT · RTE | accuracy: 79.1 -> 78.4; 79.1 -> 79.9 | Mixed |
Surprising findings
- Fine-tuning prompt and embedding matrix (24.8M params) underperforms ADePT on RTE (76.3 vs 82.0).
- ADePT without token offsets (soft prompt only) drops to 58.3 on RTE, showing the offset network is critical, while the paper also reports ADePT can fail on few-shot CB (32.1 at 4-shot) and GSM8K instruction tuning.
Limitations
- Underperforms LoRA on T5-3B and MBPP with Llama3-8B despite fewer parameters.
- Weak in few-shot settings (e.g., CB) and failed to improve GSM8K instruction tuning on Llama3-8B, per the authors' unsuccessful attempts.
- Per-token offset computation adds real-time latency relative to DePT's fixed offsets.
Experimental setup
Datasets: MNLI QQP QNLI SST-2 STS-B MRPC RTE CoLA MultiRC BoolQ WiC WSC
Benchmarks: GLUE SuperGLUE MRQA 2019 WinoGrande Yelp-2 SciTail PAWS-Wiki MBPP ReCoRD
Metrics: accuracy F1 Pearson correlation exact match pass@1 inference samples per second
Baselines: Full fine-tuning Prompt Tuning DePT Adapter AdapterDrop BitFit LoRA LST SPoT ATTEMPT MPT HyperFormer
Builds on: Prompt Tuning DePT LoRA Residual Prompt Tuning
Closest papers by meaning
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.93)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.91)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.90)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.88)
- Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models (NeurIPS 2024 · similarity 0.88)
- No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.