Research map / Architectures and efficiency / Pruning and efficient tuning
DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DePT, which decomposes a soft prompt into a shorter soft prompt plus a low-rank update added to frozen word embeddings, trained with two learning rates, to cut prompt-tuning time and memory at equal trainable parameter count.
Problem
Prompt tuning appends extra soft-prompt tokens that lengthen input sequences, increasing training/inference time and memory due to Transformer quadratic cost, and it converges slowly and is sensitive to initialization.
Approach
Decompose the soft prompt into a shorter soft prompt (m tokens) and a low-rank pair (B A) that updates the frozen word embeddings element-wise, keeping parameter count equal to vanilla PT while shortening the input sequence; the two components are optimized with different learning rates.
Key results
- On GLUE with T5-Base, DePT averages 85.9 versus 84.8 for vanilla PT at the same 76.8k trainable parameters, and exceeds full fine-tuning's 84.9.
- DePT reduces training time and memory by about 25% for T5 on GLUE (37% on SST-2 with T5-3B), and inference speed gains grow with model size (6.5% for T5-Small vs 18.1% for T5-Large).
- In few-shot CB with k=4, DePT with transfer learning reaches 75.0% vs 53.5% for vanilla PT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla prompt tuning with DePT | T5-base · GLUE | mean score: 84.8 -> 85.9 | Helped |
| Replace vanilla prompt tuning with DePT | T5-base · SuperGLUE | mean score: 60.0 -> 76.5 | Helped |
| Replace vanilla prompt tuning with DePT | T5-base · MRQA 2019 Shared Task | mean F1: +3.6% | Helped |
| Replace vanilla prompt tuning with DePT | T5-base · Other NLP datasets | mean accuracy: +14.2% | Helped |
| Use DePT instead of full fine-tuning | CLIP-T5 · VQA Karpathy test | accuracy (%): 67.1$_{0.1}$ -> 59.8$_{0.4}$ | Hurt |
| Replace vanilla prompt tuning with DePT | CLIP-T5 · MSCOCO Karpathy test | CIDEr: 112.2$_{0.3}$ -> 113.7$_{0.3}$ | Helped |
| Decompose the prompt and shorten it to m=20 | T5-small · GLUE inference | samples per second: 167.3 -> 178.3 | Helped |
| Decompose the prompt and shorten it to m=20 | T5-large · GLUE inference | samples per second: 21.0 -> 24.8 | Helped |
| Use mixed learning rates: larger for the soft prompt and lower for low-rank matrices (ablation) | DePT · GLUE | average performance: 40.8 / 54.7 -> 85.7 | Helped |
| Remove the short soft prompt, setting m=0 (ablation) | DePT · GLUE | average performance | Hurt |
| Add PETL transfer learning to DePT | DePT · SuperGLUE CB, 4-shot | accuracy: 53.5% -> 75.0% | Helped |
| Add PETL transfer learning to DePT | DePT · SciTail, 4-shot | accuracy: 57.7% -> 78.1% | Helped |
Surprising findings
- Using only the low-rank embedding update without a soft prompt (m=0) causes a notable performance drop.
- Vanilla prompt tuning reaches only 40.8 average on GLUE with a single high learning rate, showing that two learning rates are essential for DePT.
Limitations
- Introduces extra hyperparameters (two learning rates, m and r), raising hyperparameter-search cost.
- Number of trainable parameters depends on maximum sequence length s, so it may scale poorly for very long inputs.
- Still lags full fine-tuning on some tasks such as VQA and MRQA.
Experimental setup
Datasets: GLUE SuperGLUE MRQA 2019 WinoGrande Yelp-2 SciTail PAWS-Wiki VQA MSCOCO SQuAD ReCoRD
Benchmarks: GLUE SuperGLUE MRQA 2019 WinoGrande Yelp-2 SciTail PAWS-Wiki VQA MSCOCO
Metrics: accuracy F1 Pearson correlation CIDEr training time memory inference samples per second
Baselines: Fine-tuning Adapter AdapterDrop BitFit LoRA LST PT SPoT ATTEMPT MPT HyperFormer HyperDecoder
Builds on: prompt tuning LoRA ATTEMPT SPoT
Closest papers by meaning
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.93)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.93)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.89)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.88)
- Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models (NeurIPS 2024 · similarity 0.88)
- Soft Prompt Recovers Compressed LLMs, Transferably (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.