Research map / Architectures and efficiency / Pruning and efficient tuning
EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EPT, an efficient prompt tuning method that decomposes a soft prompt into a shorter prompt plus two low-rank matrices, fuses them via attention, and projects the prompt into multiple gated subspaces, evaluated on 13 GLUE/SuperGLUE tasks with T5.
Problem
Prompt tuning faces a trade-off between accuracy and training efficiency (long soft prompts are accurate but slow), and its performance is inconsistent across downstream tasks with different requirements.
Approach
Decompose the soft prompt into a short trainable prompt and two low-rank matrices (initialized via SVD of the original prompt) that update the frozen input embeddings; fuse short prompt with low-rank knowledge via Einstein-summation attention; and project the short prompt into multiple subspaces combined by a gating network.
Key results
- On T5-Base, EPT reaches 86.8 mean on GLUE and 77.5 mean on SuperGLUE, outperforming vanilla PT (84.3 / 67.3) and DEPT (85.5 / 76.2) with the same 76.8K trainable parameters.
- EPT reduces training time by 14% compared with vanilla PT at short prompt length 60 on GLUE.
- On T5-3B, EPT achieves an 81.5 mean across the 8 SuperGLUE tasks shown versus 77.2 for PT and 80.6 for DEPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prompt decomposition to vanilla PT (ablation) | EPT · GLUE | mean score (%): 84.3 -> 85.8 | Helped |
| Add prompt decomposition to vanilla PT (ablation) | EPT · SuperGLUE | mean score (%): 67.3 -> 76.3 | Helped |
| Add multi-space projection to prompt decomposition (ablation) | EPT · GLUE | mean score (%): 85.8 -> 86.4 | Helped |
| Add multi-space projection to prompt decomposition (ablation) | EPT · SuperGLUE | mean score (%): 76.3 -> 77.1 | Helped |
| Add prompt fusion to prompt decomposition (ablation) | EPT · GLUE | mean score (%): 85.8 -> 86.5 | Helped |
| Add prompt fusion to prompt decomposition (ablation) | EPT · SuperGLUE | mean score (%): 76.3 -> 76.8 | Helped |
| Add multi-space projection and prompt fusion to prompt decomposition (ablation) | EPT · GLUE | mean score (%): 85.8 -> 86.8 | Helped |
| Add multi-space projection and prompt fusion to prompt decomposition (ablation) | EPT · SuperGLUE | mean score (%): 76.3 -> 77.5 | Helped |
Surprising findings
- Full fine-tuning is best on some datasets (MNLI, QQP, SST-2), so EPT does not dominate all tasks.
- Using a random Gaussian/zero initialization for the low-rank matrices, as in DEPT, loses prompt information, motivating SVD-based initialization from the original prompt.
Limitations
- Requires tuning two learning rates for soft prompt and low-rank matrices, adding search overhead (acknowledged by authors).
- Full fine-tuning remains best on several large datasets such as MNLI and QQP.
- Results for PT variants on larger models (T5-11B, Llama2-7B) are shown mainly in figures with less pronounced gains.
Experimental setup
Datasets: MNLI QQP QNLI SST-2 STS-B MRPC RTE CoLA MultiRC BoolQ WiC WSC
Benchmarks: GLUE SuperGLUE
Metrics: accuracy F1 Pearson correlation Matthews correlation training time
Baselines: Fine-tuning LoRA Adapter AdapterDrop AdaMix BitFit PT SPoT ATTEMPT MPT DEPT DPT
Builds on: prompt tuning DEPT LoRA ATTEMPT MPT
Closest papers by meaning
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.93)
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.91)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.89)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.89)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.88)
- Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.