Research map / Large language models / Instruction and fine-tuning
Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoPA, an instance-aware soft prompt method that gates a task-specific soft prompt with a low-rank instance-specific prompt, matching LoRA and full fine-tuning on GLUE and code tasks without server-side adapters.
Problem
Prompt tuning is parameter-efficient and needs no server-side modification, but it underperforms LoRA and other PEFT methods, so a prompt-based customization method that stays competitive is needed.
Approach
Builds the soft prompt as a sigmoid-gated Hadamard product of a shared task-specific prompt and an instance-specific prompt, where the instance component is produced by two MLP heads on an encoder's output and factorized as a low-rank product u v^T to save parameters.
Key results
- On GLUE with RoBERTa-355M, LoPA averages 90.53 vs 91.26 for LoRA and 91.35 for FFT, and 61.91 for prompt tuning, with 1.60M tunable parameters.
- LoPA outperforms LoRA in 11 of 24 model-task cases, and on average is outperformed by LoRA by only 0.54% relative.
- On CruxEval-I with DeepSeek-Coder-7B, LoPA reaches 50.0 pass@1 versus 47.5 for LoRA; LoPA beats prompt tuning on CruxEval-O by 8-11 points on larger models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoPA instead of standard prompt tuning | RoBERTa · GLUE average | average score: 61.91 -> 90.53 | Helped |
| Use LoPA instead of S-IDPG | RoBERTa · GLUE average | average score: 85.17 -> 90.53 | Helped |
| Use LoPA instead of LoRA | RoBERTa · GLUE average | average score: 91.26 -> 90.53 | Hurt |
| Use LoPA instead of LoRA | code models · CruxEval-I | pass@1 | Mixed |
| Use LoPA instead of LoRA | code models · CruxEval-O | pass@1 | Mixed |
| Increase LoPA soft-prompt length (ablation) | LoPA · GLUE and CruxEval-O | performance | No clear effect |
| Increase low-rank decomposition rank from 1 to 4 (ablation) | LoPA · SST-2 | performance: 1% to 2 % | Helped |
| Increase low-rank decomposition rank from 1 to 4 (ablation) | LoPA · MRPC | performance: up to 8% | Helped |
| Increase low-rank decomposition rank (ablation) | LoPA · CruxEval | performance | No clear effect |
| Use CodeSAGE instead of CodeBERT as the LoPA encoder (ablation) | LoPA with DeepseekCoder-1.3B · CruxEval | pass@1: 2-point improvement | Helped |
| Tune the encoder while learning soft prompts (ablation) | LoPA · CruxEval | performance | No clear effect |
| Replace gated combination with concatenation of task- and instance-specific prompts (ablation) | LoPA · GLUE average | average score: 90.53 -> 83.44 | Hurt |
Surprising findings
- IDPG (instance-only soft prompt) performs worse than standard prompt tuning on several code models, e.g. CodeGen and DeepSeek-Coder-1.3B on CruxEval.
- Simple concatenation of task and instance prompt components performs worst (83.44 avg) versus max or gating (88.68 and 90.53).
Limitations
- Effectiveness shown only on benchmark tasks; may not hold for obscure real-world user tasks where LoRA or full fine-tuning is needed.
- Only prefix placement of the soft prompt was studied; suffix or random placement left to future work.
- Code benchmarks have only a few hundred samples, so higher rank tends to overfit rather than improve.
Experimental setup
Datasets: GLUE SST-2 MNLI MRPC QNLI QQP RTE MBPP CruxEval
Benchmarks: GLUE MBPP CruxEval-I CruxEval-O
Metrics: accuracy F1 pass@1
Baselines: full fine-tuning LoRA p-tuning-v2 prefix-tuning prompt tuning DePT S-IDPG
Builds on: prompt tuning IDPG LoRA DePT prefix-tuning
Closest papers by meaning
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.87)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.87)
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.87)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.86)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.86)
- Batched Low-Rank Adaptation of Foundation Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.