Research map / Large language models / Instruction and fine-tuning
All You Need is One: Capsule Prompt Tuning with a Single Vector
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Capsule Prompt-Tuning (CaPT), which uses a single learnable vector per layer added to the mean of instance representations to act as an "attention anchor," removing the need for prompt-length grid search.
Problem
Soft prompt tuning relies on grid search for prompt length, uses many prompt tokens, and task-aware soft prompts interact weakly with input tokens because they lack instance-aware information.
Approach
Each layer gets one learnable capsule vector that is added to the mean of the previous layer's outputs (instance semantics), so the prompt carries both instance-aware content and a task-aware vector; the instance-aware token receives strong attention from and gives attention to input tokens ("attention anchor").
Key results
- CaPT reaches 84.03% average SuperGLUE score on T5-Large, above full fine-tuning (83.60%) with 0.003% parameters.
- On T5-Base CaPT scores 77.51% average, versus 76.94% for P-Tuning v2 and 77.85% for full fine-tuning.
- On Llama3.2-1B CaPT averages 67.51% versus 61.06% for P-Tuning v2, and it cuts training time to 1.00x vs 8.37x for P-Tuning v2 on T5-Base.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine instance-aware mean representations with the task-aware capsule vector | CaPT on SuperGLUE · SuperGLUE validation, T5-Large | average score: 83.79 -> 84.03 | Helped |
| combine instance-aware mean representations with the task-aware capsule vector | CaPT on CSQA · CSQA, Llama3.2-1B | score: 45.29 -> 61.60 | Helped |
| avoid prompt-length grid search using a single capsule prompt | CaPT on SuperGLUE · SuperGLUE validation, T5-Base | training time: 8.77× -> 1.00× | Helped |
| use addition rather than projection to integrate the two guidance signals (ablation) | CaPT on SuperGLUE · SuperGLUE validation, T5-Base | average score: 77.64% -> 77.51% | Hurt |
| prepend the mean instance representation as independent tokens instead of using addition (ablation) | CaPT on SuperGLUE · SuperGLUE validation, T5-Base | average score: 77.51% -> 77.44% | Hurt |
| use convolutional feature extraction instead of addition (ablation) | CaPT on SuperGLUE · SuperGLUE validation, T5-Base | average score: 77.51% -> 77.21% | Hurt |
| remove task-aware learnable guidance and use instance-aware signal only (ablation) | CaPT · SuperGLUE validation, T5-Base BoolQ | accuracy: 77.25 -> 79.54 | Hurt |
| remove instance-aware guidance and use one task-aware prompt only (ablation) | CaPT · SuperGLUE validation, T5-Base CB | F1/accuracy: 87.31 -> 94.16 | Hurt |
| increase capsule prompt length (ablation) | CaPT on Llama3.2-1B · SuperGLUE validation | average score: 67.51 -> 59.38 | Hurt |
| apply CaPT at every odd-numbered layer rather than the first half of layers (ablation) | CaPT · SuperGLUE validation, T5-Base | average score: 73.67% -> 75.28% | Helped |
| add a single instance-aware token to standard soft prompts without fine-tuning | T5-Base prompt-tuned model · RTE and COPA | accuracy | Helped |
| increase the number of prepended instance-aware tokens beyond one (ablation) | T5-Base prompt-tuned model · RTE and COPA | accuracy | Hurt |
Surprising findings
- A single instance-aware token prepended without any fine-tuning already improves accuracy over standard deep prompt-tuning on RTE and COPA.
- Increasing the number of capsule prompts beyond one degrades performance, suggesting information quality matters more than quantity.
Limitations
- Extension to other modalities such as vision is not established; the authors note visual inputs lack fixed structural templates.
- Gains are smaller on decoder-only models (Llama3.2-1B, Qwen2.5-1.5B) than on T5 models.
- Results may not generalize to all models and datasets, as the authors acknowledge.
Experimental setup
Datasets: BoolQ CB COPA MRC RTE WiC CSQA
Benchmarks: SuperGLUE CSQA
Metrics: accuracy F1 average score training time parameter fraction
Baselines: Fine-Tuning Prompt-Tuning P-Tuning v2 XPrompt ResPrompt SMoP SuperPos-Prompt VFPT DePT EPT Adapter LoRA
Builds on: P-Tuning v2 Prompt Tuning M-IDPG LoPA
Closest papers by meaning
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.90)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.89)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.89)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.