Research map / Architectures and efficiency / Pruning and efficient tuning
Understanding Prompt Tuning and In-Context Learning via Meta-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses a Bayesian meta-learning view to analyze when prompt (prefix) tuning can reach optimal prediction, and tests hard, simplex, real and soft prefixes against weight-tuning on small LSTMs and Transformers over coin-flip tasks.
Problem
Prompting and in-context learning lack a conceptual understanding; it is unclear when prompt optimization can make a pretrained model Bayes-optimal on a target task and when weight tuning is needed.
Approach
Frames prompt tuning as conditioning a meta-trained Bayesian sequential predictor, derives conditions (single pretraining-supported task vs. mixtures or novel atomic tasks) under which optimal prompting is or is not possible, and empirically compares four prefix-tuning and five weight-tuning methods on coin-flip data.
Key results
- On the Single Coin target, SoftPT lets a pretrained Transformer and LSTM reach Bayes-optimal performance, while HardPT, SimplexPT and RealPT do not.
- On the Two-Coin Mixture target, no prefix-tuning method reaches TargetBayes performance, while some weight-tuning methods do.
- Soft prompting an untrained Transformer gets close to optimal on Two-Coin Mixture and Random Coins, but soft prompting an untrained LSTM has very little effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SoftPT instead of hard-token or lower-dimensional prefix tuning | Pretrained Transformer and LSTM · Single coin (bias 0.2) | cumulative regret | Helped |
| Use SoftPT instead of hard-token prefix tuning | Pretrained Transformer and LSTM · Two-coin mixture | cumulative regret | Helped |
| Use SoftPT rather than other prefix-tuning methods | Pretrained Transformer and LSTM · Two-coin mixture | cumulative regret | Hurt |
| Tune weights rather than only a prefix | Pretrained Transformer and LSTM · Two-coin mixture | cumulative regret | Helped |
| Use SoftPT instead of leaving the network untuned | Untrained Transformer · Two-coin mixture | cumulative regret | Helped |
| Use SoftPT instead of leaving the network untuned | Untrained Transformer · Random coins | cumulative regret | Helped |
| Use SoftPT instead of leaving the network untuned | Untrained LSTM · Two-coin mixture and random coins | prediction performance | No clear effect |
| Increase SoftPT embedding dimensionality from 4 to 128 (ablation) | Prefix tuning · Coin-flip tasks | prediction performance | Helped |
| Increase prefix length from 6 to 25 (ablation) | SoftPT · Two-coin mixture | prediction performance | No clear effect |
| Increase prefix length from 6 to 25 (ablation) | Prefix tuning on pretrained networks · Two-coin mixture | cumulative regret | No clear effect |
Surprising findings
- Soft prefixes of a 128-dimensional embedding outperform even the best exhaustive hard-token prefix of the same length, and the advantage largely disappears at embedding dimensionality 4.
- Soft prompting can make an untrained Transformer behave as a well-performing sequential predictor.
Limitations
- Experiments use tiny networks and simple coin-flip data, so findings may not extrapolate to frontier-scale LLMs.
- Soft prefix results depend on embedding dimensionality being much larger than input dimensionality, which is typically reversed at LLM scale.
- The paper does not propose improved prompt tuning methods; it studies existing ones.
Experimental setup
Datasets: synthetic coin-flip sequences (Random Coins, Single Coin, Two-Coin Mixture)
Benchmarks: synthetic coin-flip sequences (Single Coin, Two-Coin Mixture)
Metrics: cumulative regret (excess log loss vs. data generator)
Baselines: HardPT SimplexPT RealPT EmbedWT UnembedWT FullWT LoRAWT NoTuning RandomPF TargetBayes
Builds on: soft prompting LoRA prefix-tuning
Closest papers by meaning
- When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations (ICLR 2024 · similarity 0.91)
- Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (ICLR 2025 · similarity 0.89)
- System Prompt Optimization with Meta-Learning (NeurIPS 2025 · similarity 0.88)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.88)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.88)
- BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.