Research map / Large language models / Language, safety and interpretability
System Prompt Optimization with Meta-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces bilevel system prompt optimization and MetaSPO, a meta-learning framework that alternates inner-loop user prompt refinement with outer-loop system prompt refinement to produce task-agnostic system prompts that generalize to unseen tasks and user prompts.
Problem
Existing prompt optimization focuses on task-specific user prompts and ignores system prompts, which are meant to be task-agnostic; the paper targets designing system prompts robust to diverse user prompts and transferable to unseen tasks and domains.
Approach
Formulate system prompt optimization as a bilevel problem, where user prompts are optimized per task in an inner loop and the system prompt is meta-optimized over many tasks and their optimized user prompts in an outer loop, using LLM-based failure analysis and candidate generation at both levels.
Key results
- In unseen generalization with Domain setting, MetaSPO averages 44.5 vs 37.0 for SPRIG and 32.2 for Default across 14 target tasks (Llama 3.2 3B).
- MetaSPO reaches Default's final test-time adaptation performance with 80% fewer optimization iterations and 75% less data.
- 85.0% of sampled user prompts improve with the MetaSPO system prompt compared to the default system prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MetaSPO instead of SPRIG for global system-prompt optimization | Llama 3.2 (3B) · Unseen Generalization (Global, Avg.) | score: 35.0 -> 41.4 | Helped |
| Use MetaSPO instead of SPRIG for domain-specific system-prompt optimization | Llama 3.2 (3B) · Unseen Generalization (Domain, Avg.) | score: 37.0 -> 44.5 | Helped |
| Apply MetaSPO-optimized system prompts rather than CoT | Llama 3.1 (8B) · Review Analysis and Reasoning (Avg.) | score: 59.6 -> 69.8 | Helped |
| Apply MetaSPO-optimized system prompts rather than CoT | Qwen 2.5 (7B) · Review Analysis and Reasoning (Avg.) | score: 65.5 -> 73.2 | Helped |
| Apply MetaSPO-optimized system prompts rather than Default | GPT-4o mini · Review Analysis and Reasoning (Avg.) | score: 77.2 -> 79.6 | Helped |
| Omit iterative inner-loop user-prompt refinement (ablation) | MetaSPO · Domain averages (Avg.) | score: 42.2 -> 38.2 | Hurt |
| Replace the default prompt-optimization strategy with APE in both loops (ablation) | MetaSPO · Domain averages (Avg.) | score: 42.2 -> 38.9 | Hurt |
| Replace the default prompt-optimization strategy with EVO in both loops (ablation) | MetaSPO · Domain averages (Avg.) | score: 42.2 -> 40.2 | Hurt |
| Increase the number of source tasks from 1 to 6 (ablation) | MetaSPO · Review Analysis domain | performance: +17.10% | Helped |
| Increase the number of source tasks from 1 to 6 (ablation) | MetaSPO · Reasoning domain | performance: +8.26% | Helped |
| Use separated system and user roles instead of concatenating both prompts into the user role (ablation) | MetaSPO prompt structure · All domains except Reasoning | win ratio | Helped |
| Use separated system and user roles instead of concatenating both prompts into the user role (ablation) | MetaSPO prompt structure · Reasoning | win ratio | No clear effect |
Surprising findings
- Cross-model transfer: the system prompt optimized on Llama 3.2 (3B) performs strongly on Llama 3.1 (8B), Qwen 2.5 (7B), and GPT-4o mini without re-optimization.
- Separating system and user roles helps in most domains, but not in reasoning, where the unified input performed better.
Limitations
- Performance depends on the capability of the optimizer LLM; results rely on GPT-4o mini as optimizer.
- Performance is still far from perfect, so mission-critical uses such as biomedical need caution.
- Possible misuse to steer LLM behavior harmfully.
Experimental setup
Datasets: MedMCQA Amazon reviews BIG-bench Tweet Eval Liar HateCheck Sarcasm Anthropic Harmless Ethos SQuAD HotpotQA TriviaQA
Benchmarks: MedMCQA Amazon reviews BIG-bench Anthropic Harmless Ethos Natural Questions WebQuestions
Metrics: accuracy F1-score exact match
Baselines: Default system prompt Chain-of-Thought Service system prompt SPRIG Outer Loop MetaSPO w/ APE MetaSPO w/ EVO
Builds on: MAML SPRIG ProTeGi APE OPRO
Closest papers by meaning
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
- Understanding Prompt Tuning and In-Context Learning via Meta-Learning (NeurIPS 2025 · similarity 0.88)
- On the Worst Prompt Performance of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Large Language Models as Optimizers (ICLR 2024 · similarity 0.88)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.88)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.