Research map / Large language models / Language, safety and interpretability
Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares instruction optimization (IO) and exemplar optimization (EO) for automatic prompt optimization on BBH and MMLU with black-box PaLM 2, Gemini, and GPT models, finding that optimized exemplars often outweigh optimized instructions and that joint optimization works best.
Problem
Automatic prompt optimization has focused on instructions while exemplars are often handcrafted or ignored; it is unclear how their relative importance and joint optimization compare under black-box access and a comparable budget.
Approach
Uses model-generated exemplars from the validation set (correctly answered examples) as a free by-product and optimizes them with simple search (random search, mutation), then combines them with SoTA IO methods such as APE, ProTeGi, PromptAgent and OPRO, showing EO can outweigh IO and that mixing both under a fixed budget yields synergy.
Key results
- On BBH with PaLM 2, the best EO strategy (Mutation) on seed instruction gives about +12.6 points over No EO, versus about +7.8 for the best IO strategy (ProTeGi).
- With Gemini 1.5 Flash, No IO + Random search reaches 83.25 average BBH accuracy, versus 77.52 for APE without EO.
- Jointly mixing IO and EO under a total budget of 32 evaluations bridges most of the gap to the double-budget combination, and optimized exemplars generalize better than optimized instructions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mutation-based exemplar optimization to the seed instruction | PaLM 2 text-bison-002 · BBH | average test accuracy (%): 60.30 -> 72.92 | Helped |
| Add ProTeGi instruction optimization without exemplar optimization | PaLM 2 text-bison-002 · BBH | average test accuracy (%): 60.30 -> 68.13 | Helped |
| Add mutation-based exemplar optimization after ProTeGi | PaLM 2 text-bison-002 · BBH | average test accuracy (%): 68.13 -> 77.29 | Helped |
| Replace no exemplar optimization with random-search exemplar optimization | PaLM 2 text-bison-002 · BBH | average test accuracy (%): 60.30 -> 71.16 | Helped |
| Replace random exemplars with mutation-optimized exemplars | PaLM 2 text-bison-002 · BBH | average test accuracy (%): 70.81 -> 77.29 | Helped |
| Add ProTeGi instruction optimization without exemplar optimization | Gemini 1.0 Pro · BBH | average test accuracy (%): 63.14 -> 65.91 | Helped |
| Add mutation-based exemplar optimization to the seed instruction | Gemini 1.0 Pro · BBH | average test accuracy (%): 63.14 -> 75.77 | Helped |
| Add mutation-based exemplar optimization after ProTeGi | Gemini 1.0 Pro · BBH | average test accuracy (%): 65.91 -> 79.01 | Helped |
| Add random-search exemplar optimization to the seed instruction | PaLM 2 text-bison-002 · MMLU | average test accuracy (%): 65.77 -> 72.75 | Helped |
| Add ProTeGi instruction optimization while using random-search exemplars | PaLM 2 text-bison-002 · MMLU | average test accuracy (%): 72.75 -> 72.31 | Hurt |
| Add ProTeGi instruction optimization without exemplar optimization | Gemini 1.5 Flash · BBH | average test accuracy (%): 75.07 -> 80.39 | Helped |
| Add random-search exemplar optimization to the seed instruction | Gemini 1.5 Flash · BBH | average test accuracy (%): 75.07 -> 83.25 | Helped |
Surprising findings
- Seed instruction with optimized exemplars (No IO + Mutation) outperforms SoTA IO with random exemplars (ProTeGi + Random) in most cases.
- SoTA IO methods such as ProTeGi often embed validation exemplars verbatim in optimized instructions, and these quasi-exemplars drive much of the performance gain.
Limitations
- Evaluated only on BBH and MMLU classification and reasoning tasks; open-ended generation and safety not covered.
- Findings may not generalize to other models or tasks, as acknowledged by authors.
- Only simple EO search strategies studied; more advanced EO methods not explored.
Experimental setup
Datasets: BIG-Bench Hard MMLU
Benchmarks: BIG-Bench Hard MMLU
Metrics: accuracy
Baselines: No IO APE ProTeGi PromptAgent OPRO PromptBreeder
Builds on: APE ProTeGi PromptAgent OPRO DSPy
Closest papers by meaning
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars (NeurIPS 2024 · similarity 0.90)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
- Select Smarter, Not More? Prompt-Aware Evaluation Scheduling with Submodular Guarantees (NeurIPS 2026 · similarity 0.88)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.88)
- PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs (NeurIPS 2025 · similarity 0.88)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.