Research map / Large language models / Reasoning and chain of thought
Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Promptbreeder evolves a population of task-prompts together with self-referentially evolved mutation-prompts using an LLM as the mutation operator, outperforming CoT and Plan-and-Solve prompting on arithmetic and commonsense benchmarks without parameter updates.
Problem
Hand-crafted prompt strategies such as Chain-of-Thought are often sub-optimal, and prior automated prompt engineering (e.g., APE) suffers diminishing returns and loss of diversity across iterations.
Approach
Use an LLM as the mutation operator in a binary-tournament genetic algorithm over units of evolution that contain task-prompts and their own mutation-prompts, with hyper-mutation operators that rewrite the mutation-prompts themselves, plus diversity-maintaining EDA and Lamarckian operators, so the search procedure is improved alongside the prompts.
Key results
- Zero-shot Promptbreeder reaches 83.9% on GSM8K with PaLM 2-L, above OPRO's 80.2% and PS+ with PaLM 2-L.
- Few-shot Promptbreeder reaches 100.0% on MultiArith and 93.7% on SVAMP, versus 80.3% for Manual-CoT on SVAMP with text-davinci-003.
- On ETHOS hate speech classification, the evolved two-prompt strategy scores 89%, versus 80% for a hand-designed prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evolve task-prompts with Promptbreeder instead of using OPRO | PaLM 2-L on GSM8K · GSM8K | accuracy: 80.2 -> 83.9 | Helped |
| include discovered correct workings as few-shot context | Promptbreeder on arithmetic and commonsense benchmarks · MultiArith, GSM8K, AddSub, AQuA-RAT, SingleEq, SVAMP, CSQA, StrategyQA | accuracy: MultiArith: 99.7 -> 100.0; GSM8K: 83.9 -> 83.5; AddSub: 87.8 -> 87.1; AQuA-RAT: 62.2 -> 64.6; SingleEq: 96.4 -> 98.9; SV | Mixed |
| remove any self-referential operator (ablation) | Promptbreeder on maths and ETHOS datasets · Maths datasets and ETHOS | accuracy | Hurt |
| add initial re-description of task-prompts during initialization (ablation) | Promptbreeder · Maths datasets and ETHOS | accuracy | Helped |
| draw randomly from the mutation-prompt set upon initialization (ablation) | Promptbreeder on GSM8K · GSM8K | accuracy | Hurt |
| use Promptbreeder's evolved two-prompt strategy instead of the hand-designed hate-speech prompt | ETHOS hate speech classification · ETHOS | score: 80% -> 89% | Helped |
Surprising findings
- The best evolved zero-shot prompt for GSM8K is the unintuitive string "SOLUTION", which beats OPRO's 'Take a deep breath' prompt.
- Telling the LLM that an ascending-fitness list is descending (a deliberate 'lie') improved sampling diversity in EDA rank-and-index mutation.
Limitations
- Prompting topology is fixed; only prompt content evolves, not the prompting algorithm itself.
- Fitness is measured with the externally specified metric only; the system does not invent its own evaluation methods.
- The LLM did not appear to interpret fitness values, so fitness is not given to the mutation operators.
Experimental setup
Datasets: GSM8K SVAMP MultiArith AddSub AQuA-RAT SingleEq CommonsenseQA StrategyQA ETHOS APE instruction induction tasks
Benchmarks: GSM8K SVAMP MultiArith AddSub AQuA-RAT SingleEq CommonsenseQA StrategyQA ETHOS
Metrics: accuracy
Baselines: Chain-of-Thought Zero-shot CoT Program-of-Thoughts Auto-CoT OPRO APE Plan-and-Solve PS+ EvoPrompt
Builds on: Chain-of-Thought prompting Plan-and-Solve prompting Automatic Prompt Engineer STaR population based training
Closest papers by meaning
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.89)
- How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation (NeurIPS 2025 · similarity 0.89)
- SePO: Self-Evolving Prompt Agent for System Prompt Optimization (NeurIPS 2026 · similarity 0.89)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.88)
- PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization (ICLR 2024 · similarity 0.88)
- PACE: Two-Timescale Self-Evolution for Small Language Model Agents (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.