Research map / Large language models / Language, safety and interpretability
Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evoke iteratively refines task prompts with two instances of the same LLM (an author that edits prompts and a reviewer that scores them) plus an LLM-selector that keeps only hard training samples.
Problem
Existing automatic prompt engineering methods often perform essentially random search over paraphrases without feedback, and hand-crafted prompts require costly human trial and error; the paper targets prompt quality for LLM task performance, including robustness to typos and fine-grained NER.
Approach
Use the same LLM in three roles: an author that edits the prompt using edit history and reviewer scores, a reviewer that rates prompts and keeps memory of edits and accuracies, and a selector that rates sample difficulty so only hard samples are used for refinement.
Key results
- On logical fallacy detection, Evoke scores above 80 while APE and Human baselines are below 20.
- On Adversarial-SST2, task accuracy rises from about 75% after one refinement iteration to above 90% after three iterations.
- Evoke outperforms APE and Human baselines on all eight Instruction Induction / BBII tasks, averaged over three runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Evoke instead of APE or Human prompts | GPT-4 · BBII logical fallacy detection | accuracy: Evoke: more than 80; APE and Human: below 20 | Helped |
| Increase the reviewer prompt score from 6 to 7 | Evoke · Common-Concept | accuracy: about 5% -> about 17% | Helped |
| Increase the reviewer prompt score from 7.5 to 8 | Evoke · Adversarial-SST2 | accuracy: barely reaches 75% -> 85% | Helped |
| Increase prompt-refinement iterations from one to three | Evoke · Adversarial-SST2 | accuracy: about 75% -> above 90% | Helped |
| Select hard samples instead of randomly selected samples (ablation) | Evoke · Common-Concept and Adversarial-SST2 | accuracy | Helped |
| Select hard samples instead of easy samples (ablation) | Evoke · Common-Concept and Adversarial-SST2 | accuracy | Helped |
Surprising findings
- Easy samples yield the worst performance during refinement, while hard samples are most helpful, and random selection underperforms hard selection.
Limitations
- Relies on GPT-4 via API for all roles, so cost and reproducibility depend on a closed model.
- Reviewer scores are coarse 1-10 ratings and the reviewer's judgment may not match task accuracy in all cases.
Experimental setup
Datasets: Instruction Induction Big Bench Instruction Induction SST-2 QQP in-house multilingual NER query data
Benchmarks: Instruction Induction orthography starts with common concept rhymes Big Bench movie recommendation logical fallacy detection presuppositions as nli winowhy epistemic reasoning Adversarial SST-2 Adversarial QQP in-house multilingual NER
Metrics: accuracy
Baselines: Human-curated prompts APE RobEnc
Builds on: APE Instruction Induction RobEnc
Closest papers by meaning
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.88)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.88)
- Prompt Sketching for Large Language Models (ICML 2024 · similarity 0.87)
- Understanding the Effects of Iterative Prompting on Truthfulness (ICML 2024 · similarity 0.87)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.87)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.