Research map / Large language models / Agents, code and math
How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EGO-Prompt, which evolves an expert-built semantic causal graph and system prompts via textual gradients, generating instance-specific reasoning guidance, to improve LLM prediction on public health, traffic crash, and travel mode choice tasks.
Problem
Domain-specific LLM prompting relies on incomplete or imperfect expert knowledge and static graph databases; it is unclear how to jointly optimize prompts, reasoning, and domain causal graphs without fine-tuning.
Approach
Two-stage graph-guided reasoning: a graph description model turns the expert SCG plus each instance into deterministic causal guidance, which a predictor uses; the SCG (add/delete/edit operations) and prompts are iteratively refined with textual gradients from ground-truth labels.
Key results
- EGO-Prompt with GPT-4o mini reaches mean F1 0.410 vs 0.382 for best baseline (ProTeGi), a 7.32% gain; with Gemini 2.5 Flash mean F1 0.491 (24.6% over organized prompt).
- EGO-Prompt (GPT-4o mini) matches o4-mini performance at roughly one-sixth the cost and costs about $0.057 per 100 samples vs $0.33 for o4-mini.
- Open-source models improve by 49.9% mean F1 with EGO-Prompt (e.g., DeepSeek-V3 0.300 to 0.445).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add instance-specific reasoning guidance decomposition (ablation) | EGO-Prompt · Pandemic | F1: 0.397 -> 0.399 | Helped |
| add instance-specific reasoning guidance decomposition (ablation) | EGO-Prompt · TrafficSafe | F1: 0.247 -> 0.333 | Helped |
| add instance-specific reasoning guidance decomposition (ablation) | EGO-Prompt · Swissmetro | F1: 0.445 -> 0.498 | Helped |
| optimize the prompt and graph instead of removing optimization (ablation) | EGO-Prompt · Pandemic | F1: 0.359 -> 0.399 | Helped |
| optimize the prompt and graph instead of removing optimization (ablation) | EGO-Prompt · TrafficSafe | F1: 0.293 -> 0.333 | Helped |
| optimize the prompt and graph instead of removing optimization (ablation) | EGO-Prompt · Swissmetro | F1: 0.434 -> 0.498 | Helped |
| use iterative optimization (ablation) | EGO-Prompt · Pandemic | F1: 0.369 -> 0.399 | Helped |
| use iterative optimization (ablation) | EGO-Prompt · TrafficSafe | F1: 0.272 -> 0.333 | Helped |
| use iterative optimization (ablation) | EGO-Prompt · Swissmetro | F1: 0.447 -> 0.498 | Helped |
| use EGO-Prompt instead of the strongest listed baseline | GPT-4o mini · Pandemic | F1: 0.361 -> 0.399 | Helped |
| use EGO-Prompt instead of the strongest listed baseline | GPT-4o mini · TrafficSafe | F1: 0.304 -> 0.333 | Helped |
| use EGO-Prompt instead of the strongest listed baseline | GPT-4o mini · Mode Choice | F1: 0.481 -> 0.498 | Helped |
Surprising findings
- Reversing the expert SCG degrades pandemic F1 from 0.359 to 0.303, showing incorrect priors can misguide optimization.
- A partially randomized (33%) SCG yields lower mean F1 than an empty SCG, suggesting a broken causal graph may be worse than no guidance.
Limitations
- Results vary up to about 20% across runs due to API stochasticity and textual gradient instability; best-of-three reporting.
- Small validation and test sets (100 each) risk overfitting the validation set.
- Requires additional optimization compute that scales with sample set size.
Experimental setup
Datasets: Pandemic Hospitalization dataset TrafficSafe crash dataset Swissmetro travel mode choice dataset
Benchmarks: Pandemic TrafficSafe Swissmetro
Metrics: weighted F1 accuracy inference cost per 100 samples
Baselines: Organized Prompt Zero-Shot-CoT ProTeGi Auto-CoT PHP TextGrad o1 o3 o3-mini o4-mini Gemini 2.5 pro
Builds on: TextGrad ProTeGi PHP Zero-Shot-CoT Auto-CoT
Closest papers by meaning
- PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization (ICLR 2024 · similarity 0.90)
- MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimization (AAAI 2026 · similarity 0.89)
- Eliciting Causal Abilities in Large Language Models for Reasoning Tasks (AAAI 2025 · similarity 0.89)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.89)
- Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution (ICML 2024 · similarity 0.89)
- StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.