Research map / Large language models / Agents, code and math
Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explores using LLMs (GPT-3.5-turbo and GPT-4) for model-space edits in AI planning, both standalone and combined with combinatorial search as pre- or post-processor, on unsolvability and executability tasks in four novel and two IPC-derived domains.
Problem
Model space problems in automated planning (fixing unsolvable or non-executable PDDL models) admit many logically equivalent sound edits, but combinatorial search cannot tell which edits are more likely in real-world domains.
Approach
Uses an LLM as a statistical signal over model edits, either as a standalone end-to-end editor (LLM-only), as a selector over sound candidates from combinatorial search (post-processor), or as a ranker of edits that guides search (pre-processor), and evaluates soundness and reasonableness against ground truth.
Key results
- LLM-only with GPT-4 produced sound solutions for 399/564 unsolvability instances (about 71%) versus 276/484 for GPT-3.5-turbo.
- On unsolvability, GPT-4 LLM-only yielded 136/399 preferred (reasonable) sound solutions, while the post-processor approach with GPT-4 found sound solutions for 198/564 instances.
- On executability, GPT-4 LLM-only achieved 493/563 sound solutions versus 174/484 for GPT-3.5-turbo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Upgrade the LLM from GPT-3.5-turbo to GPT-4 in the LLM-only approach | Unsolvability model updates · Unsolvability overall | Sound solutions: 276/484 -> 399/564 | Helped |
| Upgrade the LLM from GPT-3.5-turbo to GPT-4 in the LLM-only approach | Unsolvability model updates · Unsolvability overall | Preferred sound solutions: 9/276 -> 136/399 | Helped |
| Upgrade the LLM from GPT-3.5-turbo to GPT-4 in the LLM-only approach | Executability model updates · Executability overall | Sound solutions: 174/484 -> 493/563 | Helped |
| Upgrade the LLM from GPT-3.5-turbo to GPT-4 in the LLM-only approach | Executability model updates · Executability overall | Preferred sound solutions: 53/174 -> 264/493 | Helped |
| Use the more explicit prompt about easiest-to-realize changes (ablation) | LLM-only · Executability, Barman and Logistics | Sound solutions: 30/88 (GPT-3.5-turbo); 57/58 (GPT-4) | Hurt |
| Use the more explicit prompt about easiest-to-realize changes (ablation) | LLM-only · Executability, Barman and Logistics | Preferred sound solutions: 0/30 (GPT-3.5-turbo); 27/57 (GPT-4) | Mixed |
| Use GPT-4 rather than GPT-3.5-turbo | LLM-only · Unsolvability and executability domains | Sound solutions | Helped |
Surprising findings
- The LLM-only setting outperformed both combinatorial search + LLM configurations, contrary to the hypothesis that CS+LLM would dominate.
- Gains in soundness from GPT-3.5 to GPT-4 did not carry over proportionally to reasonable (preferred) edits.
Limitations
- Only two use cases (unsolvability and executability), one edit type (adding initial-state predicates), and few LLMs evaluated.
- Ground truth for 'reasonable' edits is defined by the authors on their own designed domains; no user study.
- LLM-only approach limited by context window size, causing instances to be unprocessable, notably in Roomba with GPT-3.5.
Experimental setup
Datasets: Travel (novel) Roomba (novel) Barman-simple (derived from IPC barman) Logistics-simple (derived from IPC logistics) Logistics (IPC) Barman (IPC)
Benchmarks: Travel Roomba Barman Logistics Barman-simple Logistics-simple
Metrics: soundness rate preferred (reasonable) edit rate solution ratio
Baselines: combinatorial search
Builds on: PDDL model reconciliation model space search
Closest papers by meaning
- Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · similarity 0.91)
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.91)
- Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks (ICML 2024 · similarity 0.90)
- PlanU: Large Language Model Reasoning through Planning under Uncertainty (NeurIPS 2025 · similarity 0.90)
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.90)
- Thought of Search: Planning with Language Models Through The Lens of Efficiency (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.