Research map / Large language models / Agents, code and math
On the Planning Abilities of Large Language Models - A Critical Investigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| obfuscate action and predicate names (deceptive Mystery Blocksworld) (ablation) | GPT-4 zero-shot NL · Blocksworld | instances correct: 210/600 -> 1/600 | Hurt |
| use PDDL instead of natural language prompts | GPT-4 one-shot · Blocksworld | instances correct: 206/600 -> 75/600 | Hurt |
| add state-tracking chain of thought prompting | GPT-4 · Blocksworld | instances correct: 206/600 -> 214/600 | No clear effect |
| add one example (one-shot vs zero-shot) | GPT-4 natural language · Blocksworld | instances correct: 210/600 -> 206/600 | No clear effect |
| fine-tune GPT-3 on 1,000 Blocksworld instances | GPT-3 · Blocksworld | instances correct: 6/600 -> 122/600 | Helped |
| seed LPG with GPT-4 plan instead of empty plan | LPG · Blocksworld | avg search steps: 15.8 -> 8.9 | Helped |
| seed LPG with GPT-4 plan instead of empty plan | LPG · Logistics | avg search steps: 77.5 -> 51.3 | Helped |
| seed LPG with GPT-4 plan instead of empty plan | LPG · Mystery BW | avg search steps: 15.8 -> 16.09 | Hurt |
| seed LPG with Instruct-GPT3 plan instead of empty plan | LPG · Blocksworld | avg search steps: 15.8 -> 14.5 | No clear effect |
| add VAL verifier feedback backprompting to GPT-4 | GPT-4 · Blocksworld (failed instances) | instances correct: 82% solved within 15 rounds | Helped |
| add VAL verifier feedback backprompting to GPT-4 | GPT-4 · Mystery BW (failed instances) | instances correct: 5/50 (10%) | No clear effect |
| provide GPT-4 plan suggestions to human planners | human participants · Blocksworld user study | plan accuracy, time, cognitive load: p = 0.22, 0.92, 0.52 | No clear effect |
Closest papers by meaning
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.93)
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.92)
- Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · similarity 0.91)
- SayCanPay: Heuristic Planning with Large Language Models Using Learnable Domain Knowledge (AAAI 2024 · similarity 0.91)
- Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning (AAAI 2024 · similarity 0.90)
- Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python Code (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.