Research map / Large language models / Agents, code and math
Generalized Planning in PDDL Domains with Pretrained Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prompts GPT-4 to synthesize Python generalized planners for PDDL domains, adding chain-of-thought domain summarization and automated debugging with four feedback types, and evaluates on seven domains against PG3-style baselines.
Problem
Generalized planning: synthesizing a program from a few training tasks in a PDDL domain that efficiently produces valid plans for held-out tasks with many more objects.
Approach
Use a pretrained LLM (GPT-4) as a generalized planner by prompting it to summarize the PDDL domain, propose a search-free strategy, and implement it as Python, then iteratively debug the program using validator feedback (exceptions, timeouts, plan syntax, VAL plan semantics) on training tasks.
Key results
- GPT-4 with CoT summarization and automated debugging solves 0.90 of evaluation tasks in Delivery, 1.00 in Forest, 0.90 in Gripper, and 0.80 in Ferry, and 0.60 in Heavy where PG3 scores 0.00.
- Without automated debugging, performance drops substantially (e.g., Delivery 0.10, Spanner 0.00, Ferry 0.26).
- GPT-4 generally beats GPT-3.5 (which scores 0.00 in Delivery and Gripper) and synthesized programs run faster than Fast Downward on evaluation tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove CoT summarization (ablation) | GPT-4 generalized planner · Delivery | fraction of evaluation tasks solved: 0.90 -> 0.70 | Hurt |
| remove CoT summarization (ablation) | GPT-4 generalized planner · Forest | fraction of evaluation tasks solved: 1.00 -> 1.00 | No clear effect |
| remove CoT summarization (ablation) | GPT-4 generalized planner · Gripper | fraction of evaluation tasks solved: 0.90 -> 0.80 | Hurt |
| remove CoT summarization (ablation) | GPT-4 generalized planner · Miconic | fraction of evaluation tasks solved: 0.01 -> 0.13 | Helped |
| remove CoT summarization (ablation) | GPT-4 generalized planner · Ferry | fraction of evaluation tasks solved: 0.80 -> 0.20 | Hurt |
| remove CoT summarization (ablation) | GPT-4 generalized planner · Spanner | fraction of evaluation tasks solved: 0.10 -> 0.00 | Helped |
| remove CoT summarization (ablation) | GPT-4 generalized planner · Heavy | fraction of evaluation tasks solved: 0.60 -> 1.00 | Helped |
| remove automated debugging (ablation) | GPT-4 generalized planner · Delivery | fraction of evaluation tasks solved: 0.90 -> 0.10 | Hurt |
| remove automated debugging (ablation) | GPT-4 generalized planner · Forest | fraction of evaluation tasks solved: 1.00 -> 0.62 | Hurt |
| remove automated debugging (ablation) | GPT-4 generalized planner · Gripper | fraction of evaluation tasks solved: 0.90 -> 0.50 | Hurt |
| remove automated debugging (ablation) | GPT-4 generalized planner · Ferry | fraction of evaluation tasks solved: 0.80 -> 0.26 | Hurt |
| replace PDDL names with nondescriptive identifiers (ablation) | GPT-4 generalized planner · Forest | fraction of evaluation tasks solved: 1.00 -> 0.11 | Hurt |
Surprising findings
- CoT summarization has non-uniform impact: it helps in most domains but hurts in Miconic and Heavy.
- Just two training tasks are used in the vast majority of successful trials, showing strong data efficiency.
Limitations
- Hand-designed generalized plans are easy for these domains, limiting the significance of the benchmarks.
- Using training tasks may be insufficient to express the full task distribution.
- Experiments used the ChatGPT browser interface with manual copy-paste, so results are not fully automated and performance is averaged over conversations; Miconic and Spanner remain unsolved.
Experimental setup
Datasets: Delivery Forest Gripper Miconic Ferry Spanner Heavy
Benchmarks: Delivery Forest Gripper Miconic Ferry Spanner Heavy
Metrics: fraction of evaluation tasks solved program runtime vs Fast Downward (LAMA)
Baselines: PG3 Policy Evaluation Plan Compare Random GPT-3.5
Builds on: Chain-of-Thought prompting PG3 Inner Monologue VAL plan validator
Closest papers by meaning
- Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models (NeurIPS 2024 · similarity 0.90)
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.89)
- Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · similarity 0.87)
- Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python Code (NeurIPS 2025 · similarity 0.87)
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.87)
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.