Research map / Large language models / Agents, code and math
Large Language Models as Planning Domain Generators (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates whether LLMs can generate PDDL planning domains from natural language action descriptions, proposing an action-by-action generation task and a plan-set-based domain equivalence evaluation across 7 models and 9 domains.
Problem
Creating PDDL planning domain models manually is labor-intensive; the paper asks whether LLMs can generate planning domain models from textual descriptions and how to evaluate the generated domains automatically.
Approach
Generate PDDL actions one at a time via in-context prompting with natural-language action descriptions and predicate descriptions, reconstruct domains by swapping in each generated action, and judge equivalence to the ground-truth domain by comparing plan sets on problems using a planner and plan validator.
Key results
- LLaMA-70B base model performs best, with the highest rate of heuristically equivalent generated domains (28.80%).
- Syntax errors are rare for most models, with the dominant error class being semantic predicate-argument count mismatches for many models.
- Most valid generated domains that are not equivalent fail to yield plans, with the planner unable to find plans in many reconstructed domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaMA 70B instead of LLaMA 13B | LLaMA domain generation · 9 planning domains | Equivalent domains (%): 8.85 -> 28.80 | Helped |
| Use LLaMA 70B instead of LLaMA 13B | LLaMA domain generation · 9 planning domains | Syntax errors (%): 1.30 -> 0.36 | Helped |
| Use LLaMA 70B instead of LLaMA 13B | LLaMA domain generation · 9 planning domains | Semantic errors (%): 14.64 -> 7.08 | Helped |
Surprising findings
- The chat-tuned LLaMA models did not uniformly outperform their base counterparts; e.g., 7b chat had higher syntax errors than 7b base (22.03% vs 15.31%).
Limitations
- Action-by-action generation ignores interactions between actions and predicates in the whole domain.
- Equivalence check is a necessary-condition heuristic based on a subset of plans, not full domain equivalence.
- Evaluation covers only 9 domains and a limited set of open models.
Experimental setup
Datasets: 5 International Planning Competition domains 4 domains from Silver et al. 2023
Benchmarks: 9 PDDL planning domains (IPC and Silver et al. 2023)
Metrics: syntax error rate semantic error rate different-domain rate heuristic equivalence rate
Baselines: StarCoder LLaMA-7B LLaMA-13B LLaMA-70B LLaMA-2-chat 7B LLaMA-2-chat 13B LLaMA-2-chat 70B
Builds on: Guan et al. 2023 (LLMs for PDDL planning) LLaMA StarCoder
Closest papers by meaning
- Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · similarity 0.92)
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.91)
- Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks (ICML 2025 · similarity 0.90)
- Improving Large Language Model Planning with Action Sequence Similarity (ICLR 2025 · similarity 0.90)
- Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning (AAAI 2024 · similarity 0.90)
- SayCanPay: Heuristic Planning with Large Language Models Using Learnable Domain Knowledge (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.