Research map / Large language models / Language, safety and interpretability
Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLMFP, a zero-shot framework where LLMs formalize planning problems as constrained optimization (Definer, Formulator, Code Generator, Result Formatter, Self-Assessment) and solve them with the Z3 SMT solver, evaluated on 9 planning tasks.
Problem
Zero-shot LLM planning for complex multi-constraint and long-horizon tasks typically fails, while existing solutions rely on task-specific examples or critics that hurt cross-task generalization.
Approach
Recast planning problems as constrained optimization: LLMs extract goals, decision variables, and constraints (including implicit ones), formulate a task-agnostic JSON representation, generate Z3 SMT code, and self-assess/revise, with an external solver guaranteeing optimal plans.
Key results
- LLMFP with GPT-4o reaches 83.7% average optimal rate across 9 tasks (79.1% multi-constraint, 87.5% multi-step), versus 29.7%/68.1% for o1-preview direct planning.
- LLMFP with Claude 3.5 Sonnet reaches 86.8% average optimal rate across 9 tasks (80.7% multi-constraint, 91.8% multi-step).
- Removing the Formulator drops multi-step optimal rate from 87.5% to 0.1% with GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLMFP instead of direct planning with o1-preview | GPT-4o planning on 9 tasks · 9 planning tasks | optimal rate: 37.6% improvements | Helped |
| Use LLMFP instead of direct planning with o1-preview | Claude 3.5 Sonnet planning on 9 tasks · 9 planning tasks | optimal rate: 40.7% improvements | Helped |
| Add the Definer component (ablation) | LLMFP GPT-4o · 5 multi-constraint problems | optimal rate: 63.7% -> 79.1% | Helped |
| Add the Formulator component (ablation) | LLMFP GPT-4o · 5 multi-constraint problems | optimal rate: 56.9% -> 79.1% | Helped |
| Add Self Assess & Modification (ablation) | LLMFP GPT-4o · 5 multi-constraint problems | optimal rate: 57.2% -> 79.1% | Helped |
| Add the Formulator component (ablation) | LLMFP GPT-4o · 4 multi-step problems | optimal rate: 0.1% -> 87.5% | Helped |
| Add Self Assess & Modification (ablation) | LLMFP GPT-4o · 4 multi-step problems | optimal rate: 75.1% -> 87.5% | Helped |
| Add one task-specific example to the Formulator | LLMFP GPT-4o · Coffee | optimal rate: 61.2% -> 85.4% | Helped |
| Add the Definer component (ablation) | LLMFP GPT-4o · Coffee | optimal rate: 8.6 -> 64.7 | Helped |
| Add the Formulator component (ablation) | LLMFP GPT-4o · Warehouse | optimal rate: 2.0 -> 72.0 | Helped |
| Add Self Assess & Modification (ablation) | LLMFP GPT-4o · Workforce | optimal rate: 27.3 -> 92.2 | Helped |
Surprising findings
- Code_SMT with GPT-4o performs very poorly (2.7% average on multi-constraint) despite Claude 3.5 Sonnet reaching 62.4%, showing instability across LLMs.
- Direct o1-preview with a formal mathematical definition of Coffee only reaches 34.2% optimal rate, suggesting LLMs struggle to solve optimization directly even when given formal specifications.
Limitations
- Requires clear and detailed task descriptions and queries; ambiguous descriptions hurt goal and constraint extraction.
- Solver speed becomes slow for massive search spaces with many feasible plans, as seen in Sokoban.
Experimental setup
Datasets: Coffee Workforce Facility Task Allocation Warehouse Blocksworld Mystery Blocksworld Movie Gripper Sokoban (15-query eval set)
Benchmarks: Coffee Workforce Facility Task Allocation Warehouse Blocksworld Mystery Blocksworld Movie Gripper Sokoban
Metrics: optimal rate success rate wall time cost per query
Baselines: Direct CoT Code Code_SMT o1-preview direct planning
Builds on: chain-of-thought prompting Z3 SMT solver PDDL planning
Closest papers by meaning
- Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python Code (NeurIPS 2025 · similarity 0.91)
- Thought of Search: Planning with Language Models Through The Lens of Efficiency (NeurIPS 2024 · similarity 0.89)
- SPIRAL: Symbolic LLM Planning via Grounded and Reflective Search (AAAI 2026 · similarity 0.88)
- SayCanPay: Heuristic Planning with Large Language Models Using Learnable Domain Knowledge (AAAI 2024 · similarity 0.88)
- LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language (NeurIPS 2025 · similarity 0.88)
- Non-myopic Generation of Language Models for Reasoning and Planning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.