Research map / Large language models / Agents, code and math
Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a fully automated LLM-symbolic planning pipeline that builds a diverse action schema library from multiple LLM instances, filters schemas with conformal-prediction-thresholded sentence-encoder similarity, and ranks plans, removing expert-in-the-loop refinement.
Problem
Hybrid LLM-symbolic planning pipelines need extensive expert intervention to refine and validate generated PDDL action schemas, which limits scalability and risks biased single-expert interpretations of ambiguous natural language.
Approach
Generate many candidate action schemas from multiple high-temperature LLM instances into a library, keep only semantically aligned schemas via sentence-encoder cosine similarity with a conformal prediction threshold, enumerate solvable schema sets with a classical planner, and rank the resulting plans by cumulative semantic similarity.
Key results
- Conformal prediction filtering (confidence 0.8) reduced candidate combinations to 3.3% of the original (1051 of 31483) and raised the solvable ratio from 10.9% to 23.0%.
- In a blind human ranking of plans, the pipeline's plans averaged rank 2.97 versus 3.58 for ToT (gold plans 1.79).
- Ten LLM instances produced solvable schema sets across all test domains, whereas seven instances sometimes failed on Libraryworld and Minecraft.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed LLM-symbolic pipeline instead of Tree-of-Thought direct planning | Planning pipeline · Human plan ranking evaluation | average rank: 3.58 -> 2.97 | Helped |
| Apply conformal prediction filtering to schema candidates (ablation) | Proposed pipeline · Schema set candidates | number of combinations: 31483 -> 1051 | Helped |
| Apply conformal prediction filtering to schema candidates (ablation) | Proposed pipeline · Schema set candidates | ratio of solvable schemas: 10.9% -> 23.0% | Helped |
| Increase the number of LLM instances to 10 | Proposed pipeline · Libraryworld, Minecraft, and Dungeon | generation of solvable schema sets | Helped |
| Use layman rather than detailed task descriptions | Proposed pipeline · Test domains | number of distinct solvable schema sets: from 3419 to 8039 | Helped |
| Fine-tune the sentence encoder with hard negative samples (ablation) | all-roberta-large-v1 · Hard-negative action schema matching | identification of mismatched pairs | Helped |
| Fine-tune the sentence encoder with negative samples (ablation) | all-roberta-large-v1 · Semi-hard-negative action schema matching | identification of mismatched pairs | No clear effect |
Surprising findings
- Human assessors preferred ToT plans in the Dungeon domain over the reference plan, because ToT included an unnecessary sword-grabbing step that matched common game strategy.
- ToT with GPT-4o and GPT-3.5 failed on the Sussman Anomaly, with GPT-3.5 wrongly asserting task completion.
Limitations
- No direct metric for evaluating the quality of generated action schema sets, since bisimulation-style metrics need matching parameters.
- Open-source LLM (GLM) yields lower solvable-schema success than proprietary models, with failures at 7 LLM instances.
- Human evaluation uses only four assessors and a small number of problems.
Experimental setup
Datasets: IPC domains PDDLGym Libraryworld (new test domain) Minecraft (new test domain) Dungeon
Benchmarks: Libraryworld Minecraft Dungeon Sussman Anomaly
Metrics: solvable schema set ratio number of solvable schema sets cosine similarity separation blind plan ranking (average rank)
Baselines: Tree-of-Thought (ToT) previous expert-in-the-loop LLM-symbolic pipeline
Builds on: Tree-of-Thought chain-of-thought conformal prediction DUAL-BWFS
Closest papers by meaning
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.92)
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.92)
- Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning (AAAI 2024 · similarity 0.91)
- Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verification (ICML 2025 · similarity 0.90)
- Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks (ICML 2025 · similarity 0.90)
- SPIRAL: Symbolic LLM Planning via Grounded and Reflective Search (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.