Research map / Large language models / Agents, code and math
ACPBench: Reasoning About Action, Change, and Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ACPBench, a PDDL-grounded benchmark of 7 planning-reasoning tasks over 13 domains, evaluates ~20 LLMs and OpenAI o1, and shows QLoRA fine-tuning of Granite 8B closes much of the gap.
Problem
LLMs are increasingly used as agents for planning and multi-step decisions, but there is no systematic evaluation of their atomic reasoning skills about actions, state change, and planning.
Approach
Distill planning into 7 atomic reasoning tasks (applicability, progression, reachability, action reachability, validation, justification, landmarks) generated automatically from PDDL domains with provably correct answers in Bool and MCQ formats, and fine-tune a small model on them.
Key results
- GPT-4o achieves 81.84% mean Bool and 74.97% mean MCQ accuracy with 2-shot CoT; LLAMA-3.1 405B reaches 77.42% mean MCQ.
- OpenAI o1-preview reaches 87.31% mean MCQ accuracy but no notable gain on Bool questions (89.59% Bool mean, comparable to best LLMs).
- Fine-tuned Granite 8B raises mean accuracy on training domains from 51.43% to 95.71% (Bool) and 19.18% to 94.29% (MCQ), with gains transferring to unseen domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Granite-code 8B with QLoRA on ACPBench tasks | Granite-code 8B · ACPBench training domains | mean accuracy: 51.43% to 95.71% on boolean questions; 19.18% to 93.39% on multiple-choice questions | Helped |
| Fine-tune Granite-code 8B with QLoRA on ACPBench tasks | Granite-code 8B · ACPBench unseen domains | mean accuracy: 44.71 to 81.21 Bool; 17.00 to 70.86 MC | Helped |
| Fine-tune Granite-code 8B on ACPBench tasks | Granite-code 8B · PlanBench Blocksworld (600) | score: 24 -> 44 | Helped |
| Fine-tune Granite-code 8B on ACPBench tasks | Granite-code 8B · PlanBench Logistics (285) | score: 14 -> 15 | Helped |
| Use 2-shot Chain-of-Thought prompting instead of zero-shot IO prompting (ablation) | OpenAI o1-preview · ACPBench mean Bool | accuracy: 87.02 -> 89.59 | Mixed |
| Use 2-shot Chain-of-Thought prompting instead of zero-shot IO prompting (ablation) | OpenAI o1-preview · ACPBench mean MCQ | accuracy: 88.22 -> 87.31 | No clear effect |
| Use Chain-of-Thought with two examples instead of IO prompts (ablation) | Granite-code 8B base · ACPBench training domains mean Bool | accuracy: 51.43 -> 54.18 | Helped |
| Use Chain-of-Thought with two examples instead of IO prompts (ablation) | Granite-code 8B base · ACPBench training domains mean MCQ | accuracy: 20.21 -> 27.57 | Helped |
| Use Chain-of-Thought with two examples instead of IO prompts (ablation) | Granite-code 8B base · ACPBench unseen domains mean Bool | accuracy: 44.71 -> 49.83 | Helped |
| Use Chain-of-Thought with two examples instead of IO prompts (ablation) | Granite-code 8B base · ACPBench unseen domains mean MCQ | accuracy: 17.00 -> 31.43 | Helped |
| Increase model generation from LLAMA-2 70B to LLAMA-3 70B (ablation) | LLAMA 70B · ACPBench mean Bool | accuracy: 55.72 -> 78.71 | Helped |
| Use OpenAI o1-preview instead of the strongest conventional LLM | ACPBench · multiple-choice questions | accuracy: 87.31% average for o1-preview; GPT-4o 74.97% average | Helped |
Surprising findings
- OpenAI o1 gives large MCQ gains but no notable progress on boolean questions.
- Fine-tuned 8B model generalizes to unseen domains and to PlanBench plan generation.
Limitations
- o1 comparison is not even-handed: 1024 token limit for LLMs vs unlimited for o1, and LLMs evaluated single-generation only.
- Most LLM results averaged over 5 runs but GPT family and LLAMA-3.1 405B run once due to resource constraints.
- Templates for natural language were hand-crafted by researchers, limiting scale of language variation.
Experimental setup
Datasets: ACPBench training set generated from 8 PDDL domains PlanBench plan generation (t1)
Benchmarks: ACPBench
Metrics: accuracy
Baselines: GPT-4o GPT-4o Mini LLAMA-3 70B LLAMA-3.1 405B Mixtral 8x22B Codestral 22B OpenAI o1-preview o1-mini Phi-3 128K
Builds on: PlanBench AutoPlanBench ActionReasoningBench Chain-of-Thought prompting
Closest papers by meaning
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.88)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.88)
- Chain of Thoughtlessness? An Analysis of CoT in Planning (NeurIPS 2024 · similarity 0.88)
- Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning (AAAI 2024 · similarity 0.88)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- CodePlan: Unlocking Reasoning Potential in Large Language Models by Scaling Code-form Planning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.