Research map / Large language models / Agents, code and math
ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning
Harsha Kokel, Michael Katz, Kavitha Srinivas, Shirin Sohrabi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PDDL representation to the NL prompt (ablation) | DeepSeek V3 · ACPBench Hard avg | accuracy: 0.39 -> 0.47 | Improved |
| use PDDL only instead of NL (ablation) | DeepSeek V3 · ACPBench Hard avg | accuracy: 0.39 -> 0.44 | Improved |
| use PDDL+NL instead of PDDL only (ablation) | DeepSeek V3 · ACPBench Hard, just task | accuracy: 0.74 -> 0.68 | Hurt |
| add PDDL+NL context (ablation) | DeepSeek V3 · ACPBench Hard, val task | accuracy: 0.56 -> 0.61 | No clear effect |
| use reasoning model o1-preview instead of GPT-4o | o1-preview · ACPBench Hard, app task | accuracy: 0.25 -> 0.44 | Improved |
| use reasoning model o1-preview instead of DeepSeek V3 | o1-preview · ACPBench Hard, just task | accuracy: 0.65 -> 0.46 | Hurt |
| use reasoning model o1-preview instead of o1 mini | o1-preview · ACPBench Hard, val task | accuracy: 0.78 -> 0.26 | Hurt |
| raise token cap to 4,000 for GPT OSS reasoning models | GPT OSS 20B/120B · ACPBench Hard | accuracy | No clear effect |
| scale from Llama 3.1 70B to 405B | Llama 3.1 · ACPBench Hard, just task | accuracy: 0.44 -> 0.59 | Improved |
| use generative format instead of boolean/MCQ | GPT-4o · ACPBench vs ACPBench Hard | error | Hurt |
| use less strict Jaccard scoring instead of exact set match | o1-preview · ACPBench Hard, app task | score: 0.44 -> 0.57 | Improved |
| use reasoning models instead of LLMs | OpenAI reasoning models · ACPBench Hard, areach | accuracy: 0.12 | No clear effect |
Closest papers by meaning
- ACPBench: Reasoning About Action, Change, and Planning (AAAI 2025 · similarity 0.94)
- Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · similarity 0.88)
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.88)
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.87)
- Graph-enhanced Large Language Models in Asynchronous Plan Reasoning (ICML 2024 · similarity 0.87)
- Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
