Research map / Large language models / Agents, code and math
ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ActionReasoningBench, a diagnostic benchmark of 3,498 test questions across 8 IPC planning domains that evaluates LLMs on six reasoning-about-actions dimensions, including new ramification constraints, and fine-tunes Llama-3.1-8B on a 149k-question training split.
Problem
LLMs' ability on reasoning about actions and change (RAC), including indirect effects (ramifications) of actions, is underexplored and not systematically decomposed into categories.
Approach
Decomposes RAC into six question categories and fluent types (static, base, derived, self-derived), and introduces ramification constraints into natural-language IPC domain descriptions, generating questions from PDDL/ASP templates with paraphrasing.
Key results
- Average accuracy on the four basic RAC categories is 73.55%, 65.63%, 58.73%, and 62.38%, with Numerical RAC and Composite Questions at 33.16% and 51.19% per the abstract.
- GPT-4o answered none of the ramification free-answer questions correctly, while o1-preview scored only 18.4% on ramification questions.
- Fine-tuned Llama-3.1-8B gains an average of 33.68% over the base model on the test set and outperforms GPT-4o by 4.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3.1-8B on benchmark training data | Llama-3.1-8B-Instruct · ActionReasoningBench test set | accuracy: 33.68% | Helped |
| Increase model size from 8B to 70B | Llama-3.1-Instruct · ActionReasoningBench | accuracy: 20.84% | Helped |
| Use GPT-4o instead of Llama-3.1-70B-Instruct | ActionReasoningBench · ActionReasoningBench | accuracy: 8.64% | Helped |
| Add three few-shot examples instead of zero-shot-CoT | GPT-4o and Llama-3.1-70B-Instruct · ActionReasoningBench binary questions | accuracy: approximately 3.5% | Hurt |
| Add three few-shot examples instead of zero-shot-CoT | Open-source LLMs · ActionReasoningBench free-answer questions | accuracy | Helped |
| Add ramification constraints to the domain descriptions | GPT-4o · ActionReasoningBench ramifications, free-answer | accuracy: 00.00 | Hurt |
| Use o1-preview instead of GPT-4o | ActionReasoningBench ramifications · ActionReasoningBench ramifications, free-answer | accuracy: 18.4% | Helped |
| Increase action-sequence length | LLMs · ActionReasoningBench | accuracy | Hurt |
| Ask questions involving negative rather than exclusively true fluents | LLMs · ActionReasoningBench | accuracy: 12.16% | Hurt |
| Combine true and false fluents in the question | LLMs · ActionReasoningBench | accuracy: 12.16% | Hurt |
| Use composite questions instead of the first four basic categories | LLMs · ActionReasoningBench complex categories | accuracy: 17.88% | Hurt |
| Paraphrase templated questions with Llama-3.1-70B-Instruct | ActionReasoningBench questions · Data Verification | naturalness score: 4.2 to 4.5 | Helped |
Surprising findings
- GPT-4o failed to produce any correct answers on ramification-related questions, often omitting ramification fluents even when described in the domain.
- Few-shot-3 prompting did not significantly improve binary accuracy and slightly reduced it for GPT-4o and Llama-70B.
Limitations
- Limited to English-language questions.
- Domains are drawn from IPC planning domains, which may bias the benchmark toward planning-specific scenarios.
- Free-answer evaluation relies on Llama-3.1-70B-Instruct as judge, which is imperfect.
Experimental setup
Datasets: IPC domains (Blocksworld, Depots, Driverlog, Grippers, Mystery, Satellite, Spanner, Visitall)
Benchmarks: ActionReasoningBench
Metrics: accuracy standard error of the mean
Baselines: GPT-4o Llama-3.1-70B-Instruct Llama-3.1-8B-Instruct o1-preview PlanBench TRAC ACPBench
Builds on: PDDL Answer Set Programming Chain-of-thought prompting
Closest papers by meaning
- V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoning (ICML 2026 · similarity 0.86)
- VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.85)
- ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response (ICML 2026 · similarity 0.85)
- ACPBench: Reasoning About Action, Change, and Planning (AAAI 2025 · similarity 0.85)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.85)
- DynaAct: Large Language Model Reasoning with Dynamic Action Spaces (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.