Research map / Large language models / Agents, code and math
Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Refines the StepGame benchmark by correcting template errors, shows a logic-based template-to-relation mapping plus ASP solver achieves 100% accuracy, and improves GPT spatial reasoning with customized CoT and ToT prompting.
Problem
StepGame, a benchmark for multi-hop spatial reasoning in text, contains crowd-sourced sentence templates whose meanings do not match their intended labels, distorting evaluations of LLMs such as ChatGPT.
Approach
Corrects the erroneous StepGame templates and filters unfixable examples, proposes a non-LLM solution (template-based sentence-to-relation mapping plus ASP reasoning), and customizes chain-of-thought and tree-of-thoughts prompting for LLM spatial reasoning via object-linking chains.
Key results
- Template-to-relation mapping plus ASP reasoning reaches 100% accuracy across k=1 to 10 on the corrected StepGame clean set.
- With ToT prompting, GPT-4 maintains around 90% accuracy at higher hops, while Davinci stays near 50% and Turbo near 30%.
- Base GPT-4 accuracy drops from 100% at k=1 to 25% at k=10; CoT raises GPT-4 accuracy at k=10 to about 65%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use customized CoT instead of the base prompt | GPT-4 · Corrected StepGame k=10 | accuracy: 25 -> 65 | Helped |
| Build the linking chain with ToT, then apply CoT instead of using the base prompt | GPT-4 · Corrected StepGame k=10 | accuracy: 25 -> 95 | Helped |
| Build the linking chain with ToT before applying CoT | GPT-4 with CoT · Corrected StepGame k=10 | accuracy: 65 -> 95 | Helped |
| Use customized CoT instead of the base prompt | Davinci · Corrected StepGame k=10 | accuracy: 22 -> 32 | Helped |
| Build the linking chain with ToT, then apply CoT instead of using the base prompt | Davinci · Corrected StepGame k=10 | accuracy: 22 -> 50 | Helped |
| Build the linking chain with ToT before applying CoT | Davinci with CoT · Corrected StepGame k=10 | accuracy: 32 -> 50 | Helped |
| Use customized CoT instead of the base prompt | GPT-3.5-Turbo · Corrected StepGame k=10 | accuracy: 20 -> 24 | Helped |
| Build the linking chain with ToT, then apply CoT instead of using the base prompt | GPT-3.5-Turbo · Corrected StepGame k=10 | accuracy: 20 -> 35 | Helped |
| Build the linking chain with ToT before applying CoT | GPT-3.5-Turbo with CoT · Corrected StepGame k=10 | accuracy: 24 -> 35 | Helped |
| Use sentence-to-relation mapping plus ASP reasoning instead of the SOTA approach | StepGame reasoning system · Corrected StepGame k=10 | accuracy: 88.3 -> 100 | Helped |
| Use Davinci for semantic parsing with ASP reasoning instead of Curie for semantic parsing with ASP reasoning | GPT parser + ASP · Corrected StepGame k=10 | accuracy: 61 -> 100 | Helped |
| Select examples across 1-, 3-, 5-, 7-, and 10-hop tasks rather than one example from each 1–10-hop task | Few-shot prompting · Corrected StepGame | accuracy | Helped |
Surprising findings
- 14 of 214 StepGame templates contain meaning errors, and 4 of them cannot be repaired, so affected examples had to be removed.
- Clean 5-shot examples drawn from hops 1,3,5,7,10 outperform a 10-shot set covering hops 1 to 10 at almost every hop level.
Limitations
- ToT experiments on GPT-4 use only 20 test instances, so estimates are noisy.
- The template-based solution requires manual template updates and does not adapt to novel phrasing.
- Results are limited to StepGame, a synthetic controlled-language benchmark.
Experimental setup
Datasets: StepGame
Benchmarks: StepGame (corrected clean set)
Metrics: accuracy
Baselines: ChatGPT Curie+ASP SOTA (GPT-3 parsing + ASP) base input-output prompting
Builds on: chain-of-thought tree-of-thoughts StepGame Clingo
Closest papers by meaning
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.91)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- ChainSpace: A Chained-Reasoning Paradigm for Spatial Intelligence (NeurIPS 2026 · similarity 0.90)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
- LTD-Bench: Evaluating Large Language Models by Letting Them Draw (NeurIPS 2025 · similarity 0.89)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.