Research map / Large language models / Reasoning and chain of thought
MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MuSR, a 756-instance benchmark of multistep soft reasoning over LLM-generated natural-language narratives (murder mysteries, object placements, team allocation), built with a neurosymbolic reasoning-tree-to-story pipeline, and evaluates GPT-4, GPT-3.5, Llama 2 and Vicuna with chain-of-thought variants.
Problem
LLM reasoning benchmarks are static and either too artificial/rule-solvable or lack natural text combined with commonsense and multistep reasoning, so they fail to test the limits of chain-of-thought on complex reasoning from narratives.
Approach
Generate reasoning trees with gold facts via a validated GPT-4 recursive deduction process, then produce long natural narratives chunk by chunk ("chaptering") that embed only the scenario facts, so the reasoning must be recovered from text; the generator can be scaled as models improve.
Key results
- GPT-4 with CoT+ reaches 80.4 on murder mysteries, 60.9 on object placements and 68.4 on team allocation, below human majority vote (94.1, 95.0, 100.0).
- 1-shot CoT+ raises GPT-4 to 86.0 (murder mysteries), 72.3 (object placements) and 88.4 (team allocation).
- Rule-based baselines score near chance (50.0 murder mystery, 35.9 object placements), and fact recall rises from 67 to 95 with chaptering and validators.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 1-shot CoT+ instead of CoT+ | GPT-4 · Murder Mystery | accuracy: 80.4 -> 86.0 | Helped |
| use 1-shot CoT+ instead of CoT+ | GPT-4 · Object Placements | accuracy: 60.9 -> 72.3 | Helped |
| use 1-shot CoT+ instead of CoT+ | GPT-4 · Team Allocation | accuracy: 68.4 -> 88.4 | Helped |
| add three examples to 1-Shot CoT+ | GPT-4 · Murder Mystery | accuracy: 86.0 -> 84.8 | Hurt |
| add three examples to 1-Shot CoT+ | GPT-4 · Object Placements | accuracy: 72.3 -> 71.5 | Hurt |
| add three examples to 1-Shot CoT+ | GPT-4 · Team Allocation | accuracy: 88.4 -> 89.6 | Helped |
| add chaptering to story generation (ablation) | MuSR dataset creation · Murder Mystery | fact recall: 67 -> 93 | Helped |
| add chaptering to story generation (ablation) | MuSR dataset creation · Object Placements | fact recall: 83 -> 82 | No clear effect |
| add tree validators (ablation) | MuSR dataset creation · Murder Mystery | accuracy: 60 -> 84 | Helped |
| add tree validators (ablation) | MuSR dataset creation · Object Placements | accuracy: 65 -> 58 | Hurt |
| use chaptering and tree validators instead of removing both (ablation) | MuSR dataset creation · Object Placements | accuracy: 78 -> 58 | Hurt |
| use chaptering and tree validators instead of removing both (ablation) | MuSR dataset creation · Murder Mystery | accuracy: 60 -> 84 | Helped |
Surprising findings
- Llama 2 70B and Mixtral could not follow the construction prompts to generate valid reasoning trees, while GPT-3.5-turbo could only do so with prompt edits and produced hallucinated deductions.
- Adding more few-shot examples did not reliably help; PAL 1-shot outperformed the end-to-end GPT-4 approach on team allocation.
Limitations
- Generation relies on GPT-4, so the benchmark's validity depends on GPT-4-quality narratives; weaker generators failed.
- Reasoning trees are shallow (depth one or two) because GPT-4 struggled to produce deeper recursive deductions.
- Human evaluation used a small panel of 7 annotators on a subset of instances and Mechanical Turk attempts failed.
Experimental setup
Datasets: True Detective
Benchmarks: MuSR
Metrics: accuracy
Baselines: GPT-4 CoT+ GPT-3.5 Llama 2 70B Chat Vicuna 33B v1.3 Decomposed Prompting SymbolicTOM PAL
Builds on: chain-of-thought prompting Decomposed Prompting Program-Aided Language Models SymbolicTOM
Closest papers by meaning
- Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning (AAAI 2026 · similarity 0.90)
- LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking (ICLR 2026 · similarity 0.89)
- Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space (NeurIPS 2025 · similarity 0.89)
- Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.89)
- Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization (AAAI 2026 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.