Research map / Large language models / Reasoning and chain of thought
Premise Order Matters in Reasoning with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs are brittle to the order of premises in deductive reasoning (modus ponens logic proofs) and math word problems (R-GSM), performing best when premise order matches the ground-truth proof.
Problem
LLM reasoning performance may depend on the order of premises even though reordering premises does not change the underlying deductive task or the answer.
Approach
Systematically varies premise order using Kendall tau distance to the forward proof order (SimpleLogic-derived benchmark with distracting rules) and builds R-GSM, a GSM8K variant with reordered sentences, showing accuracy drops of up to 30% or more, with model-specific preferences for backward or other orders.
Key results
- Accuracy decreases from premise reordering can exceed 30% on logical reasoning; for GPT-3.5-turbo and Gemini 1.0 Pro, changing order from forward can drop accuracy from over 65% to below 25%.
- On R-GSM, GPT-4-turbo accuracy falls from 94.1% to 85.0% and GPT-3.5-turbo from 67.3% to 51.8% on the full set; on initially solved problems, GPT-3.5-turbo drops by more than 35%.
- GPT-4-turbo and GPT-3.5-turbo perform better with the backward order than random shuffles, while PaLM 2-L performs worst with the backward order.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Arrange premises in forward proof order instead of shuffled order | LLMs on logical reasoning with 12 relevant rules and no distractors · Logical reasoning | accuracy: 80.8% -> 96.5% (GPT-4-turbo); 66.5% -> 88.0% (PaLM 2-L); 0.2% -> 16.5% (Gemini 1.0 Pro); 1.2% -> 30.0% (GPT-3.5-turbo) | Helped |
| Arrange premises in backward proof order instead of shuffled order | GPT-4-turbo on logical reasoning with 12 relevant rules and no distractors · Logical reasoning | accuracy: 80.8% -> 84.0% | Helped |
| Arrange premises in backward proof order instead of shuffled order | PaLM 2-L on logical reasoning with 12 relevant rules and no distractors · Logical reasoning | accuracy: 66.5% -> 57.5% | Hurt |
| Reorder problem-description sentences | LLMs on R-GSM, full dataset · R-GSM | accuracy: 94.1% -> 85.0% (GPT-4-turbo); 86.4% -> 79.5% (PaLM 2-L); 80.5% -> 69.1% (Gemini 1.0 Pro); 67.3% -> 51.8% (GPT-3.5-turbo) | Hurt |
| Reorder problem-description sentences | LLMs on R-GSM problems their original versions correctly solve · R-GSM, initially solved subset | accuracy: 100% -> 89.9% (GPT-4-turbo); 100% -> 87.9% (PaLM 2-L); 100% -> 74.6% (Gemini 1.0 Pro); 100% -> 64.9% (GPT-3.5-turbo) | Hurt |
| Add distracting rules | GPT-4-turbo and PaLM 2-L on logical reasoning · Logical reasoning | accuracy | Hurt |
| Increase the number of rules required in the proof | LLMs on logical reasoning · Logical reasoning | accuracy | Hurt |
| Increase the number of reasoning steps or problem sentences | LLMs on R-GSM · R-GSM | accuracy | Hurt |
| Move relevant rules among the beginning, middle, and end while keeping order and number fixed (ablation) | PaLM 2-L with 10 distracting rules · Logical reasoning | accuracy: 68.0% / 67.0% / 67.0% (8 rules); 36.5% / 36.5% / 35.0% (12 rules) | No clear effect |
Surprising findings
- PaLM 2-L performs worst with the exact reverse (backward) order, unlike GPT-4-turbo which prefers it over other non-forward orders.
- Lost-in-the-middle position effects were found to be negligible on these short inputs; order, not position, drives the performance gap.
Limitations
- Evaluation limited to modus ponens problems and a small set of closed API models.
- R-GSM is small (220 problem pairs) and manually rewritten.
- Explanation of the ordering effect (auto-regressive design or training data bias) is left untested.
Experimental setup
Datasets: SimpleLogic GSM8K
Benchmarks: R-GSM logical reasoning premise-order benchmark (SimpleLogic-derived)
Metrics: accuracy error type breakdown (wrong refutation, rule hallucination,
Baselines: GPT-3.5-turbo PaLM 2-L Gemini 1.0 Pro
Builds on: SimpleLogic GSM8K Kendall tau distance lost-in-the-middle
Closest papers by meaning
- Are Language Models Efficient Reasoners? A Perspective from Logic Programming (NeurIPS 2025 · similarity 0.89)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.89)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.89)
- The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models (ICML 2026 · similarity 0.88)
- RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation (ICML 2025 · similarity 0.88)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.