Research map / Large language models / Agents, code and math
MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MIRAGE, a synthetic inductive-reasoning dataset with rule-induction and example-inference tasks across four scenarios, and argues LLMs are poor rule-based reasoners but good neighbor-based reasoners.
Problem
Existing inductive reasoning benchmarks evaluate only rule induction or only example inference, and use fixed test sets, so the full inductive-deductive process of LLMs and its mechanisms cannot be analyzed.
Approach
Synthesizes meta-rules from five vector operations (add, copy, map, pad, swap) and instantiates facts so input distribution, form, and difficulty can be varied; evaluates both rule induction and example inference, and attributes LLM success to nearby observed facts in feature space (neighbor-based reasoning) rather than correct rules.
Key results
- GPT-4o rule induction accuracy is consistently below example inference accuracy (e.g., 0.41 RI vs 0.68 EI on LT with D=3), indicating deduction without correct induced rules.
- Real-world problem (RP) scenario is hardest; GPT-4o reaches only 0.16 RI and 0.17 EI at D=8.
- Restricting observed facts to in-neighborhood facts raises GPT-4o LT accuracy from 0.52 baseline to 0.78 at N=3 and out-neighborhood facts drop it to 0.46.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide in-neighborhood observed facts rather than randomly sampled facts (ablation) | GPT-4o · Mirage, N=5, LT example inference | accuracy: 0.66 -> 0.84 | Helped |
| provide in-neighborhood observed facts rather than randomly sampled facts (ablation) | GPT-4o · Mirage, N=5, RP example inference | accuracy: 0.36 -> 0.52 | Helped |
| provide in-neighborhood observed facts rather than randomly sampled facts (ablation) | GPT-4o · Mirage, N=5, CG example inference | accuracy: 0.71 -> 0.84 | Helped |
| provide in-neighborhood observed facts rather than randomly sampled facts (ablation) | GPT-4o · Mirage, N=5, ST example inference | accuracy: 0.46 -> 0.63 | Helped |
| restrict observed facts to out-of-neighborhood facts rather than random sampling (ablation) | GPT-4o · Mirage, N=5, LT example inference | accuracy: 0.66 -> 0.49 | Hurt |
| restrict observed facts to out-of-neighborhood facts rather than random sampling (ablation) | GPT-4o · Mirage, N=5, RP example inference | accuracy: 0.36 -> 0.23 | Hurt |
| restrict observed facts to out-of-neighborhood facts rather than random sampling (ablation) | GPT-4o · Mirage, N=5, CG example inference | accuracy: 0.71 -> 0.57 | Hurt |
| restrict observed facts to out-of-neighborhood facts rather than random sampling (ablation) | GPT-4o · Mirage, N=5, ST example inference | accuracy: 0.46 -> 0.36 | Hurt |
| use CoT rather than direct input-output prompting | GPT-4o · Mirage, multiple scenarios | accuracy: up to 18% and 22% | Hurt |
| increase vector dimension from D=3 to D=8 | LLMs · Mirage real-world problems | accuracy | Hurt |
| use real-world natural-language scenarios instead of symbolic task forms | LLMs · Mirage | accuracy: 0.16 and 0.17 | Hurt |
| increase observed facts while tracking correction thresholds | LLMs · Mirage example inference versus rule induction | correction threshold | Helped |
Surprising findings
- Chain-of-thought-style methods often reduce accuracy relative to few-shot input-output prompting on MIRAGE.
- Example inference accuracy exceeds rule induction accuracy across most models, scenarios, and fact numbers, suggesting final answers rarely depend on the induced rule.
Limitations
- Analysis relies on black-box performance comparisons without internal mechanistic analysis.
- Experiments mainly use a limited set of models and prompting setups, with zero-shot IO and simple instructions rather than best-performing methods.
- Fine-tuning experiments show poor generalization across scenarios with different forms.
Experimental setup
Datasets: MIRAGE
Benchmarks: MIRAGE
Metrics: accuracy change rate inductive correction threshold deductive density
Baselines: chain-of-thought self-consistency self-refine hypothesis refinement input-output prompting inductive-deductive prompting
Builds on: chain-of-thought self-consistency self-refine hypothesis refinement
Closest papers by meaning
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.93)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.92)
- On the Inductive Bias of Stacking Towards Improving Reasoning (NeurIPS 2024 · similarity 0.88)
- Improving Reasoning Performance in Large Language Models via Representation Engineering (ICLR 2025 · similarity 0.88)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.88)
- Inductive Learning of Logical Theories with LLMs: A Expressivity-graded Analysis (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.