Research map / Large language models / Agents, code and math
Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies LMs' inductive reasoning via iterative hypothesis refinement (propose, select with a symbolic interpreter, refine), finding LMs are strong hypothesis proposers but puzzling reasoners that fail to apply their own rules, are brittle to noisy exemplars, and produce human-unlike rules.
Problem
Whether language models can perform inductive reasoning, i.e., inducing general rules from a few input-output observations and applying them to unseen examples, and how their behavior compares to humans.
Approach
Evaluate LMs by having them propose textual rule hypotheses that are scored and selected by a task-specific symbolic interpreter over seen examples, then iteratively refined with execution feedback, rather than predicting outputs directly with IO prompting; then probe the induced rules for application ability, robustness, and human-likeness.
Key results
- Iterative hypothesis refinement (T=3, N=5) with GPT-4 reaches 82.5 raw / 59.0 task accuracy on ACRE vs. 64.0 / 28.0 for IO prompting.
- On MiniSCAN, applying GPT-4's own proposed rules with the LM as interpreter drops task accuracy from above 80% to near zero, while the symbolic interpreter retains high accuracy.
- Adding 12.5% noisy exemplars to List Functions lowers GPT-4 raw accuracy from 51.6 to 49.6 at T=1,N=1 is not the case; larger noise (50%) drops it to 23.1 raw accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iterative hypothesis refinement with symbolic interpreters instead of standard IO prompting | GPT-4 · ACRE | raw accuracy: 64.0 -> 82.5 | Helped |
| Use iterative hypothesis refinement with symbolic interpreters instead of standard IO prompting | GPT-4 · MiniSCAN | task accuracy: 0.0 -> 95.0 | Helped |
| Use iterative hypothesis refinement with symbolic interpreters instead of standard IO prompting | GPT-4 · MiniARC | raw accuracy: 33.1 -> 18.7 | Hurt |
| Increase hypotheses per iteration from N=1 to N=5 (ablation) | Hypothesis refinement with GPT-4 · ACRE | raw accuracy: 77.8 -> 82.5 | Helped |
| Increase hypotheses per iteration from N=1 to N=5 (ablation) | Hypothesis refinement with GPT-4 · MiniSCAN | raw accuracy: 98.2 -> 93.3 | Hurt |
| Replace the symbolic interpreter with the LM as interpreter (ablation) | GPT-4 rule application · MiniSCAN | task accuracy: more than 80% -> almost-zero | Hurt |
| Use Python hypotheses instead of natural-language hypotheses (ablation) | Hypothesis refinement with GPT-4 · List Functions | raw accuracy: 71.2 -> 72.5 | No clear effect |
| Add task-specific visual and object heuristics to hypothesis-generation prompts (ablation) | Hypothesis refinement with GPT-4 · MiniARC | raw accuracy: 17.8% versus 18.7 | No clear effect |
| Perturb 12.5%, 25%, and 50% of the exemplars (ablation) | Hypothesis refinement with GPT-4 (T=1, N=1) · List Functions | raw accuracy: 51.6 -> 49.6; 36.7; 23.1 | Hurt |
| Use pseudowords instead of English words as outputs (ablation) | Hypothesis refinement with GPT-4 (T=1, N=1) · MiniSCAN | raw accuracy: 77.0 -> 72.0 | Hurt |
| Use GPT-4V instead of a symbolic interpreter to apply proposed rules (ablation) | GPT-4V rule prompting · MiniARC | raw accuracy: 6.0 -> 0.5 | Hurt |
| Add zero-shot chain-of-thought prompting for LM rule application (ablation) | GPT-4 rule application · ACRE | task accuracy: 47.0 -> 25.0 | Hurt |
Surprising findings
- LMs frequently fail to apply the rules they themselves proposed, with the gap between symbolic and LM interpretation being very large on MiniSCAN.
- Increasing sampling temperature for self-consistency does not improve performance, while increasing the number of hypotheses with selection does.
Limitations
- Tasks are synthetic and symbolic, limiting generalization to everyday human induction.
- Hyperparameters T and N were limited by context length and not exhaustively tuned.
- Possible confound from pretraining contamination makes it hard to separate induction from recall.
Experimental setup
Datasets: ACRE MiniSCAN List Functions MiniARC
Benchmarks: ACRE MiniSCAN List Functions MiniARC
Metrics: raw accuracy task accuracy clarity supportiveness
Baselines: IO prompting self-consistency prompting self-refine GPT-3.5 Claude-2 LLaMA2-70B GPT-4V
Builds on: self-refine self-consistency prompting zero-shot chain-of-thought ACRE MiniSCAN List Functions MiniARC
Closest papers by meaning
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.93)
- MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models (ICLR 2025 · similarity 0.93)
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.90)
- Reasoning Scaffolding: Distilling the Flow of Thought from LLMs (ICLR 2026 · similarity 0.90)
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning (ICML 2025 · similarity 0.90)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.