Research map / Large language models / Agents, code and math
Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| test on compositional proofs with non-compositional demonstrations (OOD) instead of compositional demonstrations (ID) (ablation) | GPT-3.5, PaLM, LLaMA, FLAN-T5 (8-shot CoT) · PrOntoQA-OOD compositional proofs | proof accuracy | No clear effect |
| omit demonstrations of the test deduction rule (conjunction rules, disjunction introduction) (ablation) | LLMs with CoT · PrOntoQA-OOD single rules | proof accuracy | No clear effect |
| omit demonstrations of disjunction elimination or proof by contradiction (ablation) | LLMs with CoT · PrOntoQA-OOD single rules | proof accuracy | Hurt |
| omit proof-by-contradiction demonstrations (model can use modus tollens) (ablation) | GPT-3.5 · PrOntoQA-OOD proof by contradiction | proof accuracy | No clear effect |
| increase test proof depth beyond demonstration depth 2 (ablation) | LLMs with CoT · PrOntoQA-OOD | proof accuracy | Hurt |
| increase test proof width beyond demonstration width 2 (ablation) | LLMs with CoT · PrOntoQA-OOD conjunction rules | proof accuracy | Mixed |
| remove distractors from in-context examples (ablation) | LLMs with CoT · PrOntoQA-OOD single rules | proof accuracy | No clear effect |
| include distractors in in-context examples (ablation) | GPT-3.5 · PrOntoQA-OOD implication and disjunction elimination | proof accuracy | Hurt |
| use larger model size | LLMs with CoT · PrOntoQA-OOD | proof accuracy | No clear effect |
Closest papers by meaning
- Are Language Models Efficient Reasoners? A Perspective from Logic Programming (NeurIPS 2025 · similarity 0.91)
- MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs (ICLR 2025 · similarity 0.91)
- Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Trace (NeurIPS 2026 · similarity 0.90)
- Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation (ICLR 2025 · similarity 0.90)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.89)
- Premise Order Matters in Reasoning with Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.