Research map / Large language models / Language, safety and interpretability
Measuring Systematic Generalization in Neural Proof Generation with Transformers
Nicolas Gontier, Koustuv Sinha, Siva Reddy, Chris Pal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on backward-chaining (reversed) proofs instead of forward-chaining proofs (ablation) | 2.5M Transformer decoder · CLUTRR facts and amt stories | answer accuracy | Improved |
| use long exhaustive proofs instead of short proofs (ablation) | 2.5M Transformer decoder · CLUTRR | answer accuracy | Improved |
| generate a proof before the answer vs no-proof (answer directly) (ablation) | 2.5M Transformer decoder · CLUTRR facts stories | answer accuracy | Hurt |
| forward-chaining vs backward-chaining proofs (ablation) | 2.5M Transformer decoder · CLUTRR | proof validity | Improved |
| provide ground-truth proof in the input (ablation) | Transformer decoder · CLUTRR inductive and interpolation levels | answer accuracy | Improved |
| extrapolate to unseen harder levels (7-10) even when proof is given | Transformer decoder · CLUTRR extrapolation | answer accuracy | Hurt |
| train on noisy amt-template stories instead of facts-template | Transformer decoder · CLUTRR | answer accuracy and proof validity | Hurt |
| scale model from 2.5M to 145M parameters (ablation) | Transformer decoder · CLUTRR | answer accuracy | No clear effect |
Closest papers by meaning
- Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Trace (NeurIPS 2026 · similarity 0.91)
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.91)
- Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization (NeurIPS 2025 · similarity 0.89)
- When can transformers reason with abstract symbols? (ICLR 2024 · similarity 0.89)
- Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation (ICLR 2025 · similarity 0.89)
- Self-Verifying Reflection Helps Transformers with CoT Reasoning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
