Research map / Large language models / Retrieval and knowledge
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on generated hypernymy data and give no explicit hypernym rule (implicit reasoning) vs hypothesis only | RoBERTa-Large · Hypernymy test | accuracy (%): 65.2 -> 88.8 | Improved |
| add RuleTaker+20Q-only training instead of hypernymy training (no direct implicit training) (ablation) | RoBERTa-Large · Hypernymy test, Implicit Reasoning | accuracy (%): 65.4 -> 79.1 (vs 88.8 with direct training) | Improved |
| replace RoBERTa-Large with ESIM + GloVe (ablation) | hypernymy reasoning · Hypernymy Implicit Reasoning | accuracy (%): 88.8 -> 76.1 | Hurt |
| intervene by adding correct rules for the model's incorrect implicit beliefs (ablation) | RoBERTa implicit reasoning · Hypernymy test | accuracy (%): 88.8 -> 98.3 | Improved |
| intervene by adding correct rules for incorrect beliefs (ablation) | RoBERTa implicit reasoning · Imaginary | accuracy (%): 76.9 -> 94.1 | Improved |
| add explicit member facts and counting training to hypothesis-only | RoBERTa-Large · Counting (1,K-1) subset | accuracy (%): 64.1 -> 73 | Improved |
| drop the quantity fact from test examples (ablation) | RoBERTa counting model · Counting (1,K-1) subset | accuracy (%): 73 -> 64.4 | Hurt |
| train on all four datasets instead of subsets (ablation) | RoBERTa-Large · Multi-skill-set | overall accuracy (%): 73.5 -> 85.4 | Improved |
| add Counting data to RuleTaker+20Q training (ablation) | RoBERTa-Large · Multi-skill-set counting skill | accuracy (%): 55.3 -> 83.0 | Improved |
| add member facts in ascending (least-known first) rather than descending model-probability order (ablation) | RoBERTa counting model · Counting analysis, c=0.25 | accuracy: +5.5 points at c=0.25, +2.7 at c=0.75 | Improved |
| test with explicit knowledge present but no hypernym rule vs no context (distractors) (ablation) | RoBERTa · Hypernymy Hypothesis-only vs Statement-only-no-context | accuracy (%): 65.2 vs 66.7 | No clear effect |
Closest papers by meaning
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.90)
- Logically Consistent Language Models via Neuro-Symbolic Integration (ICLR 2025 · similarity 0.89)
- How do Transformers Learn Implicit Reasoning? (NeurIPS 2025 · similarity 0.89)
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.89)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.89)
- When can transformers reason with abstract symbols? (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
