Research map / Large language models / Agents, code and math
Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Additional Logic Training (ALT), fine-tuning LLaMA-3.1 on a program-generated synthetic first-order logic corpus (PLD) built under four design principles, with Recall Adam to limit knowledge forgetting, yielding gains on logic, math, code, and NLI benchmarks.
Problem
LLMs struggle with reasoning on novel, unknown facts and rely on memorized knowledge; it is unclear how to design synthetic logic training samples to improve general reasoning of state-of-the-art LLMs.
Approach
Derives four design principles (unknown facts, illogical distractors, diverse deduction rules from axioms/theorems, diverse linguistic expressions) from symbolic logic theory and builds a large synthetic deduction corpus (PLD) that extends FLD, then trains LLMs on it with Recall Adam to prevent knowledge forgetting.
Key results
- ALT on PLD improves LLaMA-3.1-70B average from 60.0 to 64.4 across 31 benchmarks, with logic gains up to ~30 points (e.g., RobustLR 49.6 to 81.6).
- On LLaMA-3.1-8B, PLD-ALT reaches 52.0 average vs 47.9 for the baseline.
- Ablating any design principle generally degrades performance; training without Recall Adam underperforms the non-ALT baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Additional Logic Training on PLD | LLaMA-3.1-8B · 31-benchmark suite | micro-average: 47.9 -> 52.0 | Helped |
| Apply Additional Logic Training on PLD | LLaMA-3.1-70B · 31-benchmark suite | micro-average: 60.0 -> 64.4 | Helped |
| Use a large vocabulary for unknown facts (ablation) | PLD ALT on LLaMA-3.1-8B · 31-benchmark suite | micro-average: 51.4 -> 52.0 | Helped |
| Add adversarial distractors (ablation) | PLD ALT on LLaMA-3.1-8B · 31-benchmark suite | micro-average: 50.6 -> 52.0 | Helped |
| Include diverse axioms and representative theorems (ablation) | PLD ALT on LLaMA-3.1-8B · 31-benchmark suite | micro-average: 50.7 -> 52.0 | Helped |
| Include varied numbers of logical steps (ablation) | PLD ALT on LLaMA-3.1-8B · 31-benchmark suite | micro-average: 51.1 -> 52.0 | Helped |
| Use diverse linguistic expressions per formula (ablation) | PLD ALT on LLaMA-3.1-8B · 31-benchmark suite | micro-average: 51.3 -> 52.0 | Helped |
| Apply Additional Logic Training on PLD | LLaMA-3.1-70B · RobustLR | accuracy: 49.6 -> 81.6 | Helped |
| Apply Additional Logic Training on PLD | LLaMA-3.1-70B · GSM8K | accuracy: 80.9 -> 83.3 | Helped |
| Apply Additional Logic Training on PLD | LLaMA-3.1-70B · HumanEval | score: 32.3 -> 42.6 | Helped |
| Apply Additional Logic Training on PLD | LLaMA-3.1-70B · HELP | accuracy: 45.8 -> 51.3 | Helped |
| Apply Additional Logic Training on PLD | LLaMA-3.1-70B · SNLI | accuracy: 82.6 -> 82.3 | No clear effect |
Surprising findings
- Training without Recall Adam (knowledge-forgetting prevention) leads to performance below the untrained baseline.
- Abductive reasoning (AbductionRules) improved substantially despite training only on deductive samples.
Limitations
- Only deductive reasoning samples were used; abductive and inductive samples not explored.
- Only first-order predicate logic was examined; modal and linear logic left for future work.
- Gains on knowledge-heavy and multi-step procedural benchmarks (e.g., ARC, GPQA) are modest.
Experimental setup
Datasets: PLD (ProntoQA-style synthetic corpus, 100k train) FLD
Benchmarks: bAbI FOLIO LogicNLI RobustLR AR-LSAT LogiQA2 ReClor AbductionRules ART HELP MultiNLI RTE
Metrics: accuracy pass@1
Baselines: LLaMA-3.1-8B LLaMA-3.1-70B RuleTaker-ALT PARARULE-Plus-ALT FLD-ALT
Builds on: FLD RuleTaker RecAdam
Closest papers by meaning
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.91)
- Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.91)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.91)
- Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling (ICLR 2025 · similarity 0.91)
- Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation (ICLR 2025 · similarity 0.90)
- CRANE: Reasoning with constrained LLM generation (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.