Research map / Large language models / Agents, code and math
Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures abstract reasoning (AbsAcc) of LLMs over generic facts, builds the AbsR dataset of GPT-4-generated generic-fact-guided examples, and trains LLMs with a LoRA-based "meaningful learning" paradigm that implicitly uses facts.
Problem
LLMs answer simple questions supported by a shared generic fact inconsistently, showing a large gap between vanilla accuracy and abstract reasoning accuracy (AbsAcc), raising doubts about whether they reason or memorize.
Approach
Create a dataset (AbsR) whose examples pair questions, generic facts, and fact-guided explanations, then train LLMs with meaningful learning that jointly models conditional likelihoods with and without the fact (K-examples and R-examples), so the model internalizes facts without needing them at inference.
Key results
- MeanLearn 7B reaches 65.28 average vanilla accuracy vs 61.84 for Orca-2 7B, and 53.39 vs 48.86 AbsAcc average.
- MeanLearn 8B improves LLaMA-3 average vanilla accuracy from 66.20 to 67.12 and AbsR AbsAcc from 72.92 to 73.96.
- Preliminary study: all tested LLMs show over 17% gap between vanilla accuracy and AbsAcc on e-CARE, while humans show a minimal gap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with MeanLearn instead of the base Orca-2 model | Orca-2 7B · AbsR | vanilla accuracy: 73.50 -> 77.00 | Helped |
| train with MeanLearn instead of the base Orca-2 model | Orca-2 7B · Average | AbsAcc: 48.86 -> 53.39 | Helped |
| train with MeanLearn instead of the base LLaMA-3 model | LLaMA-3 8B · AbsR | AbsAcc: 72.92 -> 73.96 | Helped |
| train with MeanLearn instead of the base Orca-2 model | Orca-2 13B · AbsR | AbsAcc: 50.00 -> 45.82 | Hurt |
| add generic facts to training examples (ablation) | MeanLearn 7B · AbsR | vanilla accuracy: 73.50 -> 77.00 | Helped |
| add generic facts to training examples (ablation) | MeanLearn 7B · Average | AbsAcc: 49.84 -> 53.39 | Helped |
| add generic-fact-guided explanations (ablation) | MeanLearn 7B · AbsR | vanilla accuracy: 72.50 -> 77.00 | Helped |
| add generic-fact-guided explanations (ablation) | MeanLearn 7B · Average | AbsAcc: 47.91 -> 53.39 | Helped |
| use fact-guided explanations rather than unguided explanations (ablation) | MeanLearn 7B · Average | AbsAcc: 48.57 -> 53.39 | Helped |
| add generic facts to training examples (ablation) | MeanLearn 13B · AbsR | AbsAcc: 46.88 -> 45.82 | Hurt |
| add generic-fact-guided explanations (ablation) | MeanLearn 13B · Average | AbsAcc: 42.86 -> 50.51 | Helped |
| train with MeanLearn instead of Mistral | Mistral · Com. | vanilla accuracy: 66.53 -> 74.56 | Helped |
Surprising findings
- Orca-2 knows fewer generic facts as model size increases (12.78% for 13B vs 26.97% for 7B on the fact probing task).
- Training Orca-2 on AbsR* (same examples without generic-fact guidance) improves vanilla accuracy but not AbsAcc consistently.
Limitations
- Training data is small, so results are less stable and could improve with more data.
- Does not demonstrate that generated explanations are meaningful for the model; qualitative examples are missing.
- Focused on simple commonsense scenarios; complex generic facts such as causal chain or logical reasoning remain unexplored.
Experimental setup
Datasets: AbsR e-CARE GenericsKB MMLU AGIEval RACE BBH ARC
Benchmarks: AbsR AGIEval RACE BBH Commonsense (7 datasets) MMLU ARC-e ARC-c
Metrics: vanilla accuracy AbsAcc perplexity-based accuracy
Baselines: LLaMA-2 LLaMA-3 Vicuna WizardLM Orca-2 GPT-3.5 PaLM-2 Mistral
Builds on: Orca-2 LoRA GenericsKB e-CARE
Closest papers by meaning
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.91)
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking (ICLR 2026 · similarity 0.90)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.