Research map / Large language models / Agents, code and math
Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a cognitive framework decomposing LLM learning into learning from instructor, concept, and experience, runs empirical studies on each, and releases the LearnArena benchmark built on TextArena games.
Problem
LLMs' general learning ability (acquiring and generalizing new knowledge from guidance, concepts, and experience) is underexplored compared with their static task performance.
Approach
Decomposes learning into three cognitively motivated dimensions (instructor, concept, experience), operationalizes each with targeted experiments (tutor-learner clarification, concept injection in TextArena games, in-context experience), and consolidates them into a multi-environment benchmark (LearnArena).
Key results
- Interactive clarification learners outperform passive consumption across all eight math benchmarks; e.g., Qwen2.5-32B learner gains +25.7% normalized on NumGLUE interactive vs +18.2% passive.
- GPT-4o achieves the highest average win rate of 0.77 on LearnArena, while Qwen3-14B reaches 0.60 among open models.
- Few-shot ICL performance peaks and then declines as examples grow beyond roughly 900, while 3 examples match tuning on 5k instances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow one clarification question after each instructor response instead of passive solution consumption | learner models trained from matched instructors · Eight math benchmarks | performance | Helped |
| increase learner size from Qwen2.5-1.5B to Qwen2.5-32B | Qwen2.5 learner trained by Qwen2.5-72B · Eight math benchmarks | normalized learning gain: +25.7% and +23.8% normalized improvement; +18.2% and +16.8% | Helped |
| use interactive clarification rather than passive instruction | Qwen2.5-1.5B learner · GSM8K and SAT | normalized improvement: +5.0% and +3.5%; +3.8% and +2.8% | Helped |
| use interactive clarification from a very small Qwen2.5-1.5B instructor rather than passive instruction | Qwen2.5-7B learner · GSM8K, MATH, and SAT | performance | Hurt |
| add static conceptual hints | Qwen2.5-7B · LogicGrid | accuracy: 0.36 -> 0.42 | Helped |
| add static conceptual hints | Qwen2.5-72B · LogicGrid | accuracy: 0.54 -> 0.57 | Helped |
| add static conceptual hints | Qwen2.5-7B · Plan | accuracy: 0.14 -> 0.17 | Helped |
| add static conceptual hints | Qwen2.5-1.5B · TextArena competitive environments | win rate | Hurt |
| include prior game trajectories as in-context experience | Qwen2.5-1.5B · Checkers | win rate: 0.30 -> 0.11 | Hurt |
| include prior game trajectories as in-context experience | Qwen2.5-1.5B · Stratego | win rate: 0.45 -> 0.20 | Hurt |
| include prior game trajectories as in-context experience | Qwen2.5-32B · Stratego | win rate: 0.6 to 0.8 | Helped |
| include prior game trajectories as in-context experience | Qwen2.5-32B · Truth and Deception | win rate: 0.6 to 0.9 | Helped |
Surprising findings
- Concept and experience injection degrade performance for the smallest model (Qwen2.5-1.5B), with win rates dropping (e.g., Checkers 0.30 to 0.11).
- Interactive clarification with a very small instructor (Qwen2.5-1.5B) underperforms passive learning on GSM8K, MATH, and SAT.
Limitations
- Evaluation focuses on mathematics and games; broader domains are left to future work.
- Results rely on fixed TextArena game settings with 20 matches per pair, so variance may be substantial.
- Concept hints are static and generated by a single large model, limiting generality.
Experimental setup
Datasets: MagpieMath GSM8K SVAMP MATH NumGLUE SimulEq AQuA SAT MMLU-Math TextArena Intlaw Willidea
Benchmarks: LearnArena GSM8K SVAMP MATH NumGLUE SimulEq AQuA SAT MMLU-Math TextArena
Metrics: accuracy win rate normalized accuracy
Baselines: GPT-4o GPT-4o-mini Llama-3.1-8B-Instruct Mistral-7B-Instruct-v0.3 Qwen2.5-32B-Instruct
Builds on: TextArena Magpie LlamaFactory
Closest papers by meaning
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.89)
- Can Large Language Models Derive High-Level Cognition from Low-Level and Fragmented Foundational Information? (AAAI 2025 · similarity 0.88)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.88)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.88)
- Core Knowledge Deficits in Multi-Modal Language Models (ICML 2025 · similarity 0.88)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.