Research map / Large language models / Agents, code and math
SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SocraticLM, a ChatGLM3-6b model fine-tuned on SocraTeach, a GPT-4-simulated multi-agent (Dean-Teacher-Student) Socratic tutoring dataset, plus a five-dimension teaching evaluation system.
Problem
Current LLM-based personalized teaching follows a question-answering paradigm that gives students answers directly rather than engaging them in Socratic, thought-provoking dialogue.
Approach
A Dean-Teacher-Student multi-agent pipeline (GPT-4 agents, with a supervisory Dean that revises Teacher turns) generates Socratic multi-round dialogues over six simulated student cognitive states, augmented with single-round data for four teaching abilities, and three training strategies protect problem-solving ability while fine-tuning.
Key results
- SocraticLM achieves Overall teaching quality of 0.62 versus 0.50 for GPT-4, outperforming GPT-4 by 12% on the overall win-rate metric.
- SocraticLM improves SRR to 0.78 versus GPT-4's 0.55 and IARA to 0.83 versus 0.76 for GPT-4.
- Fine-tuning without problem-solving data drops GSM8K accuracy by 31.2% relative to ChatGLM3-6b, and the three balancing strategies mitigate this.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune ChatGLM3-6b on SocraTeach | ChatGLM3-6b · Socratic teaching evaluation | Overall: 0.50 -> 0.62 | Helped |
| add single-round DiaS teaching dialogues (ablation) | SocraticLM · Socratic teaching evaluation | Overall: 0.54 -> 0.62 | Helped |
| add Irrelevant-response dialogues (ablation) | SocraticLM · Socratic teaching evaluation | Overall: 0.57 -> 0.62 | Helped |
| add Questioning-response dialogues (ablation) | SocraticLM · Socratic teaching evaluation | Overall: 0.58 -> 0.62 | Helped |
| add Incorrect-reply dialogues (ablation) | SocraticLM · Socratic teaching evaluation | Overall: 0.51 -> 0.62 | Helped |
| add Correct-reply dialogues (ablation) | SocraticLM · Socratic teaching evaluation | CARA: 0.58 -> 0.98 | Helped |
| add problem-solving training data (ablation) | SocraticLM · GSM8K | accuracy: 0.312 -> 0.606 | Helped |
| use separate training for dialogue data and problem-solving data (ablation) | SocraticLM · GSM8K | accuracy: 0.159 -> 0.606 | Helped |
| use instruction tuning (ablation) | SocraticLM · Socratic teaching evaluation | Overall: 0.02 -> 0.62 | Helped |
| use mixed-prompt setting (ablation) | SocraticLM · GSM8K | accuracy: 0.605 -> 0.606 | No clear effect |
| increase multi-round dialogue training data from 75% to 100% (ablation) | SocraticLM · Socratic teaching evaluation | Overall | Helped |
| increase problem-solving data beyond the balanced ratio (ablation) | SocraticLM · GSM8K | accuracy: -1.9% | Hurt |
Surprising findings
- Training only on dialogue data without problem-solving data sharply degrades math accuracy, while the balanced model slightly improves MAWPS accuracy over ChatGLM3-6b.
- Adding only "w/o Correct" single-round data lowers CARA below the model with no single-round data at all, suggesting overfitting to incorrect-reply patterns.
Limitations
- SER of 0.74 shows room for improvement in responding to complex student questions.
- Only mathematical word problems at primary-school level were studied; other subjects need new data and training.
- Problem-solving evaluation uses the same GSM8K and MAWPS problems used to build dialogues, so it does not fully measure reasoning change.
Experimental setup
Datasets: MAWPS GSM8K MOOC student questions
Benchmarks: SocraTeach test split GSM8K MAWPS
Metrics: Overall quality (normalized win rate) IARA CARA SER SRR BLEU-4 ROUGE accuracy
Baselines: ChatGPT GPT4 Vicuna-7b Llama2-7b Llama2-13b Llama3-8b ChatGLM3-6b EduChat-32b
Builds on: Socratic method P-Tuning MathDial EduChat
Closest papers by meaning
- SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction (ICLR 2025 · similarity 0.87)
- Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.86)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.86)
- ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Education (ICML 2026 · similarity 0.86)
- MASTER: Enhancing Large Language Model via Multi-Agent Simulated Teaching (NeurIPS 2025 · similarity 0.86)
- MIND: Math Informed syNthetic Dialogues for Pretraining LLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.