Research map / Large language models / Retrieval and knowledge
Language Models as Science Tutors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TutorEval, an expert-written long-context science QA benchmark graded by GPT-4 with key points, and TutorChat, an 80K-dialogue synthetic textbook dialogue dataset used to fine-tune Llemma-7B/34B into long-context math/science tutors.
Problem
Existing LM evaluations do not test real-life science tutoring, which requires free-form answers over long scientific textbook chapters, and fine-tuning on existing dialogue data performs poorly on such tasks.
Approach
Combine expert-written questions and key-point ground truths over long textbook chapters, grade free-form answers with GPT-4 using those key points, and build a long-context synthetic textbook dialogue dataset (TutorChat) that, mixed with MetaMath as MathMix, yields strong science/math tutors.
Key results
- Llemma-7B-32K-MathMix scores 50.5 on TutorEval and 46.6 on ClosedBook, competitive with Mistral-7B-V2 (50.9 / 53.8); GPT-4-Turbo scores 85.2.
- Llemma-34B-MathMix reaches 57.2 on TutorEval and 57.6 on ClosedBook.
- GPT-4 correctness human-grade of 91.9 vs GPT-4 auto-grade 85.2; GPT-4 correctness Pearson correlation with humans is 60.0 with key points versus 56.7 without.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llemma-7B-32K with TutorChat instead of UltraChat (ablation) | Llemma-7B-32K · TutorEval | correctness: 39.9 -> 48.4 | Helped |
| Add MetaMath to TutorChat-STEM (ablation) | Llemma-7B-32K · TutorEval | correctness: 47.8 -> 50.5 | Helped |
| Add MetaMath to TutorChat-STEM (ablation) | Llemma-7B-32K · TutorEval-ClosedBook | correctness: 48.2 -> 46.6 | Hurt |
| Add MetaMath to TutorChat (ablation) | Llemma-7B-32K · TutorEval | correctness: 48.4 -> 48.2 | No clear effect |
| Add MetaMath to UltraChat (ablation) | Llemma-7B-32K · TutorEval | correctness: 39.9 -> 35.0 | Hurt |
| Add 32K context extension (ablation) | Llemma-7B-MathMix · TutorEval | correctness: 46.4 -> 50.5 | Helped |
| Use Llemma instead of CodeLlama as the base model (ablation) | MathMix fine-tuning · TutorEval | correctness: 43.2 -> 50.5 | Helped |
| Use Llemma instead of Llama-2 as the base model (ablation) | MathMix fine-tuning · TutorEval | correctness: 30.8 -> 50.5 | Helped |
| Use GPT-4-generated TutorChat data instead of GPT-3.5-generated data (ablation) | Llemma-7B-32K fine-tuning · TutorEval | correctness | Helped |
| Use open-book dialogues instead of closed-book dialogues (ablation) | Llemma-7B-32K fine-tuning · TutorEval | correctness | Helped |
| Use dialogues simulating weak students instead of strong students (ablation) | Llemma-7B-32K fine-tuning · TutorEval misleading questions | correctness | Helped |
| Remove textbook chapters used to construct TutorEval from TutorChat-STEM training (ablation) | Llemma-7B-32K fine-tuning · TutorEval | correctness: 47.8 -> 50.4 | No clear effect |
Surprising findings
- Fine-tuning on UltraChat or MetaMath alone yields poor TutorEval scores, and adding MetaMath to TutorChat does not improve math performance; only the STEM subset combined with MetaMath did.
- Math-specialized models such as WizardMath-70B and MAmmoTH-70B score near zero on closed-book TutorEval, showing that math training does not transfer to free-form tutoring.
Limitations
- TutorEval is biased toward math and computer science (362 math and 205 CS questions of 834).
- GPT-4 judge correlates only moderately with humans (Pearson about 60 for correctness), and evaluation is tied to a specific GPT-4 version.
- Llemma-34B was not context-extended due to compute limits, and TutorChat is largely GPT-generated.
Experimental setup
Datasets: TutorChat MetaMath UltraChat OpenWebMath libretexts textbooks
Benchmarks: TutorEval TutorEval-ClosedBook GSM8K MATH
Metrics: correctness score (0-3, GPT-4 graded) presentation score Pearson correlation with human grades win/tie rates
Baselines: GPT-4 GPT-3.5-Turbo Mistral-7B-V2 Llama-3-70B Mixtral-8x7B Vicuna-13B-16K Zephyr-7B Tulu-2-DPO-70B CodeLlama-32K-MathMix Llemma-7B-32K-Ultrachat
Builds on: Llemma UltraChat MetaMath AlpacaFarm OpenWebMath
Closest papers by meaning
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.89)
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.88)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.87)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.86)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.