Research map / Large language models / Agents, code and math
SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciInstruct, a 254k-instruction scientific dataset (physics, chemistry, math, Lean proofs) built via a self-reflective GPT-4 chain-of-thought annotation pipeline, and fine-tunes ChatGLM3, Llama3-8B-Instruct and MetaMath-7B into SciGLM and related models.
Problem
LLMs struggle with college-level scientific reasoning, and high-quality instruction data with step-by-step reasoning for science is scarce because most available scientific QA pairs lack reasoning chains.
Approach
A self-reflective annotation pipeline in which GPT-4 generates CoT solutions for QA pairs, filters by answer correctness, and then iteratively critiques and revises failed solutions (with the reference answer as a hint in the final stage), followed by a trained instruction-quality classifier to filter noisy data.
Key results
- SciGLM (ChatGLM3-6B-Base) improves Avg. Sci from 38.12 to 41.40 and Avg. Sci+Math from 40.34 to 45.32 over the base model.
- SciGLM (ChatGLM3-32B-Base) reaches Avg. Math 57.46 versus 53.49 for ChatGLM3-32B-Base and Avg. Sci 45.48 versus 43.74.
- Fine-tuning on SciInstruct keeps general language performance roughly unchanged (MMLU 61.38 vs 61.32 for 6B), with MBPP code dropping from 55.80 to 45.00 for the 6B model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune ChatGLM3-6B-Base on SciInstruct | ChatGLM3-6B-Base · Scientific reasoning (weighted average) | accuracy: 38.12 -> 41.40 | Helped |
| Fine-tune ChatGLM3-32B-Base on SciInstruct | ChatGLM3-32B-Base · Scientific reasoning (weighted average) | accuracy: 43.74 -> 45.48 | Helped |
| Fine-tune ChatGLM3-6B-Base on SciInstruct | ChatGLM3-6B-Base · Mathematical reasoning (weighted average) | accuracy: 42.56 -> 49.24 | Helped |
| Fine-tune ChatGLM3-32B-Base on SciInstruct | ChatGLM3-32B-Base · Mathematical reasoning (weighted average) | accuracy: 53.49 -> 57.46 | Helped |
| Fine-tune Llama3-8B-Instruct on SciInstruct | Llama3-8B-Instruct · Scientific reasoning (weighted average) | accuracy: 34.66 -> 34.54 | No clear effect |
| Fine-tune Llama3-8B-Instruct on SciInstruct | Llama3-8B-Instruct · Mathematical reasoning (weighted average) | accuracy: 41.18 -> 37.55 | Hurt |
| Fine-tune Mistral-7B: MetaMATH on SciInstruct | Mistral-7B: MetaMATH · Scientific reasoning (weighted average) | accuracy: 28.76 -> 34.81 | Helped |
| Fine-tune Mistral-7B: MetaMATH on SciInstruct | Mistral-7B: MetaMATH · Mathematical reasoning (weighted average) | accuracy: 39.09 -> 41.02 | Helped |
| Fine-tune ChatGLM3-6B-Base on SciInstruct | ChatGLM3-6B-Base · General language and coding tasks | accuracy: 61.40 -> 57.85 | Hurt |
| Fine-tune ChatGLM3-32B-Base on SciInstruct | ChatGLM3-32B-Base · General language and coding tasks | accuracy: 69.06 -> 68.78 | No clear effect |
| Exclude the lowest 10% of samples using the filter step (ablation) | SciInstruct-trained SciGLM-6B · Filter-step evaluation (Ave. Sci) | weighted accuracy: 43.57 -> 43.85 | Helped |
| Exclude the lowest 10% of samples using the filter step (ablation) | SciInstruct-trained SciGLM-6B · Filter-step evaluation (Ave. Math) | weighted accuracy: 48.50 -> 49.24 | Helped |
Surprising findings
- Translating Chinese questions into English improved LLM solution correctness for some problems that failed in Chinese.
- Fine-tuning on SciInstruct markedly reduced MBPP code generation performance for the 6B model (55.80 to 45.00).
Limitations
- Dataset (254k) and model scale (6-8B, 32B) are limited; larger scale expected to help.
- Correctness labels rely on GPT-4 outcome-based (ORM-style) labeling rather than process-level verification, so intermediate reasoning errors may remain.
- Instruction-quality classifier was trained on a small labeled set and its error analysis indicates noise sensitivity.
Experimental setup
Datasets: SciInstruct LeanDojo MetaMath MAmmoTH
Benchmarks: CEval-Hard CEval-Sci MMLU-Sci SciEval SciBench GPQA Diamond GSM8K MATH Mathematics SAT-Math MMLU-Math CEval-Math
Metrics: accuracy pass@1 pass@K weighted average accuracy
Baselines: GPT-4 GPT-3.5-turbo Claude-v1.3 Galactica LLaMA-2 ChatGLM3-Base MAmmoTH WizardMath MetaMath-7B
Builds on: chain-of-thought prompting MetaMath LeanDojo DeepSpeed
Closest papers by meaning
- Training a Scientific Reasoning Model for Chemistry (NeurIPS 2025 · similarity 0.90)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.90)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.90)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.89)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- SciReason: A Controllable Benchmark for Scientific Reasoning in LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.