Research map / Large language models / Instruction and fine-tuning
SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SciLitLLM, a Qwen2.5-based scientific literature model trained with a hybrid of continual pre-training on filtered scientific PDFs and SFT on a synthetic instruction set (SciLitIns), reaching state-of-the-art among open models under 10B and competitive at 14B on SciAssess and SciRIFF.
Problem
LLMs lack scientific knowledge and familiarity with specialized scientific literature understanding tasks, and building domain corpora from PDFs and domain instruction data is costly and noisy.
Approach
Combine CPT on a PDF-extracted, LLM-corrected, classifier-filtered scientific corpus with SFT on synthetic scientific instructions generated by GPT-4o from keyword-sampled contexts and task lists, followed by LLM-based instruction quality filtering.
Key results
- SciLitLLM-7B averages 51.7 on SciAssess vs 47.7 for Llama3.1-8B (about +4.0) and 60.6 on SciRIFF vs 50.5 for Llama3.1-8B (+10.1).
- SciLitLLM-14B reaches 58.2 on SciAssess and 64.4 on SciRIFF, above Llama3.1-70B (57.1, 58.5) and Qwen2.5-72B (57.0, 58.2).
- Ablation: CPT adds +1.9 on SciAssess and +3.6 on SciRIFF over Qwen2.5-Base+SFT; SciLitIns with SciRIFF improves SciRIFF from 56.7 to 60.6 when added to SFT data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CPT before SFT (ablation) | Qwen2.5-7B-Base · SciAssess | score: 49.8 -> 51.7 | Helped |
| Apply CPT before SFT (ablation) | Qwen2.5-7B-Base · SciRIFF | score: 57.0 -> 60.6 | Helped |
| Add SciRIFF to Infinity-Instruct (ablation) | Qwen2.5-7B-Base + CPT SFT · SciAssess | score: 47.1 -> 46.8 | Hurt |
| Add SciRIFF to Infinity-Instruct (ablation) | Qwen2.5-7B-Base + CPT SFT · SciRIFF | score: +5.5% | Helped |
| Add SciLitIns to Infinity-Instruct instead of SciRIFF (ablation) | Qwen2.5-7B-Base + CPT SFT · SciAssess | score: 47.1 -> 50.2 | Helped |
| Add SciLitIns to Infinity-Instruct instead of SciRIFF (ablation) | Qwen2.5-7B-Base + CPT SFT · SciRIFF | score: 51.2 -> 54.8 | Helped |
| Combine SciRIFF and SciLitIns with Infinity-Instruct (ablation) | Qwen2.5-7B-Base + CPT SFT · SciAssess | score: 50.2 -> 51.7 | Helped |
| Combine SciRIFF and SciLitIns with Infinity-Instruct (ablation) | Qwen2.5-7B-Base + CPT SFT · SciRIFF | score: 54.8 -> 60.6 | Helped |
| Apply the instruction quality filter to SciLitIns (ablation) | SciLitLLM-7B · SciAssess | score: +0.8% | Helped |
| Apply the instruction quality filter to SciLitIns (ablation) | SciLitLLM-7B · SciRIFF | score: +0.5% | Helped |
| Vary the CPT data filtering threshold from 15% to 25% to 35% (ablation) | SciLitLLM-7B-Base · MMLU-Pro-Bio, MMLU-Pro-Chem, MMLU-Pro-Heal, MaScQA (Avg.) | accuracy (%): 58.32; 58.75; 57.67 | Mixed |
| Use SciLitLLM domain adaptation rather than the Qwen2.5 base model | Qwen2.5-7B-Base · MMLU-Pro-Bio, MMLU-Pro-Chem, MMLU-Pro-Heal, MaScQA (Avg.) | accuracy (%): 56.15 -> 58.75 | Helped |
Surprising findings
- SFT on general Infinity-Instruct alone gives the lowest scores, and adding SciRIFF improves SciRIFF but slightly hurts SciAssess (-0.3).
- The LLM-based instruction quality filter yields only modest gains (+0.8 SciAssess, +0.5 SciRIFF), suggesting synthetic data was already fairly high quality.
Limitations
- CPT corpus is small (about 12.7B tokens after filtering) relative to typical pre-training datasets.
- No chain-of-thought or tree-of-thought reasoning enhancement is explored.
- No RLHF or preference alignment due to budget constraints.
Experimental setup
Datasets: In-house textbooks In-house journal papers RedPajama SciLitIns SciRIFF Infinity-Instruct MMLU-Pro MaScQA
Benchmarks: SciAssess SciRIFF MMLU-Pro MaScQA
Metrics: accuracy F1 ROUGE mean score across tasks
Baselines: Qwen2.5-7B-Instruct Llama3.1-8B Mistral-7B SciTulu-7B Qwen2.5-14B Llama3.1-70B Qwen2.5-72B GPT-3.5 GPT-4o Galactica-6.7B
Builds on: Qwen2.5 fineweb-edu-classifier SciRIFF Phi educational-value filtering vLLM
Closest papers by meaning
- Adapting While Learning: Grounding LLMs for Scientific Problems with Tool Usage Adaptation (ICML 2025 · similarity 0.88)
- Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains (ICML 2024 · similarity 0.87)
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.87)
- Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.87)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.87)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.