Research map / Large language models / Instruction and fine-tuning
Fine-tuning generalization and delta compression
Study fine-tune deltas, emulation, and mixtures to preserve generality when specializing LLMs.
159 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Instruction and fine-tuning, keyword cluster “instruction tuning · instructions · instruction following”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 3 tracked benchmarks.
35 papers in 2023–24 and 124 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 16.8% to 19.0%.
Open questions the data points to
- GSM8K (accuracy): best went from 46.3 (up to 2024) to 94 in 2025 — Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms (NeurIPS 2025 · 0 citations)
- MMLU (accuracy): best went from 61.8 (up to 2024) to 76 in 2025 — The Elicitation Game: Evaluating Capability Elicitation Techniques (ICML 2025 · 0 citations)
- Papers report LLaMA2-13B on GSM8K (accuracy) anywhere from 28.7 to 33.89: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs (NeurIPS 2023 · 21 citations)
- 2024 · Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · 110 citations)
- 2025 · SWIFT: A Scalable Lightweight Infrastructure for Fine-Tuning (AAAI 2025 · 40 citations)
- 2026 · Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning (ICML 2026 · 1 citation)
Earliest papers here
- The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs (NeurIPS 2023 · 21 citations)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · 110 citations)
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR 2024 · 105 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
GSM8K · accuracy (higher is better)
8 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 46.3 | Interpreting and Improving Large Language Models in Arithmetic Calculation (ICML 2024 · 1 citation) |
| 2025 | 94 | Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms (NeurIPS 2025 · 0 citations) |
MMLU · accuracy (higher is better)
5 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 61.8 | LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · 14 citations) |
| 2025 | 76 | The Elicitation Game: Evaluating Capability Elicitation Techniques (ICML 2025 · 0 citations) |
MT-Bench · score (higher is better)
4 papers, 2024–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 7.3 | HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · 2 citations) |
What has been tried
89 papers here report 1,042 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 7%, hurt 21%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add continuous pre-training on medical corpus | CMtMedQA-test · Professionalism | Helped | Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · 110 citations) | |
| Use ShareGPT instead of Alpaca as seed data for Evol-Instruct | Automatic benchmarks · Avg. | 58.96 -> 61.87 | Helped | WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR 2024 · 105 citations) |
| Add loss-scale | ToolBench in-domain · Act.EM | 55.71 -> 58.15 | Helped | SWIFT: A Scalable Lightweight Infrastructure for Fine-Tuning (AAAI 2025 · 40 citations) |
| Use autoregressive generation (ARG) instead of MAE pre-training | MIMIC-CXR · BLEU-4 | 0.104 -> 0.130 | Helped | CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset (CVPR 2025 · 20 citations) |
| Increase the KL penalty coefficient | TL;DR and CNN/DailyMail · GPT-4 preference vs reference | Hurt | Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR 2024 · 14 citations) | |
| use Adam instead of SGD for instruction tuning | MMLU · accuracy | 45.5 -> 47.2 | Helped | LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · 14 citations) |
| Increase URIAL examples from three to eight | URIAL · safety | 4.29 to 4.97 | Helped | The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · 10 citations) |
| Replace entity or noun-chunk extraction with atomic question extraction | Biographies and Medical QA · %Correct | 0.693 -> 0.810; 0.660 -> 0.711 | Helped | Fine-Tuning Language Models for Factuality (ICLR 2024 · 9 citations) |
| Tune the soft prompt and model jointly rather than in two stages | TriviaQA 1-shot · Exact match | 17.82 -> 0.75 | Hurt | Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · 8 citations) |
| Use approximate C instead of exact Ĉ projection | PureBad · MT-Bench | 5.96 -> 6.34 | Helped | Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · 8 citations) |
| Increase model size from NeuroLM-B to NeuroLM-XL | TUAB, TUEV, SEED, HMC, Workload, and TUSL · Balanced Acc. | TUAB 0.7826±0.0065 -> 0.7969±0.0091; TUEV 0.4560±0.0048 -> 0.4679±0.0356; SEED 0.5554±0.0075 -> 0.60 | Mixed | NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals (ICLR 2025 · 7 citations) |
| Fine-tune on 2% refusal examples and 98% standard conversations, including past-tense examples | JBB-Behaviors test set · attack success rate | 0% → 74%; FT 2%/98%: 2% → 24% | Helped | Does Refusal Training in LLMs Generalize to the Past Tense? (ICLR 2025 · 7 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| LLaMA2-13B | GSM8K | accuracy | 28.7 – 33.89 | 2 papers, e.g. ICML 2024 and ICML 2024 |
What reviewers will expect
Datasets and baselines named in the 63 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 12 of 63 papers
- MMLU · 8 of 63 papers
- MATH · 5 of 63 papers
- SST-2 · 5 of 63 papers
- TriviaQA · 4 of 63 papers
- WinoGrande · 4 of 63 papers
- GLUE · 4 of 63 papers
- HumanEval · 4 of 63 papers
Baselines
- GPT-4 · 4 of 63 papers
- GPT-4o · 4 of 63 papers
- AdaptLLM · 2 of 63 papers
- Random Selection · 2 of 63 papers
- FT · 2 of 63 papers
- ROME · 2 of 63 papers
- MEMIT · 2 of 63 papers
- LoRA · 2 of 63 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 159 papers; from OpenAlex and the papers' affiliations).
- Alibaba Group · 7 papers
- University College London · 4 papers
- Google DeepMind · 4 papers
- Google · 4 papers
- University of Maryland, College Park · 4 papers
- Amazon · 4 papers
- Tsinghua University · 4 papers
- Zhejiang University · 3 papers
- Chinese Academy of Sciences · 3 papers
- Hong Kong University of Science and Technology · 3 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient LLM fine-tuning systems (219 papers)
- Fine-tuning for reasoning data (158 papers)
- Safety alignment after fine-tuning (120 papers)
- Low-rank adaptation variants (43 papers)
Elsewhere in the topic
- Calibration and knowledge in LLM tuning (85 papers)
- Instruction data selection (63 papers)
- Gradient-free adaptation and merging (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).