Research map / Large language models / Instruction and fine-tuning
Efficient LLM fine-tuning systems
Cut memory and compute of fine-tuning through sparsity, projections, and distributed training.
219 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Instruction and fine-tuning, keyword cluster “instruction tuning · instructions · instruction following”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 3 of 5 tracked benchmarks.
55 papers in 2023–24 and 158 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 26.4% to 24.2%.
Open questions the data points to
- HumanEval (Pass@1): best went from 31.23 (up to 2024) to 69.1 in 2025 — Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025)
- MMLU (accuracy): best went from 59.99 (up to 2024) to 64.6 in 2025 — Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · 0 citations)
- Winogrande (accuracy): best went from 73.72 (up to 2024) to 87.2 in 2025 — Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · 0 citations)
What looks settled
- GSM8K (accuracy): best up to 2024 76.3, best in 2025–26 67.8 from 3 papers, at most 0.2% better, if at all.
- BoolQ (accuracy): best up to 2024 86.99, best in 2025–26 76.4 from 3 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Language Models are Few-Shot Learners (NeurIPS 2020 · 2,966 citations)
- 2021 · Scalable Diverse Model Selection for Accessible Transfer Learning (NeurIPS 2021 · 6 citations)
- 2022 · On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets (AAAI 2022 · 15 citations)
- 2023 · To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis (NeurIPS 2023 · 20 citations)
- 2024 · LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · 39 citations)
- 2025 · JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · 18 citations)
- 2026 · LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning (ICML 2026 · 1 citation)
Earliest papers here
- Language Models are Few-Shot Learners (NeurIPS 2020 · 2,966 citations)
- Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation (AAAI 2020 · 69 citations)
- Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (AAAI 2020 · 29 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
GSM8K · accuracy (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 76.3 | SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models (NeurIPS 2024 · 7 citations) |
BoolQ · accuracy (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 86.99 | BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · 11 citations) |
HumanEval · Pass@1 (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 31.23 | Getting the most out of your tokenizer for pre-training and domain adaptation (ICML 2024 · 0 citations) |
| 2025 | 69.1 | Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025) |
MMLU · accuracy (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 59.99 | Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · 11 citations) |
| 2025 | 64.6 | Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · 0 citations) |
Winogrande · accuracy (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 73.72 | Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · 11 citations) |
| 2025 | 87.2 | Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · 0 citations) |
What has been tried
132 papers here report 1,534 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 9%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Train normalization and embedding layers alongside LoRA rank 8 | PG19 validation, context 32768 · perplexity | 11.44 -> 8.12 | Helped | LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · 39 citations) |
| add CCR and HQAR fine-tuning | Amazon (Movie & TV) · CMR | 2.171 -> 2.353 | Helped | Fine-Tuning Large Language Model Based Explainable Recommendation with Explainable Quality Reward (AAAI 2024 · 28 citations) |
| Add monolingual fine-tuning before filtered parallel data fine-tuning | WMT’22/’21 en→xx, average · COMET | 74.35 -> 83.98 | Helped | A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · 28 citations) |
| Add BalanceMix instead of directly fine-tuning on generated and original data | ImageNet-LT · Overall accuracy | 58.3 -> 80.6 | Helped | LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content (CVPR 2024 · 27 citations) |
| SFT on preferred data instead of CPO | WMT'21/'22 en→xx (average) · XCOMET | 92.76 -> 92.54 | Hurt | Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (ICML 2024 · 19 citations) |
| Add swap augmentation | JudgeLM val, without reference · consistency with swapped answers | 73.45 -> 78.89 | Helped | JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · 18 citations) |
| Use YaRN instead of NTK-by-parts | Proof-pile, 8192 context · perplexity | 3.62 -> 3.30 | Helped | YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · 18 citations) |
| Use CFG during inference | ARC-C · score | 45.9 -> 47.9 | Helped | Large Language Diffusion Models (NeurIPS 2025 · 17 citations) |
| add CommitPackFT to instruction tuning data | HumanEvalPack Code Repair (Python) · pass@1 (%) | Helped | OctoPack: Instruction Tuning Code Large Language Models (ICLR 2024 · 17 citations) | |
| Remove the rescale operation from DARE | GSM8K · embedding cosine similarity | about 0.85/0.68 when p is 0.5/0.9 | Hurt | Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (ICML 2024 · 13 citations) |
| Extend Phi-2 to 4k with SelfExtend (group size 4, neighbor window 512) | LongBench NarrativeQA · score | 4.46 -> 6.49 (+45.52\%) | Helped | LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · 13 citations) |
| Add row-causal tabular masking (RCTM) instead of per-sample causal attention | Full test suite, few-shot · accuracy | Helped | Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · 12 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| Pre-LN | LLaMA-1B pre-training | perplexity | 17.02 – 18.65 | 2 papers, e.g. NeurIPS 2025 and ICLR 2025 |
What reviewers will expect
Datasets and baselines named in the 94 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 16 of 94 papers
- MMLU · 11 of 94 papers
- BoolQ · 9 of 94 papers
- C4 · 8 of 94 papers
- HellaSwag · 8 of 94 papers
- WinoGrande · 7 of 94 papers
- Commonsense170K · 6 of 94 papers
- MATH · 6 of 94 papers
Baselines
- LoRA · 16 of 94 papers
- DoRA · 5 of 94 papers
- IA3 · 4 of 94 papers
- Zero-shot · 4 of 94 papers
- ICL · 3 of 94 papers
- GPT-4o-mini · 3 of 94 papers
- Adapter · 3 of 94 papers
- GaLore · 3 of 94 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 213 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 9 papers
- Microsoft · 8 papers
- Massachusetts Institute of Technology · 8 papers
- Tsinghua University · 8 papers
- National University of Singapore · 8 papers
- Alibaba Group · 7 papers
- Stanford University · 7 papers
- University of Chinese Academy of Sciences · 6 papers
- Peking University · 6 papers
- University of Texas at Austin · 6 papers
Use it
Download BibTeX of 27 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Fine-tuning generalization and delta compression (159 papers)
- Fine-tuning for reasoning data (158 papers)
- Safety alignment after fine-tuning (120 papers)
- Low-rank adaptation variants (43 papers)
Elsewhere in the topic
- Calibration and knowledge in LLM tuning (85 papers)
- Instruction data selection (63 papers)
- Gradient-free adaptation and merging (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).