Research map / Large language models / Instruction and fine-tuning
Instruction data selection
Select and structure instruction-tuning data and evaluate robustness of instruction-tuned models.
63 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2022–2026) · an approach in Instruction and fine-tuning, keyword cluster “fine tuning · lora · adaptation”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 2 tracked benchmarks.
22 papers in 2023–24 and 40 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.6% to 6.1%.
Open questions the data points to
- HumanEval+ (pass@1): best went from 67.1 (up to 2024) to 76.8 in 2025 — InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · 2 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2022 · Finetuned Language Models are Zero-Shot Learners (ICLR 2022 · 69 citations)
- 2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (ICML 2023 · 113 citations)
- 2024 · PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · 29 citations)
- 2025 · XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning (AAAI 2025 · 12 citations)
- 2026 · TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning (ICML 2026 · 0 citations)
Earliest papers here
- Finetuned Language Models are Zero-Shot Learners (ICLR 2022 · 69 citations)
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (ICML 2023 · 113 citations)
- Exploring the Benefits of Training Expert Language Models over Instruction Tuning (ICML 2023 · 19 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MT-Bench · score (higher is better)
5 papers, 2024–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 7.55 | What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024) |
HumanEval+ · pass@1 (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 67.1 | SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · 5 citations) |
| 2025 | 76.8 | InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · 2 citations) |
What has been tried
46 papers here report 498 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 9%, hurt 17%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Include evaluation reasons and reference responses alongside the evaluation label | Human-annotated test set · accuracy | 0.4725 -> 0.5926 | Helped | PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · 29 citations) |
| freeze the gating function during instruction tuning | Average across MMLU, BBH, and GSM8K · average score | 37.7 -> 38.0 | Helped | Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · 20 citations) |
| Use a 4.0 rather than 4.5 filtering threshold, yielding 39k examples | Vicuna and Self-Instruct · GPT-4 evaluation | No clear effect | AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR 2024 · 15 citations) | |
| Replace demonstration inputs with random text | SuperNI (12 tasks) · average score | 44.24 -> 44.27 | No clear effect | Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · 13 citations) |
| Increase average tag number in the 6K training subset while keeping tag coverage at 100% | MT-Bench · score | Helped | #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models (ICLR 2024 · 13 citations) | |
| Add xICL code-switched context | MGSM · accuracy (Avg.) | 30.1 -> 34.8; +4.7 points | Helped | XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning (AAAI 2025 · 12 citations) |
| Use soft prompt parameters and KL alignment loss | MMLU, unobserved instructions · accuracy | +0.4 | Helped | Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · 11 citations) |
| Add a small set of safety examples during instruction tuning | I-MaliciousInstructions, Q-Harm, I-CoNa, I-Controversial, and I-PhysicalSafetyUnsafe · harmfulness | Helped | Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (ICLR 2024 · 11 citations) | |
| Increase coding data from 0% to 50% | 12 reasoning tasks · average accuracy (%) | 41.4 -> 42.9 | Helped | Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning (AAAI 2025 · 6 citations) |
| use passing responses only instead of random responses | HumanEval+ · pass@1 | 61.6 -> 65.2 | Helped | SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · 5 citations) |
| Use concise composed instructions instead of the regular CoI instructions | CoI₂-test · Rouge-L | 70.76 -> 59.19 | Hurt | Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models (AAAI 2025 · 4 citations) |
| Replace Qwen2-72B with GPT-4 as supervision model | IFEval prompt loose · accuracy | 44.5 -> 52.9 | Helped | Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 27 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 7 of 27 papers
- MMLU · 4 of 27 papers
- MBPP · 3 of 27 papers
- ShareGPT · 3 of 27 papers
- Alpaca GPT4 · 2 of 27 papers
- UltraChat · 2 of 27 papers
- Code Alpaca 20K · 2 of 27 papers
- Super-NaturalInstructions · 2 of 27 papers
Baselines
- Random · 4 of 27 papers
- BM25 · 3 of 27 papers
- LESS · 3 of 27 papers
- IFD · 2 of 27 papers
- DEITA · 2 of 27 papers
- GPT-3.5-Turbo · 2 of 27 papers
- LoRA · 2 of 27 papers
- RFT · 2 of 27 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 62 papers; from OpenAlex and the papers' affiliations).
- University of Illinois, Urbana Champaign · 6 papers
- University of Maryland, College Park · 5 papers
- Google · 5 papers
- Amazon · 4 papers
- Massachusetts Institute of Technology · 4 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- University of California, Berkeley · 3 papers
- Alibaba Group · 3 papers
- Tencent · 2 papers
Use it
Download BibTeX of 24 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Calibration and knowledge in LLM tuning (85 papers)
- Gradient-free adaptation and merging (24 papers)
Elsewhere in the topic
- Efficient LLM fine-tuning systems (219 papers)
- Fine-tuning generalization and delta compression (159 papers)
- Fine-tuning for reasoning data (158 papers)
- Safety alignment after fine-tuning (120 papers)
- Low-rank adaptation variants (43 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).