Research map / Large language models / Instruction and fine-tuning
Low-rank adaptation variants
Improve LoRA with adaptive rank, scaling, contextual, and multi-adapter designs.
43 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Instruction and fine-tuning, keyword cluster “instruction tuning · instructions · instruction following”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 1 tracked benchmark.
8 papers in 2023–24 and 35 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.8% to 5.4%.
Open questions the data points to
- GSM8K (accuracy): best went from 74.28 (up to 2024) to 92.65 in 2025 — NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML 2025)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · Accurate LoRA-Finetuning Quantization of LLMs via Information Retention (ICML 2024 · 42 citations)
- 2025 · LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · 2 citations)
- 2026 · A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search (ICML 2026 · 0 citations)
Earliest papers here
- Accurate LoRA-Finetuning Quantization of LLMs via Information Retention (ICML 2024 · 42 citations)
- HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · 24 citations)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · 21 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
GSM8K · accuracy (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 74.28 | BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models (ICML 2024 · 0 citations) |
| 2025 | 92.65 | NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML 2025) |
What has been tried
27 papers here report 310 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 7%, hurt 18%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add ICQ to Vanilla QLoRA | MMLU, 4-bit · accuracy | 38.4 -> 40.3 | Helped | Accurate LoRA-Finetuning Quantization of LLMs via Information Retention (ICML 2024 · 42 citations) |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) | MMLU · score | 46.59 -> 46.94 | Helped | HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · 24 citations) |
| Use group size 32 rather than group size 128 | MMLU 0-shot · accuracy (Avg.) | 38.0 -> 39.4 | Helped | QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · 21 citations) |
| Perform SVD before quantization in each alternating step rather than quantization before SVD | MNLI development, 2-bit uniform · accuracy (m / mm) | 88.0/88.1 -> 87.8/87.7 | Hurt | LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR 2024 · 18 citations) |
| Add task grouping to multi-task scheduling | 100 tasks · SLO Attainment | +1.16× on average | Helped | Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters (NeurIPS 2024 · 4 citations) |
| Use token-masked synthetic images for meta-training | CIFAR-FS · accuracy | 89.69 -> 91.03; 89.69 -> 90.91; 89.69 -> 89.70 | Mixed | LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · 2 citations) |
| Use a broader rank range (4–32) instead of fixed rank 8 | HumanEval · PASS@1 | 44.75±3.97 -> 48.98±2.14 | Helped | GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · 2 citations) |
| use addition rather than projection to integrate the two guidance signals | SuperGLUE validation, T5-Base · average score | 77.64% -> 77.51% | Hurt | All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · 1 citation) |
| Replace SaLoRA's task-specific initialization with LoRA initialization | Commonsense-15k · commonsense reasoning performance | Hurt | SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation (ICLR 2025 · 1 citation) | |
| Use last-layer Laplace approximation on output-layer LoRA weights instead of all LoRA weights | Common-sense reasoning benchmarks · ECE | Mixed | Bayesian Low-rank Adaptation for Large Language Models (ICLR 2024 · 1 citation) | |
| Increase LoPA soft-prompt length | GLUE and CruxEval-O · performance | No clear effect | Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation (NeurIPS 2024 · 1 citation) | |
| Use JD-Full rather than JD-Diag | LoRA collections · task performance | Helped | Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 24 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Alpaca · 6 of 24 papers
- BoolQ · 5 of 24 papers
- GSM8K · 4 of 24 papers
- MetaMathQA · 4 of 24 papers
- Code-Feedback · 4 of 24 papers
- EuroSAT · 3 of 24 papers
- ARC-c · 3 of 24 papers
- ARC-e · 3 of 24 papers
Baselines
- LoRA · 13 of 24 papers
- DoRA · 8 of 24 papers
- PiSSA · 7 of 24 papers
- AdaLoRA · 4 of 24 papers
- Full fine-tuning · 3 of 24 papers
- LoraHub · 3 of 24 papers
- TIES · 3 of 24 papers
- MiLoRA · 3 of 24 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 43 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- IBM · 3 papers
- Peking University · 3 papers
- Microsoft · 2 papers
- Georgia Institute of Technology · 2 papers
- Beihang University · 2 papers
- Google · 2 papers
- MIT-IBM Watson AI Lab · 2 papers
- University of Edinburgh · 2 papers
- The Chinese University of Hong Kong · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient LLM fine-tuning systems (219 papers)
- Fine-tuning generalization and delta compression (159 papers)
- Fine-tuning for reasoning data (158 papers)
- Safety alignment after fine-tuning (120 papers)
Elsewhere in the topic
- Calibration and knowledge in LLM tuning (85 papers)
- Instruction data selection (63 papers)
- Gradient-free adaptation and merging (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).