Research map / Architectures and efficiency / Pruning and efficient tuning
Adapter-based parameter-efficient tuning
Adds lightweight adapters and reparameterized modules for memory- and parameter-efficient fine-tuning.
149 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Pruning and efficient tuning, keyword cluster “fine tuning · lora · adaptation”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 3 tracked benchmarks.
41 papers in 2023–24 and 90 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 14.7% to 16.2%.
Open questions the data points to
- BoolQ (accuracy): best went from 73.8 (up to 2024) to 85.7 in 2025 — Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · 2 citations)
- RTE (accuracy): best went from 79.1 (up to 2024) to 82.1 in 2025 — Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · 2 citations)
- Papers report DoRA on GLUE (Avg.) anywhere from 67.9 to 88.49: comparisons against it depend on whose number you take.
- Papers report LoRA on GLUE (Avg. Score) anywhere from 76.9 to 85.3: comparisons against it depend on whose number you take.
What looks settled
- CIFAR-100 (accuracy): best up to 2024 93.25, best in 2025–26 91.8 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (ICML 2016 · 87 citations)
- 2019 · Data Fine-Tuning (AAAI 2019 · 11 citations)
- 2020 · Stochastic Normalization (NeurIPS 2020 · 21 citations)
- 2021 · On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (ICLR 2021 · 207 citations)
- 2022 · Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (NeurIPS 2022 · 288 citations)
- 2023 · On the Effectiveness of Parameter-Efficient Fine-Tuning (AAAI 2023 · 139 citations)
- 2024 · Gradient-based Parameter Selection for Efficient Fine-Tuning (CVPR 2024 · 29 citations)
- 2025 · Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · 9 citations)
- 2026 · RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models (AAAI 2026 · 1 citation)
Earliest papers here
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (ICML 2016 · 87 citations)
- Data Fine-Tuning (AAAI 2019 · 11 citations)
- Stochastic Normalization (NeurIPS 2020 · 21 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
BoolQ · accuracy (higher is better)
5 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 73.8 | Fine-Tuning Language Models with Just Forward Passes (NeurIPS 2023 · 36 citations) |
| 2025 | 85.7 | Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · 2 citations) |
CIFAR-100 · accuracy (higher is better)
5 papers, 2023–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 93.25 | Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone (NeurIPS 2023 · 4 citations) |
RTE · accuracy (higher is better)
5 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 79.1 | Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (ICML 2022 · 205 citations) |
| 2025 | 82.1 | Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · 2 citations) |
What has been tried
95 papers here report 1,136 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 20%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Select the top-gradient connections per neuron instead of selecting highest-gradient parameters across the network or by layer | FGVC · accuracy | 86.86 -> 87.17; 87.30 -> 88.07 | Helped | Gradient-based Parameter Selection for Efficient Fine-Tuning (CVPR 2024 · 29 citations) |
| Use convolution-layer parameter space only for reconstruction instead of Transformer-layer parameter space | NWPU VHR-10 · mIoU (%) | 83.1 -> 84.2 | Helped | SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · 28 citations) |
| add a second parallel adapter, so adapters operate before and after message passing | Molecular benchmarks (6 datasets), AttrMasking · ROC-AUC average | 70.4 -> 71.2 | Helped | AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs (AAAI 2024 · 19 citations) |
| add last-layer interaction guidance | Flickr30K · Rsum | Helped | UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory (CVPR 2024 · 16 citations) | |
| Add dual coefficient sets | ADOME · DSC | 90.1 -> 90.1 (+0.0) | No clear effect | Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model (CVPR 2024 · 13 citations) |
| Activate a nonzero drop path rate rather than setting it to 0 | VTAB-1K · top-1 accuracy | Helped | Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · 9 citations) | |
| replace tanh approximation with l0 approximation | SST-2 (BERT Base) · accuracy | {92.36 ± 0.36} -> 92.13 ± 0.30 | No clear effect | From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers (AAAI 2025 · 7 citations) |
| Add LoRA in both attention and feedforward layers rather than attention layers only | DOTA tail classes · accuracy | 91.1 -> 92.4 | Helped | Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning (CVPR 2025 · 7 citations) |
| use warm-up LoRA gradient-magnitude mask instead of initialization gradient-magnitude mask | GSM8K 1 Epoch · accuracy | 30.10 -> 32.16 | Helped | RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation (ICML 2024 · 7 citations) |
| Use residual parameterization I + α instead of direct α | Few-shot image classification · Avg. Score | 44.28 -> 73.49 | Helped | Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models (CVPR 2025 · 6 citations) |
| Add a trainable mask instead of using a frozen mask | 14-task zero-shot average, 25% pruning · accuracy | 54.97 -> 60.02 | Helped | PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · 5 citations) |
| Replace dynamic rank with fixed rank 8, 16, or 32 | VTAB-1k · Mean Acc | 76.8 -> 75.0; 75.1; 75.4 | Hurt | Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · 5 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| DoRA | GLUE | Avg. | 67.9 – 88.49 | 2 papers, e.g. ICML 2025 and ICLR 2025 |
| LoRA | GLUE | Avg. Score | 76.9 – 85.3 | 2 papers, e.g. ICML 2025 and NeurIPS 2022 |
| FULL-FT | CIFAR-100 | accuracy | 85.35 – 93 | 2 papers, e.g. NeurIPS 2024 and CVPR 2024 |
What reviewers will expect
Datasets and baselines named in the 54 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- VTAB-1K · 11 of 54 papers
- GLUE · 9 of 54 papers
- BoolQ · 7 of 54 papers
- MNLI · 6 of 54 papers
- CIFAR-10 · 6 of 54 papers
- CIFAR-100 · 6 of 54 papers
- SST-2 · 5 of 54 papers
- RTE · 5 of 54 papers
Baselines
- LoRA · 36 of 54 papers
- DoRA · 14 of 54 papers
- BitFit · 13 of 54 papers
- Full fine-tuning · 13 of 54 papers
- Adapter · 10 of 54 papers
- Linear Probing · 9 of 54 papers
- AdaptFormer · 7 of 54 papers
- VPT · 6 of 54 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 131 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 5 papers
- Shanghai Jiao Tong University · 5 papers
- Peking University · 5 papers
- Microsoft · 5 papers
- Google · 5 papers
- Georgia Institute of Technology · 4 papers
- National University of Singapore · 4 papers
- The Chinese University of Hong Kong · 3 papers
- Carnegie Mellon University · 3 papers
- Hong Kong University of Science and Technology · 3 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Low-rank compression and merging (226 papers)
- Parameter-efficient transfer analysis (97 papers)
- LoRA variants and theory (121 papers)
- Efficient ViT adaptation (62 papers)
Elsewhere in the topic
- Structured network pruning (162 papers)
- Sparse training from scratch (95 papers)
- Layer-wise pruning of LLMs (91 papers)
- Joint pruning and quantization (78 papers)
- Lottery ticket discovery (42 papers)
- Pruning at initialization (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).