Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter-efficient transfer analysis
Analyzes and unifies transfer methods, from theoretical studies to multi-task model fusion.
97 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Pruning and efficient tuning, keyword cluster “fine tuning · lora · adaptation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
19 papers in 2023–24 and 51 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.8% to 9.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally (CVPR 2017 · 429 citations)
- 2018 · Efficient Parametrization of Multi-Domain Deep Neural Networks (CVPR 2018 · 45 citations)
- 2019 · Attentive Single-Tasking of Multiple Tasks (CVPR 2019 · 199 citations)
- 2020 · Towards Making the Most of BERT in Neural Machine Translation (AAAI 2020 · 111 citations)
- 2021 · TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning (AAAI 2021 · 58 citations)
- 2022 · Towards a Unified View of Parameter-Efficient Transfer Learning (ICLR 2022 · 274 citations)
- 2023 · CLIPPING: Distilling CLIP-Based Models With a Student Base for Video-Language Retrieval (CVPR 2023 · 33 citations)
- 2024 · Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · 16 citations)
- 2025 · No More Tuning: Prioritized Multi-Task Learning with Lagrangian Differential Multiplier Methods (AAAI 2025 · 4 citations)
- 2026 · DIET: Machine Unlearning on a Data-Diet (AAAI 2026 · 1 citation)
Earliest papers here
- Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally (CVPR 2017 · 429 citations)
- Growing a Brain: Fine-Tuning by Increasing Model Capacity (CVPR 2017 · 158 citations)
- Efficient Parametrization of Multi-Domain Deep Neural Networks (CVPR 2018 · 45 citations)
What has been tried
70 papers here report 763 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 10%, hurt 17%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use LoSA with rank 8 instead of rank 16 | VTAB-1K · Average accuracy | 78.4 -> 78.3 | Hurt | Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · 16 citations) |
| Add NMT to FAMO gradient-based MTL | TikTok · Like AUC | ShareBottom +0.21% to +0.41% range | Helped | No More Tuning: Prioritized Multi-Task Learning with Lagrangian Differential Multiplier Methods (AAAI 2025 · 4 citations) |
| Use medium efficiency modifications | LargeScaleTables (29 datasets) · mean accuracy | 0.830 -> 0.828 | Hurt | TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks (NeurIPS 2024 · 4 citations) |
| Use a smaller maximum learning rate instead of the default maximum learning rate | ImageNet-1k · validation accuracy | Hurt | LEMON: Lossless model expansion (ICLR 2024 · 3 citations) | |
| Add layer-wise least-squares fitting to the permutation-matched merged model | DomainNet, shared label space (Avg) · accuracy | 17.8 -> 38.6 | Helped | PLeaS - Merging Models with Permutations and Least Squares (CVPR 2025 · 2 citations) |
| Use stitch-agnostic LoRA in self-attention layers instead of full fine-tuning | Five FGVC tasks · accuracy | Helped | Efficient Stitchable Task Adaptation (CVPR 2024 · 2 citations) | |
| Mask attention to instructions and examples after the task-recognition layer | FLORES en↔fr · translation performance | No clear effect | Where does In-context Learning Happen in Large Language Models? (NeurIPS 2024 · 2 citations) | |
| Use frequency-based rather than conventional time-domain pre-training | PPG · accuracy | 58.57 -> 65.25 | Helped | CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning (AAAI 2025 · 1 citation) |
| Use Top-1 instead of Greedy prompt selection | Pascal VOC 2012 single object detection · mIOU | 28.25 (2.94) -> 28.25 (2.94) | No clear effect | Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · 1 citation) |
| Add Wasserstein distance constraint and parameter update sparsification to MEMIT | zsRE, batch size 50, 100 iterations · overall editing score | 39.66 -> 54.67 | Helped | Wasserstein Distance Constraint and Parameter Sparsification for Batched and Iterative Knowledge Editing (AAAI 2025 · 1 citation) |
| Add affine transformation to zero-shot input | CIFAR10 · accuracy | 88.93 -> 90.17 | Helped | Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · 1 citation) |
| Replace linear layer scaling with quadratic scaling | Vision benchmarks, 8, 14, and 20 tasks · accuracy | 74.2 -> 73.8; 69.1 -> 69.2; 63.4 -> 64.6 | Mixed | LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging (ICLR 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 32 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 7 of 32 papers
- ImageNet · 6 of 32 papers
- CIFAR-100 · 6 of 32 papers
- DTD · 6 of 32 papers
- EuroSAT · 6 of 32 papers
- GTSRB · 6 of 32 papers
- Cars · 5 of 32 papers
- RESISC45 · 5 of 32 papers
Baselines
- Ties-Merging · 3 of 32 papers
- Random · 2 of 32 papers
- LoRA · 2 of 32 papers
- RegMean · 2 of 32 papers
- Git Re-Basin · 2 of 32 papers
- Task Arithmetic · 2 of 32 papers
- Consensus Merging · 2 of 32 papers
- AdaMerging · 2 of 32 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 70 papers; from OpenAlex and the papers' affiliations).
- Google · 6 papers
- University of California, Berkeley · 3 papers
- Amazon · 3 papers
- The Chinese University of Hong Kong · 3 papers
- NTT · 3 papers
- Monash University · 2 papers
- Zhejiang University · 2 papers
- University of Washington · 2 papers
- University of Illinois, Urbana Champaign · 2 papers
- ByteDance · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Low-rank compression and merging (226 papers)
- Adapter-based parameter-efficient tuning (149 papers)
- LoRA variants and theory (121 papers)
- Efficient ViT adaptation (62 papers)
Elsewhere in the topic
- Structured network pruning (162 papers)
- Sparse training from scratch (95 papers)
- Layer-wise pruning of LLMs (91 papers)
- Joint pruning and quantization (78 papers)
- Lottery ticket discovery (42 papers)
- Pruning at initialization (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).