Research map / Architectures and efficiency / Pruning and efficient tuning
Low-rank compression and merging
Compresses and adapts models with low-rank factorization and weight merging.
226 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Pruning and efficient tuning, keyword cluster “fine tuning · lora · adaptation”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
37 papers in 2023–24 and 152 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 13.3% to 27.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Low-Shot Learning With Imprinted Weights (CVPR 2018 · 558 citations)
- 2019 · Superposition of many models into one (NeurIPS 2019 · 75 citations)
- 2020 · Comparing Rewinding and Fine-tuning in Neural Network Pruning (ICLR 2020 · 195 citations)
- 2021 · AutoLR: Layer-wise Pruning and Auto-tuning of Learning Rates in Fine-tuning of Deep Networks (AAAI 2021 · 26 citations)
- 2022 · Merging Models with Fisher-Weighted Averaging (NeurIPS 2022 · 43 citations)
- 2023 · DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (ICLR 2023 · 393 citations)
- 2024 · Better by default: Strong pre-tuned MLPs and boosted trees on tabular data (NeurIPS 2024 · 41 citations)
- 2025 · TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · 39 citations)
- 2026 · Forgetting by Pruning: Data Deletion in Join Cardinality Estimation (AAAI 2026 · 1 citation)
Earliest papers here
- Low-Shot Learning With Imprinted Weights (CVPR 2018 · 558 citations)
- Context-Aware Deep Feature Compression for High-Speed Visual Tracking (CVPR 2018 · 246 citations)
- A Simple Cache Model for Image Recognition (NeurIPS 2018 · 8 citations)
What has been tried
131 papers here report 1,501 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 8%, hurt 21%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use PBLD numerical embeddings rather than PLR embeddings | Meta-train regression · SGM error | 19.0 [13.7, 24.5] error increase with PLR | Helped | Better by default: Strong pre-tuned MLPs and boosted trees on tabular data (NeurIPS 2024 · 41 citations) |
| Add full-stage knowledge distillation loss | COCO val zero-shot instance segmentation · AP (%) | 40.7 -> 41.4 | Helped | TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · 39 citations) |
| Add the new scaling factor | ISIC, zero-shot, resolution variation · performance degradation (∆diff) | 17.8 -> 16.4 | Helped | BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · 20 citations) |
| Add adaptive LORS only | COCO 2017 val · AP | 42.5† -> 42.6 | No clear effect | LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking (CVPR 2024 · 16 citations) |
| add random features (RF) | HyperFast ablation · balanced accuracy (%) | 75.387 -> 81.496 | Helped | HyperFast: Instant Classification for Tabular Data (AAAI 2024 · 15 citations) |
| add low-rank approximation without interference reduction | 8 tasks · average absolute accuracy | 76.5 to 75.2 (-1.3) | Hurt | Task Singular Vectors: Reducing Task Interference in Model Merging (CVPR 2025 · 12 citations) |
| add feature distillation | ImageNet classification · accuracy | 61.08 -> 80.90 | Helped | PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · 11 citations) |
| Increase ReLoRA rank from 128 to 512 | C4, 1.3B, 10K iterations after 5K-step warm start · perplexity | 19.16 -> 19.00 | No clear effect | ReLoRA: High-Rank Training Through Low-Rank Updates (ICLR 2024 · 11 citations) |
| remove CCI | Kodak · encoding time | 1.4s -> 15.2s | Hurt | CALLIC: Content Adaptive Learning for Lossless Image Compression (AAAI 2025 · 8 citations) |
| Replace ABMIL with ABMILX | CAMELYON · Performance | 91.78 -> 95.88 | Helped | Revisiting End-to-End Learning with Slide-level Supervision in Computational Pathology (NeurIPS 2025 · 5 citations) |
| Discard low-magnitude parameters rather than randomly discarding parameters | Eight vision datasets (average) · accuracy | At α=0.1: 69.31 vs 69.06; at α=0.8: 70.99 vs 66.09 | Mixed | Less is More: Efficient Model Merging with Binary Task Switch (CVPR 2025 · 4 citations) |
| Add SimCSE fine-tuning | STS-Average · Spearman's correlation | 87.60 -> 87.83 | No clear effect | Some Like It Small: Czech Semantic Embedding Models for Industry Applications (AAAI 2024 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 87 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 15 of 87 papers
- CIFAR-100 · 14 of 87 papers
- ImageNet-1K · 12 of 87 papers
- RESISC45 · 10 of 87 papers
- EuroSAT · 10 of 87 papers
- SVHN · 10 of 87 papers
- DTD · 10 of 87 papers
- ImageNet · 10 of 87 papers
Baselines
- LoRA · 16 of 87 papers
- TIES-Merging · 9 of 87 papers
- Task Arithmetic · 8 of 87 papers
- DARE · 8 of 87 papers
- RegMean · 5 of 87 papers
- DoRA · 5 of 87 papers
- Adam · 5 of 87 papers
- Weight Averaging · 4 of 87 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 189 papers; from OpenAlex and the papers' affiliations).
- Amazon · 7 papers
- Peking University · 7 papers
- Nanyang Technological University · 4 papers
- Chinese Academy of Sciences · 4 papers
- University of California, Los Angeles · 4 papers
- Microsoft · 4 papers
- Google · 4 papers
- Alibaba Group · 4 papers
- Google DeepMind · 4 papers
- University of Illinois, Urbana Champaign · 3 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Adapter-based parameter-efficient tuning (149 papers)
- Parameter-efficient transfer analysis (97 papers)
- LoRA variants and theory (121 papers)
- Efficient ViT adaptation (62 papers)
Elsewhere in the topic
- Structured network pruning (162 papers)
- Sparse training from scratch (95 papers)
- Layer-wise pruning of LLMs (91 papers)
- Joint pruning and quantization (78 papers)
- Lottery ticket discovery (42 papers)
- Pruning at initialization (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).