Research map / Architectures and efficiency / RNNs and normalization
Efficient network compression and training
Accelerates and compresses networks via memory-efficient backprop, structured shrinking, and parameter prediction.
229 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “memory · scaling · cost”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Few recent reports The tracked benchmarks have at most one report from 2025–26 that could beat the old record, too few to say whether they are still moving.
67 papers in 2023–24 and 38 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.3% to 4.1%.
No longer reported by papers here: CIFAR-100 (accuracy, last in 2024); ImageNet (Top-1 accuracy, last in 2024); ImageNet (accuracy, last in 2024); CIFAR-100 (Top-1 accuracy (%), last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Binarized Neural Networks (NeurIPS 2016 · 907 citations)
- 2017 · Large-Scale Evolution of Image Classifiers (ICML 2017 · 534 citations)
- 2018 · MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (CVPR 2018 · 344 citations)
- 2019 · GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (NeurIPS 2019 · 256 citations)
- 2020 · Resolution Adaptive Networks for Efficient Inference (CVPR 2020 · 249 citations)
- 2021 · Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation (CVPR 2021 · 153 citations)
- 2022 · JFB: Jacobian-Free Backpropagation for Implicit Networks (AAAI 2022 · 52 citations)
- 2023 · Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks (CVPR 2023 · 2,281 citations)
- 2024 · Model-Driven Deep Neural Network for Enhanced AoA Estimation Using 5G gNB (AAAI 2024 · 12 citations)
- 2025 · Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters (AAAI 2025 · 6 citations)
- 2026 · Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks (ICML 2026 · 0 citations)
Earliest papers here
- Binarized Neural Networks (NeurIPS 2016 · 907 citations)
- FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (CVPR 2016 · 298 citations)
- Dropout distillation (ICML 2016)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-10 · accuracy (higher is better)
20 papers, 2018–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 94.75 | Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework (AAAI 2018 · 44 citations) |
| 2019 | 99 | GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (NeurIPS 2019 · 256 citations) |
CIFAR-100 · accuracy (higher is better)
14 papers, 2018–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 68.45 | Learning Time/Memory-Efficient Deep Architectures With Budgeted Super Networks (CVPR 2018 · 15 citations) |
| 2019 | 91.3 | GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (NeurIPS 2019 · 256 citations) |
ImageNet · Top-1 accuracy (higher is better)
10 papers, 2016–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 47.1 | Binarized Neural Networks (NeurIPS 2016 · 907 citations) |
| 2018 | 73.2 | HydraNets: Specialized Dynamic Architectures for Efficient Inference (CVPR 2018 · 133 citations) |
| 2021 | 79.35 | Minimally Invasive Surgery for Sparse Neural Networks in Contrastive Manner (CVPR 2021 · 2 citations) |
| 2023 | 83.61 | DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training (ICLR 2023 · 2 citations) |
ImageNet · accuracy (higher is better)
7 papers, 2018–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 75.2 | MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (CVPR 2018 · 344 citations) |
| 2020 | 78.3 | Structured Multi-Hashing for Model Compression (CVPR 2020 · 11 citations) |
| 2022 | 80 | Sharpness-Aware Training for Free (NeurIPS 2022 · 13 citations) |
MNIST · accuracy (higher is better)
6 papers, 2018–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 99 | Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework (AAAI 2018 · 44 citations) |
| 2021 | 99.25 | Faster Neural Network Training with Approximate Tensor Operations (NeurIPS 2021) |
| 2022 | 99.4 | JFB: Jacobian-Free Backpropagation for Implicit Networks (AAAI 2022 · 52 citations) |
CIFAR-100 · Top-1 accuracy (%) (higher is better)
4 papers, 2018–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 76.25 | HydraNets: Specialized Dynamic Architectures for Efficient Inference (CVPR 2018 · 133 citations) |
| 2023 | 91.3 | DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training (ICLR 2023 · 2 citations) |
What has been tried
201 papers here report 2,214 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 11%, hurt 20%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove dataset distance from the regression model | Transfer-learning data · statistical significance | No clear effect | A Model for Estimating the Economic Costs of Computer Vision Systems That Use Deep Learning (AAAI 2024 · 8 citations) | |
| Increase trigger area from 7.91% to 14.06% | CIFAR-10 · ASR | 91.29 -> 99.07 | Helped | Deep-TROJ: An Inference Stage Trojan Insertion Algorithm through Efficient Weight Replacement Attack (CVPR 2024 · 6 citations) |
| Increase retained explained variance from ε=0.8 to ε=0.9 | ImageNet-1k, 2 layers · accuracy | 61.7 -> 62.0 | Helped | Activation Map Compression through Tensor Decomposition for Deep Learning (NeurIPS 2024 · 5 citations) |
| replace WS normalization with GN | CIFAR10C L1 · accuracy | 83.2 -> 73.4 | Hurt | TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices (NeurIPS 2024 · 5 citations) |
| Add quantifiable physical information to error-compensation training | MNIST · accuracy | 93.5% -> 96.5% | Helped | Physics-Constrained Comprehensive Optical Neural Networks (NeurIPS 2024 · 4 citations) |
| replace MobileBlock with FuseV2 MB | MPII, 20-frame videos · activation count | 9.4M to 4.5M | Helped | MEET: Towards Memory-Efficient Temporal Sparse Deep Neural Networks (CVPR 2025 · 2 citations) |
| distill three ancestry-model blocks instead of only the last block | ImageNet-1K · accuracy | 44.27 -> 53.82 (6 blocks); 54.98 -> 60.44 (9 blocks) | Helped | Building Variable-Sized Models via Learngene Pool (AAAI 2024 · 2 citations) |
| Replace the linear trajectory objective with cosine or DDPM trajectories | CIFAR-10 test, 1-Step · Top-1 accuracy (%) | 95.4±0.01 vs 94.7±0.01; 94.9±0.01 | Hurt | Layer Compression of Deep Networks with Straight Flows (AAAI 2024 · 2 citations) |
| set alpha to 2 instead of 4 | ImageNet · test error (%) | 24.32 -> 24.86 | Hurt | When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks (AAAI 2024 · 2 citations) |
| place the nonlinearity before the weights (pre-nonlinear ResNet) instead of after them | CIFAR-10 · minimal log loss | Hurt | Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · 2 citations) | |
| Apply late-stage rank adaptation with target ratio 0.8 | MNLI validation · accuracy | 63.3% -> 62.2% | No clear effect | CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization (NeurIPS 2024 · 2 citations) |
| Use three memory layers rather than one | training set · nll | 2.11 -> 2.08 | Helped | Memory Layers at Scale (ICML 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 27 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 5 of 27 papers
- CIFAR-10 · 4 of 27 papers
- C4 · 3 of 27 papers
- CIFAR-100 · 3 of 27 papers
- MNIST · 2 of 27 papers
- GLUE · 2 of 27 papers
- COCO · 2 of 27 papers
- FashionMNIST · 2 of 27 papers
Baselines
- AdamW · 2 of 27 papers
- PABEE · 2 of 27 papers
- ZTW · 2 of 27 papers
- MuE · 2 of 27 papers
- JEI-DNN · 2 of 27 papers
- Adam · 2 of 27 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 105 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Meta · 4 papers
- Huawei · 4 papers
- Seoul National University · 4 papers
- Peking University · 3 papers
- University of Texas at Austin · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- University of Illinois at Chicago · 3 papers
- Google · 3 papers
- Amazon · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Scalable approximation and equivariant compute (120 papers)
Elsewhere in the topic
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Differentiable optimization layers (125 papers)
- Non-autoregressive translation (102 papers)
- RNN training dynamics and gradients (102 papers)
- Neural-symbolic reasoning (98 papers)
- Sparse dictionary representations (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).