Research map / Architectures and efficiency / RNNs and normalization
Dataset distillation and data-efficient training
Condenses datasets and reuses pretrained models to cut training cost.
158 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RNNs and normalization, keyword cluster “memory · scaling · cost”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
47 papers in 2023–24 and 46 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.2% to 4.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Learned Optimizers that Scale and Generalize (ICML 2017 · 108 citations)
- 2018 · Learning Memory Access Patterns (ICML 2018 · 78 citations)
- 2019 · PyTorch: An Imperative Style, High-Performance Deep Learning Library (NeurIPS 2019 · 15,937 citations)
- 2020 · Radioactive data: tracing through training (ICML 2020 · 34 citations)
- 2021 · MiniSeg: An Extremely Minimum Network for Efficient COVID-19 Segmentation (AAAI 2021 · 122 citations)
- 2022 · Dataset Distillation by Matching Training Trajectories (CVPR 2022 · 236 citations)
- 2023 · Revisiting Reverse Distillation for Anomaly Detection (CVPR 2023 · 259 citations)
- 2024 · DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing (AAAI 2024 · 23 citations)
- 2025 · Dataset Distillation with Neural Characteristic Function: A Minmax Perspective (CVPR 2025 · 13 citations)
- 2026 · Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles (ICML 2026 · 0 citations)
Earliest papers here
- Learned Optimizers that Scale and Generalize (ICML 2017 · 108 citations)
- Scaling Up Sparse Support Vector Machines by Simultaneous Feature and Sample Reduction (ICML 2017 · 28 citations)
- Adaptive Classification for Prediction Under a Budget (NeurIPS 2017)
What has been tried
138 papers here report 1,462 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 9%, hurt 18%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add random-LTD to seqtru_voc curriculum learning | 1049B-token pretraining · GLUE score | 87.60±0.34 -> 87.69±0.32; 88.17±0.48 | Mixed | DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing (AAAI 2024 · 23 citations) |
| Use zero PIP steps instead of one | Medium TSPTW-50 · Sol. infeasible rate | 4.53% -> 76.92% | Hurt | Learning to Handle Complex Constraints for Vehicle Routing Problems (NeurIPS 2024 · 20 citations) |
| use guided KD instead of supervised learning | TSP-50 · optimality gap | 9.30% -> 0.36%; 7.69% -> 0.01%; 7.35% -> 0.05% | Helped | Distilling Autoregressive Models to Obtain High-Performance Non-autoregressive Solvers for Vehicle Routing Problems with Faster Inference Speed (AAAI 2024 · 15 citations) |
| Add sampling network ψ | CIFAR-10, 50 IPC · accuracy (%) | 74.2 -> 77.4 | Helped | Dataset Distillation with Neural Characteristic Function: A Minmax Perspective (CVPR 2025 · 13 citations) |
| retain only the top 10–20% by CLIP score rather than top 40% | DataComp medium; 32M compute · average zero-shot accuracy | Helped | Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · 12 citations) | |
| Replace the cumulative gradient transform cgf with cdf | CIFAR10, noise 0.4 · sampler-search performance | Hurt | Swift Sampler: Efficient Learning of Sampler by 10 Parameters (NeurIPS 2024 · 7 citations) | |
| Replace no weight adjustment with directed weight adjustment | CIFAR-100, ipc 10, ResNet-18 · Top-1 accuracy | 30.6 ± 0.7 -> 39.6 ± 0.6 | Helped | Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment (NeurIPS 2024 · 6 citations) |
| Replace LoRA adaptation with hyper-networks | Meta-World ML45 test (no-FT) · score | 0.20±0.01 -> 0.23±0.10 | Helped | Think Before You Act: Decision Transformers with Working Memory (ICML 2024 · 5 citations) |
| Use shared rather than random initialization for fitting each neural field | Neural MNIST · test accuracy | 0.37±0.18 -> 0.72±0.17 | Helped | How to Train Neural Field Representations: A Comprehensive Study and Benchmark (CVPR 2024 · 4 citations) |
| Add feature clipping to Group Calibration | CIFAR-10 · ECE (%) | 1.24 -> 1.78 | Hurt | Feature Clipping for Uncertainty Calibration (AAAI 2025 · 3 citations) |
| Replace deterministic rules (DET) with hierarchical neural network (HNN) for drug interpretation | FL and SL drugs (Table 7) · drug interpretation accuracy (%) | RMP FL 76 to 94; INH FL 81 to 94; FLQ SL 66 to 88; KAC SL 62 to 95; LLK SL 76 to 98 | Helped | Automatic Interpretation of Line Probe Assay Test for Tuberculosis (AAAI 2024 · 3 citations) |
| Add replay-buffer samples to distillation | ImageNet-C Gaussian noise severity 3 · accuracy (%) | 47.4% -> 51.1% | Helped | Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation (ICLR 2024 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 39 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 10 of 39 papers
- CIFAR-100 · 10 of 39 papers
- ImageNet-1K · 5 of 39 papers
- TinyImageNet · 4 of 39 papers
- MNIST · 2 of 39 papers
- ImageNette · 2 of 39 papers
- OpenWebText · 2 of 39 papers
- ImageNet · 2 of 39 papers
Baselines
- LoGra · 3 of 39 papers
- MLP · 2 of 39 papers
- DenseNet · 2 of 39 papers
- ResNet · 2 of 39 papers
- GradDot · 2 of 39 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 93 papers; from OpenAlex and the papers' affiliations).
- Google · 7 papers
- Microsoft · 5 papers
- Nanyang Technological University · 4 papers
- Peking University · 4 papers
- Carnegie Mellon University · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Singapore Management University · 3 papers
- Meta · 3 papers
- University of Oxford · 3 papers
- National University of Singapore · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient network compression and training (229 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Scalable approximation and equivariant compute (120 papers)
Elsewhere in the topic
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Differentiable optimization layers (125 papers)
- Non-autoregressive translation (102 papers)
- RNN training dynamics and gradients (102 papers)
- Neural-symbolic reasoning (98 papers)
- Sparse dictionary representations (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).