Research map / Architectures and efficiency / RNNs and normalization
Distributed LLM inference systems
Optimizes LLM serving and training with parallelism, KV reuse, batching, and GPU scheduling.
147 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “memory · scaling · cost”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
33 papers in 2023–24 and 88 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 5.1% to 9.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Short-Dot: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products (NeurIPS 2016)
- 2017 · Resource-efficient Machine Learning in 2 KB RAM for the Internet of Things (ICML 2017 · 137 citations)
- 2018 · Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (ICLR 2018 · 1,018 citations)
- 2019 · Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks (ICML 2019 · 50 citations)
- 2020 · HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory (NeurIPS 2020 · 26 citations)
- 2021 · Checkerboard Context Model for Efficient Learned Image Compression (CVPR 2021 · 345 citations)
- 2022 · EtinyNet: Extremely Tiny Network for TinyML (AAAI 2022 · 27 citations)
- 2023 · AHPA: Adaptive Horizontal Pod Autoscaling Systems on Alibaba Cloud Container Service for Kubernetes (AAAI 2023 · 41 citations)
- 2024 · Pipeline Parallelism with Controllable Memory (NeurIPS 2024 · 5 citations)
- 2025 · Light-T2M: A Lightweight and Fast Model for Text-to-motion Generation (AAAI 2025 · 8 citations)
- 2026 · CloserToMe: A Unified Framework for Accurate and Transferable Latency Prediction Across Heterogeneous Devices (AAAI 2026 · 2 citations)
Earliest papers here
- Short-Dot: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products (NeurIPS 2016)
- Resource-efficient Machine Learning in 2 KB RAM for the Internet of Things (ICML 2017 · 137 citations)
- Globally Induced Forest: A Prepruning Compression Scheme (ICML 2017 · 7 citations)
What has been tried
77 papers here report 845 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 10%, hurt 20%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace Light-T2M with Transformer encoder blocks | HumanML3D test · FID | 0.040 -> 0.214 | Hurt | Light-T2M: A Lightweight and Fast Model for Text-to-motion Generation (AAAI 2025 · 8 citations) |
| Increase L3TC model size from 3.2M to 12M | enwiki9 · CR (%) | 16.23 -> 16.00 | No clear effect | L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression (AAAI 2025 · 4 citations) |
| Use binary weights and binary activations | MNIST · error (%) | 1.84±0.02 -> 2.06±0.03 | Hurt | TinyFoA: Memory Efficient Forward-Only Algorithm for On-Device Learning (AAAI 2025 · 3 citations) |
| replace uniform or Frobenius block-size assignment with loss-aware assignment | CIFAR-10, CIFAR-100, and Tiny ImageNet at 50%, 25%, and 12.5% latency limitations · Top-1 accuracy | Helped | PrivCirNet: Efficient Private Inference via Block Circulant Transformation (NeurIPS 2024 · 3 citations) | |
| add restart heuristics | Weighted 3-SAT SATLIB, n=150 · average iterations (Par 2) | 74.03 -> 21.03 | Helped | Massively Parallel Continuous Local Search for Hybrid SAT Solving on GPUs (AAAI 2025 · 2 citations) |
| Add the constraint module (without iterative training) | Val Seen · CR | 27.47 -> 28.06 | Helped | Towards Realistic Earth-Observation Constellation Scheduling: Benchmark and Methodology (NeurIPS 2025 · 2 citations) |
| Co-design the compressor tree with an optimized prefix tree instead of optimizing only the compressor tree | 8/16/32/64-bit multipliers, Min Delay · delay | 0.5180 -> 0.4905; 0.7392 -> 0.7138; 0.9960 -> 0.9685; 1.2424 -> 1.2401 | Helped | Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designs (NeurIPS 2024 · 2 citations) |
| Increase Value cache split count s_n from 1 to 4 | Value Cache, size 8192, s=4 · MSE loss | 0.259 -> 0.093 | Helped | CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation (AAAI 2025 · 1 citation) |
| Constrain the strategy space to inter-layer-only strategies | ENVB, BERT-Huge-32; T5-Large-32; ViT-Huge-32; Swin-Huge-48 · training throughput | Mixed | UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming (CVPR 2025 · 1 citation) | |
| Add intra-request parallelism | Synthetic workload, 2 images/request · TTFT | 1.46 -> 0.92 | Helped | Efficiently Serving Large Multimodal Models Using EPD Disaggregation (ICML 2025 · 1 citation) |
| Add staleness gap penalty | 100K steps · validation perplexity | 7.56 -> 7.39 | Helped | CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · 1 citation) |
| Increase kernel size R | WikiText · word perplexity | Helped | Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 51 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 5 of 51 papers
- CIFAR-100 · 4 of 51 papers
- Tiny-ImageNet · 3 of 51 papers
- wikitext · 3 of 51 papers
- MNIST · 3 of 51 papers
- ImageNet-1K · 3 of 51 papers
- BoolQ · 2 of 51 papers
- MATH · 2 of 51 papers
Baselines
- vLLM · 3 of 51 papers
- Alpa · 2 of 51 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 121 papers; from OpenAlex and the papers' affiliations).
- National University of Singapore · 9 papers
- Seoul National University · 6 papers
- Beihang University · 4 papers
- Shanghai Jiao Tong University · 4 papers
- University of Texas at Austin · 4 papers
- NVIDIA · 4 papers
- Ant Group · 3 papers
- Sea AI Lab · 3 papers
- Meta · 3 papers
- Princeton University · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Dataset condensation and flow estimation (126 papers)
- Scalable approximation and equivariant compute (120 papers)
Elsewhere in the topic
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Differentiable optimization layers (125 papers)
- Non-autoregressive translation (102 papers)
- RNN training dynamics and gradients (102 papers)
- Neural-symbolic reasoning (98 papers)
- Sparse dictionary representations (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).