Research map / Large language models / Reasoning and chain of thought
Efficient LLM serving
Schedule queries, reuse prefixes, and optimize architectures for efficient LLM inference.
126 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Reasoning and chain of thought, keyword cluster “decoding · speculative · cache”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
20 papers in 2023–24 and 106 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 15.4% to 7.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (NeurIPS 2023 · 66 citations)
- 2024 · Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · 19 citations)
- 2025 · Portcullis: A Scalable and Verifiable Privacy Gateway for Third-Party LLM Inference (AAAI 2025 · 9 citations)
- 2026 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · 8 citations)
Earliest papers here
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (NeurIPS 2023 · 66 citations)
- Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (ICML 2023 · 19 citations)
- Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline (NeurIPS 2023 · 15 citations)
What has been tried
55 papers here report 617 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 7%, hurt 24%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Classification predictor (10 buckets) instead of ranking | ShareGPT, samples generated in 5 min · Samples generated | 670 -> 479 (vs FCFS) | Hurt | Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · 19 citations) |
| use BERT-NER masking instead of Hide-and-Seek | PII · full match | 0.663 -> 0.824 | Helped | Portcullis: A Scalable and Verifiable Privacy Gateway for Third-Party LLM Inference (AAAI 2025 · 9 citations) |
| predict 6 future tokens instead of 1 | APPS/Intro · pass@1 | 2.8 -> 3.5 (+0.7%) | Helped | Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · 9 citations) |
| Use LoGra-PCA initialization instead of LoGra-random | FMNIST · data valuation accuracy | Helped | What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions (NeurIPS 2025 · 8 citations) | |
| replace GELU with ReLU in Falcon 7B FFN (stage 1) | Zero-shot tasks (average) · FLOPS (G) | 6.6 -> 4.1 | Helped | ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (ICLR 2024 · 7 citations) |
| add KV-cache eviction strategy | Alpaca · PPL | 6.08 -> 6.01 | Helped | D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models (NeurIPS 2024 · 6 citations) |
| Replace detailed profiler feedback with no additional feedback | EffiBench · ET (s) | 0.52 -> 0.79; -51.9% | Hurt | EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · 5 citations) |
| Increase GPU count from 2 to 4 | Across tested benchmarks and models · Avg_Std@top1_prob | Mixed | Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · 4 citations) | |
| Add frequency-based masking to PVE | ICL and open-domain QA · task performance | Helped | Vision-centric Token Compression in Large Language Model (NeurIPS 2025 · 4 citations) | |
| incremental upscaling applied to uniform GPTQ | Wikitext2 perplexity, Llama-2-7B 4-bit · perplexity | 5.68 -> 36.82 | Hurt | Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · 4 citations) |
| Defer early-exit forward computation to backward steps | Training iteration, 1.3B GPT · peak memory (GB) | 24.05 -> 22.56 | Helped | EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · 4 citations) |
| Add prefix sharing maximization scheduling | Azure LLM Inference trace + MMLU · offline throughput | up to 4\times offline throughput gain | Helped | HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location (NeurIPS 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 106 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 12 of 106 papers
- WikiText2 · 6 of 106 papers
- MMLU · 6 of 106 papers
- C4 · 6 of 106 papers
- HotpotQA · 5 of 106 papers
- Alpaca · 5 of 106 papers
- LongBench · 5 of 106 papers
- ShareGPT · 4 of 106 papers
Baselines
- vLLM · 7 of 106 papers
- SGLang · 3 of 106 papers
- S3 · 3 of 106 papers
- FCFS · 3 of 106 papers
- Random · 3 of 106 papers
- A-Mem · 3 of 106 papers
- Mem0 · 3 of 106 papers
- zkLLM · 2 of 106 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 126 papers; from OpenAlex and the papers' affiliations).
- Huawei · 10 papers
- Meta · 6 papers
- Carnegie Mellon University · 6 papers
- National University of Singapore · 5 papers
- Tsinghua University · 5 papers
- Stanford University · 5 papers
- Hong Kong University of Science and Technology · 5 papers
- Seoul National University · 5 papers
- Rice University · 4 papers
- University of California, Berkeley · 4 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Speculative decoding (92 papers)
- KV cache compression (45 papers)
Elsewhere in the topic
- RL with verifiable rewards for reasoning (270 papers)
- Efficient and adaptive thinking (231 papers)
- Test-time compute scaling (205 papers)
- Long CoT dynamics and efficiency (136 papers)
- Routing, pruning and distillation for reasoning (135 papers)
- Chain-of-thought training and analysis (127 papers)
- Search and self-correction in reasoning (123 papers)
- Process reward models (39 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).