Research map / Architectures and efficiency / Attention and transformers
$S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict output length and allocate KV memory accordingly instead of max length (ORCA) | LLM serving · Alpaca, offline max throughput | throughput: 1.13x to 6.49x | Helped |
| use S3 predictor-based scheduling | LLAMA 33B · Alpaca serving | average batch size: 4 -> 42.47 | Helped |
| use S3 on 6 GPUs vs ORCA on 10 GPUs | GPT-3 175B · maximum throughput | throughput | No clear effect |
| replace S3 predictor with perfect Oracle predictor | S3 · Alpaca offline throughput | throughput: up to 40.52% higher | Helped |
| train predictor on non-QA Pile data instead of Alpaca (ablation) | DistilBERT predictor · length bucket prediction | accuracy (%): 98.61 -> 65.6 | Hurt |
| train predictor on Natural Questions instead of Alpaca (ablation) | DistilBERT predictor · length bucket prediction | accuracy (%): 98.61 -> 77.13 | Hurt |
Closest papers by meaning
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.89)
- ThinK: Thinner Key Cache by Query-Driven Pruning (ICLR 2025 · similarity 0.88)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.88)
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.88)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.88)
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.