Research map / Large language models / Reasoning and chain of thought
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add heavy-hitter KV entries to the recent-token (Local) cache at 20% budget (ablation) | OPT-30B · COPA | accuracy: 48.00 -> 84.00 | Helped |
| add heavy hitters to Sparse Transformer (strided) (ablation) | OPT-30B · COPA | accuracy: 50.00 -> 83.00 | Helped |
| add heavy hitters to Sparse Transformer (fixed) (ablation) | OPT-30B · Winogrande | accuracy: 49.88 -> 64.96 | Helped |
| add heavy hitters to Top-K attention (ablation) | OPT-30B · COPA | accuracy: 80.00 -> 82.00 | Helped |
| keep only heavy hitters, drop recent tokens (ablation) | H2O · Winogrande | accuracy: 69.06 -> 47.36 | Hurt |
| keep only recent tokens, drop heavy hitters (ablation) | H2O · PiQA | accuracy: 78.45 -> 55.82 | Hurt |
| add 4-bit quantization to H2O (ablation) | OPT-30B · PiQA | accuracy: 78.45 -> 78.80 | No clear effect |
| use H2O (20%) instead of FlexGen | OPT-6.7B on T4 · 512+512 generation throughput | tokens/s: 16.8 -> 51.7 | Helped |
| use H2O (20%) instead of FlexGen at same batch size | OPT-6.7B on A100 · 2048+2048 latency | latency (s): 99.5 -> 53.5 | Helped |
| use H2O instead of full cache | OPT/LLaMA · COPA 5-shot | accuracy: 81.00 -> 85.00 | No clear effect |
| use H2O instead of SpAtten | OPT-30B · COPA | accuracy: 82.00 -> 84.00 | Helped |
| use H2O instead of full cache | LLaMA-7B · XSUM generation diversity | Self-BLEU: 0.0057 -> 0.0051 | Helped |
Closest papers by meaning
- $\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models (ICLR 2025 · similarity 0.91)
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.91)
- ThinK: Thinner Key Cache by Query-Driven Pruning (ICLR 2025 · similarity 0.90)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.90)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.90)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.