Research map / Architectures and efficiency / Attention and transformers
Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
Hui Zeng, Daming Zhao, Pengfei Yang, WenXuan Hou, Tianyang Zheng, Hui Li, Weiye Ji, Jidong Zhai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use layer- and time-adaptive pruning instead of H2O | Qwen-7B · Math500 | accuracy (%): 68.00 -> 85.40 | Improved |
| use adaptive layer allocation instead of static PyramidKV | Llama-8B · Math500 | accuracy (%): 43.00 -> 77.80 | Improved |
| use Lethe instead of StreamingLLM | Llama-8B · Math500 | accuracy (%): 34.40 -> 77.80 | Improved |
| prune KV cache with Lethe versus full cache | Qwen-32B · Math500 | accuracy (%): 88.00 -> 80.00 | Hurt |
| prune KV cache with Lethe versus full cache | Qwen-7B · Math500 | accuracy (%): 86.40 -> 85.40 | Hurt |
| prune KV cache with Lethe at batch size 8 | Llama-8B · throughput | tokens/s: 115.6 -> 217.8 | Improved |
| prune KV cache with Lethe at batch size 32 | Qwen-7B · throughput | tokens/s: OOM -> 245.0 | Improved |
| prune KV cache with Lethe at batch size 1 | Qwen-7B · throughput | tokens/s: 33.1 -> 34.6 | No clear effect |
| prune KV cache with Lethe at batch size 8 | Llama-8B · generation memory | MB: 65096 -> 5383 | Improved |
| prune KV cache with Lethe at batch size 8 | Qwen-32B · generation memory | MB: 18633 -> 18639 | No clear effect |
| increase sparse_ratio up to 400 (ablation) | Lethe · Math500 / MMLU | accuracy | Improved |
| lower sparse_ratio below 400 (ablation) | Lethe · Math500 / MMLU | accuracy | Hurt |
Closest papers by meaning
- R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.91)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.90)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.90)
- Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression (ICML 2026 · similarity 0.90)
- ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models (ICLR 2026 · similarity 0.90)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
