Research map / Large language models / Reasoning and chain of thought
Intent-Aware Caching for Efficient LLM Serving
Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, Bo Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an intent-aware caching approach for LLM serving to improve prefix cache hit rates in multi-turn conversations.
Problem
Prefix caching in LLM serving for multi-turn conversations achieves low hit rates because cache reuse depends on whether a conversation continues an earlier prefix, which is hard to predict.
Approach
Uses user intent to guide which conversation prefixes' KV states are cached and reused; details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use intent-aware eviction (IAC) instead of LRU | SGLang prefix cache · ShareChat 140K tokens | cache hit rate: 12.2% -> 34.6% | Improved |
| use IAC instead of learned LPC | SGLang prefix cache · WildChat 140K tokens | cache hit rate: 17.1% -> 24.8% | Improved |
| IAC instead of LRU | Qwen-1.5-110B serving · ShareChat 5 req/s | mean TTFT: 6.98 s -> 3.76 s | Improved |
| IAC instead of LRU at high load | Llama-3-70B serving · ShareChat 10 req/s | average latency: 7.01 s -> 3.41 s | Improved |
| IAC instead of LPC for memory use | Llama-3-70B · ShareChat | GPU memory for similar hit rate: 51.9 GB -> 24.4 GB (53% saving) | Improved |
| add reuse-probability signal to recency signal (ablation) | IAC eviction priority · ShareChat/WildChat | cache hit rate | Improved |
| order signals (recency, reuse) instead of (reuse, recency) (ablation) | IAC eviction priority · ShareChat/WildChat | cache hit rate | Improved |
| add size as final tie-breaker (ablation) | IAC eviction priority · ShareChat/WildChat | cache hit rate | Improved |
| group by intent plus turn instead of turn only (ablation) | IAC statistics · ShareChat | cache hit rate | Improved |
| use 30 intent categories instead of coarser 10 or 20 (ablation) | IAC · ShareChat | cache hit rate | Improved |
| randomize intent labels at serving time (0% to 100%) (ablation) | IAC · ShareChat 170K tokens | cache hit rate: graceful drop to 30% at full randomization | Hurt |
| use median delay as reuse window instead of mean or mode (ablation) | IAC recency signal · ShareChat | cache hit rate | Improved |
Closest papers by meaning
- Tail-Optimized Caching for LLM Inference (NeurIPS 2025 · similarity 0.89)
- Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective (ICLR 2026 · similarity 0.87)
- DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving (ICLR 2026 · similarity 0.86)
- Auditing Prompt Caching in Language Model APIs (ICML 2025 · similarity 0.85)
- EPIC: Efficient Position-Independent Caching for Serving Large Language Models (ICML 2025 · similarity 0.85)
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
