Research map / Architectures and efficiency / Attention and transformers
KVBuffer: IO-aware Serving for Linear Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KVBuffer, an IO-aware serving mechanism that buffers recent keys/values for linear attention so decoding can use chunkwise, parallel-verification, and KV-only forms instead of per-step recurrent state updates, implemented in SGLang for Qwen3-Next.
Problem
Serving linear attention models recurrently reads and writes a large fixed-size state every decoding step, which is memory-bandwidth-inefficient, and speculative decoding multiplies state memory by the number of draft tokens.
Approach
Buffer recent keys and values and update the linear attention state in batch (chunkwise), verify draft tokens in parallel by buffering their KVs and committing only accepted ones, and for short contexts (L < d) compute outputs directly from buffered KVs without any state.
Key results
- Chunkwise decoding reduces linear attention decoding latency by up to 45.17% with a buffer size of 32 (kernel level).
- For verifying 8 draft tokens, the method achieves a 2.78x verification speedup versus recurrent verification.
- With 4 draft tokens, the maximum number of serving requests increases by 5x and end-to-end throughput improves by up to 1.46x.
Surprising findings
- Chunkwise decoding can be slower than recurrent decoding at batch size 1 due to extra state-update kernel launch overhead.
Limitations
- Dynamic selection among decoding forms per request is not implemented; scheduling overhead of switching forms is left to future work.
- Chunkwise decoding adds kernel launch overhead that hurts small-batch latency unless CUDA Graphs are used.
- Evaluation limited to one model (Qwen3-Next) on four L40S GPUs and kernel-level plus ShareGPT throughput results.
Experimental setup
Datasets: ShareGPT
Benchmarks: ShareGPT
Metrics: kernel latency decoding latency verification latency maximum number of serving requests end-to-end throughput
Baselines: recurrent decoding (existing serving systems) chunkwise decoding recurrent verification
Builds on: Gated DeltaNet chunkwise linear attention paged attention SGLang multi-token prediction
Closest papers by meaning
- DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference (ICLR 2025 · similarity 0.87)
- Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval (NeurIPS 2025 · similarity 0.87)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.86)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.86)
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Attention-Level Speculation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.