Research map / Large language models / Reasoning and chain of thought

Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Fast KVzip, a gating-based KV cache eviction method that trains lightweight sink-attention gates on frozen LLMs using KVzip reconstruction scores, achieving near-lossless performance at 30-40% KV cache budgets.

Problem

KV cache eviction methods either degrade accuracy when overhead is negligible, or preserve accuracy only with prohibitive compression overhead (e.g., KVzip doubles prefill compute), limiting deployment for long-context LLM inference.

Approach

Lightweight low-rank sink-attention gates operating on input hidden states predict per-KV-head importance; they are trained with BCE against KVzip reconstruction-derived attention scores on frozen LLM weights using only forward passes, and applied during chunked prefill and buffered decoding for eviction.

Key results

• Fast KVzip maintains near-lossless performance while evicting up to 70% of the KV cache across Qwen2.5-1M, Qwen3, and Gemma3 families.
• Matches KVzip compression performance on prefill-intensive tasks while removing the reconstruction doubling of prefill compute.
• Gate training takes 0.59-0.83 H100 hours for Qwen3-8B/14B and 0.70 for Qwen3-30B-A3B.

Surprising findings

• Reconstruction-based targets generalize better than next-token prediction or instruction-QA targets for gate training.
• Incorporating position-encoded keys as gate inputs degrades performance; hidden states work best.

Limitations

• Gating is applied post hoc to pretrained models; the paper suggests from-scratch or pretraining integration is future work.
• Decoding-time evaluation limited to AIME24 and MATH with Qwen3 models.

Experimental setup

Datasets: FineWeb-Edu SCBench RULER LongBench OpenAI MRCR SQuAD AIME24 MATH LongAlpaca

Benchmarks: SCBench RULER-4K KVPress OpenAI MRCR SQuAD AIME24 MATH

Metrics: relative performance vs full cache accuracy

Baselines: KVzip SnapKV Expected Attention Ada-KV DuoAttention R-KV TrimKV Locret DMS

Builds on: KVzip attention sinks FastGen SnapKV Mixture-of-Depths

Closest papers by meaning

• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.93)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 · similarity 0.93)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.92)
• KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.91)
• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)