Research map / Architectures and efficiency / Attention and transformers

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EchoKV, a flexible KV cache compression method that drops part of the KV cache and trains a lightweight linear network to reconstruct it from a global (first-layer-of-group) cache and a retained local prefix of dimensions, with a two-stage (KV-MSE then attention-output MSE) fine-tuning.

Problem

KV cache memory grows linearly with context length; existing low-rank compression methods modify model projections, so they cannot switch back to full-cache inference when memory is abundant.

Approach

Instead of compress-then-decompress, keep the full cache of a group's first layer plus a prefix of the current layer's flattened KV dims, and train a lightweight linear predictor to reconstruct the dropped KV dims; original attention projections stay untouched so full-KV inference remains available.

Key results

• On LongBench with Llama-3.1-8B-Instruct at compression ratio 0.5, EchoKV averages 48.53 vs 47.13 for CommonKV and 49.88 full KV.
• At compression ratio 0.3 on Llama-3.1-8B, EchoKV scores 45.27 LongBench average vs 31.08 for CommonKV.
• On RULER at ratio 0.5 with Llama-3.1-8B, EchoKV averages 83.52 vs 87.63 full KV and 69.68 for CommonKV.

Surprising findings

• Predicting Keys from partial information is harder than predicting Values; predicting only Values performs well while predicting only Keys degrades significantly.
• Training on short sequences (~2 min on one A100) for a 7B model still yields strong long-context performance; dataset choice has little effect.

Limitations

• Local dimension selection uses a simple fixed prefix heuristic rather than adaptive selection.
• Keys are harder to reconstruct than Values, so the approach is less effective for keys; hybrid variant is exploratory and not uniformly better.
• Requires training a predictor per model and per compression configuration on a given dataset.

Experimental setup

Datasets: LongAlpaca Alpaca ShareGPT C4

Benchmarks: LongBench RULER

Metrics: LongBench average score RULER accuracy output throughput

Baselines: Palu CommonKV ThinK MiniCache TransMLA

Builds on: Palu CommonKV ThinK MiniCache Flash Attention

Closest papers by meaning

• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.92)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.92)
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.92)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.92)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.92)