Research map / Architectures and efficiency / Attention and transformers

SALS: Sparse Attention in Latent Space for KV Cache Compression

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SALS, which compresses multi-head pre-RoPE keys into a shared low-rank latent space, selects top-k tokens by scoring in that latent space, and reconstructs only the selected keys for exact sparse attention.

Problem

KV cache size and memory bandwidth limit long-context LLM inference; low-rank KV compression conflicts with RoPE (raising key rank) and full reconstruction is costly.

Approach

Apply joint multi-head low-rank projection to pre-RoPE keys, use the latent keys (first r* dimensions) to pick top-k critical tokens, and reconstruct and RoPE-rotate only those keys before sparse attention.

Key results

• On LongBench with LLaMA2-7B-chat, SALS-25% matches baseline average (32.26 vs 32.65) at 0.11 memory access ratio, while SALS-12.5% reaches 31.97 at 0.06.
• On RULER 128k-style setting with Llama3.1-8B-Instruct at 4k length, SALS-25% averages 80.81 vs baseline 81.60; SALS-12.5% averages 75.86.
• End-to-end throughput vs GPT-Fast at 32k sequence length with batch size 8 reaches 89.47 vs 19.8 tokens/s for SALS-12.5% (about 4.5x).

Measured effects reported in the paper

Surprising findings

• Applying RoPE to keys increases their variance and rank, so pre-RoPE latent compression is preferred despite reconstruction cost.
• SALS introduces overhead on short sequences (1k) where it is slower than some sparse baselines in attention latency.

Limitations

• Short-sequence overhead: reconstruction cost makes SALS slower at 1k sequence length.
• Accuracy drops noticeably at 12.5% compression on RULER (e.g., Multi-Key-2 falls to 42.2 from 73.6).
• Layers 0, 1 and 31 are manually skipped for sparsification based on empirical observation, without a principled rule.

Experimental setup

Datasets: C4 GSM8K CoQA LongBench RULER

Benchmarks: GSM8K CoQA LongBench RULER

Metrics: accuracy memory access ratio attention operator latency (ms) end-to-end throughput (tokens/s)

Baselines: Palu KIVI Double Sparse Hshare Loki Quest StreamingLLM FlashAttention2 GPT-Fast

Builds on: Palu Double Sparse Loki KIVI Hshare

Closest papers by meaning

• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)
• AttentionPredictor: Temporal Patterns Matter for KV Cache Compression (NeurIPS 2025 · similarity 0.91)
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition (NeurIPS 2025 · similarity 0.91)
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.91)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.91)