Research map / Architectures and efficiency / Attention and transformers
Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spotlight Attention, which replaces linear LSH hashing with a learned two-layer MLP hash function trained with a Bradley-Terry ranking loss to retrieve top-k KV cache tokens during decoding, reaching near-oracle retrieval with 128-bit codes and up to 3x throughput.
Problem
Token-level KV cache retrieval for LLM decoding relies on linear LSH hashing, which needs very long hash codes (~1024 bits) because queries and keys occupy two narrow, nearly orthogonal cones, causing poor encoding and storage overhead.
Approach
Replace random linear hyperplanes with a learned non-linear MLP hashing function (sign of an MLP output) trained with a Bradley-Terry pairwise ranking loss that separates top-k from non-top-k attention scores, with the LLM frozen and only the hash modules trained layer-wise on a small calibration set.
Key results
- MLP hashing with 128-bit codes raises top-2% retrieval IoU for LLaMA3-8B to 0.34 after training (LSH 0.18), with PPL near oracle at 98% pruning on PG19.
- On LongBench with LLaMA3-8B, Spotlight at 163-token budget reaches Rouge-L fidelity 0.58 vs Quest 1024-budget 0.56 and MagicPIG 0.44.
- End-to-end throughput up to 3x vanilla decoding on Qwen2.5-7B, with bitwise hashing retrieval over 512K tokens in under 100 microseconds on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace linear LSH with trained non-linear MLP hashing | Spotlight retrieval on LLaMA2-7B · PG19, top-2% retrieval | IoU: 0.05 -> 0.41 | Helped |
| Train linear LSH hashing (ablation) | LSH retrieval on LLaMA2-7B · PG19, top-2% retrieval | IoU: 0.17 -> 0.20 | No clear effect |
| Replace Quest with Spotlight at a smaller token budget | LLaMA3-8B · ProofPile, 98% versus 96.9% pruned | perplexity: 11.631 -> 3.621 | Helped |
| Use hash-based retrieval without local windows or sink tokens | LLaMA3-8B · LongBench output fidelity | Rouge-L similarity: 0.44 -> 0.58 | Helped |
| Compress decoding with Spotlight retrieval | LLaMA3-8B · 5-shot learning, fixed budget 20 | win rate: 38% -> 62% | Mixed |
| Change model size while using Spotlight Top-2% (ablation) | Qwen2.5 · PG19 | perplexity: 13.828 -> 13.510 (1.5B); 11.112 -> 9.930 (7B); 8.416 -> 8.261 (14B) | Mixed |
| Train the hashing function on C4 instead of Arxiv + Book (ablation) | Spotlight on LLaMA3-8B · PG19 | perplexity: 8.977 -> 8.958 | Helped |
| Train the hashing function on Github Code instead of Arxiv + Book (ablation) | Spotlight on LLaMA3-8B · PG19 | perplexity: 8.977 -> 8.891 | Helped |
| Train the hashing function on C4 instead of Arxiv + Book (ablation) | Spotlight on LLaMA3-8B · Math | perplexity: 3.621 -> 3.631 | Hurt |
| Train the hashing function on Github Code instead of Arxiv + Book (ablation) | Spotlight on LLaMA3-8B · Code | perplexity: 5.434 -> 5.393 | Helped |
Surprising findings
- Training with the reconstruction (MSE/cross-entropy) loss degraded performance, while the ranking loss on top-k vs non-top-k sets was needed.
- Oracle top-k retrieval with 98% of KV cache pruned (after first two layers) increased PG19 perplexity by only about 0.1.
Limitations
- IoU remains around 40% even with non-linear hashing, indicating room for improvement.
- Efficiency experiments used HuggingFace Transformers with pipeline parallelism rather than optimized serving stacks.
- Some perplexity comparisons with MagicPIG used only 10 samples due to evaluation cost.
Experimental setup
Datasets: ArXiv Books C4 GitHub Code PG19 ProofPile CodeParrot LongBench Needle-in-a-Haystack LM-Eval-Harness tasks
Benchmarks: PG19 ProofPile CodeParrot Needle-in-a-Haystack LongBench LM-Eval-Harness 5-shot
Metrics: IoU perplexity Rouge-L throughput latency accuracy
Baselines: Oracle top-k LSH top-k Quest MagicPIG H2O
Builds on: MagicPIG Quest angular LSH RankNet
Closest papers by meaning
- MagicPIG: LSH Sampling for Efficient LLM Generation (ICLR 2025 · similarity 0.91)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.90)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.90)
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.89)
- DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference (ICLR 2025 · similarity 0.88)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.