Research map / Architectures and efficiency / Attention and transformers

Efficient Low Rank Attention for Long-Context Inference in Large Language Models

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LRQK, a training-free KV cache method that jointly low-rank factorizes queries and keys to compute proxy attention for top-k token selection, with a mixed GPU-CPU hit-and-miss cache that preserves exact attention on selected tokens.

Problem

KV cache memory grows linearly with context length, limiting long-context inference on resource-constrained devices; existing quantization, pruning/eviction, and offloading methods lose precision, discard tokens, or incur CPU-GPU transfer latency.

Approach

Jointly factorize full-precision query and key matrices into rank-r factors during prefill, then use these low-dimensional projections to compute proxy attention scores at each decode step to select top-k tokens plus a fixed recent window; only missing full-precision KV pairs are fetched from CPU via a hit-and-miss GPU cache, so attention on selected tokens is exact.

Key results

• On RULER 128K with LLaMA-3-8B-1M, LRQK (r=32, top-2048, 64 lite tokens) matches or exceeds baselines on several tasks, e.g. MQ 99.25 and MV 98.00 versus ShadowKV 96.88 and 95.83, but drops to 42.00 on MK2 versus 98.96 for ShadowKV.
• On a RULER-4K subset, LRQK improves LLaMA-3-8B-1M QA-1 from 82.00 to 84.00 and Qwen2.5-7B QA-1 from 90.00 to 91.00 with rank 16 and top-256.
• Average miss rate across a grid search on Wikitext-2-v1 is about 0.40, which the authors interpret as roughly 60% less CPU-to-GPU data transfer.

Measured effects reported in the paper

Surprising findings

• Performance degrades sharply on some RULER tasks at longer contexts, e.g. VT falls to 56.80 at r=8 and Qwen2.5-7B VT drops to 70.00 and 64.20 at 8K and 16K with LRQK.
• On Qwen2.5-14B at 64K, LRQK lowers S1 from 57.00 to 99.00 and S2 from 55.00 to 96.00 but reduces QA-1 from 64.00 to 50.00, showing mixed effects.

Limitations

• CPU-side indexing operations, not PCIe bandwidth, are the main bottleneck according to the authors.
• Hyperparameters (rank, top-k, lite tokens) require task-specific tuning.
• Results are mixed across tasks and models, with sharp drops on some retrieval tasks such as MK2 and VT.

Experimental setup

Datasets: Wikitext-2-v1

Benchmarks: RULER LongBench

Metrics: accuracy miss rate throughput

Baselines: Loki InfiniGen Quest ShadowKV StreamingLLM

Builds on: StreamingLLM ShadowKV Quest InfiniGen

Closest papers by meaning

• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.92)
• QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.91)
• Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.91)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.91)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.91)
• A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.91)