Research map / Architectures and efficiency / Attention and transformers
LoCoCo: Dropping In Convolutions for Long Context Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoCoCo, a drop-in convolutional KV-cache compressor that fuses incoming tokens into a fixed-size KV cache via learned 1D-convolution mixing weights, for long-context inference and fine-tuning.
Problem
LLM KV caches grow linearly with context length, exhausting GPU memory during long-context inference and fine-tuning; prior eviction heuristics drop tokens and lose context information.
Approach
Replace KV eviction with a learnable, data-driven weighted fusion of cached KV entries and incoming tokens, where the mixing weights are produced by a shallow 1D convolution (kernel size 21) over the KV cache and new segment, keeping the cache at a fixed size while leaving pretrained weights unchanged.
Key results
- Compresses up to 3482 tokens into a 128-slot KV cache with comparable performance to the full sequence, with accuracy gains up to 0.2791 over H2O at the same cache size.
- Extends Llama-2-7B context from 4K to 32K with a fixed 512-size KV cache; perplexity on Proof-Pile-2 at 32K is 3.4408 vs 3.4012 for full-sequence fine-tuning and 3.5102 for H2O.
- On LongBench with Llama2-13B extended to 32K, LoCoCo scores 37.4% vs 36.9% for H2O and 34.7% for LongLoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace token eviction with convolutional token merging | Llama-2 with LoCoCo · SCROLLS, including QuALITY, Qasper, GovReport, QMSum, NarrativeQA, and SummScreen | score: QuALITY: 0.3461 -> 0.3528; Qasper: 0.2659 -> 0.2813; GovReport: 0.2924 -> 0.3113; QMSum: 0.1913 -> 0.2089; NarrativeQA: | Helped |
| extend context to 32K using LoCoCo | Llama2-13B · LongBench | score: 36.9% -> 37.4% (H₂O); 34.7% -> 37.4% (LongLoRA) | Helped |
| use LoCoCo instead of H₂O | Llama2-7B · Inference prefill, 16k tokens | throughput (Token/s): 32 -> 33 | Helped |
| use LoCoCo instead of LongLoRA | Llama2-7B · Training, extending context to 16k | memory usage: 49GB -> 50GB | Hurt |
| use LoCoCo instead of H₂O | Llama-2-7B · Proof-Pile-2, evaluation context 8192 | perplexity: 3.5714 -> 3.5414 | Helped |
| add StreamingLLM attention-sink tokens (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.5451 -> 3.5439 | No clear effect |
| add H₂O heavy-hitter eviction (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.5451 -> 3.5414 | Helped |
| use kernel size 3 instead of 21 (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.53 -> 3.68 | Hurt |
| use kernel size 7 instead of 21 (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.53 -> 3.57 | Hurt |
| use kernel size 17 instead of 21 (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.53 -> 3.53 | No clear effect |
| use kernel size 31 instead of 21 (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.53 -> 3.54 | No clear effect |
| use kernel size 41 instead of 21 (ablation) | LoCoCo · Proof-Pile-2, evaluation context 8192 | perplexity: 3.53 -> 3.53 | No clear effect |
Surprising findings
- Kernel-size ablation shows performance is fairly stable across kernel sizes 17 to 51, with degradation only at very small kernels (size 3).
Limitations
- Requires training convolutional heads (and LoRA for extension) on calibration data, so it is not training-free.
- Extension experiments are limited to Llama-2 and a few baselines; perplexity comparisons at 32K on 7B show a gap to full-sequence fine-tuning.
Experimental setup
Datasets: RedPajama Proof-Pile-2 RACE TriviaQA HellaSwag WinoGrande ARC SCROLLS LongBench
Benchmarks: RACE TriviaQA HellaSwag WinoGrande ARC SCROLLS Proof-Pile-2 LongBench
Metrics: accuracy perplexity throughput peak GPU memory
Baselines: H2O StreamingLLM LongLoRA full sequence fine-tuning
Builds on: Transformer-XL H2O StreamingLLM LongLoRA positional interpolation
Closest papers by meaning
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.89)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.89)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.