Research map / Architectures and efficiency / Attention and transformers
Long Context Compression with Activation Beacon
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Activation Beacon, a plug-in module that compresses long contexts into beacon-token keys/values at every layer via progressive chunk-wise compression, trained with compression-based auto-regression and random per-chunk compression ratios.
Problem
Long-context LLM inference is costly in computation and KV cache memory; existing context compression methods compress into soft tokens all at once, require re-encoding, use fixed compression ratios, or depend on the query, which limits effectiveness and efficiency in long-context and multi-turn set
Approach
Insert beacon tokens interleaved with fine-grained units of each chunk, encode chunks one by one, discard raw-token activations and accumulate beacon-token keys/values at every layer for reuse, using separate projection matrices for beacon tokens while the LLM stays frozen.
Key results
- On LongBench, Activation Beacon on Llama-2-7B reaches 34.9 Single-Doc and 25.0 Summ. at 32K, matching or exceeding Full-FT (34.8, 23.2), and outperforming ICAE, AutoCompressor, LongLLMLingua and SnapKV.
- On Multi-Needle NIAH at 128K with Qwen-2-7B, Activation Beacon (x8) achieves 9.70 accuracy versus 9.75 for Full-FT while cutting latency from 4.399s to 2.445s (about 2x speed-up) and reducing KV cache 8x.
- Removing fine-grained interleaving drops Single-Doc LongBench score from 40.5 to 35.2 on Qwen-2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Activation Beacon compression at the reported compression settings rather than an uncompressed Full-FT baseline | Llama-2-7B · LongBench, 32K | score: 34.8 -> 34.9; 27.5 -> 27.5; 23.2 -> 25.0; 61.8 -> 61.4; 57.8 -> 57.8 | Mixed |
| Use Activation Beacon compression rather than an uncompressed Full-FT baseline | Qwen-2-7B · LongBench, 32K | score: 41.0 -> 40.5; 40.6 -> 40.3; 26.8 -> 26.8; 68.5 -> 68.4; 66.1 -> 66.4 | Mixed |
| Use Activation Beacon instead of Full-FT | Qwen-2-7B · Multi-Needle-in-a-Haystack 3-Turn, 128K | Latency: 6.153 -> 2.981 | Helped |
| Use fine-grained interleaved beacon tokens instead of appending all beacons at chunk end (ablation) | Activation Beacon · Single-Doc QA, LongBench, Qwen-2-7B, 32K, x4 | score: 35.2 -> 40.5 | Helped |
| Use chunk-wise rather than instance-wise random compression ratios during training (ablation) | Activation Beacon · Single-Doc QA, LongBench, Qwen-2-7B, 32K, x4 | score: 37.7 -> 40.5 | Helped |
| Add pre-training (ablation) | Activation Beacon · Single-Doc QA, LongBench, Qwen-2-7B, 32K, x4 | score: 34.9 -> 40.5 | Helped |
| Add fine-tuning (ablation) | Activation Beacon · Single-Doc QA, LongBench, Qwen-2-7B, 32K, x4 | score: 35.5 -> 40.5 | Helped |
Surprising findings
- Activation Beacon improves Llama-2 substantially on LongBench despite the same 4K window, suggesting it introduces information from unseen context and can serve as a context-extension approach.
- Short-context performance (MMLU, ARC-C, BoolQ, GSM8K) is largely preserved, attributed to freezing the original LLM parameters.
Limitations
- Evaluation relies on ChatGPT as judge for NIAH accuracy.
- Only Llama-2-7B and Qwen-2-7B backbones are evaluated, and only 7B scale is reported.
- Fine-grained compression with interleaved beacon tokens is the default; alternative placements yield inferior quality.
Experimental setup
Datasets: RedPajama LongAlpaca BookSum Synthesized QA (GPT-3.5-turbo generated, Books3 and Arxiv/Pile) LongBench Needle-in-a-Haystack NeedleBench MMLU ARC-Challenge BoolQ GSM8K
Benchmarks: LongBench Needle-in-a-Haystack NeedleBench multi-needle MMLU ARC-Challenge BoolQ GSM8K
Metrics: accuracy (LongBench scores) NIAH score 1-10 multi-turn accuracy end-to-end latency (s) FLOPs KV cache size
Baselines: AutoCompressor ICAE LongLLMLingua SnapKV Full Full-FT
Builds on: ICAE AutoCompressor Gisting FlashAttention-2
Closest papers by meaning
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.91)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.90)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.89)
- Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission (NeurIPS 2026 · similarity 0.89)
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.