Research map / Architectures and efficiency / Attention and transformers
Compressed Context Memory for Online Language Model Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Compressed Context Memory (CCM), which continually compresses accumulating attention keys/values into a fixed or growing compact memory via a conditional LoRA that acts only on compression tokens, trained in parallel in one forward pass.
Problem
Online LLM interaction where context keeps growing, so attention KV memory and compute increase over time and reduce throughput.
Approach
Compress each new context chunk into attention keys/values of a special COMP token, update memory by concatenation (CCM-concat) or running average (CCM-merge), use a conditional LoRA applied only to COMP tokens to avoid overfitting on inputs, and train the recursive compression as a single masked parallel forward pass.
Key results
- CCM-concat matches full-context accuracy on MetaICL at time step 16 (70.0% vs 70.8%) with 178 MB peak KV vs 630 MB; CCM-merge reaches 69.6% with 66 MB.
- On DailyDialog, CCM-concat reaches perplexity 5.98 vs full context 5.59 with 24 compressed tokens, while MemoryBank gets 7.06 with 60.
- Parallelized training is about 7x faster per sample than AutoCompressor (195 ms vs 1330 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add conditional LoRA for compression tokens instead of default unconditional LoRA (ablation) | CCM-concat and CCM-merge · MetaICL, LaMP, and DailyDialog | accuracy (%) on MetaICL and LaMP; perplexity on DailyDialog: 69.4 -> 70.0; 66.3 -> 69.6; 83.9 -> 84.7; 82.6 -> 83.9; 6.01 -> 5.96; 6.42 -> 6.33 | Helped |
| replace arithmetic averaging with EMA (a_t=0.5) (ablation) | CCM-merge · DailyDialog | perplexity: 6.34 vs 6.38 | No clear effect |
| increase CCM-concat COMP token length from 1 to 8 (ablation) | CCM-concat · MetaICL | accuracy (%): 69.5 -> 70.0 | Helped |
| increase CCM-concat COMP token length from 1 to 4 (ablation) | CCM-concat · DailyDialog | perplexity: 6.51 -> 6.26 | Helped |
| use CCM-concat instead of CCM-merge | CCM · DailyDialog | perplexity: 6.27 -> 5.91 | Helped |
| use CCM-concat instead of CCM-merge | CCM · MetaICL and LaMP | accuracy (%): 69.6 vs 70.0; 83.9 vs 84.7 | Mixed |
| compress online context with CCM rather than full context | LLaMA-7B · MetaICL, LaMP, and DailyDialog | accuracy (%) on MetaICL and LaMP; perplexity on DailyDialog: 70.8 -> 70.0; 85.1 -> 84.7; 5.56 -> 5.91 | Hurt |
Surprising findings
- Conditional LoRA applied only to COMP tokens improves compression over default LoRA across methods, including Gisting (e.g., +3.3 points for CCM-merge on MetaICL).
- Training loss without context dropped from 2.69 to 1.84 while test loss stayed at 2.59, indicating input overfitting that motivated the conditional adapter.
Limitations
- Performance still below full-context method despite beating compression baselines
- Requires training a compression adapter per application or data source; generalization across sources varies
- Compression overhead from extra COMP token forward passes and RougeL drops noticeably versus full context
Experimental setup
Datasets: MetaICL LaMP DailyDialog SODA RedPajama-V2 LmSys-Chat Pile
Benchmarks: MetaICL LaMP DailyDialog PG19
Metrics: accuracy perplexity peak KV memory RougeL throughput training time per sample
Baselines: Full context No context Gisting-online Compressive Transformer AutoCompressor RMT MemoryBank StreamingLLM
Builds on: Gisting Compressive Transformer LoRA StreamingLLM AutoCompressor
Closest papers by meaning
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.91)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.90)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.90)
- In-context Autoencoder for Context Compression in a Large Language Model (ICLR 2024 · similarity 0.89)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.89)
- Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract) (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.