Research map / Architectures and efficiency / Attention and transformers
MELODI: Exploring Memory Compression for Long Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MELODI, a hierarchical compression memory that pairs multi-layer recurrent short-term memory with a single-layer compressed long-term KV memory, matching or beating Memorizing Transformer perplexity with about 8x less memory.
Problem
Efficiently processing long documents with short context windows requires memory that bridges windows over long contexts without the memory cost of storing full KV caches.
Approach
Represent short-term memory as recurrent compression of context windows across multiple layers (via summary tokens and linear token mixers) and long-term memory as further compressed KV pairs from a single middle layer stored in a FIFO queue, arranged in a sandwich architecture.
Key results
- MELODI S128+L64 reaches 10.44 perplexity on PG-19 (T5 vocab) vs 10.62 for Memorizing Transformer, using 18.5M memory vs 147.8M (about 8x less).
- MELODI S192+L96 achieves 7.91 on PG-19 Meena, 2.09 on arXiv Meena, and 17.25 on C4(4K+), best across all reported datasets.
- MELODI S192+L32 (11.0M memory) beats Transformer XL (13.6M) and Block Recurrent Transformer (13.1M) on all datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MELODI S192+L96 instead of Memorizing Transformer | long-context language modeling · PG19 (T5) | perplexity: 10.62 -> 10.29 | Helped |
| use MELODI S192+L96 instead of Memorizing Transformer | long-context language modeling · arXiv Math (Meena) | perplexity: 2.14 -> 2.09 | Helped |
| use MELODI S192+L96 instead of Memorizing Transformer | long-context language modeling · C4(4K+) (Custom) | perplexity: 17.37 -> 17.25 | Helped |
| add summary branching (ablation) | MELODI with short- and long-term memory · PG19 (T5) | perplexity: 11.24 -> 10.95 | Helped |
| add summary branching (ablation) | MELODI with short-term memory only · PG19 (T5) | perplexity: 11.68 -> 11.39 | Helped |
| generate distinct long-term tokens instead of copying short-term tokens (ablation) | MELODI S96+L96 · PG19 (T5) | perplexity: 11.00 -> 10.92 | Helped |
| generate distinct long-term tokens instead of copying short-term tokens (ablation) | MELODI S64+L64 · PG19 (T5) | perplexity: 11.36 -> 11.08 | Helped |
| generate distinct long-term tokens instead of copying short-term tokens (ablation) | MELODI S32+L32 · PG19 (T5) | perplexity: 11.42 -> 11.25 | Helped |
| increase short-term or long-term memory capacity (ablation) | MELODI · PG19 (T5) | perplexity | Helped |
| increase long-term memory coverage up to 32 windows (ablation) | MELODI S128+L64 · PG19 (T5) | perplexity | Helped |
| increase long-term memory coverage beyond 32 windows (ablation) | MELODI S128+L64 · PG19 (T5) | perplexity | No clear effect |
| increase the number of short-term layers from 1 to 4 (ablation) | MELODI · PG19 (T5) | perplexity | Helped |
Surprising findings
- Short-term memory alone outperforms gains from additional long-term memory coverage for the most recent 2-4 windows, suggesting recent context is already captured by short-term memory.
- Re-implemented baselines with cosine decay and dense cross-attention substantially outperform their originally reported perplexities.
Limitations
- Trained from scratch only; does not address fine-tuning of pre-trained models with fixed context windows.
- Evaluated only on language modeling perplexity rather than downstream long-context tasks.
- Models are small (12-13 layers, 1024 dim) relative to modern LLMs.
Experimental setup
Datasets: PG-19 arXiv Math C4
Benchmarks: PG-19 arXiv Math C4(4K+)
Metrics: token-level perplexity memory size (floats)
Baselines: Transformer XL Block Recurrent Transformer Memorizing Transformer
Builds on: Memorizing Transformer Block Recurrent Transformer Transformer-XL Recurrent Memory Transformer AutoCompressors
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.86)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.86)
- Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.86)
- UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression (NeurIPS 2025 · similarity 0.86)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.85)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.