Research map / Large language models / Language, safety and interpretability
In-context Autoencoder for Context Compression in a Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the In-context Autoencoder (ICAE), a LoRA-adapted Llama encoder that compresses long contexts into 128 memory slots read by the frozen Llama decoder, pretrained with autoencoding and continuation then fine-tuned on the GPT-4-generated PwC dataset.
Problem
Long context handling in LLMs is costly in compute and memory; prior architectural fixes degrade on long contexts, and prompt/context compression methods are either lossy, short-prompt only, or require tuning the target LLM.
Approach
Use the LLM itself, with a lightweight LoRA adapter and learnable memory tokens, as an encoder that compresses a context into a few continuous memory slots that the untouched target LLM can condition on; pretrain with autoencoding plus text continuation, then instruction fine-tune on context-prompt-response data.
Key results
- ICAE on Llama-7b with 128 memory slots (4x compression) reaches BLEU above 95% at 500-token contexts and win+tie of 73-81% against Alpaca and StableLM-7b in GPT-4 judging.
- Pretrained ICAE with 64 slots (8x) matches non-pretrained ICAE with 128 slots (4x) in GPT-4 judged comparisons.
- Inference with ICAE compression yields over 2x speedup at 512-token input, around 3.5x for batched 8x2048 and 32x512 inputs, and about 20GB GPU memory saved at 4096 tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress a 512-token context to 128 memory slots | Llama-7B text continuation · 512→128 (4×) | PPL: 9.01 -> 9.50; +0.49 | Hurt |
| Use 64 or 32 memory slots instead of k=128 (ablation) | ICAE autoencoding · Long contexts, including length 500 | BLEU | Hurt |
| Compress 512-token content to 128 slots | ICAE restoration · Normal text versus random text | BLEU: 99.3; 3.5; 0.2 | Mixed |
| Increase memory length from k=64 to k=128 (ablation) | Llama-2-7B-chat ICAE · PwC test set | win rate in pairwise comparison: 57.6% win; 19.5% lose; 22.9% tie | Helped |
| Use pretrained k=64 rather than non-pretrained k=128 (ablation) | Llama-2-7B-chat ICAE · PwC test set | win rate in pairwise comparison: 33.1% win; 28.0% lose; 38.9% tie | Helped |
| Add pretraining (ablation) | Llama-2-7B-chat ICAE, k=128 · PwC test set | win rate in pairwise comparison: 60.4% win; 9.5% lose; 30.1% tie | Helped |
| Combine AE and LM pretraining objectives instead of using AE alone (ablation) | Llama-2-7B-chat ICAE, k=128 · PwC test set | win rate in pairwise comparison: 36.4% win; 28.5% lose; 35.1% tie | Helped |
| Combine AE and LM pretraining objectives instead of using LM alone (ablation) | Llama-2-7B-chat ICAE, k=128 · PwC test set | win rate in pairwise comparison: 35.1% win; 24.9% lose; 40.0% tie | Helped |
| Use ICAE memory slots instead of a GPT-4 summary of the same 128-token length (ablation) | Llama-2-7B-chat ICAE · PwC test set | win rate in pairwise comparison: 34.1% win; 17.6% lose; 48.3% tie | Helped |
| Increase target model size from Llama-7B to Llama-2-13B | Pretrained ICAE, 512→128 · Autoencoding | BLEU: 99.1 -> 99.8 | Helped |
| Compress the context before generation | Llama-7B inference · 8×2048 input | speedup: 3.3× | Helped |
| Increase memory length from k=128 to k=256 (ablation) | Llama-2-7B-chat ICAE · PwC test set, comparison against GPT-4 | win+tie rate: 74.2 -> 79.5 | Helped |
Surprising findings
- Random token sequences are compressed and restored far worse (BLEU 0.2 for completely random text) than normal text (BLEU 99.3), with restoration errors resembling human memorization mistakes.
- Memory slots outperform a 128-token GPT-4 natural-language summary of the same length, with roughly 2x win/lose ratio.
Limitations
- Experiments limited to Llama models up to 13B due to compute constraints.
- GPT-4 judge favors longer responses, and ICAE responses are short, so reported win rates may understate performance.
- Compressed memory is lossy; quality declines versus original context and at higher compression ratios.
Experimental setup
Datasets: The Pile PwC (Prompt-with-Context, GPT-4 generated)
Benchmarks: PwC test set The Pile (autoencoding and text continuation)
Metrics: BLEU exact match cross entropy loss perplexity win/tie rate latency
Baselines: Alpaca-7b StableLM-tuned-alpha-7b Llama-2-7b-chat Llama-2-13b-chat GPT-4 summary (128 tokens) non-pretrained ICAE
Builds on: Gisting AutoCompressor LoRA
Closest papers by meaning
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.89)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- ARC-Encoder: learning compressed text representations for large language models (NeurIPS 2026 · similarity 0.88)
- Making Text Embedders Few-Shot Learners (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.