Research map / Architectures and efficiency / Attention and transformers
Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments a frozen decoder-only LLaMA with a small trained memory encoder whose output is prepended to the decoder input, to give LMs conversation/knowledge memory at low cost.
Problem
Decoder-only language models are stateless and cannot retain conversation history or external knowledge, and the brute-force fix of enlarging the context window scales quadratically with Transformer cost.
Approach
Build a small memory encoder by mirroring LLaMA's decoder blocks with the causal mask removed, train it while the LLaMA decoder stays frozen, and inject the encoded memory by prepending it to the decoder input rather than via cross-attention.
Key results
- Memory encoder reaches cross-entropy loss 7.73 on the validation set, versus 19.97 with no memory and 21.37 with memory prepended as plain input.
- The encoder injects only 16 prepended tokens, improving over plain prepending while keeping the decoder unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a memory encoder instead of omitting memory | LLaMA decoder · Validation set | cross-entropy loss: 19.97 -> 7.73 | Helped |
| Encode memory with a memory encoder instead of prepending plain memory | LLaMA decoder · Validation set | cross-entropy loss: 21.37 -> 7.73 | Helped |
Surprising findings
- Prepending the plain memory to the input performs worse than providing no memory at all (21.37 vs 19.97 cross-entropy).
Limitations
- Results are preliminary and reported only on a validation set.
- Not yet compared against state-of-the-art memory-augmented methods.
- Separately trained encoder may underperform encoders designed jointly with the model from the start.
Experimental setup
Datasets: Curiosity DREAM SGD Wizard of Wikipedia
Benchmarks: Curiosity DREAM SGD Wizard of Wikipedia
Metrics: cross-entropy loss
Baselines: No inclusion of memory Prepending plain memory to input
Builds on: LLaMA Adapters Transformer
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.90)
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.89)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.89)
- Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.88)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.