Research map / Large language models / Retrieval and knowledge
Online Adaptation of Language Models with a Memory of Amortized Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Memory of Amortized Contexts (MAC), which compresses each incoming document into PEFT (P-tuning v2) modulations via a meta-learned amortization hypernetwork, stores them in a memory bank, and aggregates them with a question-conditioned cross-attention network to adapt a frozen LM without test-time gradient updates.
Problem
Online adaptation of language models to a growing stream of new documents is costly and forgetful when done by gradient-based finetuning, while retrieval-augmented approaches scale poorly in inference cost and can be misled by counterfactual documents.
Approach
Amortize each document into a compact modulation with a single encoder-decoder forward pass (no labels or gradients at test time), store modulations in a memory bank, and learn a permutation-invariant cross-attention aggregator conditioned on the question to produce one soft-prompt modulation for a frozen LM, with backpropagation dropout and hierarchical aggregation for memory efficiency.
Key results
- On LLaMA-2 7B, MAC raises StreamingQA F1 from 18.97 (Salient Spans) to 21.79 and SQuAD-Seq F1 from 18.66 to 21.14.
- MAC retains 96.2% of initial F1 after 1,400 additional documents versus 70.8% for CaMeLS on GPT2-XL.
- Combining MAC with BM25 top-5 retrieval raises ArchivalQA-Seq F1 from 71.83 to 74.89 on LLaMA-2 7B.
- Compared with CaMeLS, MAC reduces single-document adaptation memory by 68.0% and adaptation time from 28.58 to 2.5 minutes under the same memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Salient Spans online finetuning with MAC | LLaMA-2 7B · StreamingQA | F1: 18.97 -> 21.79 | Helped |
| Replace Salient Spans online finetuning with MAC | LLaMA-2 7B · SQuAD-Seq | F1: 18.66 -> 21.14 | Helped |
| Replace Salient Spans online finetuning with MAC | LLaMA-2 7B · ArchivalQA-Seq | F1: 22.75 -> 23.90 | Helped |
| Add MAC to BM25 retrieval | LLaMA-2 7B · ArchivalQA-Seq Top-5 | F1: 71.83 -> 74.89 | Helped |
| Use backpropagation dropout to allow a larger training context batch (K=4 rather than K=1) (ablation) | MAC on LLaMA-2 7B · StreamingQA | F1: 12.43 -> 21.79 | Helped |
| Replace P-tuning v2 modulation with LoRA (ablation) | MAC on GPT2-XL · StreamingQA | F1: 15.38 -> 15.15 | Hurt |
| Use an encoder-only amortization network instead of encoder-decoder T5 (ablation) | MAC on GPT2-XL · StreamingQA | F1: 15.38 -> 15.01 | Hurt |
| Use a decoder-only amortization network instead of encoder-decoder T5 (ablation) | MAC on GPT2-XL · StreamingQA | F1: 15.38 -> 14.87 | Hurt |
| Remove 4-bit training quantization and use 16-bit bfloat16 (ablation) | MAC on LLaMA-2 · ArchivalQA | F1: 23.90 -> 26.25 | Helped |
| Use hierarchical modulation aggregation with subgroup size M=16 (ablation) | MAC · ArchivalQA-Seq | GPU memory usage: 65.6% reduction | Helped |
| Use hierarchical modulation aggregation with subgroup size M=16 (ablation) | MAC · ArchivalQA-Seq | accuracy retention: 93.2% | Hurt |
| Use MAC rather than CaMeLS during continued adaptation | GPT2-XL · StreamingQA, initial 200 documents after adapting 1,400 additional documents | F1 retention rate: 70.8% -> 96.2% | Helped |
Surprising findings
- Training LLaMA-2 without 4-bit quantization of the aggregation network raised ArchivalQA-Seq F1 from 23.90 to 26.25, suggesting quantization limited the reported gains on large models.
- Averaging nearest-neighbor modulations preserved performance under a memory bank size cap better than random pruning or random averaging.
Limitations
- Memory bank grows linearly with the number of adapted documents, which the authors identify as a key limitation.
- Large-model results were limited by 4-bit quantization of training, and CaMeLS ran out of memory on LLaMA-2 so was compared via a proxy model.
Experimental setup
Datasets: StreamingQA SQuAD ArchivalQA
Benchmarks: StreamingQA SQuAD-Seq ArchivalQA-Seq
Metrics: exact match F1 peak GPU memory adaptation time perplexity F1 retention rate
Baselines: Uniform finetuning Salient Spans CaMeLS BM25 Contriever DPR CCM
Builds on: CaMeLS P-tuning v2 T5 Retrieval-augmented generation Hypernetworks
Closest papers by meaning
- CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.90)
- Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.89)
- Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models (NeurIPS 2024 · similarity 0.87)
- Doc-to-LoRA: Learning to Instantly Internalize Contexts (ICML 2026 · similarity 0.87)
- Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.