Research map / Continual learning / Continual fine-tuning of LLMs

CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models

AAAI 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CMT, a frozen-LLM continual knowledge learning method that compresses new documents into a latent memory bank and aggregates them via cross-attention with memory-aware objective, self-matching, and top-k aggregation.

Problem

Continual knowledge learning for LLMs on streaming documents: adapting to new knowledge while retaining old knowledge, without the cost of retraining and without catastrophic forgetting.

Approach

Freeze the LLM and learn a document compressor that encodes each new document into soft memory tokens stored in a bank; a query-conditioned cross-attention aggregator maps the selected memories into cached key-value pairs for the LLM, with RoPE-based relative positions, a memory-aware contrastive logit objective, query-memory self-matching, and top-k memory filtering.

Key results

• On StreamingQA with Llama-2-7b, CMT reaches 18.36 EM and 25.98 F1, versus 14.29 EM / 21.79 F1 for MAC (the best baseline).
• With DistilGPT2 on StreamingQA, CMT achieves 6.43 EM / 12.32 F1 versus MAC's 5.59 EM / 10.18 F1.
• Ablation on StreamingQA shows removing top-k aggregation drops EM from 18.36 to 16.43 and F1 from 25.98 to 20.13.

Measured effects reported in the paper

Surprising findings

• Removing the memory-aware objective slightly increases StreamingQA EM (18.54 vs 18.36) while lowering F1, and removing self-matching gives the best ArchivalQA EM/F1 among variants, so the components are not uniformly beneficial.

Limitations

• Largest model evaluated is 7B; 70B not tested due to compute.
• Question encoder is not separate; query is compressed by the document compressor, with other applications left for future work.
• Auxiliary auto-encoding pretraining gave no significant gains per the authors, and the method's knowledge retention is only evaluated on a subset of 200 documents.

Experimental setup

Datasets: StreamingQA SQuAD ArchivalQA English Wikipedia

Benchmarks: StreamingQA SQuAD ArchivalQA

Metrics: Exact Match F1 knowledge retention F1 ratio relative F1 under irrelevant document proportion

Baselines: Uniform Salient Spans CaMeLS MAC

Builds on: Memory of Amortized Contexts (MAC) gisting AutoCompressor ICAE RoPE

Closest papers by meaning

• Online Adaptation of Language Models with a Memory of Amortized Contexts (NeurIPS 2024 · similarity 0.90)
• Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.89)
• Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.88)
• Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)
• Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.88)