Research map / Large language models / Retrieval and knowledge

Frozen Memory Is Not Enough: Rethinking External Memory as Extraction

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Retrieval-augmented reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies whether an Engram-style hashed external memory trained on one LLM can be frozen and reused by a different target LLM through a lightweight learned reader, with canonicalized addressing, reporting near same-model QA performance with a dual-layer, four-branch reader.

Problem

External hashed memory (Engram) entangles stored knowledge with the model that trained it; it is unclear whether a frozen memory table remains useful when attached to a different backbone, and what part of the system (memory or target-side reader) determines transfer success.

Approach

Freeze a source-trained Engram table, standardize addressing across tokenizers via canonicalized word-boundary n-gram hashing, and train only a target-side multi-branch, multi-layer reader that injects retrieved values into the target residual stream; reader design is shown to be the dominant factor in transfer quality.

Key results

• Dual-layer four-branch reader (LLaMA-2-7B to Mistral-7B, injection at layers 2 and 10, R=4) reaches 38.5 average QA accuracy vs 32.1 no-memory baseline; with 30/30 budget the Mistral self-transfer reaches 38.8.
• Frozen transferred memory with a compatible reader gives ~38.3 average QA without target-side training.
• Transferred frozen memory improves perplexity over no-memory baseline in all nine 3x3 source-target pairs, with relative gains from 1.6% to 15.7%.

Surprising findings

• Random memory achieves the same test perplexity as transferred memory (8.7) yet scores substantially lower on downstream QA.
• Transferred memory is mildly harmful on TruthfulQA at every target scale, and gains are task-dependent rather than universal.

Limitations

• Gains are task-dependent and can be negative (TruthfulQA), so transfer is not universal.
• Transfer depends on a compatible reader interface; direct reuse without target-side training needs matching target residual interface.
• Canonicalization is word-boundary based and degenerates for scripts without whitespace segmentation; the character-level variant is only partially covered.

Experimental setup

Datasets: WikiText-103 FineWeb-Edu Nemotron-CC Wikipedia-2021 (Dec 2021) LAMBADA C4

Benchmarks: NQ WebQA TriviaQA TruthfulQA HotpotQA RTE BoolQ OpenBookQA SciQ RACE WikiText-103 perplexity LAMBADA

Metrics: accuracy F1 test perplexity relative PPL reduction length-normalized gold-answer log-probability 95% CI

Baselines: RAG kNN-LM CPT LoRA MLP Memory FFN-only parameter-matched control random memory permuted keys

Builds on: Engram MLP Memory KNN-LM LoRA

Closest papers by meaning

• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.89)
• Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (ICLR 2024 · similarity 0.87)
• Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.87)
• Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.87)
• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.87)
• M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.87)