Research map / Architectures and efficiency / Attention and transformers

Language Model Memory and Memory Models for Language

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Language model pretraining and induction · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows causal LM embeddings retain little input information while autoencoder embeddings retain nearly all of it, and proposes parallelized memory model architectures trained with a combined causal-plus-copy objective and curriculum-trained frozen autoencoder encoders.

Problem

Hidden-layer embeddings of language models are widely used as memories, but how much input information they actually retain is poorly characterized, and memory models for language have underperformed full-context Transformers.

Approach

Measure input retention by training a frozen-encoder decoder to invert embeddings (entropy ratio and token accuracy), then build parallelizable chunked encoder-decoder memory models whose encoders are trained for input regeneration and whose decoders are trained with copy or combined causal+copy objectives.

Key results

• Causal-trained transformer encoder embeddings reach only about 5% token accuracy (H_r 0.31) at n_ctx=512, versus 76.96% (H_r 0.952) for masked mixer autoencoders.
• Pretrained BERT large, Llama 3.1 1B and Qwen 3 0.6B embeddings recover only about 5-6% of tokens at n_ctx=512 on FineWeb, while BERT large on FineMath reaches 9.6%.
• Combined causal+copy training of a frozen-encoder memory transformer yields copy accuracy of 82.7% and causal accuracy of 40.8%, and improves LongBench code completion from 13.17 to 16.48 over causal training alone.

Surprising findings

• Frozen autoencoder memory embeddings gave no copy-training benefit until a blank-copy curriculum was introduced.
• Masked mixer encoders paired with transformer decoders reduced training efficiency relative to non-memory controls.

Limitations

• Small models (around 1B or smaller) and limited data scales were tested.
• Retention measured only with trained decoders, so underestimation is possible despite a control check.
• Memory-model causal training efficiency decreases when combined objectives or frozen autoencoder embeddings are used.

Experimental setup

Datasets: FineWeb-edu FineWeb FineMath 4+

Benchmarks: IFEval LongBench SWDE ARC-Easy WikiText LAMBADA-OpenAI HellaSwag MMLU

Metrics: entropy ratio H_r token accuracy cross-entropy loss copy accuracy bits per byte

Baselines: Causal full-context decoder Recurrent memory transformer BERT large Qwen 3 0.6B Llama 3.1 1B

Builds on: Transformer Llama 3.2 InfoNCE

Closest papers by meaning

• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.90)
• Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · similarity 0.88)
• DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.88)
• Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.88)
• Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.88)
• EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.87)