Research map / Architectures and efficiency / Attention and transformers
Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows LLM fact retrieval can be hijacked by prepending semantically neutral-but-related context, and models this as associative memory via a latent concept association task solved by a one-layer transformer whose value matrix acts as associative memory.
Problem
Fact retrieval in LLMs is not robust: adding factual but misleading context (e.g., "not in Chicago") can flip outputs, so the mechanism of memory recall in transformers needs explanation.
Approach
Frames LLM fact recall as associative memory where context tokens act as latent-concept clues; proposes a latent concept association task (Hamming-distance-based latent semantic similarity) and proves a one-layer transformer can solve it by using self-attention to aggregate context and the value matrix as associative memory.
Key results
- Prepending 'Do not think of {target_false}' raises efficacy score on CounterFact for GPT-2, Gemma and LLaMA-2-7B, with more prepends increasing it further.
- On the synthetic task with L=64, freezing W_V to identity markedly lowers accuracy compared with training W_V.
- Trained value matrices are replaceable by theory-constructed ones with little accuracy loss and share close low-rank approximations with them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prepend “Do not think of {target_false}” more times | GPT-2, Gemma, and LLaMA-2-7B · CounterFact | Efficacy Score | Helped |
| prepend factually correct sentences associated with the relation and false target | GPT-2, Gemma, and LLaMA-2-7B · CounterFact, relation-ID hijacking | Efficacy Score | Helped |
| prepend sentences containing the true target instead of the false target | GPT-2, Gemma, and LLaMA-2-7B · CounterFact, reverse context hijacking | Efficacy Score | Hurt |
| train the value matrix instead of fixing it to the identity (ablation) | single-layer transformer · Latent concept association, context length 64 and 128 | accuracy | Helped |
| replace the trained value matrix with the constructed associative-memory value matrix (ablation) | single-layer transformer · Latent concept association | accuracy | No clear effect |
| train embeddings instead of freezing them (ablation) | single-layer transformer with embedding dimension smaller than vocabulary size · Latent concept association, underparameterized regime | accuracy | Helped |
| train embeddings instead of freezing them (ablation) | single-layer transformer with embedding dimension larger than vocabulary size · Latent concept association, overparameterized regime | accuracy | No clear effect |
| use self-attention to select tokens within the same cluster | single-layer transformer · Latent concept association with cluster-structured contexts | attention to same-cluster tokens | Helped |
Surprising findings
- Hijacking works even with semantically close words only (e.g., 'France' for false target 'French'), suggesting more than token copying.
- Trained embeddings with identity W_V show inner products roughly linear in Hamming distance, matching the theory.
Limitations
- Context hijacking experiments only on open-source models, not commercial models like GPT-4.
- Theory covers only single-layer transformers without residual connections or normalization.
- Latent concept association simplifies away the sequential nature of language.
Experimental setup
Datasets: CounterFact synthetic latent concept association data
Benchmarks: CounterFact latent concept association synthetic task
Metrics: efficacy score accuracy error rate smallest principal angle
Baselines: randomly constructed value matrix identity value matrix (frozen W_V) Gaussian-initialized value matrix
Builds on: Hopfield networks CounterFact transformer attention as associative memory
Closest papers by meaning
- Understanding Factual Recall in Transformers via Associative Memories (ICLR 2025 · similarity 0.89)
- Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.89)
- Geometric Factual Recall in Transformers (NeurIPS 2026 · similarity 0.88)
- Latent Concept Disentanglement in Transformer-based Language Models (ICLR 2026 · similarity 0.88)
- Scaling Laws for Associative Memories (ICLR 2024 · similarity 0.88)
- Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.