Research map / Architectures and efficiency / Attention and transformers

In-Context Learning as Conditioned Associative Memory Retrieval

ICML 2025 · Architectures and efficiency › Attention and transformers · approach: Training dynamics of in-context learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Interprets in-context learning in one-layer attention as conditional memory retrieval in a modern Hopfield model, showing prompts reshape key/value matrices, linking this to Bayesian model averaging ICL theory and recovering an epsilon-stability generalization bound, with GPT-2 and GPT-J experiments.

Problem

Understanding the mechanism by which in-context examples modulate a transformer's predictions without parameter updates, and connecting it to existing Bayesian and stability-based theories of ICL.

Approach

Treats in-context examples as queries to a modern Hopfield model so that the prompt induces an effective linear reshaping of the pre-trained key and value matrices (via ghost examples and pseudoinverses), and embeds this into the Bayesian model averaging view of ICL.

Key results

• Under covariance shift, R-squared with 15 in-context examples drops from 0.9366 (N(-2,I)) to 0.4043 (0.7N(-2,I)+0.3N(2,I)) in linear regression.
• With response accuracy 80% versus 100%, R-squared at 30 examples falls from 0.9998 to 0.8607.
• Ghost example construction via prompt tuning achieves average MSE of 0.0003 and 0.001 across prompt lengths up to 75.

Measured effects reported in the paper

Surprising findings

• Performance decreases when prompt length exceeds the pre-training length of 50, attributed to absolute positional encodings.
• The proxy softmax attention used in the theory is not the actual LLM attention, as acknowledged by the authors.

Limitations

• Analysis restricted to one-layer attention and a proxy softmax attention rather than the actual LLM attention.
• Only single next-token prediction is analyzed; multi-token extension is only sketched.
• Memory reshaping is not verified directly in the experiments; ghost-example-based reshaping is left to future work.

Experimental setup

Datasets: synthetic Gaussian mixture linear regression data TweetEval hate speech detection

Benchmarks: synthetic linear regression decision tree 2-layer neural network TweetEval hate speech detection

Metrics: R-squared mean squared error

Baselines: Least Squares Decision Tree 2-Layer NN

Builds on: Zhang et al. 2025 Bayesian model averaging ICL Ramsauer et al. 2020 modern Hopfield networks Li et al. 2023 ICL algorithm learning stability Garg et al. 2022 ICL of function classes

Closest papers by meaning

• In-Context Denoising with One-Layer Transformers: Connections between Attention and Associative Memory Retrieval (ICML 2025 · similarity 0.90)
• Training Dynamics of In-Context Learning in Linear Attention (ICML 2025 · similarity 0.89)
• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.89)
• Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.88)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.88)
• The mechanistic basis of data dependence and abrupt learning in an in-context classification task (ICLR 2024 · similarity 0.88)