Research map / Architectures and efficiency / Attention and transformers
Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains thirteen causal sequence architectures (RNNs, CNNs, SSM-inspired models, RetNet, RWKV, transformers) from scratch on synthetic ICL tasks to test whether attention is necessary for in-context learning.
Problem
Whether attention mechanisms are necessary for in-context learning to emerge, and how different model architectures compare in ICL ability, statistical efficiency, consistency, and predisposition toward memorization versus ICL.
Approach
A controlled, from-scratch comparison of thirteen causal architectures across synthetic ICL tasks (associative recall, linear regression, multiclass classification, bursty image classification, language modeling) rather than pretrained LLMs, showing that all considered architectures can in-context learn, not only transformers.
Key results
- All thirteen considered architectures can perform in-context learning on the synthetic tasks, extending evidence that ICL is not exclusive to transformers.
- Mamba, transformers, Hyena and RetNet are consistently strong performers across tasks, though no single architecture is consistent across all tasks when extrapolating to many more in-context examples.
- ICL emerges in all evaluated architectures except LightConv in the bursty image classification setting; RetNet, S4, CNNs and RNNs strongly favor memorization at P(bursty)=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mamba instead of other evaluated architectures | Mamba · Associative recall, 63 examples | accuracy | Helped |
| Use sinusoidal or no positional embeddings instead of other positional embedding options (ablation) | Transformers · Associative recall, prompt lengths beyond training | accuracy | Helped |
| Remove positional embeddings (ablation) | Transformers · Linear regression, prompt lengths beyond training | mean squared error | Helped |
| Remove positional embeddings (ablation) | Transformers · Multiclass classification | accuracy | Helped |
| Use ALiBi positional embeddings instead of other positional embedding options (ablation) | Transformers · TinyStories language modeling | validation loss | Helped |
| Remove positional embeddings (ablation) | Transformers · TinyStories language modeling | validation loss | Hurt |
| Concatenate rather than sum input and output embeddings (ablation) | Transformers · Linear regression and image classification | performance | Helped |
| Use a single token for each in-context example pair rather than two tokens (ablation) | Transformers · Associative recall and multiclass classification | performance | No clear effect |
| Remove positional embeddings (ablation) | Transformers · Permutation-invariant in-context tasks | performance | No clear effect |
| Use an encoder-only rather than decoder-only attention mask (ablation) | Transformers · Linear regression, difficult settings (d=20, 30) | performance | Helped |
| Use an encoder-only rather than decoder-only attention mask (ablation) | Transformers · Permutation-invariant in-context tasks other than difficult linear regression | performance | No clear effect |
| Increase Gaussian noise in the input-output examples | All evaluated architectures · Noisy linear regression | mean squared error | Hurt |
Surprising findings
- RetNet, a top performer on synthetic tasks, strongly resisted developing ICL in the bursty image setting, emerging in only 2 of 108 runs.
- Llama2-7B and GPT2 differences: GPT2 did not show the ICL predisposition seen in Llama2, despite similar design.
Limitations
- Most models are small (about 33M parameters for language modeling, around 500K for synthetic tasks), so results may not transfer to large LLMs.
- RNN results were inconsistent across hyperparameters, with ICL in some runs attributed to luck.
- Performance degrades when extrapolating to many more in-context examples than seen during training for most architectures.
Experimental setup
Datasets: TinyStories Omniglot synthetic associative recall synthetic linear regression synthetic multiclass classification
Benchmarks: synthetic associative recall synthetic linear regression synthetic multiclass classification bursty Omniglot image classification TinyStories language modeling simple natural-language happy/sad ICL task
Metrics: test accuracy mean squared error validation loss ICL score
Baselines: ridge regression logistic regression LSTM GRU RNN
Builds on: Chan et al. bursty data ICL Akyürek et al. linear regression ICL Mamba Hyena RetNet
Closest papers by meaning
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.90)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.90)
- Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting (ICLR 2025 · similarity 0.90)
- Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (NeurIPS 2025 · similarity 0.90)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.89)
- The mechanistic basis of data dependence and abrupt learning in an in-context classification task (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.