Research map / Architectures and efficiency / Attention and transformers
In-Context Language Learning: Architectures and Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces in-context language learning (ICLL) on random regular languages (RegBench), finds Transformers beat RNN/conv/SSM models, attributes this to higher-order n-gram heads, and shows hard-wired n-gram heads improve models on ICLL and SlimPajama perplexity.
Problem
Existing in-context learning model problems (linear regression, associative recall) are too simple to capture free-form generative ICL in LMs; it is unclear which neural sequence architectures can perform in-context learning of new formal languages and how.
Approach
Frame ICL as in-context language learning over random probabilistic finite automata, evaluate many sequence architectures, identify n-gram heads (higher-order induction heads) as the Transformer mechanism, and add static n-gram attention heads (NGH) to other architectures to improve both synthetic and natural language modeling.
Key results
- Transformers significantly outperform RNN, convolutional and SSM models on RegBench across training set sizes; most non-Transformer models underperform n-gram and Baum-Welch baselines except at very large data.
- Adding NGH n-gram layers brings RetNet and GLA to Transformer-level TVD on ICLL (GLA TVD 0.624 to 0.207 with NGH(1,2,3)); GLA reaches 0.946 accuracy vs Transformer 0.926.
- On SlimPajama with 340M-parameter models, NGH layers reduce Transformer (Llama) perplexity from 16.96 to 15.82 (6.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NGH¹, NGH², and NGH³ to RetNet (ablation) | RetNet · RegBench, N=2500 | TVD: 0.392 -> 0.217 | Helped |
| Add NGH¹, NGH², and NGH³ to RetNet (ablation) | RetNet · RegBench, N=2500 | accuracy: 0.800 -> 0.94 | Helped |
| Add NGH¹, NGH², and NGH³ to GLA (ablation) | GLA · RegBench, N=2500 | TVD: 0.624 -> 0.207 | Helped |
| Add NGH¹, NGH², and NGH³ to GLA (ablation) | GLA · RegBench, N=2500 | accuracy: 0.526 -> 0.946 | Helped |
| Add NGH¹, NGH², and NGH³ to RetNet (ablation) | RetNet · SlimPajama, 7B tokens | perplexity: 16.55 -> 15.86 | Helped |
| Add three NGH¹ layers to GLA instead of NGH¹, NGH², and NGH³ (ablation) | GLA · SlimPajama, 7B tokens | perplexity: 15.65 -> 15.54; 15.24 with NGH¹,²,³ | Mixed |
| Add NGH¹, NGH², and NGH³ to GLA (ablation) | GLA · SlimPajama, 7B tokens | perplexity: 15.65 -> 15.24 | Helped |
| Add NGH¹, NGH², and NGH³ to Transformer (Llama) (ablation) | Transformer (Llama) · SlimPajama, 7B tokens | perplexity: 16.96 -> 15.82; +6.7% | Helped |
Surprising findings
- Transformers outperform recurrent models on regular languages even though they are formally unable to recognize some regular languages in general.
- Models with convolutional (LTI/WLTI) forms perform especially poorly on ICLL.
Limitations
- Study limited to regular languages; more expressive languages left to future work.
- Out-of-distribution generalization and learning dynamics not studied.
- Interpretability methods (attention visualization, probing, black-box analysis) have known limitations.
Experimental setup
Datasets: RegBench SlimPajama associative recall
Benchmarks: RegBench associative recall SlimPajama perplexity
Metrics: accuracy total variation distance perplexity probe relative error probe classification accuracy
Baselines: n-gram model with backoff Baum-Welch in-context HMM LSTM RWKV H3 Hyena Linear Transformer Mamba GLA S4
Builds on: induction heads S4 Mamba RetNet GLA
Closest papers by meaning
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.91)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.91)
- From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When (NeurIPS 2024 · similarity 0.90)
- Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization (ICLR 2025 · similarity 0.90)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.90)
- In-Context Learning Learns Label Relationships but Is Not Conventional Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.