Research map / Architectures and efficiency / Attention and transformers
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train Transformer from scratch on random linear-function prompts | GPT-2 style Transformer · Linear functions d=20 | squared error: error 0.0006 at 40 examples vs least squares 0 | Helped |
| scale test prompt inputs by 3 (distribution shift) | linear-function Transformer · Linear, k=40 | squared error: 0.0006 -> 0.58 | Hurt |
| scale test weight vector (1/3, 1/2, 2, 3) | linear-function Transformer · Linear, k=40 | squared error: 0.0012, 0.0008, 0.0016, 0.0278 | No clear effect |
| place in-context inputs and query in different orthants | linear-function Transformer · Linear, k=40 | squared error: 0.0006 -> 0.004 | No clear effect |
| skewed covariance for test inputs | linear-function Transformer · Linear, skewed covariance | squared error | Hurt |
| add label noise to in-context examples (trained noiseless) | linear-function Transformer · Noisy linear regression | squared error | No clear effect |
| curriculum learning (growing subspace and prompt length) (ablation) | Transformer training · Linear d=50 | training speed / loss | Helped |
| curriculum learning (ablation) | Transformer training · Linear d=20 | accuracy and robustness | No clear effect |
| increase model capacity (ablation) | Transformer · Linear functions, 2d examples | squared error | Helped |
| restrict training to n_w = 10k distinct weight vectors (vs 32M) (ablation) | Transformer training · Linear d=20 | squared error | No clear effect |
| restrict training to n_p=100k prompts or n_w=1k functions (ablation) | Transformer training · Linear d=20 | squared error | Hurt |
| use Transformer instead of greedy/XGBoost for decision-tree prompts | decision tree in-context learning · Decision trees depth 4, k=100 | squared error: 0.62 -> 0.12 | Helped |
Closest papers by meaning
- Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions (ICLR 2024 · similarity 0.93)
- In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.93)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.92)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.92)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.91)
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.