Research map / Optimization / Implicit bias and generalization
Transformers Learn In-Context by Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a single linear self-attention layer on linear regression tasks | Linear self-attention Transformer · Linear regression | prediction loss | No clear effect |
| Stack self-attention layers to perform iterative curvature correction | Deep linear self-attention Transformer · Linear regression | prediction loss | Helped |
| Add an MLP before self-attention to learn on deep representations | Transformer · Sine-wave regression | prediction loss | Helped |
| Use a single softmax self-attention head instead of linear self-attention (ablation) | Single-layer self-attention Transformer · Linear regression | prediction loss | Hurt |
| Increase softmax self-attention from one head to two heads (ablation) | Softmax self-attention Transformer · Linear regression | prediction loss | Helped |
| Use two softmax heads instead of one to approximate the gradient-descent update (ablation) | Softmax self-attention Transformer · Linear regression | alignment with gradient descent | Helped |
| Add LayerNorm before self-attention (ablation) | Self-attention Transformer · Linear regression | prediction loss | Hurt |
| Use alternating input and target tokens rather than concatenated input-target tokens (ablation) | Two-layer self-attention Transformer · Linear regression | prediction loss | Mixed |
| Use linear self-attention in the first layer for alternating input and target tokens (ablation) | Two-layer self-attention Transformer · Linear regression | training success | Hurt |
| Repeat a trained single-layer update without dampening (ablation) | Single linear self-attention layer and gradient descent · Linear regression, repeated updates | prediction loss | Hurt |
| Apply dampening with λ = 0.75 to repeated updates (ablation) | Single linear self-attention layer and gradient descent · Linear regression, repeated updates | prediction loss | Helped |
Closest papers by meaning
- In-context Convergence of Transformers (ICML 2024 · similarity 0.93)
- How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression (NeurIPS 2024 · similarity 0.92)
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression (NeurIPS 2024 · similarity 0.91)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.91)
- In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.91)
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.