Research map / Optimization / Implicit bias and generalization
In-context learning as gradient descent
Shows transformers implement gradient-descent-like algorithms for in-context linear regression.
41 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Implicit bias and generalization, keyword cluster “relu · width · kernel”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
17 papers in 2023–24 and 16 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.2% to 6.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Alternating Back-Propagation for Generator Network (AAAI 2017 · 70 citations)
- 2018 · Facial Landmarks Detection by Self-Iterative Regression Based Landmarks-Attention Network (AAAI 2018 · 7 citations)
- 2019 · Learning concise representations for regression by evolving networks of trees (ICLR 2019 · 15 citations)
- 2020 · Provably Efficient Neural Estimation of Structural Equation Models: An Adversarial Approach (NeurIPS 2020 · 11 citations)
- 2022 · An Initial Alignment between Neural Network and Target is Needed for Gradient Descent to Learn (ICML 2022 · 2 citations)
- 2023 · Transformers Learn In-Context by Gradient Descent (ICML 2023 · 88 citations)
- 2024 · Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization (ICLR 2024 · 3 citations)
- 2025 · Training Dynamics of In-Context Learning in Linear Attention (ICML 2025 · 0 citations)
- 2026 · Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent (ICML 2026 · 0 citations)
Earliest papers here
- Alternating Back-Propagation for Generator Network (AAAI 2017 · 70 citations)
- Facial Landmarks Detection by Self-Iterative Regression Based Landmarks-Attention Network (AAAI 2018 · 7 citations)
- Learning concise representations for regression by evolving networks of trees (ICLR 2019 · 15 citations)
What has been tried
28 papers here report 216 measured changes (a component added, removed or swapped, compared with a baseline): helped 60%, no clear effect 15%, hurt 20%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove L2 regularizer from shared-layer transformers | Linear regression (LR) · test query MSE | PrefixLM-SL: 8.6e-3 -> 6.7e-3; CausalLM-SL: 1.9e-1 -> 5.0e-2 | Mixed | CausalLM is not optimal for in-context learning (ICLR 2024 · 2 citations) |
| Use finite rather than infinite samples per training task | New-task test tasks, mtest=20, 1-GD · excess risk | 1.78 ± 0.02 -> 0.81 ± 0.01 | Helped | First-order ANIL provably learns representations despite overparametrisation (ICLR 2024 · 1 citation) |
| Remove gate regularization and the gate–weight timescale difference | Alternating two-task curriculum · task-switch adaptation speed | Hurt | Flexible task abstractions emerge in linear networks with fast and bounded units (NeurIPS 2024 · 1 citation) | |
| Increase hidden size to 32 or 64 | In-context linear regression · ability to mimic OLS | Helped | Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression (NeurIPS 2024 · 1 citation) | |
| add the MLP component | Linear regression, β* = (0,0,...,0)^⊤ · loss | 0.024 -> 0.003 | Helped | In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization (NeurIPS 2024 · 1 citation) |
| Use rank 2 for separate key and query weights | In-context linear regression, D=8 · plateau pattern | Mixed | Training Dynamics of In-Context Learning in Linear Attention (ICML 2025 · 0 citations) | |
| add the input-dependent selection mechanism to S4 | In-context linear regression · ICL capability | Helped | Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression (NeurIPS 2025 · 0 citations) | |
| Remove the loss term corresponding to R_* | Modular addition · hidden nodes per frequency required for a global solution | 6 to 3 | Helped | Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets (NeurIPS 2025 · 0 citations) |
| Increase context length to L = 10 | Synthetic data, context length L = 10 · attention concentration on noise patches | No clear effect | On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · 0 citations) | |
| Restrict sparse-vector support to the first 10 entries | In-context sparse recovery, support restricted to first 10 entries · label prediction loss | Helped | On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery (ICLR 2025 · 0 citations) | |
| Mask non-diagonal gradients of W^KQ_32 and W^PV_12 during training | AR process, fixed all-ones initial token · vanilla gradient-descent mesa-optimization | ã b̃ = 1/(1 + (d-1)/(T-2) ∑ (t=2)^{T-1} 1/(t-1)) | Helped | On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability (NeurIPS 2024 · 0 citations) |
| Constrain attention matrices to be diagonal | Uniform σ_τ∼U(0,5) · adjusted evaluation loss | 0.047 -> 0.047 | No clear effect | Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · 0 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 33 papers; from OpenAlex and the papers' affiliations).
- Google · 5 papers
- Massachusetts Institute of Technology · 4 papers
- Meta · 2 papers
- University of Southern California · 2 papers
- University College London · 2 papers
- University of California, Berkeley · 2 papers
- Boston University · 2 papers
- RIKEN AIP · 2 papers
- Tsinghua University · 2 papers
- Shandong University · 2 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Shallow ReLU network convergence (132 papers)
- Neural tangent kernel analysis (114 papers)
- Gradient flow and mean-field dynamics (77 papers)
- Low-rank and single-index recovery (70 papers)
Elsewhere in the topic
- Edge-of-stability dynamics (116 papers)
- SGD noise and flat minima (112 papers)
- Stability-based generalization analysis (88 papers)
- Explicit and implicit regularization interplay (63 papers)
- Implicit regularization of SGD (55 papers)
- Implicit bias in linear networks (53 papers)
- Sharpness-aware minimization theory (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).