Research map / Large language models / Language, safety and interpretability
Position: Do pretrained Transformers Learn In-Context by Gradient Descent?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that the ICL≈GD equivalence claims do not hold for real pretrained LLMs, using order-sensitivity analysis and an empirical comparison of ICL vs fine-tuned GD on LLaMA-7B.
Problem
Whether in-context learning in pretrained transformers is algorithmically equivalent to gradient descent, given that prior theoretical and empirical support relies on ICL-objective training, hand-constructed weights, and assumptions that differ from real LLMs.
Approach
Critiques the assumptions of prior ICL≈GD work (ICL-objective training, hand-constructed sparse weights, single-parameter-set analysis), proves that algorithmic equivalence implies matching order sensitivity, and tests this on natural pretrained LLMs with distributional metrics (top-K token overlap, OCS, accuracy) comparing ICL against GD and sub-model GD.
Key results
- On AGNews with 8 demos, GD fine-tuned on LLaMA-7B reaches accuracy around 0.42 for 8 demos versus 0.69 with 512 demos, showing GD needs many more samples than ICL to match performance.
- ICL shows much higher order sensitivity than GD, SGD, and Adam on LLaMA-7B (AGNews, 8 demos, 10 orders), and GD sensitivity decreases as training progresses.
- Token overlap and OCS between ICL and GD outputs are consistently lower than between ICL runs with different demonstration orders, indicating different output distributions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase GD demonstrations from 8 to 512 | GD on LLaMa-7B · AGNews | accuracy: 0.42 -> 0.69 | Helped |
| Increase GD demonstrations from 8 to 512 | GD on LLaMa-7B · CB | accuracy: 0.39 -> 0.72 | Helped |
| Increase GD demonstrations from 8 to 512 | GD on LLaMa-7B · SST-2 | accuracy: 0.49 -> 0.75 | Helped |
| Increase GD demonstrations from 8 to 512 | GD on LLaMa-7B · RTE | accuracy: 0.36 -> 0.65 | Helped |
| Use ICL instead of GD | LLaMa-7B · AGNews, CB, SST-2, and RTE | accuracy | Helped |
| Increase ICL demonstrations | LLaMa-7B · AGNews, CB, SST-2, and RTE | accuracy | Helped |
| Compare ICL with GD or an implicit-submodel GD variant | LLaMa-7B · Evaluated datasets and demonstration settings | token overlap | Hurt |
| Compare ICL with GD or an implicit-submodel GD variant | LLaMa-7B · Evaluated datasets and demonstration settings | OCS | Hurt |
| Present demonstrations in different orders | ICL versus GD-based algorithms · AGNews | order sensitivity | Mixed |
| Continue pretraining across checkpoints | GPT-J · AGNews, SST-2, CB, and RTE | ICL capacity | No clear effect |
| Fine-tune W_V in a single deep layer rather than the full model (ablation) | Implicit-submodel GD on LLaMa-7B · ICL versus implicit-submodel GD | output-distribution alignment | Hurt |
| Fine-tune W_V in a single middle layer rather than the full model (ablation) | Implicit-submodel GD on LLaMa-7B · ICL versus implicit-submodel GD | output-distribution alignment | Hurt |
Surprising findings
- GD fine-tuning with 8 demos performs worse than ICL, and with 512 demos it roughly matches ICL performance with fewer demos, highlighting an empirical discrepancy.
- At N=1, one GD variant achieves almost the same accuracy as ICL on some datasets, yet output distributions still differ markedly.
Limitations
- Exhaustive search over implicit sub-models is computationally infeasible, so sub-model GD uses only intuitive W_V subsets of single layers.
- Experiments are limited by LLaMA's context window, so GD with many demos cannot be directly compared with ICL.
- The paper does not provide alternative functional explanations of how ICL works.
Experimental setup
Datasets: AGNews CB SST-2 RTE
Benchmarks: AGNews CB SST-2 RTE
Metrics: accuracy over full vocabulary token overlap (top-10) overlap cosine similarity (OCS) order sensitivity (std of output probabilities)
Baselines: Fine-tuning with GD SGD Adam GD sub-model (W_V of a middle or deep layer)
Builds on: Akyürek et al. ICL as gradient descent von Oswald et al. transformers learn in-context by gradient descent Dai et al. GPT as meta-optimizers Garg et al. what can transformers learn in-context
Closest papers by meaning
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.92)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.91)
- Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.91)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.91)
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.91)
- Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.