Research map / Learning theory and algorithms / Kernels and regression theory
What learning algorithm is in-context learning? Investigations with linear models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a sufficiently deep transformer (8 or more layers) rather than a very shallow one | Transformer ICL on noiseless linear regression · Linear regression, underdetermined region | SPD to OLS | Helped |
| Increase transformer depth from 1 layer to 2–4 layers | Transformer ICL · Noiseless linear regression | SPD to ridge regression | Helped |
| Increase transformer depth from 1 layer to 2–4 layers | Transformer ICL · Noiseless linear regression | SPD to one-step gradient descent | Hurt |
| Increase transformer depth to 8 or more layers | Transformer ICL · Noiseless linear regression | SPD to OLS | Helped |
| Increase hidden size to at least 16 for 8D problems or 32 for 16D problems | Transformer ICL · Noiseless linear regression, d=8 and d=16 | SPD to ridge regression | Helped |
| Use the Bayesian ridge regularization parameter σ²/τ² for noisy labels | Transformer ICL compared with ridge predictors · Noisy linear regression | SPD | Helped |
| Use OLS as the reference predictor instead of one-step gradient descent, SGD, or nearest neighbors | Transformer ICL · Noiseless linear regression, d=8 | SPD and ILWD: less than 0.01 | Helped |
| Decode moment vectors and OLS weight vectors with a 2-layer MLP probe instead of a linear probe | Probes of Transformer ICL hidden states · Linear regression, prefixes of the context | Probe prediction error | Helped |
| Decode intermediate quantities from deeper hidden layers rather than the input layer | Probes of Transformer ICL hidden states · Linear regression, prefixes of the context | Probe prediction error | Helped |
Closest papers by meaning
- In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.90)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.89)
- How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression (NeurIPS 2024 · similarity 0.89)
- In-context Learning in Presence of Spurious Correlations (NeurIPS 2026 · similarity 0.89)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.89)
- Does learning the right latent variables necessarily improve in-context learning? (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.