Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
In-Context Learning through the Bayesian Prism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically tests whether transformers trained in-context on single and mixed function families (hierarchical meta-ICL) behave like Bayesian posterior-mean predictors, and studies simplicity bias, OOD generalization, and forgetting.
Problem
Understanding the inductive biases of in-context learning in transformers trained on meta-learning-style function-class setups, especially when training data is a mixture of multiple function families and when generalization to unseen function classes occurs.
Approach
Generalizes meta-ICL to hierarchical meta-ICL (HMICL) where prompts come from a mixture of function classes, and argues via the Bayesian view that transformers approximate the posterior predictive mean (PME), so task recognition and solving are intertwined; tests this across many linear and nonlinear families against exact or numerical Bayesian predictors and strong baselines.
Key results
- Transformers trained on a 2-component Gaussian mixture of linear regression tasks match the exact PME (GMM) error and implied weights across prompt lengths.
- In the multi-task degree-2 monomial setting, OOD generalization improves with task diversity K, with ID and OOD error converging at large K.
- In the noisy linear regression setting, models first generalize to the full Gaussian task distribution and later forget it during pretraining for task diversities 2^7 to 2^11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on a two-component Gaussian mixture rather than one Gaussian task prior | Transformer in-context learning · GMM dense regression | MSE | Helped |
| Increase the number K of pretraining feature-set classes | Degree-2 monomial transformer · OOD feature sets | error | Helped |
| Increase the number K of pretraining feature-set classes | Degree-2 monomial transformer · ID feature sets | error | Hurt |
| Use a mixture of Fourier classes with different maximum frequencies instead of a single class | Fourier-series transformer · Fourier series, short prompts | recovered frequency coefficients | Mixed |
| Bias the pretraining distribution toward lower frequencies | Fourier-series transformer · Fourier series, short prompts | recovered frequency coefficients | Helped |
| Remove positional encodings (ablation) | Transformer · Dense regression, length generalization | error | Helped |
| Remove positional encodings (ablation) | Transformer · Sparse regression, length generalization | error | Helped |
| Increase K task diversity from the Gaussian forgetting region to very high diversity | Noisy linear regression transformer · OOD Gaussian tasks | loss | Mixed |
| Use curriculum learning | Transformer training · In-context regression tasks | convergence speed | Helped |
| Train on the artificial repeated-coordinate weight-vector task instead of sign-vector regression | Transformer · Artificial task regression | error | Helped |
Surprising findings
- Transformers trained on mixtures show a low-frequency bias at short prompt lengths that reflects the pretraining frequency distribution, and an intermediate-diversity regime where models generalize early in pretraining and then forget.
- Removing positional encodings improved length generalization for dense and sparse linear regression in this setup.
Limitations
- Focuses on continuous regression function classes rather than real-world LLMs or discrete language tasks
- Exact Bayesian predictor is intractable for several families, so proxies (MCMC, strong baselines, full-distribution transformer) are used
- Deviations from Bayesian behavior (OOD generalization, forgetting) are reported but not fully explained
Experimental setup
Datasets: synthetic function-class prompts (dense, sparse, sign-vector, low-rank regression; GMM; Fourier series; degree-2 monomia
Benchmarks: synthetic function-class regression tasks
Metrics: squared error vs prompt length MSE of probed implied weights Fourier coefficient magnitude
Baselines: OLS Ridge regression Lasso L-infinity norm minimization nuclear norm minimization NUTS MCMC PME dMMSE
Builds on: Garg et al. meta-ICL setup for function classes Prior Fitted Networks Raventos et al. pretraining task diversity GPT-2 decoder-only transformer
Closest papers by meaning
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.93)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.93)
- When can in-context learning generalize out of task distribution? (ICML 2025 · similarity 0.92)
- Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.91)
- In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.91)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.