Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
Tomoya Wakayama, Taiji Suzuki
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fit Bayes gap jointly in pretraining prompts N and context length p rather than N only (ablation) | power-law surrogate of Bayes gap · Pooled Bayes-gap fit (two-family synthetic) | R^2: 0.030 -> 0.912 | Improved |
| fit Bayes gap jointly in N and p rather than p only (ablation) | power-law surrogate of Bayes gap · Pooled Bayes-gap fit (two-family synthetic) | R^2: 0.016 -> 0.912 | Improved |
| increase number of pretraining prompts N | GPT-2 transformer · Two-family synthetic regression | Bayes gap | Improved |
| increase pretraining prompt length p | GPT-2 transformer · Two-family synthetic regression | Bayes gap | Improved |
| increase observation noise from 0.1 to 0.5 | Bayes posterior · Three-family mixture, k=3 | wrong-family posterior mass: 0.067 -> 0.093 | Hurt |
| increase observation noise from 0.1 to 0.5 | Bayes posterior · Three-family mixture, k=5 | wrong-family posterior mass: 0.0010 -> 0.0054 | Hurt |
| increase context length from k=1 to k=5 | Transformer · Three-family mixture, noise 0.1 | MSE: 19.15 -> 0.156 | Improved |
| increase context length from k=1 to k=5 | Transformer · Three-family mixture, noise 0.5 | MSE: 23.94 -> 0.338 | Improved |
| give true task family to the Bayes predictor (oracle vs mixture) | Bayes predictor · Three-family mixture, noise 0.1, k=1 | MSE: 18.027 -> 4.063 | Improved |
| give true task family to the Bayes predictor (oracle vs mixture) | Bayes predictor · Three-family mixture, noise 0.5, k=5 | MSE: 0.317 -> 0.314 | No clear effect |
Closest papers by meaning
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.92)
- An Information-Theoretic Analysis of In-Context Learning (ICML 2024 · similarity 0.91)
- In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.90)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.90)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.89)
- How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
