Research map / Architectures and efficiency / RNNs and normalization
In-Context Learning and Occam's Razor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Links in-context learning's next-token prediction loss to prequential coding and Occam's razor, showing ICL meta-learns learners that jointly minimize training error and model complexity, with experiments on synthetic regression/Mastermind/HMM tasks.
Problem
Explaining why in-context learning generalizes well from small data and why standard ML only minimizes training error, lacking a principled way to directly favor simple models (Occam's razor).
Approach
Shows the next-token prediction loss used to train in-context learners equals prequential code length, so meta-training a sequence model with it directly minimizes compression length, i.e., training error plus model complexity; contrasts with a train-risk ICL control that only minimizes training error.
Key results
- Prequential ICL outperforms train-risk ICL in generalization at short context lengths, with the gap growing with task difficulty (e.g., sinusoid regression and Mastermind).
- SGD-based learners generalize poorly in low-data regimes versus ICL, but on Mastermind SGD performs better in large-data regimes, showing ICL underfitting.
- GPT-4o performs far worse than small ICL models trained on Mastermind and even worse than a naive marginal-class baseline in prequential code length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use next-token prediction (prequential ICL) instead of past-token prediction (train-risk ICL) | ICL · Linear, sinusoidal, and Mastermind tasks, short contexts | generalization error | Helped |
| use next-token prediction (prequential ICL) instead of past-token prediction (train-risk ICL) | ICL · Linear, sinusoidal, and Mastermind tasks, large context lengths | generalization error | No clear effect |
| use next-token prediction (prequential ICL) instead of past-token prediction (train-risk ICL) | ICL · HMM | generalization error | Helped |
| train an ICL learner on the full sequence rather than only the latter half (ablation) | In-context learner · Prequential coding evaluation | prequential code length | Helped |
| use prequential ICL instead of an SGD-based learner | Learner · All tasks, low-data regimes | generalization error | Helped |
| use an SGD-based learner instead of prequential ICL | Learner · All tasks, enough training data | generalization error | No clear effect |
| use an SGD-based learner instead of prequential ICL | Learner · Mastermind, large-data regime | generalization error | Helped |
| apply regularization to SGD | SGD-based learner · Low-data regimes | prequential code length | Hurt |
| remove the bottleneck from the Transformer (ablation) | ICL architecture · Mastermind | prequential code length | Helped |
| replace the Transformer with an SSM architecture (ablation) | ICL architecture · HMM | prequential code length | Helped |
| use a small task-trained ICL model instead of GPT-4o | Mastermind predictor · Mastermind | prequential code length | Helped |
| use a small task-trained ICL model instead of a marginal-distribution baseline | Mastermind predictor · Mastermind | prequential code length | Helped |
Surprising findings
- GPT-4o's prequential code length on Mastermind is higher than a naive baseline predicting the marginal class distribution over the context.
- On Mastermind, SGD outperforms ICL in large-data regimes, indicating ICL can underfit.
Limitations
- ICL learners can underfit on difficult tasks in large-data regimes, hurting generalization.
- Task generalization to novel open-ended tasks is difficult; a large LLM fails on a novel Mastermind task.
- Experiments are on small synthetic tasks with small models; prequential code length is an upper bound on Kolmogorov complexity.
Experimental setup
Datasets: synthetic linear regression tasks synthetic sinusoidal regression tasks Mastermind synthetic HMM sequences
Benchmarks: synthetic linear regression synthetic sinusoidal regression Mastermind HMM next-token prediction noisy polynomial regression
Metrics: MSE cross-entropy prequential code length generalization error
Baselines: train-risk ICL SGD-based learner regularized SGD GPT-4o Mamba 1 Mamba 2
Builds on: prequential coding minimum description length memory-based meta-learning
Closest papers by meaning
- Context is Environment (ICLR 2024 · similarity 0.87)
- An Information-Theoretic Analysis of In-Context Learning (ICML 2024 · similarity 0.86)
- In-context Learning in Presence of Spurious Correlations (NeurIPS 2026 · similarity 0.86)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.86)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.86)
- In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.