Research map / Architectures and efficiency / Attention and transformers
MLPs Learn In-Context on Regression and Classification Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that vanilla MLPs and MLP-Mixers can learn in-context on synthetic regression and classification tasks at compute comparable to Transformers, and outperform Transformers on psychology-style relational reasoning tasks.
Problem
In-context learning is often assumed to be unique to attention-based Transformers; it is unknown whether MLPs can learn in-context and how they compare to Transformers under equal compute on synthetic ICL and relational reasoning tasks.
Approach
Evaluates vanilla MLPs and MLP-Mixers against Transformers on controlled ICL regression/classification tasks (compute-matched, online training), showing MLPs learn in-context, and on classical relational tasks (match-to-sample, sphere/line oddball) where MLPs outperform Transformers, plus a relationally bottlenecked MLP baseline.
Key results
- On ICL regression with sufficient compute, MLPs, Mixers and Transformers all reach near Bayes-optimal Ridge MSE; vanilla MLPs degrade beyond about 2^6 context exemplars while Mixers stay flat.
- On match-to-sample and sphere oddball, the vanilla MLP outperforms the Transformer by a wide compute margin and generalizes better out of distribution (e.g., to larger radii and perturbation distances).
- On simple (non-ICL) 64-dimensional regression, MLPs reach substantially lower MSE at lower compute than Transformers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a vanilla MLP instead of a Transformer | ICL regression · Unrestricted regression distribution | MSE | No clear effect |
| Use an MLP-Mixer instead of a vanilla MLP | ICL regression · Unrestricted regression with long contexts | excess MSE | Helped |
| Increase context length | Vanilla MLP · Unrestricted ICL regression | excess MSE | Hurt |
| Increase data diversity | MLP, Mixer, and Transformer · ICL regression | MSE on the unrestricted distribution | Helped |
| Use a vanilla MLP instead of a Transformer | ICL classification · Classification with k = 2048, B = 4, L = 8 | cross entropy loss | No clear effect |
| Increase context length | MLP-Mixer · ICL classification | cross entropy loss | Hurt |
| Increase data diversity | MLP, Mixer, and Transformer · ICL classification | accuracy | Helped |
| Use a vanilla MLP instead of a Transformer | Match to sample · Match-to-sample | cross entropy loss | Helped |
| Use a relationally bottlenecked MLP instead of a vanilla MLP | Match to sample · Match-to-sample | cross entropy loss | Helped |
| Test on a different circle radius after training at unit radius | Transformer · Match-to-sample OOD radius | accuracy | Hurt |
| Use a vanilla MLP instead of a Transformer | Sphere oddball · Sphere oddball | cross entropy loss | Helped |
| Test at longer oddball distances than training | Vanilla MLP or RB MLP instead of Transformer · Sphere oddball OOD distance | accuracy | Helped |
Surprising findings
- Vanilla MLPs outperform Transformers on relational oddball and match-to-sample tasks, contrary to prior claims that MLPs cannot reason relationally.
- Simple non-ICL regression and classification with long inputs show a large compute gap favoring MLPs over Transformers.
Limitations
- Only controlled synthetic tasks; no evaluation on complex real-world language or vision data.
- Online training with infinite data; behavior in data-limited settings is untested and Transformers may dominate there.
- Relational bottleneck baseline fails on line oddball without deeper layers, showing sensitivity to task alignment.
Experimental setup
Datasets: synthetic in-context regression data synthetic in-context classification (Gaussian mixture) data match-to-sample synthetic task sphere and line oddball synthetic tasks same-different task
Benchmarks: in-context regression (MSE vs Bayes-optimal Ridge) in-context classification (cross entropy, IWL vs ICL accuracy) match-to-sample sphere oddball line oddball simple regression simple classification
Metrics: mean squared error cross entropy loss accuracy excess MSE compute (PFLOPs)
Baselines: Transformer MLP-Mixer relationally bottlenecked MLP Ridge regression (Bayes optimal) dMMSE estimator
Builds on: Transformers in-context learning (Garg et al.) MLP-Mixer relational bottleneck (Webb et al.) Boix-Adsera et al. relational impossibility
Closest papers by meaning
- How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPs (NeurIPS 2025 · similarity 0.89)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.89)
- Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.89)
- In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness (NeurIPS 2024 · similarity 0.89)
- Breaking through the learning plateaus of in-context learning in Transformer (ICML 2024 · similarity 0.88)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.