Research map / Architectures and efficiency / Attention and transformers
Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes linear transformers trained by gradient descent (max-margin limit) on random Gaussian-mixture linear classification tasks, giving in-context generalization guarantees and showing benign overfitting in-context under label-flipping noise.
Problem
How many pre-training tasks and in-context examples are needed for a transformer trained on random linear classification tasks to generalize in-context, and can such transformers memorize noisy in-context labels while still generalizing?
Approach
Uses a convex linear-transformer parameterization whose trained predictor is the max-margin solution over pre-training tasks (via implicit bias of GD); shows the resulting in-context predictor behaves like a nearly uniform average of in-context examples, enabling generalization bounds and benign overfitting analysis for classification.
Key results
- Max-margin linear transformer generalizes in-context with test error near label-noise rate p when test cluster separation R~ can be as small as Omega(1) and M is large enough (bounds stated in theorem).
- Benign overfitting in-context is shown theoretically: with B=d, M a large constant, and R~=d^beta for beta in (1/4,1/2), in-context examples are memorized while clean test error is at most p+small.
- Experiments with GD (step size 0.01, 300 steps) show three regimes: overfitting without generalization (small R~, large d/M), benign overfitting (intermediate R~), and optimal test performance without overfitting (large R~).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fewer pre-training tasks B | Trained linear transformer · In-context classification | test accuracy | Hurt |
| Increase test-time cluster separation R̃ | Trained linear transformer · In-context classification | test accuracy | Helped |
| Increase test-time cluster separation R̃ | Trained linear transformer · In-context classification with label-flipping noise | in-context memorization | Hurt |
| Use high feature dimension with R̃ in the intermediate regime | Trained linear transformer · Noisy in-context classification | test error | Helped |
| Use high feature dimension with R̃ in the intermediate regime | Trained linear transformer · Noisy in-context classification | in-context memorization | Helped |
| Increase the number of pre-training tasks B | Trained linear transformer · In-context classification | test accuracy | Helped |
| Use test-time examples with lower SNR than pre-training examples | Trained linear transformer · In-context classification | test error | No clear effect |
| Add label-flipping noise at test time | Trained linear transformer · In-context classification | test error | No clear effect |
| Increase the number of test-time in-context examples M when label-flipping noise is present | Trained linear transformer · Noisy in-context classification | test error | Helped |
Surprising findings
- Transformers pre-trained only on clean, high-SNR data can tolerate smaller test-time SNR and label-flipping noise at test time they never saw in pre-training.
Limitations
- Analysis restricted to a convex linear attention parameterization with fixed w21 and w22 terms, not full softmax transformers.
- Pre-training assumed to have large SNR and no label noise; noisy pre-training not analyzed.
- Benign overfitting theory requires Lambda=I and high dimension d relative to M.
Experimental setup
Datasets: synthetic random class-conditional Gaussian mixture tasks
Benchmarks: synthetic Gaussian mixture in-context classification
Metrics: in-context test accuracy misclassification probability memorization of in-context examples
Builds on: Transformers learn in-context by gradient descent / linear regression ICL analys Soudry et al. implicit bias of GD toward max-margin benign overfitting in 2-layer networks (Frei et al.)
Closest papers by meaning
- In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.89)
- Transformers are almost optimal metalearners for linear classification (NeurIPS 2025 · similarity 0.89)
- In-context Convergence of Transformers (ICML 2024 · similarity 0.89)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.89)
- Pretrained Transformer Efficiently Learns Low-Dimensional Target Functions In-Context (NeurIPS 2024 · similarity 0.88)
- On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.