Research map / Label-efficient and robust learning / Test-time adaptation
Generalization vs Specialization under Concept Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes optimally tuned ridge regression under concept shift (changed input-label function at test time), deriving exact thermodynamic-limit prediction risk and showing more data can hurt under strong concept shift, validated with transformers on noisy linear ICL and MNIST/FashionMNIST classification.
Problem
How concept shift (a change in the input-label relationship at test time) affects generalization and data dependence of prediction risk in neural-network-like learners, where more training data may hurt.
Approach
Uses high-dimensional ridge regression with a minimal concept-shift model parametrized by coefficient alignment cos(theta) and scaling kappa to derive exact thermodynamic-limit risk; identifies a phase transition at kappa*cos(theta)=1/2 where more data stops helping, and extends to anisotropic two-scale features where nonmonotonicity depends on whether low- or high-variance features shift.
Key results
- Concept shift yields a phase transition: more data helps only when kappa*cos(theta) > 1/2 and hurts below it, independent of SNR.
- For isotropic features, R_N=0 = kappa^2 ||beta||^2 and R_N->inf = (1 - 2 kappa cos(theta) + kappa^2) ||beta||^2, so risk is monotone in data for optimally tuned ridge.
- Transformers trained for noisy linear ICL show test MSE that changes from improving to worsening with context length as concept shift grows, in good quantitative agreement with the thermodynamic-limit theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training data under weak concept shift (κ cosθ > 1/2) | Optimally tuned ridge regression · Concept shift | prediction risk: κ cosθ > 1/2 | Helped |
| Increase training data under strong concept shift (κ cosθ < 1/2) | Optimally tuned ridge regression · Concept shift | prediction risk: κ cosθ < 1/2 | Hurt |
| Increase training data at the transition boundary | Optimally tuned ridge regression · Concept shift | prediction risk: κ cosθ = 1/2 | No clear effect |
| Increase in-context context length under weak concept shift | Transformer in-context regression · Noisy linear regression, isotropic features | MSE | Helped |
| Increase in-context context length under strong concept shift | Transformer in-context regression · Noisy linear regression, isotropic features | MSE | Hurt |
| Train on anisotropic rather than isotropic data | Transformer in-context regression · Two-scale anisotropic data with concept shift on high-variance features | prediction risk | Helped |
| Use isotropic-trained transformers on anisotropic test data | Transformer in-context regression · Two-scale anisotropic data with concept shift | prediction risk | Hurt |
| Increase training sample size under strong concept shift with few robust features | Multinomial logistic regression · MNIST | test accuracy | Hurt |
| Increase training sample size under strong concept shift with few robust features | Multinomial logistic regression · FashionMNIST | test accuracy | Hurt |
| Increase training sample size when concept shift affects low-variance features | Optimally tuned ridge regression · Two-scale anisotropic features | prediction risk | Mixed |
| Increase training sample size under strong concept shift affecting high-variance features | Optimally tuned ridge regression · Two-scale anisotropic features | prediction risk | Mixed |
Surprising findings
- For sufficiently strong concept shift, ridge regression fails to generalize even with infinite data, so more training data strictly hurts.
- Nonmonotonic data dependence arises even with optimally tuned ridge, where double descent is absent.
Limitations
- Theory is exact only for ridge regression in the thermodynamic limit with Gaussian features.
- Classification experiments are qualitative and use multinomial logistic regression with a PCA-based robust/nonrobust feature construction.
Experimental setup
Datasets: MNIST FashionMNIST synthetic Gaussian linear regression tasks
Benchmarks: MNIST FashionMNIST synthetic linear regression risk (thermodynamic limit)
Metrics: prediction risk (MSE) bias-variance decomposition test accuracy
Baselines: in-distribution optimally tuned ridge regression
Builds on: ridge regression nanoGPT optimal ridge regularization
Closest papers by meaning
- When Shift Happens - Confounding Is to Blame (ICLR 2026 · similarity 0.89)
- General Quantification of Covariate and Concept Shifts (ICML 2026 · similarity 0.89)
- A Fine-Grained Analysis on Distribution Shift (ICLR 2022 · similarity 0.87)
- Beyond Invariance: Test-Time Label-Shift Adaptation for Addressing "Spurious" Correlations (NeurIPS 2023 · similarity 0.86)
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.84)
- Domain-wise Data Acquisition to Improve Performance under Distribution Shift (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.