Research map / Learning theory and algorithms / Kernels and regression theory
Model Assessment and Selection under Temporal Distribution Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an adaptive rolling-window method (Goldenshluger-Lepski-style bias-variance proxy) for estimating generalization error and selecting among models under arbitrary temporal distribution shift, with a single-elimination tournament for multi-model selection.
Problem
Assessing and selecting models whose generalization error is measured on the current data distribution when the data distribution drifts over time, with only limited current data and abundant historical data.
Approach
Adaptively choose the look-back window by estimating bias from data via a Goldenshluger-Lepski comparison of rolling-window means, with empirical Bernstein variance proxies; compare models by applying this to the loss difference and run a single-elimination tournament over pairwise comparisons.
Key results
- On the randomized non-stationary synthetic setting with sigma^2=1, the adaptive rolling window achieves mean excess risk 0.139 versus 0.157 for window 1 and 1.067 for window 256.
- On arXiv topic frequency, the adaptive method attains 2.4e-3 excess risk, matching window 16 and close to window 256 (1.9e-3), versus 6.7e-3 for window 1.
- On Dubai housing, the adaptive method attains overall MSE 0.071, tied with window 1 and better than window 256 (0.095).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive rolling-window selection instead of the best fixed window | Model selection · Synthetic stationary, σ²=1 | mean excess risk: 0.015 vs 0.010 | Hurt |
| Use adaptive rolling-window selection instead of the best fixed window | Model selection · Synthetic stationary, σ²=10 | mean excess risk: 1.293 vs 0.982 | Hurt |
| Use adaptive rolling-window selection instead of fixed window size 1 | Model selection · Synthetic non-stationary, σ²=1 | mean excess risk: 0.157 vs 0.139 | Helped |
| Use adaptive rolling-window selection instead of the best fixed window | Model selection · Synthetic non-stationary, σ²=10 | mean excess risk: 2.052 vs 1.771 | Hurt |
| Use adaptive rolling-window selection instead of fixed window size 64 | Model selection · arXiv cs.LG topic frequency | mean excess risk (× 10⁻³): 2.4 vs 1.7 | Hurt |
| Use adaptive rolling-window selection instead of fixed window size 4 | Model selection · Dubai housing prediction | overall MSE: 0.071 vs 0.069 | Hurt |
Surprising findings
- On the arXiv data the adaptive method is not best; window 64 (1.7e-3) and window 256 (1.9e-3) beat it slightly, so adaptivity has a cost there.
- In the stationary synthetic setting, the adaptive method is comparable to large fixed windows but the best fixed window differs across settings, underscoring that no single window is robust.
Limitations
- Assumes no structure on non-stationarity; does not exploit seasonality or trends.
- Model selection only covers finitely many candidate models.
- Oracle guarantees hide logarithmic factors and assume bounded losses (relaxable only with light-tail truncation).
Experimental setup
Datasets: synthetic Gaussian mean estimation data arXiv cs.LG metadata (118,883 papers) Dubai Land Department apartment transactions (211,432 sales)
Benchmarks: synthetic mean estimation arXiv topic frequency estimation Dubai housing price prediction
Metrics: excess risk mean squared error
Baselines: fixed-window model selection
Builds on: Goldenshluger-Lepski method empirical Bernstein inequality prediction from expert advice
Closest papers by meaning
- CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions (NeurIPS 2024 · similarity 0.85)
- Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line (NeurIPS 2024 · similarity 0.85)
- Quantifying Uncertainty in the Presence of Distribution Shifts (NeurIPS 2025 · similarity 0.84)
- Time-Varying Propensity Score to Bridge the Gap between the Past and Present (ICLR 2024 · similarity 0.83)
- Bayesian Weight Enhancement with Steady-State Adaptation for Test-time Adaptation in Dynamic Environments (ICML 2025 · similarity 0.83)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.