Research map / Learning theory and algorithms / Prediction and decision losses
Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and numerically that K-fold CV never beats plug-in for estimating out-of-sample performance (bias and coverage), while LOOCV has smaller bias but often no coverage advantage once variability is accounted for.
Problem
Whether cross-validation (K-fold and leave-one-out) statistically outperforms the plug-in approach (reusing training data for evaluation) in estimating out-of-sample model performance, especially for nonparametric models with slow convergence rates.
Approach
A higher-order Taylor analysis of out-of-sample evaluation that accounts for dependence between the trained model and the test data, yielding necessary and sufficient conditions for plug-in and CV interval validity (bias and coverage) across convergence-rate regimes.
Key results
- K-fold CV's bias is Theta(n^-2gamma), larger than plug-in's Theta(n^-2gamma_v), so plug-in is never worse than K-fold CV in bias or coverage.
- LOOCV has bias o(1/n), smaller than plug-in, but the improvement is negligible relative to variability for parametric models and for nonparametric models with gamma_v > 1/4.
- In simulations with 500 replications, for kNN with kn=n^1/4 only LOOCV achieved near-nominal coverage (~0.88-0.92 at n=2400-9600) while plug-in and 5-CV failed (plug-in 0.00 coverage).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use K-fold CV instead of plug-in | Out-of-sample performance evaluation · General parametric and nonparametric models | evaluation bias: Θ(n−2γv) vs Θ(n−2γ) | Hurt |
| use K-fold CV instead of plug-in | Out-of-sample performance interval · Models with γ ≤ 1/4 | coverage validity: limn→∞P(c(ẑ) ∈ Ikcv) < 1 − α | Hurt |
| use plug-in instead of K-fold CV | Out-of-sample performance interval · Nonparametric models with γ ≤ 1/4 and γv > 1/4 | coverage validity: limn→∞P(c(ẑ) ∈ Ip) = 1 − α; limn→∞P(c(ẑ) ∈ Ikcv) < 1 − α | Helped |
| use LOOCV instead of plug-in | Out-of-sample performance evaluation · General parametric and nonparametric models | evaluation bias: o(n−1) vs Θ(n−2γv) | Helped |
| use LOOCV instead of plug-in | Out-of-sample performance interval · Models with γv ≤ 1/4 | coverage probability: limn→∞P(c(ẑ) ∈ Iloocv) = 1 − α | Helped |
| use 5-CV instead of plug-in | kNN n²/³ regression · Regression—kNN n²/³, n=9600 | coverage probability (c(ẑ)): 0.88 -> 0.61 | Hurt |
| use 5-CV instead of plug-in | Forest regression · Regression—Forest, n=9600 | coverage probability (c(ẑ)): 0.85 -> 0.42 | Hurt |
| use LOOCV instead of plug-in | kNN CVaR portfolio optimization · CVaR portfolio optimization—kNN n¹/⁴, n=9600 | coverage probability (c(ẑ)): 0.00 -> 0.89 | Helped |
| use plug-in instead of 5-CV | kNN CVaR portfolio optimization · CVaR portfolio optimization—kNN n¹/⁴, n=9600 | coverage probability (c(ẑ)): 0.66 -> 0.00 | Helped |
Surprising findings
- Plug-in, despite being optimistically biased, gives valid coverage for many nonparametric models where 5-fold CV fails.
- For kNN with kn=n^1/4, plug-in coverage of c(ẑ) drops to 0.00 as n grows, despite being the default evaluation.
Limitations
- Results are asymptotic in fixed low dimension; finite-sample and high-dimensional regimes are not covered.
- Requires smoothness and LOO stability assumptions; non-smooth or less stable models are not fully covered.
- Comparisons concern model evaluation, not model selection via hyperparameter tuning.
Experimental setup
Datasets: synthetic regression data (dx=4 and dx=10) synthetic CVaR portfolio data a real-world regression dataset (Appendix F.3)
Benchmarks: synthetic regression simulations CVaR portfolio optimization simulations
Metrics: coverage probability interval width evaluation bias
Baselines: K-fold cross-validation leave-one-out cross-validation plug-in interval
Builds on: Bayes-optimal CLT for plug-in (Bayle et al.) CV asymptotic normality (Bayle et al. 2020) stability-based generalization analysis
Closest papers by meaning
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.89)
- Cross-Validated Off-Policy Evaluation (AAAI 2025 · similarity 0.84)
- Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.84)
- An Effective Theory of Bias Amplification (ICLR 2025 · similarity 0.84)
- Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data (ICLR 2025 · similarity 0.83)
- Show Me What You Don’t Know: Efficient Sampling from Invariant Sets for Model Validation (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.