Research map / Probabilistic and scientific modeling / Molecules and proteins
The Relative Instability of Model Comparison with Cross-validation
Alexandre Bayle, Lucas Janson, Lester Mackey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compare two ST (or Lasso) fits with penalties differing by a constant instead of assessing a single fit (ablation) | Cross-validation CLT intervals · Simulated linear model, p=10, tau=10, 10-fold CV | relative loss stability r(h_n): o(1) -> Omega(sqrt(n)) | Hurt |
| choose lambda by inner CV for Lasso comparisons | CV confidence intervals for Lasso · Simulated linear model, p=10, tau=10, 10-fold CV | CI coverage | No clear effect |
| replace ST by ridge regression for the comparison (ablation) | CV comparison · Simulated linear model, p=10, tau=10, 10-fold CV | relative loss stability: comparison loss stability scales as 1/n^4 | Improved |
| make the true coefficient vector fully dense (no zero entries) for ST comparison (ablation) | ST comparison · Simulated linear model, p=10, tau=10, 10-fold CV | relative loss stability: loss stability of difference scales as 1/n^4 | Improved |
| use the within-fold variance estimator for the comparison | CV CLT interval · Simulated linear model, p=10, tau=10, 10-fold CV | variance estimate | Hurt |
| build comparison interval from two separate single-algorithm intervals | CV comparison interval · Proposition and appendix illustration | asymptotic validity | Mixed |
Closest papers by meaning
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.88)
- Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance? (NeurIPS 2024 · similarity 0.87)
- Choice of V for V-Fold Cross-Validation in Least-Squares Density Estimation (JMLR 2016 · similarity 0.87)
- Cross-validation Confidence Intervals for Test Error (NeurIPS 2020 · similarity 0.85)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.84)
- Post-Hoc Reversal: Are We Selecting Models Prematurely? (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
