Research map / Learning theory and algorithms / Prediction and decision losses
Cross-validation Confidence Intervals for Test Error
Pierre Bayle, Alexandre Bayle, Lucas Janson, Lester W. Mackey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed CLT-based confidence interval instead of concentration-based intervals | Ridge regression test error · FlightDelays regression experiment with standardized features | CI width: 91 times wider | Improved |
| Use the proposed CLT-based confidence interval instead of concentration-based intervals | Ridge regression test error · FlightDelays regression experiment with unstandardized features | CI width: 5× 10^{14} times wider | Improved |
| Use the proposed CLT-based confidence interval instead of the compared alternatives | Test error assessment · Higgs classification and FlightDelays regression | CI width | Improved |
| Use the proposed test instead of the compared testing procedures | Algorithm improvement testing · Higgs classification and FlightDelays regression | power | Improved |
| Use the proposed test instead of a single hold-out test | Algorithm improvement testing · Higgs classification and FlightDelays regression | power | Improved |
| Use the proposed test with less stable neural networks or deeper random forests | Algorithm improvement testing · Destabilized learning algorithms | test size | Hurt |
| Use the proposed LOOCV interval rather than the popular alternatives | Ridge regression test error · Leave-one-out cross-validation | CI width | Improved |
| Use repeated train-validation confidence intervals | Test error assessment · Higgs classification and FlightDelays regression | coverage | Hurt |
Closest papers by meaning
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.86)
- Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance? (NeurIPS 2024 · similarity 0.85)
- Kernel conditional tests from learning-theoretic bounds (NeurIPS 2025 · similarity 0.83)
- Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning (NeurIPS 2025 · similarity 0.82)
- Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (ICML 2025 · similarity 0.82)
- Rectifying Conformity Scores for Better Conditional Coverage (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
