Research map / Learning theory and algorithms / Prediction and decision losses
Fast and Informative Model Selection Using Learning Curve Cross-Validation
Felix Mohr, Jan N. van Rijn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 5-fold CV by 80LCCV for validation | random search AutoML · AutoML benchmark (67 datasets), random search of 200 pipelines | mean runtime (hours): 5.5 -> 4 | Improved |
| replace 10-fold CV by 90LCCV for validation | random search AutoML · AutoML benchmark (67 datasets), random search of 200 pipelines | mean runtime (hours): 7 -> 4.5 | Improved |
| replace 5CV by 80LCCV on the 19 slowest datasets | random search AutoML · AutoML benchmark (19 datasets with runtime above 6h), random search | mean runtime (hours): 11 -> 8 | Improved |
| replace 10CV by 90LCCV on the 19 slowest datasets | random search AutoML · AutoML benchmark (19 datasets with runtime above 6h), random search | mean runtime (hours): 12 -> 8 | Improved |
| replace kCV by LCCV | random search AutoML · AutoML benchmark (67 datasets), random search of 200 pipelines | error rate of chosen model: within 0.015 in 90% of cases, within 0.01 in 83% | No clear effect |
| use LCCV on datasets with few instances and many attributes (Amazon) | 80LCCV · AutoML benchmark (67 datasets), random search of 200 pipelines | runtime | Hurt |
| use LCCV when it neither prunes nor skips and hits the timeout | 80LCCV · AutoML benchmark (67 datasets), random search of 200 pipelines | error rate | Hurt |
| add convexity-based pruning with confidence intervals | LCCV · AutoML benchmark (67 datasets), random search of 200 pipelines | runtime: at most twice kCV worst case, 83% of 5CV typical | Improved |
| raise confidence width requirement or number of samples per anchor (ablation) | LCCV (sensitivity analysis) · 200 pipelines, 300s timeout | runtime and error rate | Mixed |
| use IPL prediction instead of last anchor on timeout (ablation) | LCCV (sensitivity analysis) · 200 pipelines, 300s timeout | error rate | No clear effect |
| predict improvement from doubling data with IPL on LCCV curves (regression) | IPL-regressor · Portfolio of 17 classifiers | prediction error tolerance: 50% within 0.025, over 90% within 0.07 | Mixed |
| use IPL-classifier on LCCV curves to decide whether to double data | IPL-classifier · Portfolio of 17 classifiers | decision accuracy: over 80% for any threshold, mostly above 90% | Improved |
Closest papers by meaning
- Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance? (NeurIPS 2024 · similarity 0.88)
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.86)
- LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought (NeurIPS 2025 · similarity 0.84)
- Computational Budget Should Be Considered in Data Selection (NeurIPS 2025 · similarity 0.83)
- Cross-Validated Off-Policy Evaluation (AAAI 2025 · similarity 0.82)
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
