AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Fast and Informative Model Selection Using Learning Curve Cross-Validation

Felix Mohr, Jan N. van Rijn

TPAMI 2023 · Learning theory and algorithms › Prediction and decision losses · approach: Data attribution and recourse · 77 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace 5-fold CV by 80LCCV for validationrandom search AutoML · AutoML benchmark (67 datasets), random search of 200 pipelinesmean runtime (hours): 5.5 -> 4Improved
replace 10-fold CV by 90LCCV for validationrandom search AutoML · AutoML benchmark (67 datasets), random search of 200 pipelinesmean runtime (hours): 7 -> 4.5Improved
replace 5CV by 80LCCV on the 19 slowest datasetsrandom search AutoML · AutoML benchmark (19 datasets with runtime above 6h), random searchmean runtime (hours): 11 -> 8Improved
replace 10CV by 90LCCV on the 19 slowest datasetsrandom search AutoML · AutoML benchmark (19 datasets with runtime above 6h), random searchmean runtime (hours): 12 -> 8Improved
replace kCV by LCCVrandom search AutoML · AutoML benchmark (67 datasets), random search of 200 pipelineserror rate of chosen model: within 0.015 in 90% of cases, within 0.01 in 83%No clear effect
use LCCV on datasets with few instances and many attributes (Amazon)80LCCV · AutoML benchmark (67 datasets), random search of 200 pipelinesruntimeHurt
use LCCV when it neither prunes nor skips and hits the timeout80LCCV · AutoML benchmark (67 datasets), random search of 200 pipelineserror rateHurt
add convexity-based pruning with confidence intervalsLCCV · AutoML benchmark (67 datasets), random search of 200 pipelinesruntime: at most twice kCV worst case, 83% of 5CV typicalImproved
raise confidence width requirement or number of samples per anchor (ablation)LCCV (sensitivity analysis) · 200 pipelines, 300s timeoutruntime and error rateMixed
use IPL prediction instead of last anchor on timeout (ablation)LCCV (sensitivity analysis) · 200 pipelines, 300s timeouterror rateNo clear effect
predict improvement from doubling data with IPL on LCCV curves (regression)IPL-regressor · Portfolio of 17 classifiersprediction error tolerance: 50% within 0.025, over 90% within 0.07Mixed
use IPL-classifier on LCCV curves to decide whether to double dataIPL-classifier · Portfolio of 17 classifiersdecision accuracy: over 80% for any threshold, mostly above 90%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy