Research map / Learning theory and algorithms / Prediction and decision losses

Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Data attribution and recourse · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that repeated cross-validation (MCCV/repeated K-fold) substantially reduces benchmarking variance, quantified as a "sample gain" in equivalent test-set size, and proposes a study-only redundancy score to early-stop CV.

Problem

Benchmarking learning algorithms on small datasets yields unreliable, high-variance performance estimates and unstable rankings, creating a validation crisis; the paper asks how many CV splits reduce variance and when extra splits stop paying off.

Approach

Defines the variance-equivalent test sample gain (how much larger a single hold-out test set would need to be to match K-split CV variance), estimates inter-fold covariance via ANOVA over seeds, and proposes a study-only redundancy score computed from the first few splits to triage whether continuing CV is worthwhile.

Key results

• On the simulated linear dataset at N=1,000, Ridge reaches a variance-equivalent test sample gain of about 14 at K=200 splits, and MLP about 15.
• Sample gains for ExtraTrees and GradientBoosting on the simulated pipeline are at least 5 at K=20 and had not converged after K=5 splits.
• On Yelp at N=3,000, single hold-out ranks XLM-RoBERTa above BERT in roughly 70-80% of runs, while at K=20 the retrieval proportion is close to balanced, consistent with no significant difference between them.

Surprising findings

• Diminishing returns from additional CV splits set in much later than the classic heuristic that each sample needs to be held out only once suggests.
• On Yelp, single-split evaluations systematically rank XLM-RoBERTa above BERT even though BERT has the higher mean score, due to low-score outliers of XLM-RoBERTa.

Limitations

• Requires a large benchmarking set to estimate oracle scores, so real-data experiments are constrained by dataset availability and compute.
• Covers only two real-data domains (histopathology and NLP fine-tuning).
• Hyperparameters fixed for NLP to isolate the validation effect; variance estimates on real data are noisier than simulated ones.

Experimental setup

Datasets: PatchCamelyon Yelp Review Full synthetic linear regression data

Benchmarks: PatchCamelyon Yelp Review Full synthetic regression

Metrics: variance-equivalent test sample gain variance of CV estimator ranking retrieval proportion study-only redundancy score

Baselines: single hold-out evaluation repeated K-fold CV

Builds on: Monte-Carlo cross-validation Nadeau & Bengio corrected variance Nagler et al. reshuffling asymptotics Arlot & Celisse CV survey

Closest papers by meaning

• Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance? (NeurIPS 2024 · similarity 0.89)
• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.86)
• Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.86)
• Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.85)
• Uncertainty Quantification for Data-Driven Change-Point Learning via Cross-Validation (AAAI 2024 · similarity 0.85)
• Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.84)