Research map / Learning theory and algorithms / Kernels and regression theory
Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that reshuffling resampling splits independently for each hyperparameter configuration during HPO can improve final generalization, with theory (asymptotic regret bound), simulations, and random search/BO benchmarks on tabular binary classification.
Problem
Standard hyperparameter optimization uses fixed resampling splits (holdout or CV) for all configurations, which may steer optimization toward configurations overfit to those splits (overtuning); the effect of reshuffling splits per configuration on final generalization is unexplored.
Approach
Reshuffle train-validation splits independently for every hyperparameter configuration; analyze via the asymptotic Gaussian limit of the validation loss surface, showing that decorrelating validation losses trades off a penalty for bad luck against a gain in hedging, yielding an expected regret bound that depends on signal-to-noise and correlation.
Key results
- In simulations with 10,000 repetitions, reshuffling improves true risk of the selected configuration for low-curvature loss surfaces (m <= 2) when noise correlation is weak (kappa >= 1).
- In real HPO experiments, reshuffled holdout attains final test performance often on par with standard 5-fold CV while requiring fewer model fits per HPC evaluation.
- Reshuffling improved random search, HEBO and SMAC3 generalization, and narrowed the gap between BO optimizers and random search; reshuffling rarely helped for logloss, especially at small sample sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reshuffle holdout splits for each hyperparameter configuration | HPO with holdout · Tabular binary-classification HPO benchmarks | test ROC AUC | Helped |
| Use reshuffled holdout instead of fixed holdout | HPO with holdout · Tabular binary-classification HPO benchmarks | test ROC AUC | Helped |
| Reshuffle splits between hyperparameter configurations | HPO with 5-fold holdout · Tabular binary-classification HPO benchmarks | test ROC AUC | Helped |
| Reshuffle folds between hyperparameter configurations | HPO with 5-fold CV · Tabular binary-classification HPO benchmarks | test ROC AUC | No clear effect |
| Reshuffle folds between hyperparameter configurations | HPO with 5x 5-fold CV · Tabular binary-classification HPO benchmarks | test ROC AUC | No clear effect |
| Reshuffle resampling splits | Random search · Tabular binary-classification HPO benchmarks | test ROC AUC | Helped |
| Reshuffle resampling splits | HEBO · Tabular binary-classification HPO benchmarks | test ROC AUC | Helped |
| Reshuffle resampling splits | SMAC3 · Tabular binary-classification HPO benchmarks | test ROC AUC | Helped |
| Reshuffle resampling splits | HPO optimizing logloss · Tabular binary-classification HPO benchmarks, especially small sample sizes | generalization performance | Hurt |
| Increase reshuffling by reducing τ | HPO on a simulated loss surface with low curvature and weakly correlated noise · Simulation study | true risk | Helped |
| Increase reshuffling by reducing τ | HPO on a simulated loss surface with strongly correlated noise · Simulation study | true risk | Hurt |
| Increase the number of folds in M-fold holdout (ablation) | Reshuffled M-fold holdout · HPO benchmark experiments | effect of reshuffling on generalization performance | Hurt |
Surprising findings
- Reshuffling the train-validation split for every configuration often improves test performance, contrary to common practice of fixed splits.
- The choice of optimizer appears to matter less for final generalization than whether resampling is reshuffled.
Limitations
- Theory relies on asymptotic Gaussian approximation and loss stability assumptions, which fail for highly sensitive losses such as logloss.
- Analysis covers only generalization after search over a fixed finite candidate set, ignoring dynamic HPO algorithms.
- Experiments restricted to tabular data and binary classification, avoiding very small or large datasets.
Experimental setup
Datasets: AutoML benchmark tabular datasets (binary classification, <100 features, 10,000 to 1,000,000 observations)
Benchmarks: AutoML benchmark tabular binary classification tasks (ROC AUC, accuracy, logloss) controlled simulation of quadratic loss surface with Gaussian process noise
Metrics: test ROC AUC of incumbent accuracy logloss true risk of selected configuration
Baselines: fixed holdout fixed 5-fold CV 5x 5-fold CV HEBO SMAC3 random search
Builds on: Bayesian optimization cross-validation AutoML benchmark
Closest papers by meaning
- Hyperparameter Optimization through Neural Network Partitioning (ICLR 2023 · similarity 0.89)
- Random Reshuffling: Simple Analysis with Vast Improvements (NeurIPS 2020 · similarity 0.88)
- Don't Compress Gradients in Random Reshuffling: Compress Gradient Differences (NeurIPS 2024 · similarity 0.86)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.85)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.85)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.