Research map / Learning theory and algorithms / Prediction and decision losses
Reliable Algorithm Selection for Machine Learning-Guided Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes design algorithm selection, using prediction-powered inference adapted to covariate shift to produce valid p-values that select machine learning-guided design configurations whose label distributions satisfy a user-specified success criterion with high probability.
Problem
Choosing a design algorithm configuration (algorithm, hyperparameters, predictive/generative models) for machine learning-guided design so that the resulting design label distribution satisfies a success criterion, where predictions on designs are biased by distribution shift.
Approach
Frame configuration selection as multiple hypothesis testing; for each configuration compute a prediction-powered p-value that uses held-out labeled data, importance-weighted by design-to-labeled density ratios, to correct the bias of mean predictions on designs; then apply Bonferroni correction to select configurations.
Key results
- On GB1 with known density ratios, the method kept error rates below alpha=0.1 for all tau while prediction-only and CalibratedForecasts had 100% error rates for most tau.
- On RNA binder design with estimated density ratios, the method had lower error than all alternatives except GMMForecasts with q=0 across the tau range where those had nonzero selection rates, but error exceeded alpha for tau > 0.32.
- The method maintained nonzero selection rates for a broader range of tau than GMMForecasts with any q.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use held-out labeled data to correct prediction-based estimates, with known density ratios | Design algorithm selection · Protein GB1, mean design label | error rate: 100% to below the desired level of α | Helped |
| Use held-out labeled data to correct prediction-based estimates | Design algorithm selection · RNA binders, mean design label | error rate: 100% to much lower error rates | Helped |
| Use estimated density ratios rather than known density ratios | Design algorithm selection · RNA binders, mean design label | error rate | Hurt |
| Use GMM forecasts with q = 1 instead of q ∈ {0, 0.5} | GMMForecasts · Protein GB1, mean design label | selection rate: zero to high selection | Helped |
| Use GMM forecasts with q ∈ {0, 0.5} rather than q = 1 | GMMForecasts · Protein GB1, mean design label | selection rate: zero | Hurt |
| Increase GMM forecast hyperparameter q to 1 | GMMForecasts · Protein GB1, mean design label with τ ≥ 0.5 | selection rate | Hurt |
| Use GMM forecasts with q = 0 | GMMForecasts · RNA binders, mean design label | error rate: low error rates, even zero | Helped |
| Use GMM forecasts with q = 0 rather than q ∈ {0.5, 1} | GMMForecasts · RNA binders, mean design label | selection rate | Hurt |
| Use GMM forecasts with q ∈ {0.5, 1} rather than q = 0 | GMMForecasts · RNA binders, mean design label | selection rate: 100% | Helped |
| Use GMM forecasts with q ∈ {0.5, 1} rather than q = 0 | GMMForecasts · RNA binders, mean design label | error rate: much higher error rates | Hurt |
| Reserve half of labeled sequences for held-out error correction instead of using all 10k for training | Design algorithm selection · Protein GB1, mean design label | error rate: 10k; 5k | Helped |
| Increase the menu to 249 configurations | Design algorithm selection · RNA binders | selection rate | Mixed |
Surprising findings
- The conformal prediction adaptation was prohibitively conservative and never selected any configuration in experiments.
- The prediction-only baseline's selected configurations could have mean design labels considerably below the target tau.
Limitations
- Guarantees require known density ratios; with estimated density ratios, validity is only approximate and error exceeded alpha for large tau in RNA experiments.
- Bonferroni correction is conservative for large menus, reducing selection rates.
- Method focuses on single-round success; multi-round design not addressed.
Experimental setup
Datasets: GB1 binding affinity dataset (all 20^4 variants) simulated RNA binding energy labels via ViennaFold
Benchmarks: GB1 library design simulation RNA binder design simulation (ViennaFold)
Metrics: error rate selection rate mean design label
Baselines: prediction-only method GMMForecasts CalibratedForecasts conformal prediction method
Builds on: prediction-powered inference AdaLead multinomial logistic regression density ratio estimation Waudby-Smith & Ramdas mean confidence bounds
Closest papers by meaning
- Searching for High-Value Molecules Using Reinforcement Learning and Transformers (ICLR 2024 · similarity 0.86)
- Tree Search-Based Evolutionary Bandits for Protein Sequence Optimization (AAAI 2024 · similarity 0.86)
- Multi-Objective Molecular Design Through Learning Latent Pareto Set (AAAI 2025 · similarity 0.86)
- Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient (NeurIPS 2024 · similarity 0.85)
- Enhancing Ligand Validity and Affinity in Structure-Based Drug Design with Multi-Reward Optimization (ICML 2025 · similarity 0.85)
- Steering Generative Models with Experimental Data for Protein Fitness Optimization (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.