Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Learning Prediction Intervals for Model Performance
Benjamin Elder, Matthew Arnold, Anupama Murthi, Jiří Navrátil
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a transfer-trained uncertainty model instead of the strongest calibrated baseline | Performance prediction intervals with the Meta-model predictor · linear-skew train → linear-skew eval | average cost: 1.77 -> 1.33 | Improved |
| Use a transfer-trained uncertainty model instead of the strongest calibrated baseline | Performance prediction intervals with the Meta-model predictor · linear-skew train → nearest-neighbors eval | average cost: 1.15 -> 0.97 | Improved |
| Add Distance features to Prediction features (ablation) | Uncertainty model · Feature ablation; meta-model predictor | average cost (α=0.9; normalized): 1.33 -> 1.07 | Improved |
| Add Internal features to Prediction features (ablation) | Uncertainty model · Feature ablation; meta-model predictor | average cost (α=0.9; normalized): 1.33 -> 1.05 | Improved |
| Add Drift features to the feature set (ablation) | Uncertainty model · Feature ablation; meta-model predictor | average cost (α=0.9; normalized): 1.16 -> 1.0 | Improved |
| Use all feature types instead of Prediction features only (ablation) | Uncertainty model · Feature ablation; meta-model predictor | average cost (α=0.9; normalized): 1.33 -> 1.0 | Improved |
| Use all feature types instead of Noise features only (ablation) | Uncertainty model · Feature ablation; meta-model predictor | average cost (α=0.9; normalized): 1.32 -> 1.0 | Improved |
| Replace the Meta-model predictor with Crossval | Performance prediction · linear-skew scenarios | MAE: 3.84 -> 4.04 | Hurt |
| Replace the Meta-model predictor with Dropout | Performance prediction · linear-skew scenarios | MAE: 3.84 -> 4.70 | Hurt |
Closest papers by meaning
- Efficient Prediction of Pass@k Scaling in Large Language Models (NeurIPS 2026 · similarity 0.85)
- Algorithms with Calibrated Machine Learning Predictions (ICML 2025 · similarity 0.84)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.84)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.84)
- Divide-and-Aggregate Learning for Evaluating Performance on Unlabeled Data (AAAI 2024 · similarity 0.84)
- Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
