Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Beyond the Norms: Detecting Prediction Errors in Regression Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes detecting unreliable regressor predictions by learning a diversity metric over pairs of independent error samples from estimated conditional error distributions, outperforming baseline detectors on 8 UCI regression datasets.
Problem
Detecting when a regressor's prediction error exceeds a specified tolerance epsilon (unreliable behavior from aleatoric or model uncertainty), where baselines depend on accurate estimates of conditional error distributions.
Approach
Define a diversity coefficient over two independent draws of the discrepancy from an estimated conditional distribution, and learn the symmetric function h that minimizes diversity on good points and maximizes it on bad points, so the detector compensates for estimation errors in the conditional distribution.
Key results
- DV algorithms improve AUROC over the best baselines by up to 3.4/100 (absolute error) and 3.7/100 (relative error).
- DV algorithms reduce FPR at TPR 90% by up to 50% on some datasets while performing comparably in the worst case.
- DV-Y variants over the same conditional estimator gain up to 18.5/100 average AUROC in some cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diversity-based detectors instead of the strongest baseline | Regression error detection · 8 UCI regression datasets, absolute error | AUROC: up to 3.4/100 | Helped |
| Use diversity-based detectors instead of the strongest baseline | Regression error detection · 8 UCI regression datasets, relative error | AUROC: up to 3.7/100 | Helped |
| Use diversity-based detectors instead of competing algorithms | Regression error detection · Some UCI datasets | FPR at TPR 90%: up to 50% | Helped |
| Use DV algorithms rather than their counterparts with the same conditional estimators | Regression error detection · UCI regression datasets | AUROC: up to 18.5/100 | Helped |
| Add conformal correction to the baseline | Baseline regression error detectors · 144 experiments across 8 datasets, 3 epsilons and 6 baselines | AUROC: 52% | Mixed |
| Estimate the conditional distribution of Y given X rather than the discrepancy D given X | Baseline and diversity detectors · UCI regression datasets | Detection performance | Mixed |
Surprising findings
- Conformal correction of baselines improves AUROC in only 52% of experiments, so conformal intervals are not a reliable fix for detection.
Limitations
- Performance depends on the quality of the underlying conditional distribution estimator and on the chosen epsilon and gamma, which must be tuned per task.
- Conformal extension has no theoretical guarantees; the paper notes conformal methods cannot directly solve the problem.
- Evaluated only on small-to-medium tabular UCI datasets with MLP regressors.
Experimental setup
Datasets: yacht energy concrete kin8nm naval power wine Boston
Benchmarks: yacht energy concrete kin8nm naval power wine Boston
Metrics: AUROC FPR at TPR 90%
Baselines: B1-SQR B1-CG B1-KNIFE B2-SQR B2-CG B2-KNIFE RIO CF conformal baselines
Builds on: simultaneous quantile regression diversity coefficient conformalized quantile regression neyman-pearson lemma
Closest papers by meaning
- Calibrated Reliable Regression using Maximum Mean Discrepancy (NeurIPS 2020 · similarity 0.87)
- Distribution-Free Model-Agnostic Regression Calibration via Nonparametric Methods (NeurIPS 2023 · similarity 0.87)
- PAGER: Accurate Failure Characterization in Deep Regression Models (ICML 2024 · similarity 0.86)
- probly: Uncertainty-Aware Machine Learning (NeurIPS 2026 · similarity 0.85)
- Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.84)
- Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.