Research map / Learning theory and algorithms / Prediction and decision losses
Reconciling Model Multiplicity for Downstream Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReDCal, an algorithm that reconciles pairs of equally accurate predictors using decision calibration so they agree on best-response actions as well as individual predictions, with finite-sample guarantees and ImageNet/HAM10000 experiments.
Problem
Model multiplicity in downstream decision-making: two predictors with equivalent accuracy can induce different best-response actions, and prior reconciliation (Reconcile) can increase downstream decision loss even though it makes individual predictions agree.
Approach
Reconcile predictor pairs on disagreement events defined by best-response actions: first patch the predictor on the disagreement region to reduce Brier error, then run decision calibration within that event so the updated predictor's loss estimates are accurate and its best-response actions are safe to use.
Key results
- On ImageNet, ReDCal reduces test-set decision loss gap more than Reconcile, averaged over 500 random loss functions and 10 runs (exact numbers in tables not extracted).
- Running ReDCal as post-process after Decision-Calibration further reduces decision loss and yields the largest Brier score decrease on ImageNet.
- On HAM10000 with ResNet-50 and DenseNet-121 (~88% top-1 accuracy), ReDCal decreases decision loss more than Reconcile on the test split, though it reduces Brier score by less than Reconcile.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReDCal instead of Reconcile | ImageNet · ImageNet test | decision loss | Helped |
| Use ReDCal instead of Reconcile | HAM10000 · HAM10000 test | decision loss | Helped |
| Apply ReDCal after Decision Calibration | ImageNet · ImageNet | decision loss | Helped |
| Apply ReDCal after Decision Calibration | HAM10000 · HAM10000 | decision loss | Helped |
| Apply ReDCal after Decision Calibration | ImageNet · ImageNet | Brier score | Helped |
| Use ReDCal instead of Reconcile | ImageNet · ImageNet test | Brier score | Hurt |
| Use ReDCal instead of Reconcile | HAM10000 · HAM10000 | Brier score | Hurt |
| Use ReDCal instead of Reconcile as the number of classes grows | ImageNet · ImageNet test, 10 to 1000 classes | decision loss | Helped |
| Use ReDCal instead of Reconcile at 1000 classes | ImageNet · ImageNet test, 1000 classes | convergence speed | Hurt |
| Calibrate predictors with Reconcile | ImageNet predictors · ImageNet test | decision loss | Hurt |
| Reconcile individual predictions without downstream decision calibration | Binary classification predictors · Stylized binary decision-making example | expected decision loss: + 1/2 | Hurt |
Surprising findings
- Reconcile can increase downstream decision loss even though it lowers Brier scores and makes individual predictions agree everywhere (theoretical counterexample).
- Decision-calibrated predictors can still disagree on best-response actions on a substantial mass of units.
Limitations
- Theoretical guarantees assume linear loss functions over a finite family and known or i.i.d-sampled distributions; finite-sample bounds depend on a large covering set of predictors.
- Experiments use randomly generated loss functions (ImageNet) and a single medical-inspired loss with added noise (HAM10000), so real-world decision impact is not directly assessed.
- On ImageNet with 1000 classes ReDCal converges slower than Reconcile.
Experimental setup
Datasets: ImageNet HAM10000
Benchmarks: ImageNet validation/test partition HAM10000 test split
Metrics: Brier score decision loss gap relative to optimal loss disagreement region mass
Baselines: Reconcile Decision-Calibration
Builds on: Reconcile (Roth et al. predictive multiplicity reconciliation) decision calibration (Zhao et al.) multi-calibration
Closest papers by meaning
- Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? (ICML 2025 · similarity 0.84)
- Predictive Performance Comparison of Decision Policies Under Confounding (ICML 2024 · similarity 0.84)
- Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents (ICML 2025 · similarity 0.84)
- Testable and Actionable Calibration for Full Swap Regret (NeurIPS 2026 · similarity 0.84)
- Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making (NeurIPS 2024 · similarity 0.83)
- Dissenting Explanations: Leveraging Disagreement to Reduce Model Overreliance (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.