Research map / Probabilistic and scientific modeling / Molecules and proteins
Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controls counterfactual harm of prediction-set decision support systems via conformal risk control, using SCM-based identifiability under counterfactual or interventional monotonicity assumptions, validated on ImageNet16H human-subject data.
Problem
Decision support systems based on prediction sets can improve average human accuracy but may cause counterfactual harm, where a human who would have been correct alone becomes wrong when using the system.
Approach
Formalizes counterfactual harm with an SCM and shows it is identifiable from human predictions made alone under counterfactual monotonicity (and partially identifiable under interventional monotonicity), then uses conformal risk control to pick a threshold lambda whose expected harm is below a user bound alpha.
Key results
- Decision support systems based on prediction sets always cause some counterfactual harm for lambda < 1, and for alpha=0.01 the accuracy-optimal threshold is not harm-controlling.
- The framework's harm-controlling sets are typically valid (do not include lambda with H(lambda) > alpha) but conservative due to calibration estimation error.
- A trade-off between accuracy and counterfactual harm is observed consistently across five pre-trained classifiers and on both datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restrict human predictions to a prediction set with λ < 1 | Decision support systems based on prediction sets · ImageNet16H and ImageNet16H-PS | counterfactual harm | Hurt |
| Use the conformal-risk-control framework to select harm-controlling λ values | Decision support systems based on prediction sets · ImageNet16H and ImageNet16H-PS | counterfactual harm control | Helped |
| Increase accuracy while controlling counterfactual harm with prediction sets | Decision support systems based on prediction sets · ImageNet16H and ImageNet16H-PS | accuracy and counterfactual harm | Mixed |
| Use mixture of multinomial logit models to estimate human accuracy | Decision support systems based on prediction sets · ImageNet16H-PS | accuracy trend | No clear effect |
Surprising findings
- The accuracy-maximizing threshold yields decision support systems that are not harm-controlling at alpha=0.01.
Limitations
- Assumes i.i.d. data, noiseless ground-truth labels and fixed threshold; no distribution shift or label noise.
- Treats all incorrect predictions as equally harmful.
- Under interventional monotonicity the lower and upper bounds have a large gap; evaluated on a single image benchmark.
Experimental setup
Datasets: ImageNet16H ImageNet16H-PS ILSVRC 2012
Benchmarks: ImageNet16H ImageNet16H-PS
Metrics: average counterfactual harm average accuracy harm-controlling set coverage
Baselines: Richens et al. counterfactual harm definition accuracy-optimal threshold lambda*
Builds on: conformal risk control Straitouri et al. prediction-set decision support counterfactual harm (Richens et al.) multinomial logit mixture models
Closest papers by meaning
- Designing Decision Support Systems using Counterfactual Prediction Sets (ICML 2024 · similarity 0.91)
- Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents (ICML 2025 · similarity 0.86)
- Conformal Prediction Sets Improve Human Decision Making (ICML 2024 · similarity 0.85)
- Towards Human-AI Complementarity with Prediction Sets (NeurIPS 2024 · similarity 0.85)
- Conformal Prediction Sets Can Cause Disparate Impact (ICLR 2025 · similarity 0.84)
- Regretful Decisions under Label Noise (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.