Research map / Learning theory and algorithms / Prediction and decision losses
Conformal Prediction Sets Can Cause Disparate Impact
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-registered randomized human-subject experiments show conformal prediction sets can cause disparate impact on human accuracy, and that Equalized Coverage (Mondrian conditional CP) amplifies it, so the authors propose equalizing set sizes instead.
Problem
Conformal prediction sets given to human decision makers may produce unequal accuracy gains across protected groups, and the prevailing fairness standard (Equalized Coverage) may not ensure fair outcomes.
Approach
Shows via randomized human trials that equalizing coverage with Mondrian conformal prediction increases disparate impact relative to marginal sets, and that set size differences (not coverage differences) correlate strongly with disparate impact; recommends Equalized Set Size or Equalized Singleton Frequency as fairness targets.
Key results
- Conditional (Equalized Coverage) treatment gives maxROR 1.51 on FACET vs 1.26 for marginal; on BiosBias 1.33 vs 1.12; on RAVDESS 1.28 vs 1.01.
- Coverage differences show no apparent correlation with disparate impact, while set size differences correlate strongly with it.
- 600 participants (200 per task, 50 per treatment) gave about 30,000 responses across three tasks; the conditional treatment had the largest set-size gap and most disparate impact in each task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Conditional instead of Marginal prediction sets | Human decisions assisted by conformal prediction sets · FACET | maxROR: 1.26 -> 1.51 | Hurt |
| Use Conditional instead of Marginal prediction sets | Human decisions assisted by conformal prediction sets · BiosBias | maxROR: 1.12 -> 1.33 | Hurt |
| Use Conditional instead of Marginal prediction sets | Human decisions assisted by conformal prediction sets · RAVDESS | maxROR: 1.01 -> 1.28 | Hurt |
| Provide Marginal prediction sets rather than no model assistance | Human classification decisions · FACET | accuracy improvement across groups | Mixed |
| Provide prediction sets rather than no model assistance | Human classification decisions · BiosBias | accuracy improvement across groups | Mixed |
| Provide Avg-k or Marginal prediction sets rather than no model assistance | Human classification decisions · RAVDESS | disparate impact: 1.02; 1.01 | No clear effect |
| Equalize set sizes across groups instead of equalizing coverage | Human decision pipelines using prediction sets · FACET, BiosBias, and RAVDESS | disparate impact | Helped |
Surprising findings
- Equalizing coverage across groups increased disparate impact compared with marginal sets, contrary to the prevailing fairness assumption.
- Experiments with larger coverage differences (e.g., RAVDESS-Marginal) sometimes had the fairest outcomes.
Limitations
- Some group-level statistics were not significant at 5% due to limited observations per group, especially FACET's Older and Unknown groups.
- Only classification tasks were studied; regression, time-series, and NLP settings were not tested.
- Each task used a single model and a forced-choice setup with a 50-trial test per participant.
Experimental setup
Datasets: FACET BiosBias RAVDESS
Benchmarks: FACET BiosBias RAVDESS
Metrics: accuracy odds ratio maximum ratio of odds ratios (maxROR) coverage difference set size difference top-1 accuracy
Baselines: marginal conformal prediction avg-k set prediction control (no assistance)
Builds on: conformal prediction Mondrian conformal prediction RAPS SAPS average-k set prediction
Closest papers by meaning
- On the Burden of Achieving Fairness in Conformal Prediction (NeurIPS 2026 · similarity 0.90)
- A Generic Framework for Conformal Fairness (ICLR 2025 · similarity 0.87)
- Conformal Classification with Equalized Coverage for Adaptively Selected Groups (NeurIPS 2024 · similarity 0.87)
- Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making (NeurIPS 2024 · similarity 0.86)
- Conformal Prediction Sets Improve Human Decision Making (ICML 2024 · similarity 0.85)
- Counterfactual Fairness by Combining Factual and Counterfactual Predictions (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.