Research map / Label-efficient and robust learning / Recognition and long-tail
Conformal Classification with Equalized Coverage for Adaptively Selected Groups
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptively Fair Conformal Prediction (AFCP), which automatically selects at most one sensitive attribute per test point via a leave-one-out bias test and builds prediction sets with valid coverage conditional on that adaptively chosen group.
Problem
Conformal classification sets with marginal coverage can undercover groups affected by algorithmic bias, while exhaustive equalized coverage over all sensitive attributes yields overly conservative, uninformative sets when calibration subsets become small.
Approach
For each test point, use a leave-one-out procedure with placeholder labels to select the sensitive attribute whose worst-off group has the highest miscoverage (only if a one-sided t-test flags bias), then calibrate conformal sets within that group while always containing the marginal set, giving adaptive equalized coverage on the selected attribute.
Key results
- On synthetic data, AFCP achieves valid coverage for the Blue group with prediction sets not much larger than the marginal benchmark, while the exhaustive benchmark is overly conservative.
- On Nursery data, AFCP produces smaller prediction sets than the partial equalized benchmark while maintaining valid coverage for the hardest group (Parents' occupation level one).
- AFCP consistently selects the Color attribute as the sample size grows, correctly identifying the biased group in the synthetic experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AFCP instead of marginal conformal prediction | Conformal classification · Synthetic medical diagnosis (Blue group) | conditional coverage | Helped |
| Use AFCP instead of marginal conformal prediction | Conformal classification · Synthetic medical diagnosis | average prediction set size | Mixed |
| Use AFCP instead of exhaustive equalized conformal prediction | Conformal classification · Synthetic medical diagnosis | average prediction set size | Helped |
| Use AFCP instead of partial equalized conformal prediction | Conformal classification · Nursery | prediction set size | Helped |
| Use AFCP instead of marginal conformal prediction | Conformal classification · Nursery (Parents' occupation, level one) | conditional coverage | Helped |
| Use AFCP instead of exhaustive equalized conformal prediction | Conformal classification · Nursery | prediction set size | Helped |
| Always select one sensitive attribute, regardless of the significance test (AFCP1) (ablation) | AFCP · Synthetic medical diagnosis (small sample sizes) | conditional coverage | Helped |
| Always select one sensitive attribute, regardless of the significance test (AFCP1) (ablation) | AFCP · Synthetic medical diagnosis (Color-Blue group) | conditional coverage | Helped |
Surprising findings
- AFCP can fail to select any attribute at small sample sizes, leading to slight undercoverage for the biased group, though it still outperforms marginal prediction.
Limitations
- Does not always identify the most relevant sensitive attribute, especially with limited sample sizes.
- Cannot guarantee equalized coverage for all sensitive groups simultaneously, only for the selected attribute.
- Selection stability and correctness are shown empirically, without theoretical guarantees of correct selection.
Experimental setup
Datasets: Nursery COMPAS Adult Income synthetic medical diagnosis data
Benchmarks: synthetic medical diagnosis data Nursery
Metrics: marginal coverage conditional coverage by protected group average prediction set size
Baselines: marginal conformal prediction exhaustive equalized coverage partial equalized coverage AFCP1
Builds on: equalized coverage conformal prediction adaptive conformity scores (APS) split conformal prediction
Closest papers by meaning
- Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.89)
- A Generic Framework for Conformal Fairness (ICLR 2025 · similarity 0.88)
- Valid Selection among Conformal Sets (NeurIPS 2025 · similarity 0.88)
- Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration (NeurIPS 2024 · similarity 0.87)
- Probabilistic Conformal Prediction with Approximate Conditional Validity (ICLR 2025 · similarity 0.87)
- Conformity Score Averaging for Classification (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.