Research map / Learning theory and algorithms / Prediction and decision losses
A Generic Framework for Conformal Fairness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes Conformal Fairness and proposes a post-hoc framework that searches a conformal score threshold so that coverage gaps across sensitive groups (and labels) meet a fairness closeness criterion, without needing group labels at inference.
Problem
Conformal prediction gives marginal coverage guarantees that ignore sensitive attributes, so coverage can differ substantially across groups; existing group-conditional fair CP methods need group membership at test time or are model-specific and mostly limited to IID data.
Approach
Calibrate on group- (and label-) filtered calibration subsets, use exact-interval bounds on inverse-quantile coverage to compute per-group coverage for a fixed positive label, and pick the smallest threshold lambda from a search space whose worst pairwise coverage gap satisfies a fairness closeness criterion; relies only on exchangeability so it extends to graph data.
Key results
- On Credit with TPS, disparate impact rose from 0.252 (base) to 0.793 (CF), with efficiency going from 2.268 to 2.558.
- On ACSIncome with APS, the four-fifths disparate impact improved from 0.397 to 0.797, at a cost in efficiency from 2.212 to 2.674.
- On ACSEducation with the Predictive Parity Proxy at c=0.05, APS max fairness disparity was 0.044 versus a base average of 0.411.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CF to APS | APS conformal predictor · Credit | disparate impact: 0.646 -> 0.821 | Helped |
| apply CF to RAPS | RAPS conformal predictor · Credit | disparate impact: 0.586 -> 0.768 | Helped |
| apply CF to TPS | TPS conformal predictor · Credit | disparate impact: 0.252 -> 0.793 | Helped |
| apply CF to CFGNN | CFGNN conformal predictor · Credit | disparate impact: 0.922 -> 0.922 | No clear effect |
| apply CF to DAPS | DAPS conformal predictor · Credit | disparate impact: 0.539 -> 0.809 | Helped |
| apply CF to APS | APS conformal predictor · ACSIncome | disparate impact: 0.397 -> 0.797 | Helped |
| apply CF to RAPS | RAPS conformal predictor · ACSIncome | disparate impact: 0.387 -> 0.790 | Helped |
| apply CF to TPS | TPS conformal predictor · ACSIncome | disparate impact: 0.356 -> 0.798 | Helped |
| apply CF with Predictive Parity Proxy | APS conformal predictor · ACSEducation | Max Fairness Disparity: 0.411 -> 0.044 | Helped |
| apply CF with Predictive Parity Proxy | RAPS conformal predictor · ACSEducation | Max Fairness Disparity: 0.268 -> 0.043 | Helped |
| apply CF with Predictive Parity Proxy | TPS conformal predictor · ACSEducation | Max Fairness Disparity: 0.319 -> 0.038 | Helped |
| use classwise lambdas (ablation) | CF APS · ACSIncome | disparate impact: 0.815 -> 0.797 | Hurt |
Surprising findings
- The CF framework with a single shared lambda gave better fairness disparity than the classwise variant on ACSEducation, though with larger prediction sets.
- Intersectional fairness on Pokec-n sometimes left the CF disparity slightly above the closeness threshold c=0.1 due to small per-group calibration sets.
Limitations
- Intersectional fairness multiplies groups and classes, shrinking calibration subsets so guarantees weaken in practice (e.g., Pokec-n).
- Guarantees for predictive parity require assumptions on label distribution; the arbitrary-closeness result relies on a proxy metric unless labels are independent of group.
- Current scope is classification; regression is left to future work.
Experimental setup
Datasets: Pokec-n Pokec-z Credit ACSIncome ACSEducation
Benchmarks: Pokec-n Pokec-z Credit ACSIncome ACSEducation
Metrics: worst fairness disparity efficiency (average prediction set size) disparate impact
Baselines: APS RAPS TPS CFGNN DAPS BatchGCP
Builds on: conformal prediction APS RAPS TPS DAPS
Closest papers by meaning
- FedCF: Fair Federated Conformal Prediction (NeurIPS 2026 · similarity 0.92)
- On the Burden of Achieving Fairness in Conformal Prediction (NeurIPS 2026 · similarity 0.89)
- Conformal Classification with Equalized Coverage for Adaptively Selected Groups (NeurIPS 2024 · similarity 0.88)
- Conformal Prediction Sets Can Cause Disparate Impact (ICLR 2025 · similarity 0.87)
- An Information Theoretic Perspective on Conformal Prediction (NeurIPS 2024 · similarity 0.87)
- Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.