Research map / Probabilistic and scientific modeling / Molecules and proteins
Conformalized Multiple Testing after Data-dependent Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCPV, a selective conformal p-value method that adaptively picks a labeled calibration set matching the data-dependent selection rule, combined with BH to control FDR over the selected subset.
Problem
Controlling false discovery rate in conformalized multiple testing when the test units are first chosen by a data-dependent selection rule, which distorts the null distribution of conformal p-values and makes the selected set random.
Approach
Build selective conformal p-values by picking a calibration subset from the labeled data using the same selection rule, ensuring exchangeability between the selected test unit and calibration data; use BH on these p-values with FDR guarantees for strongly stable selection rules (joint-exchangeable, top-K) and extend via adaptive calibration and conditional calibration to weakly stable rules (mean
Key results
- On real datasets with target FDR 10%, SCPV keeps empirical FDR near 6-9% while achieving higher power than OMT (which inflates FDR to 13-25%) and AMT(BH) (more conservative), e.g., Census exchangeable setting power 15.3% vs 5.48% for AMT(BH).
- Under the mean selection rule, SCPV with adaptive p-values and conditional calibration controls FDR near the 10% target on synthetic Case A (about 9.9%) and Case B (about 9.8%).
- On the diabetes task (target FDR 20%), SCPV achieves 19.99% FDR with 63.99% power under constant threshold, while OMT exceeds the FDR target at 23.88%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCPV instead of OMT | Conformalized multiple testing · Abalone, EXCH | FDR: SCPV 6.68(0.75); OMT 24.8(0.51) | Helped |
| Use SCPV instead of AMT(BH) | Conformalized multiple testing · Abalone, EXCH | Power: SCPV 11.3(0.13); AMT(BH) 8.30(1.0) | Helped |
| Use SCPV instead of OMT | Conformalized multiple testing · Census, EXCH | FDR: SCPV 7.12(0.80); OMT 13.9(0.68) | Helped |
| Use SCPV instead of AMT(BH) | Conformalized multiple testing · Census, EXCH | Power: SCPV 15.3(1.5); AMT(BH) 5.48(0.86) | Helped |
| Use SCPV instead of OMT | Conformalized multiple testing · Credit, EXCH | FDR: SCPV 8.85(0.54); OMT 12.6(0.62) | Helped |
| Use SCPV instead of AMT(BH) | Conformalized multiple testing · Credit, EXCH | Power: SCPV 85.9(0.33); AMT(BH) 84.9(0.32) | Helped |
| Use SCPV instead of OMT | Conformalized multiple testing · Promotion, EXCH | FDR: SCPV 7.44(0.61); OMT 19.7(0.68) | Helped |
| Use SCPV instead of AMT(BH) | Conformalized multiple testing · Promotion, EXCH | Power: SCPV 19.6(1.1); AMT(BH) 16.5(1.0) | Helped |
Limitations
- Theory requires i.i.d. labeled and test data; distribution shift is not handled.
- Guarantees require selection rules to be (strongly or weakly) stable; unstable rules such as clustering on test data only are not covered.
- Adjusted baselines (AMT, BY) are conservative and lose power; the method also relies on a holdout labeled set for calibration.
Experimental setup
Datasets: Abalone Census Credit card fraud Promotion Diabetes Health Indicators (BRFSS) synthetic regression data (Cases A and B)
Benchmarks: synthetic Case A/B Abalone Census Credit Promotion Diabetes Health Indicators
Metrics: FDR power
Baselines: OMT (ordinary multiple testing) AMT (BH) AMT (BY) SCOP SCA InfoSCOP
Builds on: conformal p-value (Bates et al.) Benjamini-Hochberg procedure SCOP (Bao et al.) InfoSCOP (Gazin et al.) conditional calibration (Fithian & Lei) boosted e-BH (Nguyen et al.)
Closest papers by meaning
- Multivariate Conformal Selection (ICML 2025 · similarity 0.89)
- Conditional Testing based on Localized Conformal $p$-values (ICLR 2025 · similarity 0.88)
- Calibration without labels in multiple testing (NeurIPS 2026 · similarity 0.87)
- Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach (ICML 2025 · similarity 0.87)
- Rectifying Conformity Scores for Better Conditional Coverage (ICML 2025 · similarity 0.86)
- Backward Conformal Prediction (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.