Research map / Label-efficient and robust learning / Recognition and long-tail
Class conditional conformal prediction for multiple inputs by p-value aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes class-conditional conformal prediction for multi-input (multiple images of one instance) classification by aggregating per-observation conformal p-values using their exact joint distribution, with score-based envelopes and refined majority voting.
Problem
Class-conditional conformal prediction sets become large when multiple observations of the same instance are available at test time and naive aggregation of softmax scores breaks exchangeability; classes with few calibration points make class-conditional calibration hard.
Approach
Perform class-conditional conformal p-values per observation, derive their exact joint distribution (with ties handled via randomization), and aggregate the p-value vector with envelope score functions (quantile, l_q-area, l_q distance to identity) calibrated by Monte Carlo; also refine majority voting with Beta-Binomial quantiles.
Key results
- On the synthetic data with m=20 observations and the l2-envelope method, average set size drops to 3.6 versus 6.6 for the original baseline at alpha=0.1.
- On LifeCLEF 2015 (K=688 classes), p-value aggregation methods give consistently narrower sets than majority vote after a few observations, while majority vote with Temp=1 tends to predict nearly all classes.
- The exchangeability assumption fails on the original LifeCLEF multi-input structure (correlated images from same user submission), causing coverage loss for p-value methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate the class-conditional conformal p-values with the ℓ2 envelope | Conformal prediction with multiple inputs · Synthetic data, m=20 | average prediction set size: 6.6 -> 3.6 | Helped |
| Use p-value aggregation instead of majority-vote aggregation | Multiple-input conformal prediction · Synthetic data | prediction set size | Helped |
| Replace the ordinary majority-vote threshold with the Beta-Binomial quantile | Majority-vote conformal prediction · Synthetic data | prediction set size | Helped |
| Approximate the Beta-Binomial quantile with a Binomial quantile | Majority-vote conformal prediction · Synthetic data, large m | class-conditional coverage | Hurt |
| Use temperature 1 rather than temperature 20 (ablation) | LifeCLEF prediction methods · LifeCLEF Plant Identification Task 2015 | prediction set size | Mixed |
| Replace the softmax score with the APS score (ablation) | P-value aggregation methods · LifeCLEF Plant Identification Task 2015 | prediction set size | No clear effect |
| Use p-value aggregation when multi-input observations violate exchangeability (ablation) | P-value aggregation methods · LifeCLEF Plant Identification Task 2015, correlated multi-inputs | coverage | Hurt |
| Use majority-vote methods instead of p-value aggregation under strong within-input dependence | Multiple-input conformal prediction · LifeCLEF Plant Identification Task 2015, correlated multi-inputs | coverage | Helped |
| Calibrate on aggregated multi-input scores rather than individual observations (ablation) | Class-conditional conformal prediction · LifeCLEF Plant Identification Task 2015 | prediction set length | Hurt |
Surprising findings
- Naive conformal prediction on aggregated softmax scores or aggregated calibration scores severely breaks exchangeability and yields excessively large or uninformative sets.
- On the original LifeCLEF multi-input data, the exchangeability assumption does not hold and p-value-based methods lose coverage, while majority-vote methods remain more robust.
Limitations
- Validity relies on exchangeability between multi-inputs and same-class calibration scores, which fails for strongly correlated images within one submission (e.g., LifeCLEF).
- Class-conditional calibration requires enough calibration points per class; classes with few points make sets uninformative, so the Binomial approximation loses coverage for large m or small classes.
- Simulation-based calibration of the envelope threshold adds Monte Carlo variability and computation.
Experimental setup
Datasets: LifeCLEF Plant Identification Task 2015 Pl@ntNet-300K synthetic Gaussian mixture
Benchmarks: LifeCLEF Plant Identification Task 2015 synthetic Gaussian mixture
Metrics: marginal coverage class-conditional coverage average prediction set size
Baselines: Majority vote Exchangeable majority Binomial quantile voting Naive aggregation of softmax scores
Builds on: split conformal prediction class-conditional conformal prediction conformal majority voting (Gasparin et al.) template methods for multiple testing
Closest papers by meaning
- Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation (NeurIPS 2025 · similarity 0.87)
- Conformity Score Averaging for Classification (ICML 2025 · similarity 0.87)
- Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.86)
- Conformal Classification with Equalized Coverage for Adaptively Selected Groups (NeurIPS 2024 · similarity 0.86)
- Conformal Prediction with Learned Features (ICML 2024 · similarity 0.86)
- Conformal Structured Prediction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.