Research map / Learning theory and algorithms / Prediction and decision losses
Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reliable prediction sets (RPS), combining hash-partitioned smoothed score functions over k_t classifiers with majority-vote merging of k_c calibration-partition prediction sets, to give provable pointwise coverage and size reliability under training and calibration poisoning.
Problem
Conformal prediction sets can be significantly altered by data poisoning of training and calibration data, leading to overly conservative or empty sets; existing reliability work covers only evasion or calibration-only coverage certification.
Approach
Partition training data by a deterministic hash so each point affects one classifier; use a smoothed vote-count score (softmax over vote fractions) for conformal scores; partition calibration data into k_c subsets, build a prediction set per subset, and take a binomial-quantile majority vote over them, yielding certifiable pointwise coverage and size reliability.
Key results
- Majority prediction sets with k_c=22 on CIFAR10 reach 90.2% empirical coverage and 0.94 average size, with over 93% coverage-reliable at r_c=4 poisoned calibration points.
- Smoothed score function with k_t=100 on CIFAR10 gives 90.7% coverage and 3.18 average size, guaranteeing over 34% coverage-reliable at r_t=4.
- Under r_t=3 training and r_c=3 calibration poisoning with k_t=100, k_c=40, over 31% of prediction sets remain coverage reliable at 92% coverage and 3.41 average size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Partition calibration data and merge HPS sets by majority vote | HPS on CIFAR10 · CIFAR10, calibration poisoning | coverage reliability ratio: over 93% remain reliable at rc=4 | Helped |
| Partition calibration data and merge HPS sets by majority vote | HPS on CIFAR10 · CIFAR10, calibration poisoning | size reliability ratio: over 87% remain size reliable at rc=15 | Helped |
| Use a smoothed score from classifiers trained on separate partitions | Conformal prediction on CIFAR10 · CIFAR10, training poisoning | coverage reliability ratio: over 34% remain reliable at rt=4 | Helped |
| Use majority calibration sets together with the smoothed score | Conformal prediction on CIFAR10 · CIFAR10, training and calibration poisoning | coverage reliability ratio: over 31% remain reliable at rt=3 and rc=3 | Helped |
| Smooth the voting function with a softmax instead of using the vote function π alone (ablation) | RPS score function · CIFAR10 | average set size | Helped |
| Use only 2 or 3 training partitions (ablation) | RPS smoothed score · Image classification | empirical coverage | Mixed |
| Increase training partitions (ablation) | RPS smoothed score · Image classification | average set size | Hurt |
| Train classifiers on partitions of CIFAR100 rather than the full training set (ablation) | ResNet18 · CIFAR100 | classifier accuracy: at least 70% -> approximately 30% | Hurt |
| Train with 10 partitions (ablation) | RPS smoothed score · CIFAR100 | average set size: above 42 | Hurt |
| Train with 100 partitions (ablation) | RPS smoothed score · CIFAR100 | average set size: 42.96 | Hurt |
| Fine-tune an ImageNet-pretrained model instead of training from scratch (ablation) | RPS smoothed score · CIFAR100, 100 partitions | average set size: 42.96 -> 18.16 | Helped |
| Use more than 4 training classifiers (ablation) | RPS smoothed score · Image classification | average set size | No clear effect |
Surprising findings
- Using only k_t=2 or 3 classifiers makes the smoothed score unstable, causing spikes in coverage and set size.
- Prior majority-vote uncertainty-set merging violates marginal coverage because its threshold uses a supremum, so the paper corrects the threshold to a discrete binomial quantile.
Limitations
- Training on subsets of larger datasets such as CIFAR100 or ImageNet causes large accuracy loss, leading to excessively large prediction sets (average size above 42 with 10 partitions on CIFAR100).
- Calibration partitioning requires n >= k_c(1/alpha - 1) calibration points, bounding the number of partitions and reducing size reliability for too many partitions.
- Training-poisoning guarantees are non-trivial only with many training partitions (e.g., k_t=100), and reliability ratios drop substantially at larger poisoning radii.
Experimental setup
Datasets: SVHN CIFAR10 CIFAR100 ImageNet (pretrained-model experiments)
Benchmarks: CIFAR10 CIFAR100 SVHN
Metrics: coverage reliability ratio size reliability ratio robustness ratio empirical coverage average set size
Baselines: homogeneous prediction sets (HPS) vanilla conformal prediction
Builds on: split conformal prediction homogeneous prediction sets deep partition aggregation Chernoff-style majority voting of uncertainty sets
Closest papers by meaning
- Robust Yet Efficient Conformal Prediction Sets (ICML 2024 · similarity 0.90)
- Robust Conformal Outlier Detection under Contaminated Reference Data (ICML 2025 · similarity 0.89)
- Data Poisoning Attacks against Conformal Prediction (ICML 2024 · similarity 0.88)
- Verifiably Robust Conformal Prediction (NeurIPS 2024 · similarity 0.88)
- Conformal Prediction with Learned Features (ICML 2024 · similarity 0.88)
- Direct Prediction Set Minimization via Bilevel Conformal Classifier Training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.