Research map / Learning theory and algorithms / Prediction and decision losses
Robust Conformal Outlier Detection under Contaminated Reference Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that conformal outlier detection with contaminated calibration data is conservative, and proposes Label-Trim, which spends a small labeling budget to annotate and remove outliers among the highest-scoring calibration points to recover power while approximately controlling type-I error.
Problem
Conformal outlier detection relies on a clean inlier reference set for type-I error control, but practical reference sets contain a small fraction of unlabeled outliers, which makes p-values conservative and reduces power.
Approach
Use a pre-trained outlier detector to rank calibration points, spend a limited labeling budget to annotate the top-scoring ones, and remove only those confirmed as outliers before computing conformal p-values; theory shows approximate validity when the budget is small relative to the calibration size.
Key results
- On six visual datasets at alpha=0.01 with 5% contamination, Label-Trim reaches relative power 1.786 vs 1.0 for Standard, with type-I error 0.008, while Naive-Trim inflates type-I error to 0.036.
- On the shuttle dataset, Label-Trim with a budget of m=50 achieves power nearly indistinguishable from the Oracle at low contamination rates.
- Standard conformal p-values are conservative under contamination, with type-I error decreasing as the outlier proportion grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Label-Trim instead of Standard, selectively labeling the 50 largest calibration scores | Conformal outlier detection with ReAct and ResNet-18 · Visual datasets, α=0.01, contamination 1% | Power (relative to Standard): 1.0 ($\pm$ 0.0317) -> 1.166 ($\pm$ 0.0336) | Helped |
| Use Label-Trim instead of Standard, selectively labeling the 50 largest calibration scores | Conformal outlier detection with ReAct and ResNet-18 · Visual datasets, α=0.01, contamination 3% | Power (relative to Standard): 1.0 ($\pm$ 0.0354) -> 1.517 ($\pm$ 0.042) | Helped |
| Use Label-Trim instead of Standard, selectively labeling the 50 largest calibration scores | Conformal outlier detection with ReAct and ResNet-18 · Visual datasets, α=0.01, contamination 5% | Power (relative to Standard): 1.0 ($\pm$ 0.0408) -> 1.786 ($\pm$ 0.0498) | Helped |
| Use Label-Trim instead of Standard, selectively labeling the 50 largest calibration scores | Conformal outlier detection with ReAct and ResNet-18 · Visual datasets, α=0.03, contamination 1% | Power (relative to Standard): 1.0 ($\pm$ 0.0174) -> 1.041 ($\pm$ 0.0177) | Helped |
| Use Label-Trim instead of Standard, selectively labeling the 50 largest calibration scores | Conformal outlier detection with ReAct and ResNet-18 · Visual datasets, α=0.03, contamination 3% | Power (relative to Standard): 1.0 ($\pm$ 0.0189) -> 1.139 ($\pm$ 0.019) | Helped |
| Use Label-Trim instead of Standard, selectively labeling the 50 largest calibration scores | Conformal outlier detection with ReAct and ResNet-18 · Visual datasets, α=0.03, contamination 5% | Power (relative to Standard): 1.0 ($\pm$ 0.0212) -> 1.215 ($\pm$ 0.0226) | Helped |
| Remove the labeling-guided selection and use a small clean reference set (ablation) | Label-Trim · Visual datasets, α=0.03, contamination 1% | Power (relative to Standard): 1.041 ($\pm$ 0.0177) -> 0.714 ($\pm$ 0.0448) | Hurt |
| Remove outliers by score rank without checking their labels (Naive-Trim) | Conformal outlier detection with contaminated reference data · Visual datasets, α=0.03, contamination 5% | Type-I error: Label-Trim 0.021 ($\pm$ 0.0006); Naive-Trim 0.052 ($\pm$ 0.0008) | Hurt |
Surprising findings
- Label-Trim still controls type-I error in practice for labeling budgets beyond the theoretical condition m <= alpha(n+1).
- Small-Clean, which uses the same labeling budget to build a small clean set, has lower power than the Standard contaminated-set method at some settings.
Limitations
- Label-Trim remains conservative when the labeling budget is small or the outlier detector is weak, leaving power gains on the table.
- Relies on actively collecting new annotations, which may be unavailable in some settings.
- Theory assumes i.i.d. inliers and test points from the same distribution.
Experimental setup
Datasets: shuttle credit card KDDCup99 CIFAR10 MNIST SVHN Texture Places365 TinyImageNet CIFAR100
Benchmarks: shuttle credit card KDDCup99 CIFAR10 inlier with MNIST, SVHN, Texture, Places365, TinyImageNet, CIFAR100 outliers
Metrics: empirical type-I error power relative power vs Standard
Baselines: Standard conformal Oracle (clean reference) Naive-Trim Small-Clean One-Class SVM Local Outlier Factor SCALE
Builds on: split conformal prediction weighted conformal prediction ReAct
Closest papers by meaning
- Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach (ICML 2025 · similarity 0.89)
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning (ICLR 2025 · similarity 0.89)
- Robust Conformal Prediction Using Privileged Information (NeurIPS 2024 · similarity 0.86)
- Contamination-Resilient Anomaly Detection via Adversarial Learning on Partially-Observed Normal and Anomalous Data (ICML 2024 · similarity 0.86)
- One Sample is Enough to Make Conformal Prediction Robust (NeurIPS 2025 · similarity 0.86)
- Robust Yet Efficient Conformal Prediction Sets (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.