Research map / Learning theory and algorithms / Prediction and decision losses
Active, anytime-valid risk controlling prediction sets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends risk controlling prediction sets (RCPS) to anytime-valid, active-labeling settings via betting e-processes, with label policies and predictors that reduce the labels needed to find the risk-controlling threshold.
Problem
Calibrating black-box model prediction sets to control risk when data arrive sequentially and labels are costly, so that the risk guarantee holds at all times under a label budget.
Approach
Builds anytime-valid RCPS by testing each candidate threshold with betting e-processes (supermartingales) whose increments use inverse-propensity-weighted labeled risk under a randomized labeling policy, plus a model-based control variate; derives log-optimal labeling policy and predictor under a label budget and regret bounds linking estimation error to wealth growth.
Key results
- On the simulation and ImageNet, the 'pretrain' and 'learned' label policies reach lower beta-hat than 'all' and 'oblivious' baselines at every number of labels queried.
- All methods keep the safety violation rate below the target alpha=0.05 on both simulation and ImageNet.
- Experiments use theta=0.1, alpha=0.05, and label budget B=0.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the pretrained-model-based active labeling policy instead of labeling all data | Active anytime-valid RCPS · Numerical simulation | average β̂ | Helped |
| Use the learned active labeling policy instead of labeling all data | Active anytime-valid RCPS · Numerical simulation | average β̂ | Helped |
| Use the pretrained-model-based active labeling policy instead of random sampling | Active anytime-valid RCPS · Numerical simulation | average β̂ | Helped |
| Use the learned active labeling policy instead of random sampling | Active anytime-valid RCPS · Numerical simulation | average β̂ | Helped |
| Use the pretrained-model-based active labeling policy instead of labeling all data | Active anytime-valid RCPS · ImageNet | average β̂ | Helped |
| Use the learned active labeling policy instead of labeling all data | Active anytime-valid RCPS · ImageNet | average β̂ | Helped |
| Use the pretrained-model-based active labeling policy instead of random sampling | Active anytime-valid RCPS · ImageNet | average β̂ | Helped |
| Use the learned active labeling policy instead of random sampling | Active anytime-valid RCPS · ImageNet | average β̂ | Helped |
| Use the learned policy instead of the pretrained policy | Active anytime-valid RCPS · ImageNet | average β̂ | No clear effect |
Limitations
- Assumes i.i.d. data; distribution shift is not handled.
- Requires a bounded label policy assumption (q bounded below by epsilon) and an existence condition for the optimal policy.
- Only evaluated on a simple simulation and ImageNet classification.
Experimental setup
Datasets: ImageNet simulated Bernoulli data
Benchmarks: ImageNet numerical simulation (FPR)
Metrics: average beta-hat vs labels queried safety violation rate final beta-hat
Baselines: all labeling oblivious random labeling
Builds on: RCPS (Bates et al.) testing by betting prediction-powered inference Ville's inequality online newton step
Closest papers by meaning
- Performative Prediction with Selective Labels (NeurIPS 2026 · similarity 0.86)
- Prediction-Powered Active Testing (NeurIPS 2026 · similarity 0.85)
- Regretful Decisions under Label Noise (ICLR 2025 · similarity 0.84)
- Performative Risk Control: Calibrating Models for Reliable Deployment under Performativity (NeurIPS 2025 · similarity 0.84)
- Uncertainty Herding: One Active Learning Method for All Label Budgets (ICLR 2025 · similarity 0.84)
- Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.