Research map / Probabilistic and scientific modeling / Molecules and proteins
Real-Time Selection Under General Constraints via Predictive Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes II-COS, a real-time sample selection procedure that uses predictive inference (local FDR estimates) to jointly control an individual cost constraint (e.g., FSR) and an interactive pairwise-similarity constraint over a stopping time.
Problem
Online sample selection from a sequential stream of covariates where responses are unobserved, requiring control of individual constraints (FSR or expected cost) and interactive pairwise constraints (e.g., expected similarity/diversity) simultaneously up to an arbitrary stopping time.
Approach
Estimate local FDR from predicted responses via a two-group model and a calibration split, then make sequential accept/reject decisions that check both a running individual-cost ratio and a running pairwise-interaction ratio against thresholds, extending SAST-style online FDR control to general constraints.
Key results
- On the Census income data, II-COS achieved FSR 0.16 and ES 5.56e-3 versus SAST's FSR 0.19 and ES 30.90e-3, with II-COS keeping ES under the target K=6e-3.
- On synthetic classification with ncal=4000, II-COS controlled ES below K=0.045 at stopping time while LOND, SAFFRON and ADDIS did not control ES.
- Conformal-p-value online FDR methods (LOND, SAFFRON, ADDIS) suffered alpha-death, selecting far fewer than m=100 samples with small calibration sets, while II-COS always reached m.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use II-COS instead of SAST | Online selection on Candidate dataset · Candidate dataset | ES (×10^-3): 8.73 -> 0.98 | Helped |
| Use II-COS instead of SAST | Online selection on Candidate dataset · Candidate dataset | Tm: 310 -> 2227 | Hurt |
| Use II-COS instead of SAST | Online selection on Income dataset · Income dataset | ES (×10^-3): 30.90 -> 5.56 | Helped |
| Use II-COS instead of SAST | Online selection on Income dataset · Income dataset | Tm: 1200 -> 2760 | Hurt |
| Disable the interactive constraint by setting K = +∞ (ablation) | II-COS · Synthetic classification | ES: 0.044 -> 0.073 | Hurt |
| Disable the individual constraint by setting α = 1 (ablation) | II-COS · Synthetic classification | FSR: 0.102 -> 0.849 | Hurt |
| Disable the individual constraint by setting α = 1 (ablation) | II-COS · Synthetic classification | Tm: 628.82 -> 107.79 | Helped |
| Vary calibration size from ncal = 200 to ncal = 800 (ablation) | II-COS · Synthetic classification | FSR (%): 9.22(0.73) -> 9.36(0.69) | No clear effect |
| Vary calibration size from ncal = 200 to ncal = 800 (ablation) | II-COS · Synthetic classification | ES (×10^-2): 4.41(0.08) -> 4.40(0.08) | No clear effect |
| Vary calibration size from ncal = 200 to ncal = 800 (ablation) | II-COS · Synthetic regression | FSR (%): 5.74(0.18) -> 5.79(0.19) | No clear effect |
| Vary calibration size from ncal = 200 to ncal = 800 (ablation) | II-COS · Synthetic regression | ES (×10^-2): 1.49(0.03) -> 1.49(0.04) | No clear effect |
Surprising findings
- Conformal-p-value-based online FDR methods frequently stop early (alpha-death) and select very few samples when calibration sets are small.
- The offline CP baseline on the income dataset had FSR of 0.42, well above the 0.2 target.
Limitations
- Interactive constraints are restricted to pairwise binary functions; other constraints such as the Gini index are not handled.
- Theoretical guarantees hold only asymptotically in the calibration size and require Hölder continuity and density lower bounds that may fail when classifier probabilities concentrate near 0 or 1.
- Feedback after decisions is not incorporated.
Experimental setup
Datasets: synthetic Gaussian classification and regression data Kaggle recruitment dataset 1994 US Census income dataset
Benchmarks: synthetic classification synthetic regression Kaggle recruitment Census income
Metrics: FSR expected similarity (ES) stopping time Tm expected cost (EC)
Baselines: SAST LOND SAFFRON ADDIS CP (offline conformal p-value selection)
Builds on: SAST (structure-adaptive sequential testing) alpha-investing conformal p-values LOND local FDR
Closest papers by meaning
- Statistical Inference Under Constrained Selection Bias (ICML 2024 · similarity 0.86)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.84)
- Compute-Constrained Data Selection (ICLR 2025 · similarity 0.83)
- Towards a statistical theory of data selection under weak supervision (ICLR 2024 · similarity 0.83)
- Online Prediction with Limited Selectivity (NeurIPS 2025 · similarity 0.83)
- Online Algorithms with Uncertainty-Quantified Predictions (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.