Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Robust Sampling for Active Statistical Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes robust sampling for active statistical inference that interpolates between uniform and uncertainty-based active sampling along a budget-preserving path, with a robust minimax constraint on estimated model error, guaranteeing variance no worse than either endpoint.
Problem
Active statistical inference with inaccurate model uncertainty scores can produce highly noisy estimates with variance worse than uniform sampling, so labeling budget may be wasted when the sampling rule is poorly calibrated.
Approach
Find a budget-preserving path (linear, geometric, or Hellinger geodesic) between the given active sampling rule and uniform sampling, choose the path parameter rho by minimizing an estimated variance, and make this choice robust via a minimax problem over bounded perturbations of the estimated squared model error.
Key results
- On Pew survey data with a poorly fit model, standard active sampling yields ESS below the uniform budget while robust active sampling outperforms both baselines across burn-in sizes.
- On the toy example with n=7000, n_h=1400, robust active (global) achieves ESS 1491 (+6.5%) versus uniform 1400 and active 1213 (-13.3%).
- Across social science text annotation tasks using GPT-4o confidences, the robust approach is essentially never worse than uniform or active inference, in some cases outperforming both by a large margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use robust active sampling with a global robustness constraint instead of uniform sampling | Active inference · Toy example | ESS: 1400 -> 1491 | Helped |
| Use robust active sampling with a structured hard-region constraint instead of uniform sampling | Active inference · Toy example | ESS: 1400 -> 1495 | Helped |
| Sample actively using the estimated error without robust optimization instead of sampling uniformly | Active inference · Toy example | ESS: 1400 -> 1213 | Hurt |
| Add a structured hard-region constraint instead of using the global robustness constraint (ablation) | Robust active sampling · Toy example | ESS: 1491 -> 1495 | Helped |
| Tune the sampling rule along a budget-preserving path (ablation) | Pew post-election survey · Pew post-election survey | effective sample size | Helped |
| Add robust optimization over the uncertainty set when sampling proportional to the estimated error (ablation) | Pew post-election survey · Pew post-election survey | effective sample size | Helped |
| Use the geometric path instead of the linear or Hellinger path (ablation) | Pew post-election survey · Pew post-election survey | effective sample size | Helped |
| Use robust active sampling instead of standard active sampling or uniform sampling | Robust active inference · US Census California 2019 age-income analysis | effective sample size | Helped |
| Use robust active sampling instead of standard active sampling or uniform sampling | Robust active inference · Political bias, politeness, and misinformation text annotation | effective sample size | Helped |
| Increase burn-in data used to fit the error function (ablation) | Active inference · Pew post-election survey | effective sample size | Helped |
| Use grid-search step size 0.1 instead of 0.01 for tuning the path parameter (ablation) | Robust active inference · Pew post-election survey | effective sample size | No clear effect |
Surprising findings
- Setting the sampling rule proportional to the estimated error without robustness yields a trivial rho=0, so the robust constraint is necessary in that case.
- Standard active inference with pi proportional to estimated error can have effective sample size below uniform sampling.
Limitations
- Theoretical guarantees are asymptotic; finite-sample validity shown only empirically.
- Performance is sensitive to the choice of uncertainty set C, which is tuned by cross-validation and may give sampling rules too close or far from uniform.
- Choice of budget-preserving path is heuristic; geometric path recommended without theoretical optimality.
Experimental setup
Datasets: Pew Research Center post-election survey 2020 American Community Survey PUMS California 2019 media political bias dataset politeness dataset (Wikipedia and StackExchange requests) news headline misinformation dataset GPT-4o annotations and verbalized confidences
Benchmarks: Pew post-election survey data US Census ACS data social science text annotation datasets
Metrics: effective sample size coverage variance
Baselines: uniform sampling (prediction-powered inference) standard active inference linear path Hellinger path
Builds on: active statistical inference prediction-powered inference augmented inverse probability weighting active learning
Closest papers by meaning
- Active Statistical Inference (ICML 2024 · similarity 0.91)
- Balanced Active Inference (NeurIPS 2025 · similarity 0.88)
- Uncertainty for Active Learning on Graphs (ICML 2024 · similarity 0.87)
- Post-ADC Inference: Valid Inference After Active Data Collection (NeurIPS 2026 · similarity 0.86)
- Uncertainty Herding: One Active Learning Method for All Label Budgets (ICLR 2025 · similarity 0.86)
- Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.