Research map / Probabilistic and scientific modeling / Molecules and proteins
Active Measurement: Efficient Estimation at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces active measurement, a human-in-the-loop framework that combines importance-sampled human labeling of units, iterative AI model adaptation, and new weighting schemes and variance estimators to produce unbiased, calibrated estimates of total scientific measurements.
Problem
Scientific measurement workflows with AI models (e.g., counting birds, roosts, malaria cells, damaged buildings) lack accuracy and statistical guarantees, and fully automated estimates are biased or imprecise without costly validation.
Approach
Use an AI model's predictions as an importance-sampling proposal for sampling units without replacement for human labeling, iteratively adapt the model with new labels, and combine per-step unbiased estimates with novel weighting (COMB, conditional inverse variance) and martingale-based variance estimators to obtain valid confidence intervals.
Key results
- On the sky and reeds images, active measurement (DIS+AIS+WOR) outperforms DISCount with as little as 1% of tiles labeled, averaged over 10,000 runs.
- On the radar task, fine-tuning reduces the raw detector's fractional error from 3.78 to about 2, and active measurement continues improving and beats DISCount at all 11 stations.
- The COMB weighting is proven within a factor 9/8 of optimal variance, and conditional inverse-variance weighting with gamma=0.5 can achieve lower error than COMB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use active measurement (adaptive importance sampling with sampling without replacement) instead of DISCount | Scientific counting with object detectors · Tree Swallow sky and reeds | fractional error | Helped |
| Add sampling without replacement (ablation) | DIS+AIS · Tree Swallow sky and reeds | fractional error | Helped |
| Fine-tune on station-specific data | Radar detector · Great Lakes radar stations | fractional error: 3.78 to approximately 2 | Helped |
| Use active measurement instead of DISCount | Radar counting · Great Lakes radar stations | fractional error | Helped |
| Use COMB weights instead of LURE weights (ablation) | Active measurement · Tree Swallow sky and reeds | fractional error | Helped |
| Use COMB weights instead of SQRT weights (ablation) | Active measurement · Tree Swallow sky and reeds | fractional error | Helped |
| Use inverse variance weights with γ=0.3 instead of COMB weights (ablation) | Active measurement · Tree Swallow sky and reeds | fractional error | No clear effect |
| Use inverse variance weights with γ=0.5 instead of COMB weights (ablation) | Active measurement · Tree Swallow sky and reeds | fractional error | Helped |
| Use inverse variance weights with γ=0.9 instead of COMB weights (ablation) | Active measurement · Tree Swallow sky and reeds | fractional error | Hurt |
| Adapt the detector and use sampling without replacement (ablation) | DIS baseline · Malaria Cell BBBC041v1 | fractional error | Helped |
| Adapt the detector and use sampling without replacement (ablation) | DIS baseline · Palu Tsunami xBD | fractional error | Helped |
| Use the proposed variance estimators with adaptation rather than DISCount | Radar confidence intervals · Radar counting | CI coverage | Helped |
Surprising findings
- Inverse-variance weighting with estimated variances, previously advised against in adaptive importance sampling, can outperform the COMB weights when gamma is chosen moderately (0.5).
- Detector fine-tuning saturates quickly for the raw detector, yet the proposal distribution keeps improving, which continues to benefit the AIS estimator.
Limitations
- Estimates may not be precise enough for all high-stakes applications; careful validation required.
- Model adaptation (fine-tuning deep detectors) incurs substantial computational overhead that limits interactivity.
- Inverse-variance weighting makes estimator biased and may hurt confidence interval coverage.
Experimental setup
Datasets: sky and reeds Tree Swallow images (925 and 1,426 tiles) weather radar roost annotations (Great Lakes stations, 2015-2019) BBBC041v1 Malaria Cell dataset xBD Palu Tsunami subset
Benchmarks: sky and reeds bird counting radar roost counting (11 Great Lakes stations) malaria-infected cell counting damaged building counting (xBD Palu)
Metrics: fractional error confidence interval coverage confidence interval width
Baselines: DISCount simple Monte Carlo MC without replacement raw detector predictions prediction-powered inference style estimator square root law weights LURE weights
Builds on: DISCount active testing adaptive importance sampling prediction-powered inference Faster R-CNN
Closest papers by meaning
- Robust Sampling for Active Statistical Inference (NeurIPS 2025 · similarity 0.84)
- Human Expertise in Algorithmic Prediction (NeurIPS 2024 · similarity 0.83)
- Active Target Discovery under Uninformative Priors: The Power of Permanent and Transient Memory (NeurIPS 2025 · similarity 0.83)
- Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges (AAAI 2025 · similarity 0.82)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.82)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.