Research map / Probabilistic and scientific modeling / Bayesian optimization
Post-ADC Inference: Valid Inference After Active Data Collection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes post-ADC inference, a selective-inference framework that conditions on the joint selection event of active data collection (GP-UCB, TPE) and data-driven target construction to give exact valid p-values and confidence intervals for Gaussian noise.
Problem
Statistical inference on data collected adaptively by sequential model-based optimization is invalid when the inferential target is also constructed from the same data after collection, causing inflated type-I error and undercoverage.
Approach
Treats the entire pipeline (ADC trajectory plus data-dependent target weights) as one selection event and conditions on it; under Gaussian noise the resulting conditional law of the linear test statistic is a truncated normal on a line, giving exact selective p-values and CIs without assumptions on the black-box function or surrogate.
Key results
- Under the global null, post-ADC stays near nominal type-I error and 90% coverage across horizons while naive and partially conditioned ablations are anti-conservative and Bonferroni is conservative.
- On four UCI datasets with UCB, post-ADC rejection probability reaches 0.432 (Concrete), 0.995 (Gas Turbine CO), 0.911 (Gas Turbine NOx), 0.999 (Power Plant), while Bonferroni is near 0 on most.
- With TPE, post-ADC rejection probability is 0.396 (Concrete), 0.989 (CO), 0.918 (NOx), 0.929 (Power Plant), versus Bonferroni at 0.000-0.058.
Surprising findings
- Bonferroni correction over all possible trajectories and region selections is essentially powerless on real datasets.
Limitations
- Exact theory requires a finite candidate set, so continuous domains need discretization.
- Assumes known Gaussian noise variance; estimated variance or non-Gaussian noise remains open.
- Applies only to ADC rules and target constructions expressible as compositions of finitely many piecewise-linear maps, not every ADC algorithm.
Experimental setup
Datasets: Concrete Gas Turbine (CO) Gas Turbine (NOx) Power Plant synthetic objective families (sinc, cos, chirp, bump, peak, negative_forrester)
Benchmarks: synthetic objective families under global null UCI regression datasets as black-box objectives
Metrics: type-I error (rejection probability at alpha=0.05) coverage of 90% intervals power rejection probability
Baselines: naive Z-test Bonferroni correction w/o-eta ablation w/o-T ablation
Builds on: selective inference tree-structured Parzen estimator GP-UCB
Closest papers by meaning
- Active Statistical Inference (ICML 2024 · similarity 0.88)
- Robust Sampling for Active Statistical Inference (NeurIPS 2025 · similarity 0.86)
- Task-Agnostic Machine-Learning-Assisted Inference (NeurIPS 2024 · similarity 0.84)
- ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition (NeurIPS 2025 · similarity 0.84)
- Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization (ICML 2025 · similarity 0.84)
- ABC3: Active Bayesian Causal Inference with Cohn Criteria in Randomized Experiments (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.