Research map / Label-efficient and robust learning / Recognition and long-tail

Prediction-Powered Active Testing

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Deep active learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Prediction-Powered Active Testing (PPAT), which combines the unbiased LURE estimator with a prediction-powered control variate from a cheap black-box proxy, plus a tailored acquisition rule and asymptotically valid confidence intervals for risk estimation.

Problem

Estimating a model's test-pool risk with few labels when cheap black-box predictions are available for the whole pool, while keeping unbiasedness and valid uncertainty estimates.

Approach

Use proxy predictions to residualise the loss (subtract a centred proxy loss scaled by lambda) before applying LURE importance weighting, so the estimator stays unbiased while variance drops as a control variate; derive a matching acquisition proposal for the residualised objective and an online lambda estimate.

Key results

• PPAT achieves lower median squared error than Random, LURE and ASE across the UCI regression datasets for all three lambda choices, with ASE sometimes better only at very small budgets (e.g. Keggundirected).
• On classification, PPAT beats baselines on CIFAR-10 and CIFAR-100 for all lambda; on Tiny-ImageNet PPAT with lambda=1 performs comparably to Random.
• PPAT confidence intervals reach the nominal 0.9 coverage at the same or faster rate than LURE and Random, with substantially narrower widths.

Surprising findings

• Random sampling outperforms LURE on the classification datasets, attributed to a poorly tracking surrogate inflating LURE's importance-weight variance.
• ASE results in biased estimates and was excluded from coverage experiments because it cannot provide asymptotically valid intervals.

Limitations

• Variance reduction depends critically on choosing lambda; wrong sign or large magnitude inflates variance.
• Variance bound is only tight when budget is small relative to pool and proposals are not concentrated; asymptotic normality rests on an assumed convergence condition.
• Plug-in lambda estimate introduces finite-sample bias; experiments use a fixed proxy and fixed surrogate per dataset.

Experimental setup

Datasets: Keggundirected Keggdirected Sml Bike CIFAR-10 CIFAR-100 Tiny-ImageNet

Benchmarks: Keggundirected Keggdirected Sml Bike CIFAR-10 CIFAR-100 Tiny-ImageNet

Metrics: median squared error mean error (bias) confidence interval coverage confidence interval width

Baselines: LURE ASE Random sampling without replacement

Builds on: LURE Active Surrogate Estimators prediction-powered inference PPI++

Closest papers by meaning

• Scaling Up Active Testing to Large Language Models (NeurIPS 2025 · similarity 0.88)
• Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression (ICML 2025 · similarity 0.86)
• Anytime-valid, Bayes-assisted, Prediction-Powered Inference (NeurIPS 2025 · similarity 0.85)
• Active Statistical Inference (ICML 2024 · similarity 0.85)
• Active, anytime-valid risk controlling prediction sets (NeurIPS 2024 · similarity 0.85)
• Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.85)