Research map / Reinforcement learning / Policy gradient and actor-critic
Adaptive Labeling for Efficient Out-of-distribution Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adaptive labeling as an MDP over posterior beliefs, optimized with autodiff pathwise policy gradients through simulated one-step lookahead rollouts, to evaluate models under out-of-support selection bias.
Problem
Evaluating ML model performance under severe selection bias and out-of-support distribution shift when ground-truth labels are costly, so one must choose which inputs to label in batches to minimize posterior uncertainty about model error.
Approach
Formulates batch label selection as an MDP whose states are posterior beliefs over the data-generating function, and optimizes a differentiable K-subset sampling policy with pathwise (autodiff) gradients through simulated posterior updates instead of high-variance REINFORCE; the framework is agnostic to the UQ method (GPs or Ensemble+).
Key results
- On synthetic data, Autodiff 1-lookahead gives substantial improvements over active learning baselines and more robust performance than REINFORCE across horizons.
- On eICU, planning policies (REINFORCE and autodiff) consistently outperform heuristics by a large margin, with active learning baselines sometimes worse than random.
- With Ensemble+ on a toy setting, REINFORCE and Autodiff 1-lookahead both reach variance 1697.1 versus 7129.8 for random sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Autodiff 1-lookahead instead of random sampling | Ensemble+ adaptive labeling · Ensemble+ toy setting | Variance of ℓ2 loss estimate: 7129.8 ± 1027.0 -> 1697.1 ± 0.0 | Helped |
| Use Autodiff 1-lookahead instead of random sampling | Ensemble+ adaptive labeling · Ensemble+ toy setting | Error of ℓ2 loss estimate: 136.2 ± 8.28 -> 45.27 ± 0.0 | Helped |
| Use Autodiff 1-lookahead instead of static uncertainty sampling | Ensemble+ adaptive labeling · Ensemble+ toy setting | Variance of ℓ2 loss estimate: 10852 ± 0.0 -> 1697.1 ± 0.0 | Helped |
| Use Autodiff 1-lookahead instead of static uncertainty sampling | Ensemble+ adaptive labeling · Ensemble+ toy setting | Error of ℓ2 loss estimate: 162.156 ± 0.0 -> 45.27 ± 0.0 | Helped |
| Use Autodiff 1-lookahead instead of sequential uncertainty sampling | Ensemble+ adaptive labeling · Ensemble+ toy setting | Variance of ℓ2 loss estimate: 8585.5 ± 898.9 -> 1697.1 ± 0.0 | Helped |
| Use Autodiff 1-lookahead instead of sequential uncertainty sampling | Ensemble+ adaptive labeling · Ensemble+ toy setting | Error of ℓ2 loss estimate: 144 ± 6.93 -> 45.27 ± 0.0 | Helped |
| Replace REINFORCE policy gradients with pathwise gradients in Autodiff 1-lookahead | Ensemble+ adaptive labeling · Ensemble+ toy setting | Variance of ℓ2 loss estimate: 1697.1 ± 0.0 -> 1697.1 ± 0.0 | No clear effect |
| Replace REINFORCE policy gradients with pathwise gradients in Autodiff 1-lookahead | Ensemble+ adaptive labeling · Ensemble+ toy setting | Error of ℓ2 loss estimate: 45.27 ± 0.0 -> 45.27 ± 0.0 | No clear effect |
Surprising findings
- Active learning heuristics perform poorly in small-horizon batched problems and can be worse than random sampling on eICU.
- One-step lookahead remains surprisingly effective even at long horizons.
Limitations
- Deep ensemble UQ requires retraining across sampled actions, which is computationally expensive; only a toy Ensemble+ experiment is shown.
- Posterior beliefs are hard to keep accurate as more data is collected.
- Assumes known GP hyperparameters in main experiments and a naive predictor psi = 0.
Experimental setup
Datasets: synthetic GP data (51 clusters) eICU
Benchmarks: synthetic GP data eICU
Metrics: variance of MSE under posterior error of estimated MSE vs oracle
Baselines: REINFORCE uncertainty sampling (static) uncertainty sampling (sequential) random sampling
Builds on: REINFORCE K-subset sampling (Xie and Ermon) Ensemble+ knowledge gradient
Closest papers by meaning
- Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging (NeurIPS 2026 · similarity 0.86)
- Uncertainty Herding: One Active Learning Method for All Label Budgets (ICLR 2025 · similarity 0.86)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.86)
- From Biased Selective Labels to Pseudo-Labels: An Expectation-Maximization Framework for Learning from Biased Decisions (ICML 2024 · similarity 0.85)
- Performative Prediction with Selective Labels (NeurIPS 2026 · similarity 0.85)
- Enhancing Treatment Effect Estimation via Active Learning: A Counterfactual Covering Perspective (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.