Research map / Reinforcement learning / Policy gradient and actor-critic

Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Importance-weighted OPE estimators

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formulates budgeted allocation of counterfactual annotations from multiple noisy sources for DM+-IS off-policy evaluation and solves it with a majorization-minimization algorithm using DP subroutines.

Problem

Choosing which context-action pairs to annotate and which costly, biased, noisy annotation sources (experts, LLMs) to query under a budget so as to minimize OPE estimator MSE in contextual bandits.

Approach

Derives an allocation-dependent MSE decomposition for the DM+-IS estimator (bias and variance terms in the reward model), a first-annotation threshold and four annotation-value regimes, and an MM algorithm that majorizes the cross-action bias coupling so the surrogate separates by pair with DP across actions per context.

Key results

• Four-source MM allocation reduces scaled MSE from 84.23 to 66.90 (20.58%) on the synthetic clinical bandit relative to no annotation.
• On the ASSISTments LLM-annotated education bandit, MM reaches scaled MSE 79.062 vs 88.608 with no annotation (10.77% reduction).
• Greedy-initialized MM has mean optimality gap 0.05% vs Gurobi on 30 small instances, compared with 0.30% for max-gain greedy and 3.67% for cost-aware greedy.

Surprising findings

• Source-level bias does not bias the OPE estimate but enters variance through the reward model, so a biased source can still be worth buying in some regimes.
• Single-annotation gains can be uninformative in regimes where a source is locally harmful, so greedy allocation can miss joint gains.

Limitations

• Limited to contextual bandits, not sequential MDPs.
• Assumes known or pilot-estimated source bias and excess-variance profiles.
• ASSISTments experiment is semi-synthetic with LLM annotations, not real human annotations.

Experimental setup

Datasets: ASSISTments (6.1M student-problem interactions) synthetic clinical bandit

Benchmarks: synthetic clinical bandit ASSISTments-based semi-synthetic education bandit

Metrics: scaled MSE relative MSE reduction optimality gap to Gurobi

Baselines: Greedy (max-gain) Greedy (cost-aware) No annotation Gurobi exact MILP

Builds on: DM+-IS CANDOR APRIL majorization-minimization

Closest papers by meaning

• Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation (NeurIPS 2023 · similarity 0.91)
• Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits (ICLR 2025 · similarity 0.89)
• OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.87)
• Cross-Validated Off-Policy Evaluation (AAAI 2025 · similarity 0.86)
• A General Framework for Off-Policy Learning with Partially-Observed Reward (ICLR 2025 · similarity 0.86)
• Off-policy estimation with adaptively collected data: the power of online learning (NeurIPS 2024 · similarity 0.85)