Research map / Reinforcement learning / RL from feedback
A General Framework for Off-Policy Learning with Partially-Observed Reward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyPeR, an off-policy learning method that uses densely observed secondary rewards alongside partially observed target rewards to get a lower-variance, unbiased policy gradient, plus a bootstrap-based tuning of the mixing weight.
Problem
Off-policy learning in contextual bandits when the target reward is only partially observed (missing, delayed, censored), making policy gradient estimates high-variance; secondary rewards are dense but may be misaligned with the target.
Approach
A hybrid policy gradient estimator that combines a DR-style estimator using secondary rewards as a control variate with the partially observed target reward, remaining unbiased under the same conditions as r-DR; the mixing weight gamma is deliberately tuned (via bootstrapped validation) to trade bias for variance, even if it differs from the true objective weight beta.
Key results
- On synthetic and KuaiRec data, HyPeR(gamma=beta) gives the highest combined policy value in nearly all settings and beats r-DR and s-DR across observation probabilities, data sizes, and correlation levels.
- HyPeR with bootstrapped tuned gamma (Tuned gamma-hat*) consistently outperforms HyPeR(gamma=beta) and other feasible methods, especially with sparse observation and small data.
- Theory shows the HyPeR target-gradient estimator is unbiased under the same condition as r-DR and has variance no larger than r-DR when the secondary-reward q-hat is more accurate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyPeR with γ=β instead of the reported baselines | OPL with partially observed rewards · Synthetic data, varying target-reward observation probability | combined policy value | Helped |
| Use HyPeR with γ=β instead of the reported baselines | OPL with partially observed rewards · Synthetic data, varying target-reward observation probability | target policy value | Helped |
| Add secondary-reward information to the target-reward gradient estimator (ablation) | HyPeR(γ=0) · Synthetic data, varying target-reward observation probability | target policy value | Helped |
| Tune γ using the proposed data-driven procedure instead of setting γ=β (ablation) | HyPeR · Synthetic data, varying β | combined policy value | Helped |
| Tune γ using sampling with replacement rather than without replacement (ablation) | HyPeR weight tuning · Synthetic data, varying β | combined policy value | Helped |
| Use HyPeR(γ=β) instead of the reported baselines | OPL on KuaiRec · KuaiRec, varying target-reward observation probability, data size, and β | combined policy value | Helped |
| Tune γ instead of setting γ=β (ablation) | HyPeR · KuaiRec, especially sparse observation and small training data | combined policy value | Helped |
| Use only secondary rewards as surrogates for the target reward | s-IPS and s-DR · Synthetic data, varying target-secondary reward correlation | policy value | Mixed |
| Use secondary rewards as surrogates rather than combining their information with target rewards | s-IPS and s-DR · Synthetic data, varying secondary-reward noise | policy value | Hurt |
| Use HyPeR instead of methods relying exclusively on target rewards | OPL with partially observed rewards · Synthetic data, varying target-reward noise | policy value | Helped |
| Use HyPeR instead of the reported baselines | OPL with estimated observation probabilities · Synthetic data, noisy estimated p(o|x) | combined policy value | Helped |
| Use bootstrapping with replacement when tuning γ (ablation) | HyPeR weight tuning · Synthetic data, varying β | tuned γ: 0.5595 vs 0.3971 vs 0.4788; 0.6075 vs 0.5762 vs 0.5067; 0.7390 vs 0.7241 vs 0.5542; 0.8681 vs 0.8198 vs 0.6371; 0.9019 v | Mixed |
Surprising findings
- Deliberately using an incorrect mixing weight (gamma-hat* not equal to beta) can yield better combined policy value than the unbiased choice gamma=beta, because of variance reduction.
Limitations
- Real-world evaluation uses the semi-synthetic KuaiRec fully observed matrix with simulated partial observation and synthetic secondary-reward definitions; no live deployment.
- Tuning gamma requires a validation split and bootstrapping; the procedure's benefit is shown empirically without a finite-sample guarantee.
- Unbiasedness relies on full support of the logging policy and known or estimated observation probability p(o|x).
Experimental setup
Datasets: KuaiRec synthetic data
Benchmarks: KuaiRec synthetic contextual bandit simulations
Metrics: relative policy value (combined, target, secondary)
Baselines: r-IPS r-DR s-IPS s-DR r-DM s-DM HyPeR(gamma=0) DR with F(s,r)
Builds on: doubly robust estimator inverse propensity score policy gradient for off-policy learning
Closest papers by meaning
- Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits (ICLR 2025 · similarity 0.90)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.88)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.88)
- Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning (NeurIPS 2024 · similarity 0.87)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.87)
- Off-policy estimation with adaptively collected data: the power of online learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.