Research map / Reinforcement learning / Policy gradient and actor-critic
Policy Learning for Balancing Short-Term and Long-Term Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes policy learning that balances short-term and long-term rewards when long-term outcomes can be missing, deriving efficient (doubly/quadruply robust) estimators and a policy learning approach with regret rates.
Problem
Learning treatment policies that balance short-term and long-term rewards under confounding and missing long-term outcomes that depend on short-term outcomes.
Approach
Defines a weighted objective V(π;s)+λV(π;y) with identifiability under a missing-at-random-given-(X,A,S) assumption, derives efficient influence functions and semiparametric efficiency bounds, and builds doubly/quadruply robust estimators used for policy learning; shows short-term outcomes improve long-term reward estimation.
Key results
- On JOBS and PRODUCT (50 trials, missing ratio 0.1, T=10), Proposed (Ours, λ=0.5) achieves the highest balanced reward on JOBS (2895.4 vs 2845.4 Naive-S and 2866.2 Naive-Y), and on PRODUCT (5183.4 vs 5176.6 and 5171.8).
- Theory: efficiency bound of long-term reward is lower when short-term outcomes are associated with Y given X; estimators are doubly robust for short-term reward and quadruply robust for long-term reward.
- Regret and estimation error of the learned policy are O_P(1/sqrt(n)) for parametric policy classes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Balance short-term and long-term rewards instead of optimizing short-term reward alone | Proposed policy learning · JOBS | Balanced reward: 2845.4 -> 2895.4 | Helped |
| Balance short-term and long-term rewards instead of optimizing long-term reward alone | Proposed policy learning · JOBS | Balanced reward: 2866.2 -> 2895.4 | Helped |
| Balance short-term and long-term rewards instead of optimizing short-term reward alone | Proposed policy learning · PRODUCT | Balanced reward: 5176.6 -> 5183.4 | Helped |
| Balance short-term and long-term rewards instead of optimizing long-term reward alone | Proposed policy learning · PRODUCT | Balanced reward: 5171.8 -> 5183.4 | Helped |
| Use short-term outcome S associated with long-term outcome Y, rather than an unassociated S | Proposed policy learning · IHDP and JOBS | Balanced reward | Helped |
| Increase the missing ratio of long-term outcome Y | Naive-Y and proposed policy · IHDP and JOBS | Balanced reward | Hurt |
| Include short-term outcome S in estimation of the long-term reward | Long-term reward estimator · Long-term reward estimation | Semiparametric efficiency bound | Helped |
Surprising findings
- On JOBS, the Naive-S baseline scores best on the short-term reward for several estimators, and the OR Naive-S long-term reward beats the proposed method, showing the trade-off is not uniformly won by the proposed method.
Limitations
- Assumes no unmeasured confounders; the identification assumptions fail when unmeasured confounding affects treatment and outcomes.
- Experiments rely on simulated long-term outcomes and missingness, since the benchmarks lack long-term effects.
Experimental setup
Datasets: IHDP JOBS PRODUCT (KuaiRec)
Benchmarks: IHDP JOBS PRODUCT
Metrics: balanced reward welfare change (ΔW) policy error
Baselines: DM IPW OR Naive-S Naive-Y
Builds on: efficient influence function theory doubly robust estimation potential outcome framework
Closest papers by meaning
- Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards (NeurIPS 2024 · similarity 0.92)
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · similarity 0.88)
- Off-policy estimation with adaptively collected data: the power of online learning (NeurIPS 2024 · similarity 0.87)
- A General Framework for Off-Policy Learning with Partially-Observed Reward (ICLR 2025 · similarity 0.87)
- Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning (NeurIPS 2024 · similarity 0.86)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.