Research map / Reinforcement learning / Offline RL
Uncertainty-Guided Reward Labeling for Reinforcement Learning under Limited Feedback
Renhao Zhang, Shreyas Chaudhari, Bruno Silva
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an uncertainty-guided method for choosing which offline samples to reward-label under limited feedback in RL.
Problem
In reinforcement learning from limited feedback (RLLF), only a small fraction of an offline dataset can be reward-labeled, so the question is which samples to label to learn a strong policy. Prior work treats downstream policy learning as a black box.
Approach
Selects reward-labeling samples using uncertainty guidance, coupling the selection stage with downstream policy learning rather than treating policy learning as a black box (based on abstract only; details beyond the abstract are not stated).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| multiply reachability by uncertainty-reduction instead of either alone (ablation) | SURE · maze2d-large | percentage feedback to reach 95%: 2.51% (Reach) -> 0.15% | Improved |
| multiply reachability by uncertainty-reduction instead of uncertainty alone (ablation) | SURE · halfcheetah | percentage feedback to reach 95%: 2.49% -> 0.71% | Improved |
| replace exact one-step gain with computable SURE score (ablation) | SURE · exact linear MDP | labels to reach 95%: 12 -> 12 | No clear effect |
| use SURE instead of uniform label selection | SURE · exact linear MDP (48 states) | labels to reach 95%: 132 -> 12 | Improved |
| use SURE instead of uniform selection with CQL backbone | SURE · HalfCheetah | percentage feedback to reach 90%: 2.44% -> 0.55% | Improved |
| use SURE instead of uniform selection with TD3+BC backbone | SURE · Maze2d-Medium | percentage feedback to reach 90%: 1.92% -> 0.14% | Improved |
| use multi-step successor reachability instead of one-step predecessor guidance | Reach vs RLLF-Guided · maze2d-medium | percentage feedback to reach 90%: 1.45% -> 0.50% | Improved |
| use SURE instead of IDRL | SURE · halfcheetah | percentage feedback to reach 95%: 6.18% -> 0.71% | Improved |
| use SURE instead of Reach at the 80% level | SURE · maze2d-medium | percentage feedback to reach 80%: 0.06% (Reach) vs 0.05% | No clear effect |
| use SURE instead of Uncertainty at 80% level | SURE · halfcheetah | percentage feedback to reach 80%: 0.31% (Uncertainty) vs 0.30% | No clear effect |
Experimental setup
Builds on: reward selection under limited feedback
Closest papers by meaning
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.89)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Deep Reinforcement Learning from Hierarchical Preference Design (ICML 2025 · similarity 0.89)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.89)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.89)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
