Research map / Reinforcement learning / Visual and representation RL
DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DrS, which learns reusable dense rewards for multi-stage robot manipulation tasks by training per-stage success-vs-failure discriminators from sparse rewards and stage indicators, then reuses them to train SAC agents on unseen ManiSkill task variants.
Problem
Human-engineered dense rewards require heavy domain expertise and trial and error, and existing learned rewards (AIL discriminators, offline classifiers) are not reusable across tasks within a task family.
Approach
Train discriminators to separate success from failure trajectories (instead of agent from demonstration trajectories) so the gap does not vanish at convergence; for multi-stage tasks, train one discriminator per stage and combine them as k + alpha*tanh(D_k(s')) with stage indicators used only during training.
Key results
- Learned DrS rewards outperform semi-sparse rewards and VICE-RAQ/ORIL across all three ManiSkill task families when reused on unseen test objects (5 seeds).
- Learned rewards achieve performance comparable to human-engineered rewards on Pick-and-Place and Turn Faucet.
- VICE-RAQ and ORIL obtain no success on any task; semi-sparse rewards yield limited success within the training budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reuse DrS rewards instead of semi-sparse rewards | SAC · Pick-and-Place, Turn Faucet, and Open Cabinet Door test tasks | success rate | Helped |
| Reuse DrS rewards instead of human-engineered rewards | SAC · Pick-and-Place and Turn Faucet test tasks | performance | No clear effect |
| Use semi-sparse rewards instead of DrS rewards | SAC · Pick-and-Place, Turn Faucet, and Open Cabinet Door test tasks | success rate | Hurt |
| Use VICE-RAQ rewards instead of DrS rewards | SAC · Pick-and-Place, Turn Faucet, and Open Cabinet Door test tasks | success rate | Hurt |
| Use ORIL rewards instead of DrS rewards | SAC · Pick-and-Place, Turn Faucet, and Open Cabinet Door test tasks | success rate | Hurt |
| Reduce the Pick-and-Place stage configuration from 3 stages to 2 (ablation) | DrS · Pick-and-Place test tasks | sample efficiency | Hurt |
| Reduce the Pick-and-Place stage configuration from 3 stages to 1 (ablation) | DrS · Pick-and-Place test tasks | success rate | Hurt |
| Preserve the stationary-object stage rather than the approach-object stage when using 2 stages (ablation) | DrS · Pick-and-Place test tasks | performance | Helped |
| Increase the object-to-goal stage threshold from 2.5 cm to 5cm or 10cm (ablation) | DrS · Pick-and-Place test tasks | RL training efficiency | No clear effect |
| Fine-tune the reward-learning policy with DrS rewards rather than human-engineered rewards (ablation) | DrS byproduct policy · New tasks | performance | Helped |
| Use semi-sparse rewards to fine-tune the reward-learning policy instead of DrS rewards (ablation) | DrS byproduct policy · New tasks | performance | Hurt |
| Use point cloud inputs instead of state inputs for the reward function (ablation) | DrS · Pick-and-Place test tasks | performance | No clear effect |
Surprising findings
- Discriminator-based rewards from GAIL, even with stage indicators, and from one-stage success/failure learning without stage indicators fail to be reusable on test tasks.
- Fine-tuning the byproduct policy with the learned reward slightly outperforms the human-engineered reward.
Limitations
- Requires stage indicators during RL training, and their acquisition for real-world training is not investigated.
- Evaluated only on simulated ManiSkill manipulation families (plus simple navigation), with limited task diversity per family.
- Reliance on success signals; VICE-RAQ and ORIL fail, showing sensitivity to learning setup.
Experimental setup
Datasets: ManiSkill YCB EGAD PartNet-Mobility
Benchmarks: ManiSkill Pick-and-Place ManiSkill Turn Faucet ManiSkill Open Cabinet Door
Metrics: success rate sample efficiency
Baselines: Human-engineered reward Semi-sparse reward VICE-RAQ ORIL GAIL w/ stage indicators Ours w/o stage indicators
Builds on: GAIL SAC VICE ORIL ManiSkill
Closest papers by meaning
- Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning (ICML 2025 · similarity 0.89)
- Learning Process Rewards via Visitation Matching for Efficient RL (NeurIPS 2026 · similarity 0.87)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.86)
- Reward Machines for Deep RL in Noisy and Uncertain Environments (NeurIPS 2024 · similarity 0.86)
- Subtask-Aware Visual Reward Learning from Segmented Demonstrations (ICLR 2025 · similarity 0.86)
- Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.