Research map / Reinforcement learning / Visual and representation RL
Episodic Return Decomposition by Difference of Implicitly Assigned Sub-trajectory Reward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diaster, an episodic return decomposition method that splits each trajectory at a cut point and derives step-wise proxy rewards from differences of implicitly assigned sub-trajectory rewards, implemented with a GRU and SAC.
Problem
Reinforcement learning with extremely delayed rewards, where only an episodic return is observed at episode end, leading to poor sample efficiency and inefficient long-horizon credit assignment.
Approach
Cut each trajectory at any time step into two sub-trajectories whose rewards sum to the episodic return; a GRU-based sub-trajectory reward function is learned, and step-wise proxy rewards are obtained as differences of sub-trajectory rewards in expectation, combining temporal structure with attribution.
Key results
- Diaster outperforms RUDDER on all seven MuJoCo tasks and generally shows better sample efficiency than RRD and IRCR in most environments.
- Only on HumanoidStandup does Diaster show a slight disadvantage relative to RRD and IRCR.
- Ablation over the number of cut points shows performance rises then falls as m grows from 0 to T-1, with m=1 or m=5 performing best across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Diaster instead of RUDDER | Diaster with SAC · MuJoCo: InvertedPendulum, Hopper, Swimmer, Walker2d, Ant, Humanoid, HumanoidStandup | episodic return | Helped |
| use Diaster instead of RRD or IRCR | Diaster with SAC · Most evaluated MuJoCo tasks | sample efficiency | Helped |
| use Diaster instead of RRD or IRCR | Diaster with SAC · HumanoidStandup | episodic return | Hurt |
| vary the number of cut points and use an intermediate value (ablation) | Diaster · MuJoCo: InvertedPendulum, Hopper, Swimmer, Walker2d | episodic return | Helped |
| use m=0 cut points, limiting reward attribution (ablation) | Diaster · MuJoCo: InvertedPendulum, Hopper, Swimmer, Walker2d | episodic return | Hurt |
| use m=T-1 cut points, limiting long-term representation (ablation) | Diaster · MuJoCo: InvertedPendulum, Hopper, Swimmer, Walker2d | episodic return | Hurt |
| use a relatively small intermediate number of cut points (ablation) | Diaster · MuJoCo: InvertedPendulum, Hopper, Swimmer, Walker2d | episodic return | Helped |
| remove learning the Markovian step-wise proxy reward and optimize with sub-trajectory reward differences (ablation) | Diaster · MuJoCo: InvertedPendulum, Hopper, Swimmer, Walker2d | episodic return | Hurt |
Surprising findings
- Removing the learned step-wise proxy reward and using differences of the sub-trajectory reward directly degrades performance, since the non-Markovian signal makes policy optimization non-stationary.
- Setting the number of cut points to 0 or T-1 (the extremes) performs poorly, indicating both attribution and representation are needed.
Limitations
- Gains are not overwhelming; only a slight improvement on most tasks and a disadvantage on HumanoidStandup.
- Assumes episodic reward equals the sum of step-wise rewards.
- Performance sensitive to the number of cut points, which depends on the task.
Experimental setup
Datasets: MuJoCo PointMaze
Benchmarks: MuJoCo InvertedPendulum MuJoCo Hopper MuJoCo Swimmer MuJoCo Walker2d MuJoCo Ant MuJoCo Humanoid MuJoCo HumanoidStandup PointMaze_UMaze
Metrics: average episodic return learning curves over environment steps
Baselines: RUDDER IRCR RRD
Builds on: RUDDER IRCR RRD SAC potential-based reward shaping
Closest papers by meaning
- Learning Long-Term Reward Redistribution via Randomized Return Decomposition (ICLR 2022 · similarity 0.92)
- Detecting Rewards Deterioration in Episodic Reinforcement Learning (ICML 2021 · similarity 0.87)
- Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach (NeurIPS 2023 · similarity 0.87)
- Episodic Novelty Through Temporal Distance (ICLR 2025 · similarity 0.86)
- Reinforcement Learning with Segment Feedback (ICML 2025 · similarity 0.86)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.