Research map / Reinforcement learning / Offline RL
Horizon Reduction Makes RL Scalable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that standard offline goal-conditioned RL methods scale poorly on long-horizon OGBench tasks with up to 1B transitions, attributes this to horizon (bias accumulation in TD and complex policy mappings), and proposes SHARSA, a minimal horizon-reduction method using rejection sampling with flow BC and SARSA.
Problem
Offline RL methods fail to scale with more data on complex, long-horizon tasks, saturating well below maximum performance.
Approach
Identifies the effective decision horizon as the main barrier to offline RL scaling, and proposes SHARSA, which reduces both value and policy horizons using n-step SARSA for value learning and rejection sampling over a flow BC high-level subgoal policy with goal-conditioned flow BC low-level policy.
Key results
- Standard offline RL methods (flow BC, IQL, CRL, SAC+BC) fail to master the tasks even with 1B data, and fail completely on cube-octuple.
- Horizon reduction methods such as n-step SAC+BC and hierarchical methods substantially improve scaling; SHARSA is the only method achieving non-trivial performance on all four tasks.
- Reward-based singletask experiments show n-step SAC+BC and SHARSA improve over SAC+BC; e.g., humanoidmaze-giant navigate SHARSA reaches 87% vs 44% for SAC+BC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use n-step returns instead of one-step TD updates | SAC+BC · complex goal-conditioned OGBench tasks | scaling and asymptotic performance | Helped |
| Use a hierarchical policy rather than a flat policy | Flow BC · complex goal-conditioned OGBench tasks | performance | Helped |
| Reduce both value and policy horizons with SHARSA | offline RL methods · four challenging goal-conditioned tasks | performance | Helped |
| Replace SARSA value learning with IQL (κ=0.7) (ablation) | SHARSA · benchmark tasks | performance | Mixed |
| Replace high-level rejection sampling with DDPG+BC policy extraction (ablation) | SHARSA · especially puzzle tasks | performance | Hurt |
| Replace high-level rejection sampling with FQL policy extraction (ablation) | SHARSA · especially puzzle tasks | performance | Hurt |
| Use binary cross-entropy rather than regression for value losses (ablation) | SHARSA · benchmark tasks | performance | Helped |
| Increase network capacity up to 591M parameters (ablation) | SAC+BC · complex long-horizon tasks | performance | Mixed |
| Replace MLPs with Transformers (ablation) | SAC+BC · four benchmark tasks | performance | Mixed |
| Use a flow policy instead of the SAC+BC policy class (ablation) | SAC+BC (FQL) · four benchmark tasks | performance | No clear effect |
| Increase the Q ensemble from 2 to 10 networks (ablation) | SAC+BC · four benchmark tasks | performance | No clear effect |
| Add weight decay (ablation) | SAC+BC · humanoidmaze-giant versus the other three tasks | performance | Mixed |
Surprising findings
- Larger models (up to 591M parameters), Transformers, larger Q ensembles, and many other hyperparameter changes did not substantially improve offline RL scaling.
- Standard 1-step DQN accumulates Q-value bias more than 64-step DQN even with similar TD errors.
Limitations
- Methods only mitigate, not fully solve, TD error accumulation with two-level hierarchies; none masters all four tasks at 100%.
- SHARSA and n-step methods assume dataset trajectories are near-optimal within short segments.
- Performance does not always scale monotonically with dataset size; idealized state-based setting with oracle goal representations.
Experimental setup
Datasets: OGBench cube-octuple OGBench puzzle-4x5 OGBench puzzle-4x6 OGBench humanoidmaze-giant OGBench cube-double OGBench puzzle-4x4
Benchmarks: OGBench cube-octuple OGBench puzzle-4x5 OGBench puzzle-4x6 OGBench humanoidmaze-giant OGBench singletask reward-based tasks
Metrics: success rate scaling curves over 1M-1B data
Baselines: IQL CRL SAC+BC flow BC HIQL FQL n-step SAC+BC
Builds on: OGBench HIQL CRL FQL implicit Q-learning
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset (ICLR 2025 · similarity 0.90)
- Scaling Offline RL via Efficient and Expressive Shortcut Models (NeurIPS 2025 · similarity 0.90)
- Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction (AAAI 2024 · similarity 0.90)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.