Research map / Reinforcement learning / Offline RL

Horizon Adaptive Offline Policy Learning via Value Stitching

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VAST, an offline RL method that learns a horizon-conditioned cumulative-return (G) function and a diffusion/flow stitching policy to choose adaptive horizons and sub-goals for value stitching, evaluated on 50 OGBench singletask tasks.

Problem

Fixed-horizon value backups (n-step TD, action chunking) are rigid and can wash out short-range decisive signals and bias value estimates in long-horizon offline RL.

Approach

Learn a horizon- and target-state-conditioned return function G(s, s+, k) with MC and compositional losses, and a stitching Bellman operator that maximizes over horizon and sub-goal chosen by a flow-based stitching policy; execution policy is goal-conditioned flow BC with rejection sampling on Q.

Key results

• VAST is the strongest method across all Hard OGBench domains, with an overall score reported as 2.5x the best baseline.
• Across 50 OGBench singletask tasks (8 seeds, 1M offline steps), VAST performs best or on par with n-step, chunking, and generative-value baselines.
• Horizon limit K around 10-25 works best in most manipulation domains; performance drops mildly for larger K.

Surprising findings

• Rejection sampling over the execute policy (N_rej>1) adds gains beyond high-level goal selection alone, though the stitching policy alone already provides reward maximization.

Limitations

• Performance depends on the maximum horizon K, which is domain-specific and must be tuned.
• Gains are reported mainly on OGBench singletask state-based tasks; no visual or real-robot evaluation.

Experimental setup

Datasets: OGBench scene-play OGBench cube-single-play OGBench cube-double-play OGBench puzzle-4x4-play OGBench antmaze-large-navigate OGBench antmaze-giant-navigate OGBench cube-triple-play OGBench cube-quadruple-play OGBench puzzle-4x5-play OGBench puzzle-4x6-play

Benchmarks: OGBench

Metrics: success rate

Baselines: IQL FQL IFQL FQL-n IFQL-n QC VALUE-FLOWS FLOQ

Builds on: IQL FQL IFQL Q-chunking advantage-weighted regression

Closest papers by meaning

• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.90)
• Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
• From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.89)