Research map / Reinforcement learning / Offline RL

Scaling Offline RL via Efficient and Expressive Shortcut Models

NeurIPS 2025 · Reinforcement learning › Offline RL · approach: Simple practical offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SORL, a one-stage offline RL method that trains a shortcut-model policy with flow-matching and self-consistency losses, enabling sequential and best-of-N (Q-verified) test-time scaling.

Problem

Applying expressive generative policies (diffusion/flow) to offline RL is hard because iterative sampling makes policy optimization via backpropagation through time expensive and inference slow, and existing approaches require two-stage distillation or lose inference scalability.

Approach

Uses shortcut models conditioned on step size so the policy can be trained with few backprop steps, regularized to data with many discretization steps, and sampled with any number of inference steps; the learned Q-function serves as a verifier for best-of-N sampling.

Key results

• SORL achieves best performance on 5 of 8 OGBench environments, e.g., antmaze-large singletask 89 vs. 81 for ReBRAC and 79 for FQL.
• On antsoccer-arena singletask SORL reaches 69 vs. 60 for FQL.
• Sequential and best-of-N parallel test-time scaling improves performance, with SORL generalizing to more inference steps than backprop steps used in training (up to about 4 backprop steps).

Measured effects reported in the paper

Surprising findings

• Best-of-N sampling with the learned Q-function is not theoretically guaranteed to help but empirically acts as an inference-time policy-extraction step.
• Performance gains from parallel scaling do not occur consistently across all environments.

Limitations

• Q-function verifier may be inaccurate or out-of-distribution, limiting benefit of best-of-N scaling.
• Longer training runtime than FQL.
• Lacks principled exploration for online RL fine-tuning.

Experimental setup

Datasets: OGBench

Benchmarks: OGBench

Metrics: average return/success rate across 8 seeds

Baselines: BC IQL ReBRAC IDQL SRPO CAC FAWAC FBRAC IFQL FQL

Builds on: flow matching shortcut models behavior-regularized actor-critic FQL

Closest papers by meaning

• Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.90)
• A Tractable Inference Perspective of Offline RL (NeurIPS 2024 · similarity 0.90)
• A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.89)
• SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.89)
• Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.89)