Research map / Reinforcement learning / Offline RL

Reinforcement Learning with Action Chunking

NeurIPS 2025 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-chunking, running TD-based offline-to-online RL over short action chunks with behavior constraints (QC via best-of-N, QC-FQL via W2-regularized flow policy), improving sample efficiency on long-horizon sparse-reward manipulation.

Problem

Offline-to-online RL on long-horizon, sparse-reward manipulation tasks suffers from poor exploration and slow value propagation; it is unclear how offline data should guide a good exploratory policy.

Approach

Apply action chunking to TD-based actor-critic RL: the critic and policy operate on h-step action sequences, giving unbiased n-step backups (chunked Q-function over the same actions) and temporally coherent exploration constrained by a flow-matching behavior policy (implicit KL via best-of-N in QC, W2 distillation in QC-FQL).

Key results

• QC achieves 52 offline -> 86 online average success across 25 OGBench tasks, versus 37->58 for FQL and 12->... for IQL (Table summary).
• On cube-quadruple, QC reaches 73% online success from 4% offline, versus 36% for FQL-n and 3% for FQL.
• QC-FQL improves overall OGBench average from 37->58 (FQL) to 38->86.

Measured effects reported in the paper

Surprising findings

• Gaussian-policy action chunking (RLPD-AC) performs poorly, reaching only 61 overall vs 67 for RLPD, suggesting expressive flow policies are needed.
• Increasing UTD ratio to 5 does not improve QC sample efficiency.

Limitations

• Fixed action chunk size requires task-specific tuning.
• Action chunking is a limited subclass of non-Markovian policies and may hurt in high-frequency feedback settings.
• Best-of-N sampling adds computational cost.

Experimental setup

Datasets: OGBench scene-sparse OGBench puzzle-3x3-sparse OGBench cube-double OGBench cube-triple OGBench cube-quadruple-100M robomimic lift robomimic can robomimic square

Benchmarks: OGBench robomimic

Metrics: success rate offline success rate online success rate after 1M steps

Baselines: RLPD RLPD-AC SUPE-GT IQL ReBRAC IFQL IFQL-n FQL FQL-n BFN BFN-n

Builds on: FQL RLPD IFQL ReBRAC OGBench

Closest papers by meaning

• SEAR: Sample Efficient Action Chunking Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling (ICLR 2025 · similarity 0.89)
• DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.88)
• Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
• Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning (ICML 2024 · similarity 0.88)
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.88)