Research map / Reinforcement learning / Offline RL

SEAR: Sample Efficient Action Chunking Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEAR, an off-policy maximum-entropy RL method that trains a causal transformer critic with multi-horizon targets and random replanning so action-chunking policies can be learned online and deployed with receding horizons.

Problem

Online off-policy RL with action chunks is sample-inefficient: enlarged chunk action spaces give poor critic fits, and open-loop execution reduces decision frequency and state coverage.

Approach

Train a causal transformer critic on multi-horizon N-step targets (one target per chunk prefix) so every prefix gets a training signal, and collect data by executing only a random prefix of each chunk, which uniformizes actor state coverage and enables receding-horizon control at evaluation time.

Key results

• On the 20 hardest Metaworld tasks, SEAR-10 reaches about 84% IQM success versus about 60-62% for CQN-AS-5 and SimbaV2 after 1M steps.
• A naive chunked SimbaV2 and T-SAC reach only around 20% IQM success on Metaworld-Hard, showing naive chunking fails online.
• Applying SEAR's multi-horizon targets and random replanning to QC improves IQM success on OGBench cube-triple-play tasks, most clearly early in online training.

Surprising findings

• Removing multi-horizon targets and fitting only the full-chunk target drops SEAR to around 20% IQM success, close to the weakest baselines.
• Training with action chunks and deploying with a receding horizon of one outperforms both SEAR-1 and SimbaV2, suggesting chunking can serve as a training-only aid.

Limitations

• Current action chunking methods degrade in high-reactivity domains such as locomotion, as chunking during training degrades data quality.
• The transformer critic adds training cost compared with single-step methods.
• Evaluated mainly on Metaworld and a small set of OGBench tasks; QC experiments use an MLP critic rather than a transformer.

Experimental setup

Datasets: Metaworld ML1 (20 hardest tasks, plus all 50 in appendix) OGBench cube-triple-play offline dataset

Benchmarks: Metaworld OGBench cube-triple-play

Metrics: IQM success rate bootstrapped 95% confidence intervals

Baselines: SimbaV2 CQN-AS T-SAC SAC QC QC-FQL

Builds on: TOP-ERL T-SAC CQN-AS QC SimbaV2

Closest papers by meaning

• Reinforcement Learning with Action Chunking (NeurIPS 2025 · similarity 0.92)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling (ICLR 2025 · similarity 0.86)
• Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
• Action abstractions for amortized sampling (ICLR 2025 · similarity 0.86)
• FASTER: Value-Guided Sampling for Fast RL (NeurIPS 2026 · similarity 0.86)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.86)