Research map / Reinforcement learning / Policy gradient and actor-critic

FASTER: Value-Guided Sampling for Fast RL

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FASTER, which models filtering of diffusion-policy action candidates during denoising as an MDP and learns a noise-level critic to pick one seed to denoise, recovering best-of-N gains at roughly single-rollout cost.

Problem

Best-of-N test-time scaling for diffusion/flow RL policies is effective but requires denoising all N action candidates, making training and inference prohibitively expensive, especially for large VLAs.

Approach

Model denoising of multiple action candidates plus selection as a filtering MDP over noise seeds; learn a noise-level critic Q^dn(s, eps) regressed to the action-level critic so only the top-scoring initial noise is denoised, collapsing the MDP to one step in practice.

Key results

• On the pi05_libero VLA, FASTER-EXPO cuts update step time from 11.6 s to 2.5 s (about 4.5x) and inference latency from 566 ms to 335 ms (1.7x) while matching EXPO success on most tasks.
• Inference FLOPs per step drop from 3.75e13 for EXPO to 4.70e12 for FASTER-EXPO, an 8x reduction.
• FASTER-EXPO and FASTER-IDQL achieve success rates within the margin of error of EXPO and IDQL with best-of-N sampling in the online setting on Robomimic tasks.

Surprising findings

• Filtering at the initial noise seed performs comparably to learning the full adaptive filtering MDP, suggesting the one-step simplification is a good proxy.
• Distilling the value-maximizing action into a single policy performs substantially worse than FASTER, attributed to non-stationary distillation targets.

Limitations

• Not designed to improve sample efficiency; FASTER-EXPO does not surpass EXPO in sample efficiency.
• Applies only to policy classes that use initial noise seeds, excluding policies without seed structure.
• VLA experiments use only 5 held-out LIBERO-90 tasks.

Experimental setup

Datasets: Robomimic LIBERO-90 LIBERO

Benchmarks: Robomimic Lift Robomimic Can Robomimic Square Robomimic Tool Hang LIBERO-90 held-out tasks

Metrics: success rate update step time inference latency FLOPs

Baselines: EXPO IDQL RLPD QSM DSRL QAM FQL pi05_libero

Builds on: EXPO IDQL DSRL

Closest papers by meaning

• Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.89)
• PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.88)
• Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Learning to Sample From Diffusion Models via Inverse Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.88)