Research map / Reinforcement learning / Visual and representation RL

Learning Process Rewards via Visitation Matching for Efficient RL

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Reward shaping for sparse rewards

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes success visitation matching (SVM), which turns a sparse outcome reward into a dense process reward via a success-vs-failure discriminator whose log-odds reward is provably optimal-policy-preserving in deterministic settings, and shows faster RL finetuning of robot policies.

Problem

Sparse outcome rewards in RL make credit assignment hard and slow RL improvement, especially for finetuning robotic control policies, while existing reward-shaping methods need side knowledge, human supervision, or do not preserve the optimal policy.

Approach

Train a discriminator to separate state-action pairs from successful versus unsuccessful episodes collected online, and use its clipped log-odds (a KL-regularized visitation-matching term) added to the outcome reward as a dense process reward; a theorem shows optimal policies for this reward also maximize the outcome reward in deterministic environments.

Key results

• On LIBERO kitchen scenes with DSRL, SVM reaches significantly higher final success and faster convergence than baselines; final success consistently above 80%.
• Finetuning pi0 with DSRL and SVM needs roughly 2x fewer environment steps to reach 90% success than outcome-only RL on four LIBERO-90 tasks.
• On real WidowX tasks, SVM reaches over 80% success within fewer steps and enables learning on Pick and Place and Open Drawer where outcome-only RL fails.

Surprising findings

• On Robomimic from demonstrations, SVM performs essentially identically to GAIL, which uses the demonstrations without the outcome reward.
• Timestep conditioning of the discriminator does not improve performance on average.

Limitations

• Theoretical optimal-policy guarantee assumes deterministic transitions; real-world experiments are not strictly deterministic.
• Gains on Robomimic from demonstrations are matched by GAIL, so the advantage over adversarial imitation is not shown there.
• Experiments focus on robotic control; LLM/RLVR application is left to future work.

Experimental setup

Datasets: LIBERO-90 RoboCasa PnPCounterToCab Robomimic BridgeData V2

Benchmarks: LIBERO-90 Kitchen Scenes 1-3 RoboCasa PnPCounterToCab Robomimic Can and Square real-world WidowX tasks

Metrics: success rate sample efficiency (environment steps to reach success thresh

Baselines: SORS SASR GAIL GAIL-reward RND GVL outcome-only RL

Builds on: adversarial inverse reinforcement learning GAIL DSRL Residual RL RLPD

Closest papers by meaning

• Enhanced Meta Reinforcement Learning via Demonstrations in Sparse Reward Environments (NeurIPS 2022 · similarity 0.90)
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.89)
• Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning (ICML 2025 · similarity 0.88)
• Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.88)
• PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling (ICML 2024 · similarity 0.88)