Research map / Vision-language and multimodal / Multimodal LLMs

SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains SOLE-R1, a Qwen3-VL-8B video-language reasoner that gives per-timestep chain-of-thought and dense progress rewards, via SFT then GRPO-based RLVR, and uses it as the sole reward for zero-shot online RL from random initialization.

Problem

VLM-based reward models used as evaluators for robot RL fail under partial observability and distribution shift, so policies exploit perceptual errors (reward hacking) instead of solving tasks.

Approach

A video-native reasoning reward model that emits per-timestep spatiotemporal CoT and a scalar progress value, trained on synthesized non-expert/reversed trajectories with progress labels (geometric in simulation, temporal-order proxy on real video), followed by RLVR on progress accuracy.

Key results

• SOLE-R1 achieves at least 50% success on 24 of 40 tasks in zero-shot online RL, versus 7 for GPT-5 and 5 for Gemini.
• SOLE-R1 raises SmolVLA-diffusion RL steering success from 2% to a median of 12% in a long-horizon LIBERO setting where sparse reward fails.
• Inference takes about 40 s per 20-frame video batch on a single H100, versus 366 s for GPT-5 and 322 s for Gemini-3-Pro.

Surprising findings

• General-purpose VLM rewarders mainly fail through reward hacking (high perceived, low true success), while SOLE-R1 failures are mostly signal-limited.
• Non-reasoning reward models achieve near-zero success on nearly all tasks outside Meta-World.

Limitations

• Rewards can be flat, noisy, or miscalibrated when visual evidence is brief, occluded, or ambiguous.
• Learned policies can still exploit reward errors under distribution shift, so physical deployment needs external safety constraints.

Experimental setup

Datasets: Open X-Embodiment RoboCasa SSR-CoT SpaceOm/SpaceThinker via VQASynth Embodied CoT Robo2VLM-Reasoning RoboVQA From Seeing to Doing

Benchmarks: SpatialBench SSRBench CV-Bench RoboSuite ManiSkill Meta-World LIBERO real-world tabletop

Metrics: zero-shot online RL success rate perceived vs true success value-order correlation reward-ground-truth correlation inference speed

Baselines: GPT-5 Gemini-3-Pro Gemini Robotics-ER 1.5 Robometer RoboReward TOPReward LRM ReWiND VLAC LIV GVL

Builds on: GRPO DrQv2 SSR-CoT VQASynth Robometer

Closest papers by meaning

• Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.91)
• RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.91)
• GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning (ICML 2026 · similarity 0.89)
• RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.89)
• LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning (NeurIPS 2026 · similarity 0.89)