Research map / Large language models / Reasoning and chain of thought

RLVR-World: Training World Models with Reinforcement Learning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLVR-World, which post-trains autoregressive language and video world models with GRPO using task metrics (accuracy, F1, perceptual losses) as verifiable rewards, yielding gains on text games, web states, and robot video prediction.

Problem

Standard MLE training of world models misaligns with transition-prediction metrics such as accuracy, F1, and perceptual quality, and suffers from issues like repetition and blurry predictions.

Approach

Formulates world modeling for language and video as autoregressive generation over tokens, decodes predicted next states, and uses task-specific prediction metrics as verifiable rewards to fine-tune the pre-trained model with GRPO.

Key results

• Text game state prediction: RLVR-World with 7B base reaches 65.53% overall accuracy, surpassing GPT-4 (64.76%); 1.5B task-specific reward improves overall accuracy from 32.87% (SFT) to 63.24%.
• Web page state prediction F1 improves from 49.94% (SFT) to 65.11% with RLVR, and web agent success rate rises from 12.06% to 14.29%.
• Video world model on RT-1 multi-step prediction reduces repetition rate from 48.6% to 9.9% and improves LPIPS by 9.2% relative, with only hundreds of RL steps versus hundreds of thousands for MLE.

Measured effects reported in the paper

Surprising findings

• Best-of-N base model eventually surpasses RLVR-trained video models as N grows to 100, suggesting limits of current RLVR for exploration.
• Automatic VLM-based evaluation of simulated trajectories was unreliable, so policy evaluation relied on human annotators.

Limitations

• RLVR gains saturate within hundreds of steps and best-of-N base models overtake RLVR at large N.
• Video world models are trained on the same dataset in two stages; no general-purpose pretrained video world model is used.
• Web page results compare only to the authors' base models because the task formulation differs from WMA.

Experimental setup

Datasets: ByteSized32-State-Prediction WMA web page state transition dataset (WebArena) RT-1 PushT Rope Granular Open X-Embodiment

Benchmarks: ByteSized32-State-Prediction WMA web page state prediction WebArena RT-1 PushT Rope Granular SIMPLER

Metrics: accuracy precision recall F1 success rate MSE PSNR SSIM LPIPS repetition rate

Baselines: SFT GPT-4 DINO-WM AVDC SIMPLER Base world model with repetition rejection

Builds on: GRPO DeepSeek-R1 iVideoGPT WMA DINO-WM

Closest papers by meaning

• Reinforcing VLAs in Task-Agnostic World Models (NeurIPS 2026 · similarity 0.91)
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL (NeurIPS 2026 · similarity 0.90)
• Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.90)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.90)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.89)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)