Research map / Reinforcement learning / Visual and representation RL

Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares masked diffusion language models with autoregressive LLMs as steerable text-based world models on a 239k-trajectory dataset, and uses MDLM-generated rollouts for zero-shot GRPO training of small agents on ScienceWorld, ALFWorld, and AppWorld.

Problem

Hand-curated RL environments are static and become ineffective as models improve, and autoregressive text world models suffer from left-to-right bias that prevents conditioning on globally interdependent state anchors such as tool schemas and expected outcomes.

Approach

Formalizes text world modeling as steerable transition dynamics and shows masked diffusion LMs with bidirectional anchor-aware denoising yield more coherent, diverse rollouts than larger AR LLMs; uses these rollouts in a plug-and-play GRPO framework with deterministic state checks for zero-shot transfer.

Key results

• SDAR-8B surpasses every AR baseline on in-domain MAUVE (.982 vs .932 for Qwen3.5-35B-A3B) after fine-tuning despite being over 4x smaller.
• GRPO with SDAR-generated world model rollouts raises LFM2.5-1.2B on ALFWorld from 5.7% to 53.6% (+47.9 absolute) in a zero-shot setting.
• Human evaluation by four experts gives means of 4.75 realism, 4.25 outcome correctness, and 4.50 training utility, with Krippendorff's alpha above 0.89.

Surprising findings

• Small MDLMs (8B) outperform fine-tuned AR models up to 35B MoE, suggesting causal inductive bias rather than capacity is the main bottleneck in LLM-based world modeling.
• MDLMs show a 0.68x to 1.80x latency relative to non-thinking AR baseline, while AR thinking costs 3.75-4.87x.

Limitations

• MDLMs struggle to produce well-formed API keys and show pagination drift and block-level repetition at high temperature.
• Both MDLM and AR world models drift over long horizons; evaluation limited to turn-based text environments.
• Human evaluation covers only 100 samples from four annotators.

Experimental setup

Datasets: SWE-Smith Coderforge DeepSearchQA OpenResearcher TauBench BFCLv4 Toolathlon Pandora Webshop InterCode SQL OccuBench API-Bank

Benchmarks: API-Bank OccuBench InterCode (SQL) ScienceWorld ALFWorld AppWorld

Metrics: BLEU-1 ROUGE-L MAUVE Self-BLEU Distinct-N task success rate Krippendorff's alpha

Baselines: Qwen3.5-27B Qwen3.5-35B-A3B GPT-OSS-20B GLM-4.7-Flash Nemotron-3-Nano-30B LLaDA-2.1-mini WeDLM-8B Qwen3-8B

Builds on: GRPO LLaDA SDAR WeDLM Qwen3

Closest papers by meaning

• RLVR-World: Training World Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents (NeurIPS 2026 · similarity 0.90)
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.89)
• Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.89)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Reinforcing VLAs in Task-Agnostic World Models (NeurIPS 2026 · similarity 0.89)