Research map / Multi-agent systems and games / LLM agents

Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADWM, a latent diffusion world model with policy-guided score for off-policy evaluation of LLM agents, evaluated by rank correlation of value estimates across four agent benchmarks.

Problem

Estimating the value of a new LLM agent policy in multi-turn interactive environments from pre-collected trajectories, without online interaction, where discrete text actions break existing joint-diffusion OPE methods.

Approach

Factor the policy-guided trajectory law into single-step conditionals (prior × action posterior × policy continuation factor) and realize them as classifier-free-guided latent diffusion, so the evaluation LLM steers each denoising step in causal order.

Key results

• ADWM achieves positive Spearman rho in all six configurations, mean +0.82 (min +0.67 on ALFWorld-iter1).
• Best classical baseline FQE reaches mean rho +0.10 across configurations; IS and DR reach as low as -0.90.
• Ablations show removing local CFG drops WebShop rho from +0.90 to +0.10, and removing continuation guidance drops WebShop to +0.30.

Surprising findings

• Direct method yields rho=0 within every epsilon sweep and inverts on the cross-policy split (rho=-0.45).
• WIS produces spurious positive correlations driven by floating-point underflow ties.

Limitations

• Ground-truth evaluation is noisy on WebShop, so fine-grained distinctions between adjacent LEAP iterates need more episodes.
• The psi adapter is tied to the evaluation LLM's embedding space, requiring retraining for cross-family evaluation.
• Evaluated on small behavior datasets (256-744 trajectories) with a limited number of policy pairs.

Experimental setup

Datasets: HotpotQA ScienceWorld ALFWorld WebShop

Benchmarks: HotpotQA ScienceWorld ALFWorld WebShop

Metrics: Spearman rank correlation bootstrap confidence interval

Baselines: DM IS WIS FQE DR

Builds on: DDPM classifier-free guidance importance sampling fitted-Q evaluation COBS

Closest papers by meaning

• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL (NeurIPS 2026 · similarity 0.90)
• Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.89)
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.89)
• Test-Time Adaptation for LLM Agents via Environment Interaction (ICLR 2026 · similarity 0.89)
• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.89)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.88)