Research map / Large language models / Agents, code and math

WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WorldCoder-Bench, 2,026 expert-curated tasks for LLM generation of executable Three.js 3D worlds, and StateProbe, a hidden-contract state-verification protocol in headless Chromium, benchmarking nine frontier models.

Problem

Evaluating whether LLM-generated browser-native Three.js 3D worlds behave correctly, since the mechanics live inside an opaque WebGL canvas and existing screenshot, DOM, and agent-based evaluators fail to capture runtime state correctness.

Approach

Probe the runtime state of generated Three.js programs via a standardized state interface and check before/after state deltas under scripted actions against hidden, mutation-hardened behavioral contracts; add cost- and time-adjusted utility metrics (RoA, TEM).

Key results

• Best model GPT-5.4 reaches 27.8% V-Cov on WorldCoder-Core; no model exceeds 30%.
• On WorldCoder-Robust, best V-Cov is 19.9% (GPT-5.4), and the proprietary-open gap widens to 6 points.
• DOM-only scoring has Kendall tau_b of -0.021 against V-Cov, while the 8-step agent admits 45.6% of low-V-Cov outputs as passing.

Surprising findings

• DOM-only scoring is essentially uncorrelated (and in aggregate inverted, tau_b -0.238) with hidden state-level correctness.
• DeepSeek-V4-Flash tops RoA at about $19,150 per API dollar despite having one of the lowest V-Cov scores.

Limitations

• Core split is a hard subset selected for low reference-model T-Cov, so scores are not representative of the full benchmark.
• Human effort estimates rest on only seven explicit expert annotations plus a difficulty-based fallback.
• Hidden contracts and parts of the evaluation are not publicly released for the main splits.

Experimental setup

Datasets: WorldCoder-Bench (2,026 expert-curated tasks)

Benchmarks: WorldCoder-Core WorldCoder-Robust WorldCoder-Extended

Metrics: Verification Coverage Affordance Coverage State Coverage Transition Coverage Return on Automation Time Efficiency Multiplier Kendall tau_b

Baselines: DOM-only inspection Screenshot+VLM (GPT-5.4) Pure Agent (8-step inspection)

Builds on: Design2Code WebGen-Bench GameDevBench

Closest papers by meaning

• 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.89)
• WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes (NeurIPS 2026 · similarity 0.88)
• 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models (CVPR 2026 · similarity 0.87)
• SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.87)
• Generative Visual Code Mobile World Models (ICML 2026 · similarity 0.87)
• SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code (ICML 2024 · similarity 0.87)