Research map / Large language models / Agents, code and math

BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing

CVPR 2025 · Large language models › Agents, code and math · approach: Code generation benchmarks · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BlenderGym, a 245-instance benchmark that evaluates VLM systems on code-based 3D graphics editing (scene reconstruction via Blender Python), and uses it to study inference scaling of VLM verifiers and compute allocation between generation and verification.

Problem

Evaluating VLM-driven 3D graphics editing is hard: existing evaluations cover narrow tasks, rely on human or AI judges, and lack quantitative, comprehensive benchmarks.

Approach

Tasks VLM systems with reconstructing a goal Blender scene from a start scene by editing Python scripts, scored automatically with photometric loss, Chamfer distance and CLIP-based metrics against fixed start-goal pairs, and studies scaling the VLM verifier at inference time.

Key results

• Human Blender users outperform all VLM systems on BlenderGym; e.g., blend shape photometric loss of 0.934 (x10^-3) for humans vs. 9.140 for GPT-4o.
• Human-VLM verifier alignment is highest for Claude 3.5 Sonnet (0.66), below inter-human alignment (0.79).
• Scaling the VLM verifier (up to 64 reselections) lowers the metrics of selected edits; scaled InternVL2-8B outperforms unscaled GPT-4o and Claude 3.5 Sonnet on all three metrics.

Measured effects reported in the paper

Surprising findings

• Claude 3.5 Sonnet is the best-aligned VLM verifier yet still falls well short of human inter-annotator agreement.
• Certain VLM verifiers such as Qwen consistently favor the second candidate regardless of order.

Limitations

• Benchmark covers only five editing tasks and 245 instances; object sculpting, camera adjustment, and animation are not included.
• Human baseline and human-VLM alignment studies use a limited number of annotators (10 Blender users, 50 participants for alignment).
• Scaled verifiers still sometimes fail to select the best edit.

Experimental setup

Datasets: Blender prototypes from BlenderAlchemy Infinigen BlenderKit

Benchmarks: BlenderGym

Metrics: photometric loss negative CLIP score Chamfer distance human-VLM verifier alignment rate

Baselines: GPT-4o Claude 3.5 Sonnet Gemini 1.5 Flash GPT-4-Turbo Claude 3 Haiku Qwen2-VL-7B MiniCPM-V-2.6 InternVL2-8B Phi-3.5-vision Human Blender users

Builds on: BlenderAlchemy Large Language Monkeys (repeated sampling) Infinigen

Closest papers by meaning

• 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.88)
• EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing (ICLR 2025 · similarity 0.88)
• VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing (ICML 2026 · similarity 0.87)
• EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.87)
• Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing (NeurIPS 2025 · similarity 0.87)
• UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (CVPR 2026 · similarity 0.87)