Research map / Vision-language and multimodal / Multimodal LLMs

IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces IR3D-Bench, a CLEVR-based benchmark where VLM agents write Blender scripts (via BlenderMCP) to reconstruct 3D scenes from single images, scored on localization, appearance, and GPT-4o-judged semantics.

Problem

Whether VLMs truly understand 3D scenes beyond descriptive tasks like captioning or VQA; the paper proposes testing understanding by having agents recreate the scene via tool use.

Approach

Frame scene understanding as agentic inverse rendering: a VLM agent outputs a JSON scene description that is turned into a Blender script and rendered, then compared to the input image using geometric, mask, CLIP-attribute, and GPT-4o-based metrics; also tests iterative refinement with the GT image.

Key results

• Gemini-2.5-pro achieves the top count accuracy (1.00) and relation accuracy (0.55) among the evaluated models, with an overall LLM score of 2.62 and CLIP overall of 93.08.
• Qwen2.5-VL-72B obtains the highest LLM overall score (3.02) despite relatively weak bounding-box score (0.21).
• Several open models (DeepSeek-VL2, Llama-3.2-11B-Vision, H2OVL-Mississippi-2B) failed to produce valid outputs; mask IoU stays low across models (best 0.11).

Measured effects reported in the paper

Surprising findings

• Appearance attributes (color, material, shape) are recognized with high CLIP scores, yet mask IoU and relation accuracy remain low, indicating visual precision rather than tool usage is the bottleneck.
• Iterative refinement with GPT-4o over ten rounds brings its output quality to roughly Gemini-2.5-pro level.

Limitations

• Built only on synthetic CLEVR scenes, lacking real-world visual richness and noise.
• Single-view static scenes only; no multi-view or temporal consistency.
• Camera intrinsics/extrinsics fixed and illumination omitted, reducing realism.

Experimental setup

Datasets: CLEVR validation split (1,500-scene subset)

Benchmarks: IR3D-Bench

Metrics: pixel distance count accuracy bbox edge score relation accuracy IoU DICE CLIP score (color, size, material, shape) LLM score (GPT-4o)

Baselines: GPT-4o GPT-4.1 Gemini-2.5-pro Gemini-2.0-flash Claude-3.7-Sonnet Claude-3.5-Sonnet Grok-3 Qwen2.5-VL-72B Qwen2.5-VL-7B InternVL3-38B InternVL2.5-38B Pixtral-12B

Builds on: CLEVR Blender Model Context Protocol Segment Anything Model Hungarian algorithm

Closest papers by meaning

• Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.89)
• Video Perception Models for 3D Scene Synthesis (NeurIPS 2025 · similarity 0.89)
• 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.88)
• The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.88)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.88)
• Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.88)