Research map / 3D vision / 3D generation
MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces task-oriented tabletop scene generation, the MesaTask-10K dataset of manually refined tabletop scenes, and MesaTask, a Qwen3-8B-based LLM that uses a spatial reasoning chain with SFT and DPO to generate task-conforming layouts.
Problem
Generating physically plausible, realistic tabletop scenes that align with high-level task instructions is hard because of the gap between task descriptions and 3D layouts and the scarcity of manually crafted scene data.
Approach
Decompose task-to-scene generation into a spatial reasoning chain (object list completion, interrelation inference, scene graph construction, then 3D layout), train an LLM on human-refined scenes with SFT, and refine it with DPO using negatives built from collisions, wrong relations, and missing objects.
Key results
- MesaTask achieves FID 40.3 and average GPT score 8.25, versus 74.4 FID and 7.09 for GPT-4o zero-shot on 500 generated scenes.
- User study score of 6.12 for MesaTask versus 4.21 for GPT-4o, with 127 participants.
- Ablation: DPO lowers FID from 41.4 to 40.3 and raises user-study score from 5.75 to 6.12 relative to SFT-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spatial reasoning to the method (ablation) | MesaTask · Task-oriented tabletop scenes | FID: 40.8 -> 40.3 | Helped |
| add spatial reasoning to the method (ablation) | MesaTask · Task-oriented tabletop scenes | GPT Score Avg.: 8.20 -> 8.25 | Helped |
| add DPO training after SFT (ablation) | MesaTask · Task-oriented tabletop scenes | FID: 41.4 -> 40.3 | Helped |
| add DPO training after SFT (ablation) | MesaTask · Task-oriented tabletop scenes | User Study: 5.75 -> 6.12 | Helped |
Surprising findings
- GPT-4o without reasoning chain achieves a higher success rate (91.6%) than with reasoning on some metrics, and Holodeck-table reaches 99.3% success but low GPT score (5.42).
Limitations
- Relies on human annotators (10 to 20 minutes per scene) for layout refinement, limiting scalability.
- Negatives for DPO are synthetic corruptions rather than real SFT model failures.
- Success rate and FID are not computed for unseen categories; generalization is evaluated only with GPT scores.
Experimental setup
Datasets: MesaTask-10K Objaverse PartNet-Mobility Holodeck assets
Benchmarks: MesaTask-10K test split
Metrics: success rate FID GPT score (CwT, OSR, PPI, LCR, OV) user study
Baselines: GPT-4o GPT-4o w/o reasoning Holodeck-table I-Design-table ATISS DiffuScene PhyScene
Builds on: DPO Holodeck I-Design Architect
Closest papers by meaning
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.88)
- LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.86)
- RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.86)
- InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity (CVPR 2026 · similarity 0.85)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.85)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.