Research map / 3D vision / 3D generation
Functionality Understanding and Segmentation in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Fun3DU, a training-free pipeline that uses an LLM for chain-of-thought task parsing, open-vocabulary 2D segmentation plus visibility-based view selection, and a VLM (Molmo) with SAM to segment functional object parts in 3D scenes, lifted to the point cloud via multi-view agreement.
Problem
Locating fine-grained functional interactive objects (handles, buttons, switches) in 3D indoor scenes from natural language task descriptions that often do not name the object, requiring world knowledge and fine-grained spatial perception.
Approach
Combine a frozen LLM's chain-of-thought parsing of the task (contextual and functional object), open-vocabulary segmentation of the contextual object to select the best views, then a VLM pointing model plus promptable segmenter on those views, and lift 2D masks to 3D with multi-view voting, without any training.
Key results
- Fun3DU reaches 33.3 AP25 on SceneFun3D split0 and 23.1 AP25 on split1, the only method above 0.4 AP25 on either split... (AP25 0.333 on split0; 0.231 on split1)
- Fun3DU outperforms the nearest open-vocabulary baseline by 32.9 AP25 points on split0 (OpenMask3D) and by 15 mIoU points on split0
- Ablations show removing the VLM (replacing it with OWLv2 on concatenated text) drops AP25 by 19.4 on split0
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Fun3DU instead of OpenMask3D | SceneFun3D split0 functionality segmentation · SceneFun3D split0 | AP25: 0.4 -> 33.3 | Helped |
| use Fun3DU instead of OpenMask3D | SceneFun3D split1 functionality segmentation · SceneFun3D split1 | AP25: 0.0 -> 23.1 | Helped |
| add VLM-based functional-object segmentation rather than open-vocabulary segmentation (ablation) | Fun3DU · SceneFun3D split0 | AP25: 13.9 -> 33.3 | Helped |
| retain the full task description instead of prompting only “Point to all the F” (ablation) | Fun3DU · SceneFun3D split0 | AP25: 18.9 -> 33.3 | Helped |
| select views using the contextual object rather than randomly sampling 100 images (ablation) | Fun3DU · SceneFun3D split0 | AP25: 20.5 -> 33.3 | Helped |
| combine contextual-mask confidence, distance, and angle scores for view ranking (ablation) | Fun3DU · SceneFun3D split0 | AP25: 27.0 -> 33.3 | Helped |
| combine confidence, distance, and angle scores rather than using distance alone (ablation) | Fun3DU · SceneFun3D split0 | AP25: 27.2 -> 33.3 | Helped |
| combine confidence, distance, and angle scores rather than using angle alone (ablation) | Fun3DU · SceneFun3D split0 | AP25: 30.1 -> 33.3 | Helped |
| include contextual-mask confidence along with distance and angle distributions in view ranking (ablation) | Fun3DU · SceneFun3D split0 | AP25: 30.8 -> 33.3 | Helped |
Surprising findings
- Baselines achieve high recall but near-zero precision, tending to segment the contextual object (e.g., whole cabinet) instead of the functional part
- Using 10 or fewer views still yields mIoU above 10, and 4 or fewer views above 7, so view quality matters more than quantity
Limitations
- Fails when the contextual object's location alone is insufficient to identify the functional object
- Spurious masks arise from imprecise VLM point predictions for distant objects
- Performance worse on split1 with more complex scenes
Experimental setup
Datasets: SceneFun3D
Benchmarks: SceneFun3D split0 SceneFun3D split1
Metrics: AP25 AP50 mAP AR25 AR50 mAR mIoU
Baselines: OpenMask3D OpenIns3D LERF
Builds on: SceneFun3D Molmo OWLv2 SAM chain-of-thought prompting
Closest papers by meaning
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.91)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.89)
- Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces (CVPR 2025 · similarity 0.88)
- TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances (AAAI 2025 · similarity 0.87)
- A Unified Framework for 3D Scene Understanding (NeurIPS 2024 · similarity 0.87)
- Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.