Research map / Video / Video-language understanding
UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free UniFunc3D uses a unified MLLM (Qwen3-VL) as an active observer with coarse-to-fine video frame grounding, MLLM visual mask verification, and SAM3 lifting to segment 3D affordances on SceneFun3D.
Problem
Grounding implicit natural-language task descriptions into precise 3D masks of fine-grained functional parts in real indoor scenes, where existing training-free pipelines use text-only parsing and passive single-scale frame selection.
Approach
Replace fragmented multi-model pipelines with one MLLM that jointly reasons over semantics, frame selection and point grounding in a coarse (low-res multi-sampling) round and a fine (high-res temporal window) round, then verifies SAM3 masks visually before multi-view 3D lifting.
Key results
- UniFunc3D-30B reaches 31.24 AP50 and 24.30 mIoU on SceneFun3D split0, a 59.9% relative mIoU gain over Fun3DU (15.2).
- UniFunc3D-8B achieves 23.82 AP50 on split0 versus 16.9 for Fun3DU and 26.9 for training-based TASA-72B on AP50 (TASA higher in AP50 with 72B model).
- Ablation shows the temporal window gives the largest single gain and K=4 sampling iterations with verification gives the best mIoU (20.92 for 8B).
Surprising findings
- Qwen2.5-VL-72B with the training-free pipeline was reported as beating TASA-72B on AP25 and mIoU, though TASA keeps a higher AP50.
- Swapping SAM3 for SAM gave nearly identical results, suggesting the gains come from pipeline design rather than the segmenter.
Limitations
- Struggles with extremely small functional parts (<0.1% of image area) and severe occlusion.
- Per-scene runtime still around 26 minutes on 8 H100s for the 30B variant.
Experimental setup
Datasets: SceneFun3D
Benchmarks: SceneFun3D
Metrics: AP25 AP50 AR25 AR50 mIoU
Baselines: Fun3DU TASA AffordBot OpenMask3D LERF OpenIns3D
Builds on: Fun3DU SAM3 Qwen3-VL
Closest papers by meaning
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.91)
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.90)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.90)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.89)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.89)
- LASO: Language-guided Affordance Segmentation on 3D Object (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.