Research map / Vision-language and multimodal / Multimodal LLMs
AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AffordBot, which renders surround-view images of a 3D scene with projected affordance candidates and uses an MLLM with a three-step observe-then-infer chain-of-thought to output (mask, motion type, motion axis) triplets for instruction-conditioned 3D affordance reasoning on SceneFun3D.
Problem
Instruction-conditioned fine-grained 3D embodied reasoning: for each referenced affordance element in a 3D scene, predict its 3D mask, motion type, and motion axis, which prior work treats in isolation or with instruction-agnostic motion prediction.
Approach
Bridges 3D point clouds and 2D-native MLLMs by rendering 360-degree surround-view images with projected 3D affordance candidates (adaptive label placement), then applying a CoT pipeline where the MLLM first selects the most informative view, then grounds the target element IDs, then infers motion type and discretized axis direction.
Key results
- AffordBot (Qwen2.5-VL-72B) reaches 14.0 mIoU, 15.5 AP, 20.0 AP50, 23.3 AP25 on SceneFun3D grounding, versus 11.5 mIoU and 23.1 AP25 for Fun3DU.
- With motion constraints AffordBot scores 18.3 (+T) and 10.8 (+TD), above Fun3DU (+motion) at 11.5 and 4.0.
- Swapping the MLLM to GPT-o1 raises AP25 to 33.4 from 23.3 with Qwen2.5-VL-72B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AffordBot instead of Fun3DU | SceneFun3D affordance grounding · SceneFun3D | AP: 6.1 -> 15.5 | Helped |
| Add ALR to the baseline pipeline (ablation) | AffordBot grounding pipeline · SceneFun3D | AP25: 15.7 -> 16.1 (+0.4% AP$_{25}$) | Helped |
| Add EVS to the ALR variant (ablation) | AffordBot grounding pipeline · SceneFun3D | AP25: 16.1 -> 22.1 (+6.0%) | Helped |
| Add AVS to the EVS variant (ablation) | AffordBot grounding pipeline · SceneFun3D | AP25: 22.1 -> 23.3 | Helped |
| Use surround views with MLLM-selected viewpoint instead of query-aligned single-frame retrieval (ablation) | AffordBot viewpoint selection · SceneFun3D | AP25: 16.1 -> 23.3 (7.6%) | Helped |
| Replace Mask3D predicted masks with ground-truth proposals (ablation) | AffordBot · SceneFun3D | AP25: 23.3 -> 45.4 (22.1%) | Helped |
| Use ground-truth views in addition to ground-truth proposals (ablation) | AffordBot with GT proposals · SceneFun3D | AP25: 45.4 -> 47.4 (+2.0%) | Helped |
| Replace Qwen2.5-VL-72B with LLaVA-v1.6-34B (ablation) | AffordBot · SceneFun3D | AP25: 23.3 -> 16.9 | Hurt |
| Replace Qwen2.5-VL-72B with GPT-4o (ablation) | AffordBot · SceneFun3D | AP25: 23.3 -> 28.9 | Helped |
| Replace Qwen2.5-VL-72B with GPT-o1 (ablation) | AffordBot · SceneFun3D | AP25: 23.3 -> 33.4 | Helped |
Surprising findings
- Replacing Mask3D predicted masks with ground-truth masks raises AP25 from 23.3 to 45.4, indicating segmentation noise is the main bottleneck.
- Bird's-eye-view rendering performs worst (AP25 12.7), below uniformly sampled video frames.
Limitations
- Performance depends heavily on initial instance segmentation quality; missing elements break downstream reasoning.
- Scene-centric rendering can cause occlusion and poor visibility of targets.
- Motion axis is discretized into six coarse primitives, limiting precision.
Experimental setup
Datasets: SceneFun3D ScanNet200
Benchmarks: SceneFun3D
Metrics: mIoU AP AP50 AP25 +T +TD
Baselines: OpenMask3D LERF OpenMask3D-F OpenIns3D Fun3DU Fun3DU (+motion)
Builds on: SceneFun3D Fun3DU Mask3D Qwen2.5-VL
Closest papers by meaning
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.93)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.90)
- GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding (CVPR 2025 · similarity 0.90)
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.90)
- IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments (CVPR 2025 · similarity 0.90)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.