Research map / Vision-language and multimodal / Robot action and navigation
Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the task of grounding 3D object affordance from language, images and point clouds, releases the AGPIL dataset (full, partial and rotation views) and the LMAffordance3D baseline built on a frozen LLaVA-7B.
Problem
Grounding 3D object affordance regions from language instructions plus visual observations (images and point clouds), including partial and rotated object views, which prior 3D affordance methods handle poorly for unseen affordances and categories.
Approach
A multi-modal vision encoder (ResNet18 for images, PointNet++ for point clouds) fuses 2D and 3D spatial features, projects them into a VLM's text space via an adapter, and uses a cross-attention decoder with spatial queries over the VLM output to predict point-wise affordance heatmaps.
Key results
- LMAffordance3D reaches AUC 0.8895, aIoU 0.2123, SIM 0.6102 on full-view seen setting, outperforming PointRefer and other baselines across views.
- On unseen full-view setting, AUC 0.7741 vs 0.7375 for OpenAD, showing better generalization to unseen affordances.
- Removing the interaction image lowers seen full-view AUC from 0.8895 to 0.8827 and unseen from 0.7741 to 0.7604; using full GPT-4o instructions beats empty instructions (0.8263 AUC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add interaction image input (ablation) | LMAffordance3D · AGPIL full-view seen | AUC: 0.8827 -> 0.8895 | Helped |
| add interaction image input (ablation) | LMAffordance3D · AGPIL full-view unseen | AUC: 0.7604 -> 0.7741 | Helped |
| add interaction image input (ablation) | LMAffordance3D · AGPIL partial-view seen | AUC: 0.8383 -> 0.8478 | Helped |
| add interaction image input (ablation) | LMAffordance3D · AGPIL partial-view unseen | AUC: 0.7426 -> 0.7602 | Helped |
| add interaction image input (ablation) | LMAffordance3D · AGPIL rotation-view seen | AUC: 0.7732 -> 0.7823 | Helped |
| add interaction image input (ablation) | LMAffordance3D · AGPIL rotation-view unseen | AUC: 0.6126 -> 0.6303 | Helped |
| use full GPT-4o instruction instead of no instruction (ablation) | LMAffordance3D · AGPIL full-view seen | AUC: 0.8263 -> 0.8895 | Helped |
| use full GPT-4o instruction instead of no instruction (ablation) | LMAffordance3D · AGPIL full-view unseen | AUC: 0.6741 -> 0.7741 | Helped |
| use full GPT-4o instruction instead of no instruction (ablation) | LMAffordance3D · AGPIL partial-view seen | AUC: 0.8281 -> 0.8478 | Helped |
| use full GPT-4o instruction instead of no instruction (ablation) | LMAffordance3D · AGPIL partial-view unseen | AUC: 0.6513 -> 0.7602 | Helped |
| use full GPT-4o instruction instead of no instruction (ablation) | LMAffordance3D · AGPIL rotation-view seen | AUC: 0.7052 -> 0.7823 | Helped |
| use full GPT-4o instruction instead of no instruction (ablation) | LMAffordance3D · AGPIL rotation-view unseen | AUC: 0.5899 -> 0.6303 | Helped |
Surprising findings
- Unseen-setting results are sometimes higher than seen-setting results for some affordances, and the paper attributes overall gains to LLM world knowledge.
- Empty instruction ablation still gives reasonable AUC, showing the image input carries much of the signal.
Limitations
- Only rigid objects are covered; flexible objects are not studied.
- Unseen-setting performance remains low for many affordances (e.g., aIoU around 0.01 to 0.15 on several categories).
- Real-world evaluation is only qualitative with two examples.
Experimental setup
Datasets: AGPIL 3D AffordanceNet AGD20K PIAD ImageNet
Benchmarks: AGPIL
Metrics: AUC aIoU SIM MAE
Baselines: 3D AffordanceNet IAG OpenAD PointRefer
Builds on: LLaVA 3D AffordanceNet PIAD PointNet++
Closest papers by meaning
- GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding (CVPR 2025 · similarity 0.93)
- Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding (AAAI 2025 · similarity 0.92)
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.91)
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.90)
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.90)
- ScanERU: Interactive 3D Visual Grounding Based on Embodied Reference Understanding (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.