Research map / Detection and segmentation / Open-set and few-shot detection
3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates 3D affordance detection as instruction-reasoning affordance segmentation and builds 3D-ADLLM, an LLM with an <AFF> token and mask decoder, pretrained on PartNet part segmentation (ROPS) and fine-tuned on 3D AffordanceNet-based IRAS data.
Problem
Existing 3D affordance detection relies on fixed label sets and cannot comprehend complex natural-language instructions, limiting open-world generalization; 3D affordance data is also scarce for training large models.
Approach
Reframe affordance detection as instruction-reasoning segmentation driven by LLM text, where a special <AFF> token's hidden embedding is decoded into a point-wise affordance mask (LISA-style embedding-as-mask), with transfer from PartNet referring part segmentation pretraining.
Key results
- Ours-Phi reaches 30.43 mIoU(c) on full-view open-set, versus 22.41 for LASO (about 8% gain).
- On all-instance evaluation, 3D-ADLLM gets 27.80 mAP50 full-view versus 2.42 for LASO.
- On the out-of-distribution AffordPose subset, 3D-ADLLM gets 36.33 mIoU(i) versus 34.49 for LASO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3D-ADLLM instead of LASO | Open-vocabulary affordance detection · Full-view | mIoU^c: 22.41 -> 30.43 | Helped |
| use 3D-ADLLM instead of LASO | Open-vocabulary affordance detection · Partial-view | mIoU^c: 20.06 -> 27.25 | Helped |
| use 3D-ADLLM instead of LASO | OOD affordance detection · Full-view | mIoU^i: 20.47 -> 30.28 | Helped |
| use 3D-ADLLM instead of LASO | OOD affordance detection · Partial-view | mIoU^i: 11.46 -> 28.72 | Helped |
| use 3D-ADLLM instead of LASO | AffordPose open-vocabulary detection · AffordPose | mAP50^i: 8.40 -> 36.33 | Helped |
| add pretrained weights for the point backbone and affordance decoder (ablation) | 3D-ADLLM · Open-vocabulary affordance detection, full-view | mIoU^c: 24.82 -> 30.43 | Helped |
| add the sample unbalanced loss factor (ablation) | 3D-ADLLM · Open-vocabulary affordance detection, full-view | mIoU^c: 25.35 -> 30.43 | Helped |
| use Dice loss without BCE loss instead of the combined Dice and BCE losses (ablation) | 3D-ADLLM · Open-vocabulary affordance detection, full-view | mIoU^c: 30.43 -> 31.00 | Helped |
| use Dice loss without BCE loss instead of the combined Dice and BCE losses (ablation) | 3D-ADLLM · Closed-set affordance detection, full-view | mIoU^c: 42.35 -> 38.65 | Hurt |
| use BCE loss only instead of combined Dice and BCE losses (ablation) | 3D-ADLLM · Open-vocabulary affordance detection, full-view | mIoU^c: 30.43 -> 15.99 | Hurt |
| replace the Phi-3.5-mini-instruct backbone with Qwen2-1.5B (ablation) | 3D-ADLLM · Full-view vs partial-view | mIoU^c | Mixed |
| replace the ULIP2 point encoder with Uni3D (ablation) | 3D-ADLLM · Open-vocabulary affordance detection, full-view | mIoU^c: 30.43 -> 30.26 | Hurt |
Surprising findings
- ShapeLLM without fine-tuning scores under 1.5 mIoU, indicating that grounding-capable point LLMs do not transfer directly to affordance masks.
- The Uni3D point encoder gives higher mean accuracy than ULIP-2 in one setting, though ULIP-2 gives better mIoU.
Limitations
- Training data scarcity: the IRAS dataset is built only from 3D AffordanceNet and is not released.
- Closed-set mIoU is not best on all metrics, and mAcc and mPrec trail LASO in some instance-level settings.
- Hardware and training cost details are not reported.
Experimental setup
Datasets: PartNet 3D AffordanceNet AffordPose ModelNet
Benchmarks: 3D AffordanceNet (OpenAD split) AffordPose out-of-distribution subset
Metrics: mIoU Acc mAcc mPrec mRec mAP50
Baselines: ZSLPC TZSLPC 3DGenZ OpenAD-PointNet++ OpenAD-DGCNN IAGNet LASO ShapeLLM
Builds on: LISA ULIP-2 Point-BERT Point Transformer OpenAD
Closest papers by meaning
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.93)
- LASO: Language-guided Affordance Segmentation on 3D Object (CVPR 2024 · similarity 0.91)
- UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation (NeurIPS 2026 · similarity 0.91)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.90)
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.90)
- GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.