Research map / 3D vision / 3D generation
InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InteractVLM, which fine-tunes LLaVA with LoRA to produce contact tokens and uses a multi-view SAM-based localization module (Render-Localize-Lift) to predict 3D human and object contacts from single images, then applies them to 3D HOI reconstruction.
Problem
Estimating 3D contact points on human bodies and objects from single in-the-wild images, which is hard because existing methods need scarce 3D contact annotations and foundation VLMs only reason in 2D.
Approach
Render the body and object as multi-view images, use VLM contact tokens (guided by camera-based feature lifting) to prompt a SAM-based multi-view localization model that predicts 2D contact masks, and lift them back to 3D; also introduces Semantic Human Contact conditioned on object labels.
Key results
- Binary human contact on DAMON: F1 75.6% vs. 55.0% for DECO, geodesic error 2.89 cm vs. 21.32 cm.
- Object affordance on PIAD-Seen: SIM 62.7% vs. 54.5% for IAGNet; PIAD-Unseen SIM 41.4% vs. 35.2%.
- Trained on 5% of DAMON, InteractVLM reaches F1 0.58, surpassing DECO trained on full data (0.55); 3D reconstruction preferred 62% of the time over PHOSA in a perceptual study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InteractVLM instead of DECO for binary human contact prediction | DAMON · DAMON, binary human contact | F1: 55.0 -> 75.6 | Helped |
| Use InteractVLM instead of DECO for binary human contact prediction | DAMON · DAMON, binary human contact | Geodesic distance: 21.32 -> 2.89 | Helped |
| Use InteractVLM instead of Semantic-DECO for semantic human contact | DAMON · DAMON, semantic human contact across seven object categories | F1: 40.1 -> 61.1; 26.5 -> 68.6; 11.7 -> 66.4; 24.5 -> 60.5; 27.7 -> 71.8; 36.4 -> 77.9; 52.0 -> 77.8 | Helped |
| Use InteractVLM instead of IAGNet for object affordance prediction | PIAD-Seen · PIAD-Seen | SIM: 54.5 -> 62.7 | Helped |
| Use InteractVLM instead of IAGNet for object affordance prediction | PIAD-Unseen · PIAD-Unseen | SIM: 35.2 -> 41.4 | Helped |
| Use InteractVLM instead of IAGNet for object affordance prediction | PIAD-Seen · PIAD-Seen | MAE: 0.98 -> 0.81 | Helped |
| Use InteractVLM instead of IAGNet for object affordance prediction | PIAD-Unseen · PIAD-Unseen | MAE: 1.27 -> 0.99 | Helped |
| Train InteractVLM with 1% rather than 100% of DAMON training data | InteractVLM · DAMON, binary human contact | F1: 0.75 -> 0.53 | Hurt |
| Train InteractVLM with 5% rather than 100% of DAMON training data | InteractVLM · DAMON, binary human contact | F1: 0.75 -> 0.58 | Hurt |
| Use InteractVLM reconstruction instead of PHOSA reconstruction | 3D human-object reconstruction · Perceptual study, 55 images | Preference rate: 62% | Helped |
Surprising findings
- With only 1% of DAMON training data, InteractVLM reaches F1 0.53, nearly matching DECO trained on 100% of the data (0.55).
Limitations
- Object contact is learned from 3D affordance data (PIAD) as a proxy for contact, not true contact labels.
- Object shapes are retrieved from a database (OpenShape/Objaverse) at inference, so reconstruction depends on retrieval quality.
Experimental setup
Datasets: DAMON PIAD 3DIR Objaverse
Benchmarks: DAMON PIAD
Metrics: F1 precision recall geodesic distance SIM MAE AUC aIOU
Baselines: POSA PIXIE BSTRO DECO Semantic-DECO PMF ILN PFusion XMF IAGNet PHOSA
Builds on: LISA LLaVA SAM DECO PHOSA
Closest papers by meaning
- ContactField: Implicit Field Representation for Multi-Person Interaction Geometry (NeurIPS 2024 · similarity 0.90)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.89)
- PICO: Reconstructing 3D People In Contact with Objects (CVPR 2025 · similarity 0.89)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.88)
- Pose Priors from Language Models (CVPR 2025 · similarity 0.88)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.