Research map / 3D vision / 3D generation
LEMON: Learning 3D Human-Object Interaction Relation from 2D Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEMON, a unified model that jointly predicts 3D human contact, object affordance, and human-object spatial relation from a 2D image plus human mesh and object point cloud, and releases the 3DIR dataset.
Problem
Predicting 3D human-object interaction elements (contact, affordance, spatial relation) in isolation ignores correlations between interaction counterparts and suffers from intention and region uncertainty under limited views and occlusion.
Approach
Jointly anticipate the three interaction elements by (1) using multi-branch attention from image features to extract intention tokens for human and object geometries with a cosine-similarity consistency term, and (2) using normal curvatures plus intention tokens to guide cross-attention between human and object geometry, then predicting spatial relation conditioned on contact.
Key results
- On 3DIR, LEMON (DGCNN backbone) reaches human contact F1 0.78 and geodesic error 7.55 cm, versus 0.70 F1 and 15.25 cm for DECO.
- Object affordance AUC 88.51 and aIOU 41.34 versus 86.63 and 38.57 for the best prior method IAG.
- Spatial relation MSE 0.010 versus 0.027 for PopUp, with ablations showing removing intention or geometry modeling degrades all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LEMON with DGCNN instead of DECO | 3DIR human contact estimation · 3DIR test | F1: 0.69 -> 0.78 | Helped |
| Use LEMON with DGCNN instead of IAG | 3DIR object affordance estimation · 3DIR test | AUC: 86.63 -> 88.51 | Helped |
| Use LEMON with DGCNN instead of PopUp | 3DIR spatial relation prediction · 3DIR test | MSE: 0.027 -> 0.010 | Helped |
| Add interaction intention modeling (ablation) | LEMON human contact estimation · 3DIR test | F1: 0.70 -> 0.78 | Helped |
| Add semantic consistency constraint φ (ablation) | LEMON human contact estimation · 3DIR test | F1: 0.74 -> 0.78 | Helped |
| Add geometric correlation modeling (ablation) | LEMON human contact estimation · 3DIR test | F1: 0.67 -> 0.78 | Helped |
| Add curvature features (ablation) | LEMON human contact estimation · 3DIR test | F1: 0.75 -> 0.78 | Helped |
| Add contact feature φc to spatial relation prediction (ablation) | LEMON · 3DIR test | MSE: 0.037 -> 0.010 | Helped |
| Add relative-distance loss Lpr (ablation) | LEMON spatial relation prediction · 3DIR test | MSE: 0.024 -> 0.010 | Helped |
| Add affordance feature φa (ablation) | LEMON spatial relation prediction · 3DIR test | MSE: 0.019 -> 0.010 | Helped |
Limitations
- Requires a pre-inferred human mesh as input rather than predicting it end-to-end.
- Dataset is relatively small (about 5k images) and annotations are partly manual.
Experimental setup
Datasets: 3DIR BEHAVE HAKE V-COCO PIAD DAMON
Benchmarks: 3DIR DAMON BEHAVE PIAD
Metrics: precision recall F1 geodesic error (cm) AUC aIOU SIM MSE
Baselines: BSTRO DECO 3DA. IAG DJ-RN PopUp Baseline (multitask)
Builds on: DECO DJ-RN 3D-AffordanceNet DGCNN
Closest papers by meaning
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.89)
- Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation (CVPR 2024 · similarity 0.87)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.86)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.86)
- InteractVLM: 3D Interaction Reasoning from 2D Foundational Models (CVPR 2025 · similarity 0.86)
- EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.