Research map / 3D vision / 3D generation
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoChoir, which fuses egocentric video, head motion, and a 3D object point cloud with parallel cross-attention and gradient modulation to jointly predict 3D human contact and object affordance, and releases a newly annotated egocentric contact/affordance dataset from Ego-Exo4D and GIMO.
Problem
Estimating where human-object interaction manifests in 3D (human contact and object affordance) from egocentric views, where interacting parties are often partially or fully invisible, causing ambiguity between visual observations and interaction content.
Approach
Harmonizes visual appearance, relative head motion, and 3D object geometry: object affordance is first inferred from geometry queried against appearance and motion clues, then used to query head motion and appearance to model human contact; learnable modulation tokens scale features to modulate gradients so the model adaptively uses appropriate clues across hand and body interactions.
Key results
- EgoChoir reaches human contact F1 0.76 and geodesic error 12.62 cm, versus 0.67 F1 and 21.43 cm for LEMON, the strongest baseline.
- For object affordance, EgoChoir achieves AUC 78.02, aIOU 14.94 and SIM 0.436, improving aIOU by about 55% over O2O-Afford's 9.65 (baseline table lists O2O at 9.65 aIOU).
- Removing head motion drops contact F1 from 0.76 to 0.69 and AUC from 78.02 to 74.36.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EgoChoir instead of LEMON | Egocentric 3D human contact estimation · Egocentric human contact | F1: 0.67 -> 0.76 | Helped |
| Use EgoChoir instead of LEMON | Egocentric 3D object affordance estimation · Egocentric object affordance | AUC: 75.97 -> 78.02 | Helped |
| Add head motion (ablation) | EgoChoir · Egocentric human contact | F1: 0.69 -> 0.76 | Helped |
| Add affordance to contact modeling (ablation) | EgoChoir · Egocentric human contact | F1: 0.66 -> 0.76 | Helped |
| Add gradient modulation (ablation) | EgoChoir · Egocentric human contact | F1: 0.71 -> 0.76 | Helped |
| Add semantic feature (ablation) | EgoChoir · Egocentric human contact | F1: 0.73 -> 0.76 | Helped |
| Add contact-affordance cross-attention (ablation) | EgoChoir · Egocentric human contact | F1: 0.72 -> 0.76 | Helped |
| Add temporal position encoding (ablation) | EgoChoir · Egocentric human contact | Precision: 0.75 -> 0.78 | Helped |
| Pre-train the motion encoder instead of randomly initializing it (ablation) | EgoChoir · Egocentric human contact | F1: 0.70 -> 0.76 | Helped |
| Use the HRNet-based video feature extractor instead of SlowFast (ablation) | EgoChoir · Egocentric human contact | F1: 0.72 -> 0.76 | Helped |
| Use the HRNet-based video feature extractor instead of LaViLa (ablation) | EgoChoir · Egocentric human contact | F1: 0.74 -> 0.76 | Helped |
| Use joint instead of divided space-time attention (ablation) | EgoChoir · Egocentric human contact | F1: 0.75 -> 0.76 | Helped |
Surprising findings
- Pre-trained egocentric video backbones (SlowFast, LaViLa) homogenize features across frames, yielding nearly identical contact predictions over a sequence and lower precision.
- Using whole-body motion instead of head motion further improves results (F1 0.79, geo. 11.24 cm), suggesting head motion is not the upper bound.
Limitations
- May predict interaction regions slightly before or after the exact contact frame due to lacking spatial relation perception between parties.
- Cannot cover all interaction types, which may lead to confusing predictions.
- Dataset covers only 12 interactions with 18 object categories.
Experimental setup
Datasets: Ego-Exo4D GIMO 3D object datasets (over 20K instances)
Benchmarks: EgoChoir egocentric contact and affordance dataset (Ego-Exo4D and GIMO derived)
Metrics: Precision Recall F1 geodesic error AUC aIOU SIM
Baselines: BSTRO DECO LEMON O2O-Afford IAG-Net
Builds on: LEMON DECO HRNet DGCNN Ego-Exo4D
Closest papers by meaning
- Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.89)
- ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · similarity 0.89)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.87)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.87)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.87)
- SAW-Bench: Learning Situated Awareness in the Real World (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.