Research map / 3D vision / 3D generation
CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CORE4D, a large-scale 4D human-object-human interaction dataset for collaborative object rearrangement (1K mocap sequences plus 10K retargeted synthetic sequences), and benchmarks motion forecasting and interaction synthesis on it.
Problem
Modeling cooperative human-object-human rearrangement of household objects is limited by a lack of large-scale, richly annotated multi-person interaction data covering diverse object shapes.
Approach
Capture real two-person collaboration motions with a hybrid mocap/camera rig, then augment them to thousands of same-category ShapeNet objects using an iterative collaboration retargeting pipeline that combines DeepSDF-based contact retargeting, optimization-based interaction retargeting, and a learned human pose discriminator with beam search over contact candidates.
Key results
- Motion forecasting human-joint error rises for unseen objects (S2) with MDM at 186.4 vs 170.8 on seen objects (S1).
- Adding CORE4D-Synthetic to training reduces motion forecasting Je from 127.7 (1.0K real + 0.1K/0.9K synthetic setting A) to 116.2 when training on 5.0K total (C).
- Humanoid box lifting with CORE4D-trained HumanPlus reaches 26.5% success rate versus 0.0% for PPO RL baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CORE4D-Synthetic training data (training set A versus B) (ablation) | CAHMP · Motion forecasting S2 | Re: 9.48 -> 8.04 | Helped |
| Increase training data to 5.0K sequences, including 4.0K synthetic (ablation) | CAHMP · Motion forecasting S2 | Je: 127.0 -> 116.2 | Helped |
| Use CORE4D-Real contact candidates rather than only source-trajectory contact points (ablation) | Collaboration retargeting · Retargeting ablation Abl.1 | penetration distance: 0.61 -> 0.24 | Helped |
| Add the human pose discriminator to select contact candidates (ablation) | Collaboration retargeting · Retargeting ablation Abl.2 | penetration distance: 0.55 -> 0.24 | Helped |
| Select candidates with the human pose discriminator instead of selecting the motion with the most accurate contact (ablation) | Collaboration retargeting · Retargeting ablation Abl.3 | Cacc: 94.7 -> 83.3 | Mixed |
| Update contact candidates during the search (ablation) | Collaboration retargeting · Retargeting ablation Abl.4 | penetration distance: 0.24 -> 0.23 | Helped |
| Train using CORE4D demonstrations rather than no demonstration data | Humanoid box lifting with PPO · Humanoid box lifting | SR: 0.0 -> 21.0 | Helped |
Surprising findings
- In user study, synthetic data was judged comparable to real data in 43% of cases and preferred over real in 14% of cases.
Limitations
- Outdoor scenes are not included in CORE4D-Real due to the mocap system.
- CORE4D-Synthetic excludes visual signals.
- Existing motion forecasting and synthesis methods show degraded generalization to unseen object shapes.
Experimental setup
Datasets: CORE4D-Real ShapeNet CORE4D-Synthetic
Benchmarks: CORE4D motion forecasting (S1, S2) CORE4D interaction synthesis (S1, S2) Isaac Gym humanoid box lifting
Metrics: Je Te Re contact accuracy penetration rate R.Je R.Ve FID success rate user preference
Baselines: MDM InterDiff CAHMP OMOMO CHOIS PPO HumanPlus ACT
Builds on: OakInk CHAIRS DeepSDF MDM CHOIS
Closest papers by meaning
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions (NeurIPS 2024 · similarity 0.88)
- FAVOR: Full-Body AR-Driven Virtual Object Rearrangement Guided by Instruction Text (AAAI 2024 · similarity 0.88)
- AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation (ICLR 2025 · similarity 0.87)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.87)
- ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.