Research map / 3D vision / 3D generation
Scaling Up Dynamic Human-Scene Interaction Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TRUMANS, a 15-hour motion-capture human-scene interaction dataset across 100 indoor scenes, and a diffusion-based autoregressive model that generates arbitrary-length, scene- and action-conditioned HSI motions in real time.
Problem
Human-scene interaction (HSI) modeling is limited by scarce high-quality motion-capture data with object dynamics and contact, and by difficulty generating long, scene-aware, action-controllable human motions.
Approach
Combine a large VICON-captured HSI dataset with physically-consistent augmentation of object sizes, and an autoregressive conditional diffusion transformer conditioned on a local voxel scene perceiver and frame-wise action labels with progress indicators.
Key results
- On the static setting, the proposed model achieves contact 0.992 and max penetration 11.74 on TRUMANS, versus 0.969/14.33 for Wang et al.'s cVAE baseline.
- Adding TRUMANS to 3DPW training reduces MPVE from 101.3 to 78.5 (1:1 ratio) for Ma et al.'s mesh estimation.
- Human study: participants identified the MoCap sequence with about 25% success, close to the 20% random-guess rate; the model generates an episode of 1.6 s at 10 FPS in 0.7 s on an A800 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on TRUMANS rather than PROX | Ours · static HSI generation | Contact: 0.723 vs 0.992 | Helped |
| train on TRUMANS rather than PROX | Ours · static HSI generation | Penemean: 1.820 vs 4.820 | Mixed |
| add data augmentation (ablation) | Ours · TRUMANS static setting | Penemean: 2.010 -> 1.820 | Helped |
| add data augmentation (ablation) | Ours · TRUMANS static setting | Penemax: 15.52 -> 11.74 | Helped |
| add the action progress indicator Aind (ablation) | Ours · TRUMANS dynamic setting | FID: 2.104 -> 0.313 | Helped |
| add the action progress indicator Aind (ablation) | Ours · TRUMANS dynamic setting | Diversity: 1.318 -> 2.693 | Helped |
| add the action progress indicator Aind (ablation) | Ours · TRUMANS dynamic setting | Dis. suc.: 1.000 -> 0.226 | Helped |
| use Ours instead of GOAL | HSI generation · TRUMANS dynamic setting | FID: 0.512 -> 0.313 | Helped |
| use Ours instead of GOAL | HSI generation · GRAB dynamic setting | Diversity: 2.180 -> 2.150 | Hurt |
| use Ours instead of GOAL | HSI generation · GRAB dynamic setting | Penescene: 44.09 -> 34.41 | Helped |
Surprising findings
- Disabling the action progress indicator causes the dynamic-setting method to fail (FID rises from 0.313 to 2.104).
- Models trained on TRUMANS did not uniformly beat baselines; on the PROX static evaluation the method had higher mean penetration than some baselines (4.820 vs 3.267 for SceneDiff).
Limitations
- Scenes are discretized into voxel grids, a trade-off that avoids a ~300x slower training pipeline but simplifies geometry.
- High penetration rates with GRAB-trained methods indicate limited scene-adherent HOI motion.
- Only 7 participants performed the captures, limiting human diversity.
Experimental setup
Datasets: TRUMANS PROX GRAB 3DPW RICH DAMON 3D-FRONT BlenderKit
Benchmarks: TRUMANS PROX GRAB 3DPW RICH DAMON
Metrics: contact penetration mean penetration max discrimination success rate FID diversity MPJPE PA-MPJPE MPVE precision
Baselines: cVAE (Wang et al.) SceneDiff GMD GOAL IMoS SAMP DIMOS LAMA BSTRO DECO Ma et al. mesh estimation
Builds on: DDPM Shafir et al. autoregressive diffusion (anchor-based inpainting) IMoS GOAL SMPL-X
Closest papers by meaning
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.91)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.89)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.89)
- HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.87)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.87)
- GenZI: Zero-Shot 3D Human-Scene Interaction Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.