Research map / 3D vision / 3D generation
Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Predicts the sounds of human hand actions in 3D Gaussian-splatted scenes by training a rectified flow audio model on a newly captured dataset of 3D hand-scene interactions paired with sound.
Problem
Making 3D scene reconstructions interactive by predicting the sound produced when a human hand physically interacts with objects in the scene, given a user-specified 3D hand trajectory.
Approach
Condition a rectified flow audio generator (adapted from Frieren) on CLIP features of rendered hand-in-scene views plus the 3D hand pose trajectory, trained on action-sound pairs collected by lifting real hand videos into Gaussian-splatted scene reconstructions, with multi-view synthetic rendering as augmentation.
Key results
- Full model reaches about 47% misclassification rate in a 59-participant real-or-fake 2AFC study, near the 50% chance level.
- Full model achieves CLAP-acc of 28.09% overall versus 23.94% for Frieren, and lower STFT distance (0.50 vs 0.74).
- Removing hand pose conditioning drops action CLAP accuracy to 39.11%, while removing CLIP drops material CLAP accuracy to 31.80%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hearing Hands instead of Frieren | Hearing Hands · Interaction test set | STFT: 0.74 -> 0.50 | Helped |
| Use Hearing Hands instead of Frieren | Hearing Hands · Interaction test set | CLAP-acc all (%): 41.73 -> 50.50 | Helped |
| Add CLIP visual features (ablation) | Hearing Hands · Interaction test set | CLAP-acc material (%): 41.24 ± 2.62 -> 47.18 ± 2.66 | Helped |
| Add hand-pose conditioning (ablation) | Hearing Hands · Interaction test set | CLAP-acc action (%): 39.11 -> 45.62 | Helped |
| Add synthetic-view data augmentation (ablation) | Hearing Hands · Interaction test set | CLAP-acc all (%): 47.61 -> 50.50 | Helped |
Surprising findings
- Off-the-shelf Frieren fails to generate high-quality sound from simulated interaction videos even when fine-tuned on the dataset, motivating the CLIP and hand-pose modifications.
Limitations
- Assumes objects in the scene do not move or deform when manipulated.
- Errors from 3D hand detection (HaMeR) can produce inaccurate hand motion in the dataset.
- Generated audio is less synchronized when the hand is less visible or the motion is ambiguous.
Experimental setup
Datasets: self-collected 3D hand-scene interaction dataset (24 scenes, 9.1 hours)
Benchmarks: self-collected 3D hand-scene interaction dataset
Metrics: STFT distance envelope distance CDPAM FID IS CLAP accuracy real-or-fake misclassification rate (2AFC user study)
Baselines: Frieren RegNet
Builds on: Frieren Gaussian Splatting HaMeR CLIP Diff-Foley
Closest papers by meaning
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.86)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.86)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.85)
- 🎧MOSPA: Human Motion Generation Driven by Spatial Audio (NeurIPS 2025 · similarity 0.85)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.85)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.