Research map / 3D vision / 3D generation
Neural Multisensory Scene Inference
Jae Hyun Lim, Pedro O O. Pinheiro, Negar Rostamzadeh, Chris Pal, Sungjin Ahn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APoE instead of the summed-context C-GQN baseline | GMN · MESE cross-modal density estimation: HAPTIC→GRAY, HAPTIC→RGB, and RGB→HAPTIC | conditional log-likelihood | Improved |
| Use APoE instead of the summed-context C-GQN baseline | GMN · MESE scene classification from image, haptic, or combined contexts | classification accuracy | Improved |
| Use APoE instead of the summed-context C-GQN baseline | GMN · MESE, unseen context modality configurations | negative ELBO | Improved |
| Use APoE rather than separate full ConvDraw networks for each expert | Multisensory inference network · MESE, 5 modalities | parameters: 53M -> 29M | Improved |
| Use APoE rather than separate full ConvDraw networks for each expert | Multisensory inference network · MESE, 14 modalities | parameters: 131M -> 51M | Improved |
| Use APoE rather than separate full ConvDraw networks for each expert | Multisensory inference network · MESE, 5 modalities | time per iteration: 790 ms -> 679 ms | Improved |
| Use APoE rather than separate full ConvDraw networks for each expert | Multisensory inference network · MESE, 14 modalities | time per iteration: 2059 ms -> 1189 ms | Improved |
| Use APoE rather than the summed-context C-GQN baseline | Multisensory inference network · MESE, 2 modalities | time per iteration: 346 -> 587 | Hurt |
| Use APoE rather than the summed-context C-GQN baseline | Multisensory inference network · MESE, 5 modalities | time per iteration: 397 -> 679 | Hurt |
| Use APoE rather than the summed-context C-GQN baseline | Multisensory inference network · MESE, 8 modalities | time per iteration: 481 -> 992 | Hurt |
| Use APoE rather than the summed-context C-GQN baseline | Multisensory inference network · MESE, 14 modalities | time per iteration: 866 -> 1189 | Hurt |
| Aggregate additional haptic context observations | GMN · MESE cross-modal generation | prediction quality | Improved |
Closest papers by meaning
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.87)
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (CVPR 2024 · similarity 0.87)
- IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments (CVPR 2025 · similarity 0.84)
- SPA: 3D Spatial-Awareness Enables Effective Embodied Representation (ICLR 2025 · similarity 0.84)
- Any2Policy: Learning Visuomotor Policy with Any-Modality (NeurIPS 2024 · similarity 0.84)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
