Research map / Vision-language and multimodal / Human and embodied modeling
Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M²SE-VTTS, which uses paired RGB and depth panoramic images, Gemini-generated captions, and patch-level top-k region selection to synthesize reverberant speech for visual text-to-speech.
Problem
Visual Text-to-Speech must understand the spatial environment from an image to synthesize matching reverberant speech, but prior methods use only global RGB features and ignore local and depth cues.
Approach
Combines RGB and depth patch features with CLIP-based local region selection (top-k patches guided by Gemini captions and shared indices from RGB to depth) and a local-aware global attention that fuses multi-modal, multi-scale features into the ViT-TTS backbone.
Key results
- On test-unseen, M²SE-VTTS reaches MOS 3.849, RTE 0.0744 and MCD 4.4215, outperforming ViT-TTS-CLIP (3.651, 0.0772, 4.5871) and the other baselines.
- On test-seen, M²SE-VTTS reaches MOS 3.939, RTE 0.0642 and MCD 4.3809, the best among compared systems.
- Ablations show removing RGB or depth, the LLM captions, or the local/global attention each degrades MOS, RTE and MCD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RGB image features (ablation) | M²SE-VTTS · SoundSpaces-Speech test-unseen | MOS: 3.716 $\pm $ 0.049 -> 3.849 $\pm $ 0.025 | Helped |
| add Depth image features (ablation) | M²SE-VTTS · SoundSpaces-Speech test-unseen | MOS: 3.753 $\pm $ 0.022 -> 3.849 $\pm $ 0.025 | Helped |
| add Gemini-based spatial semantic understanding (ablation) | M²SE-VTTS · SoundSpaces-Speech test-unseen | MOS: 3.749 $\pm $ 0.026 -> 3.849 $\pm $ 0.025 | Helped |
| add local spatial understanding and Top_k region selection (ablation) | M²SE-VTTS · SoundSpaces-Speech test-unseen | MOS: 3.753 $\pm $ 0.025 -> 3.849 $\pm $ 0.025 | Helped |
| add local semantics to local-aware global spatial understanding (ablation) | M²SE-VTTS · SoundSpaces-Speech test-unseen | MOS: 3.698 $\pm $ 0.043 -> 3.849 $\pm $ 0.025 | Helped |
| add global context to local-aware global spatial understanding (ablation) | M²SE-VTTS · SoundSpaces-Speech test-unseen | MOS: 3.703 $\pm $ 0.046 -> 3.849 $\pm $ 0.025 | Helped |
| use shared RGB Top_k indices to select Depth features rather than independent indices | M²SE-VTTS · SoundSpaces-Speech test-unseen | RTE | Helped |
Limitations
- Increased computational complexity from multi-modal and multi-scale integration may hinder real-time use.
- Performance is weaker in unseen environments, indicating limited generalization.
Experimental setup
Datasets: SoundSpaces-Speech
Benchmarks: SoundSpaces-Speech
Metrics: MOS RTE MCD
Baselines: ProDiff DiffSpeech VoiceLDM ViT-TTS-ResNet18 ViT-TTS-CLIP
Builds on: ViT-TTS SRSM CLIP BigVGAN
Closest papers by meaning
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.86)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.86)
- Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.86)
- Synergistic Dual Spatial-aware Generation of Image-to-text and Text-to-image (NeurIPS 2024 · similarity 0.86)
- See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.86)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.