Research map / Video / Video generation
MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMAudio, a 157M-1.03B flow-matching transformer trained jointly on video-audio-text and audio-text data plus a frame-level synchronization module, achieving state-of-the-art video-to-audio synthesis among public models.
Problem
Video-to-audio (Foley) synthesis requires audio that is both semantically aligned and temporally synchronized with video, but audio-visual training data is scarce (VGGSound ~550 hours) and existing methods either train from scratch on it or bolt control modules onto text-to-audio models.
Approach
Jointly train a single multimodal transformer (MM-DiT style) on audio-visual-text and large audio-text datasets with missing modalities masked, using flow matching; add a conditional synchronization module that injects 24 fps Synchformer features via frame-aligned adaLN, plus aligned RoPE and ConvMLP for temporal alignment.
Key results
- MMAudio-S-16kHz (157M) reaches FD_PaSST 70.19 and DeSync 0.483 on VGGSound, improving over prior public video-to-audio methods while inferring an 8s clip in 1.23s on an H100.
- Joint training yields roughly 10% lower FD and 15% higher IS relative to the audio-visual-only ablation (FD_PaSST 77.38 to 70.19, IS 12.53 to 14.44).
- On AudioCaps text-to-audio, MMAudio-L-44.1kHz attains the highest CLAP (0.348) and IS (12.08) among the compared models without fine-tuning, while its FD_VGG is worse than GenAU-Large.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly train on audio-visual-text and audio-text data (ablation) | MMAudio · VGGSound test | FD_PaSST: 77.38 -> 70.19 | Helped |
| jointly train on audio-visual-text and audio-text data (ablation) | MMAudio · VGGSound test | IS: 12.53 -> 14.44 | Helped |
| add text to audio-visual training data (ablation) | MMAudio · VGGSound test | FD_PaSST: 72.77 -> 70.19 | Helped |
| add text to audio-text training data (ablation) | MMAudio · VGGSound test | IB-score: 28.72 -> 29.13 | Helped |
| add uncaptioned audio data (ablation) | MMAudio · VGGSound test | IS: 14.30 -> 14.44 | Helped |
| add audio-text training data (ablation) | MMAudio · VGGSound test | DeSync: 0.562 -> 0.483 | Helped |
| increase audio-text data from 10% to 100% (ablation) | MMAudio · VGGSound test | FD_PaSST: 79.21 -> 70.19 | Helped |
| use the conditional synchronization module instead of no synchronization features (ablation) | MMAudio · VGGSound test | DeSync: 0.973 -> 0.483 | Helped |
| use the conditional synchronization module instead of summing sync and visual features (ablation) | MMAudio · VGGSound test | IS: 16.70 -> 14.44 | Hurt |
| use aligned RoPE embeddings (ablation) | MMAudio · VGGSound test | DeSync: 0.509 -> 0.483 | Helped |
| replace MLP with ConvMLP (ablation) | MMAudio · VGGSound test | FD_PaSST: 73.84 -> 70.19 | Helped |
| replace MLP with ConvMLP (ablation) | MMAudio · VGGSound test | DeSync: 0.533 -> 0.483 | Helped |
Surprising findings
- Joint training with text-audio data yields competitive text-to-audio generation without any text-to-audio fine-tuning, and the authors report that adding uncaptioned audio also helps.
- Without synchronization features the DeSync score degrades sharply (0.483 to 0.973), indicating the sync module is critical for temporal alignment.
Limitations
- Generates unintelligible mumbles when prompted for human speech.
- Diminishing returns for larger models, limited by data quality and the amount of audio-visual data.
- Worse FD_VGG on text-to-audio compared with recent works.
Experimental setup
Datasets: VGGSound AudioCaps Clotho WavCaps Greatest Hits
Benchmarks: VGGSound test set AudioCaps test set Greatest Hits
Metrics: FD_PaSST FD_PANNs FD_VGG KL_PANNs KL_PaSST IS IB-score DeSync CLAP onset accuracy
Baselines: ReWaS Seeing&Hearing V-AURA VATT Frieren FoleyCrafter V2A-Mapper AudioLDM 2-L TANGO TANGO 2 Make-An-Audio 2 GenAU-Large
Builds on: MM-DiT Frieren Synchformer FLUX conditional flow matching
Closest papers by meaning
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.90)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.89)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.88)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.88)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.87)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.