Research map / Video / Video generation
Learning to Highlight Audio by Watching Movies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces visually-guided acoustic highlighting, a transformer-based dual U-Net model (VisAH) that remixes poorly mixed movie audio guided by video or captions, trained on a new movie-derived dataset (Muddy Mix).
Problem
Poorly balanced audio in user-generated videos often mismatches visual saliency; the task is to re-balance audio sources (speech, music, sound effects) so they align with the video's highlighting.
Approach
Treat acoustic highlighting as audio-to-audio translation: a dual (spectrogram + waveform) U-Net backbone with a latent transformer that cross-attends to temporal video (CLIP) or caption (InternVL2-generated, T5-encoded) context, trained on pseudo-poorly-mixed movie audio created by separate-adjust-remix from movie clips used as free supervision.
Key results
- VisAH reduces MAG from 22.69 (poorly mixed input) to 10.08 and KLD from 20.61 to 11.01 on the test set, beating Learn2Remix and LCE baselines.
- Temporal text context with 6 transformer layers gives the best result (MAG 10.08, KLD 11.01, W-dis 0.79).
- In a 9-participant subjective test, VisAH achieves a top-2 ranking rate of 77% and surpasses ground truth in 34% of videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use temporal text context instead of no context (ablation) | VisAH · THE MUDDY MIX test | KLD: 11.95 -> 11.01 | Helped |
| Use temporal text context instead of no context (ablation) | VisAH · THE MUDDY MIX test | ΔIB: 0.99 -> 0.80 | Helped |
| Use temporal vision context instead of no context (ablation) | VisAH · THE MUDDY MIX test | KLD: 11.95 -> 11.18 | Helped |
| Use semantic vision context instead of no context (ablation) | VisAH · THE MUDDY MIX test | ΔIB: 0.99 -> 0.91 | Helped |
| Use semantic text context instead of no context (ablation) | VisAH · THE MUDDY MIX test | ΔIB: 0.99 -> 0.84 | Helped |
| Increase video transformer encoder layers from 0 to 3 (ablation) | VisAH with vision context · THE MUDDY MIX test | MAG: 10.36 -> 10.24 | Helped |
| Increase video transformer encoder layers from 3 to 6 (ablation) | VisAH with vision context · THE MUDDY MIX test | MAG: 10.24 -> 10.69 | Hurt |
| Increase text transformer encoder layers from 0 to 6 (ablation) | VisAH with text context · THE MUDDY MIX test | MAG: 10.66 -> 10.08 | Helped |
| Train and test with high-level adjustment difficulty rather than the mixed-difficulty training setup (ablation) | VisAH · THE MUDDY MIX test, high difficulty | MAG: 27.70 -> 12.03 | Helped |
| Train and test with moderate-level adjustment difficulty rather than the mixed-difficulty training setup (ablation) | VisAH · THE MUDDY MIX test, moderate difficulty | MAG: 22.70 -> 8.73 | Helped |
| Train and test with low-level adjustment difficulty rather than the mixed-difficulty training setup (ablation) | VisAH · THE MUDDY MIX test, low difficulty | MAG: 16.40 -> 9.55 | Helped |
| Use VisAH rather than the poorly mixed input | THE MUDDY MIX test · THE MUDDY MIX test | MAG: 22.69 -> 10.08 | Helped |
Surprising findings
- VisAH's output sometimes is preferred over the ground-truth movie audio (34% of videos in subjective test).
- Mean normalization of the spectrogram was removed because it suppressed ambient sounds and reduced sensitivity to sound effects.
Limitations
- Dataset limited to CMD movies tagged 'Action' and expansion is left to future work.
- Poorly mixed inputs are synthetic (separation-adjust-remix), so real-world mismatch is only approximated.
- Training loss is a simple MR-STFT loss; alternative loss designs were not explored.
Experimental setup
Datasets: Condensed Movie Dataset (CMD) DnR v3
Benchmarks: THE MUDDY MIX DATASET test split
Metrics: MAG ENV KLD delta IB Wasserstein distance subjective ranking
Baselines: Poorly Mixed Input DnRv3+CDX Learn2Remix Listen, Chat and Edit (LCE)
Builds on: HybridDemucs Learn2Remix Listen, Chat and Edit MR-STFT loss CLIP
Closest papers by meaning
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video (CVPR 2025 · similarity 0.87)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.87)
- SoundBrush: Sound as a Brush for Visual Scene Editing (AAAI 2025 · similarity 0.86)
- SonicVisionLM: Playing Sound with Vision Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.