Research map / Video / Video generation
Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tri-Ergon is a diffusion-based video-to-audio model that conditions on text, audio, and pixel-level video prompts plus a LUFS loudness embedding to generate up to 60-second 44.1 kHz stereo audio, trained on a new MM-V2A dataset.
Problem
Existing video-to-audio models lack fine-grained control over loudness and multi-modal conditioning, and typically produce mono audio of fixed short duration rather than high-fidelity stereo of variable length.
Approach
Combine an unbalanced multi-modal prompting module (text, audio, and video conditions) with a LUFS loudness embedding injected into a Diffusion Transformer, so loudness can be controlled over time per channel, and generate variable-length stereo audio.
Key results
- Tri-Ergon generates 44.1 kHz stereo audio clips up to 60 seconds, versus mono fixed-duration outputs of prior V2A methods (per abstract claim).
- Claims to outperform existing state-of-the-art V2A methods in both qualitative and quantitative results (specific numbers not in the provided text).
Limitations
- Evaluation details and quantitative results are not included in the provided text excerpt.
Experimental setup
Datasets: MM-V2A
Baselines: SpecVQGAN Im2Wav Difffoley V2A-Mapper
Builds on: CLIP CLAP DINOv2 Stable Audio
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- PrismAudio: Decomposed Chain-of-Thought and Multi-dimensional Rewards for Video-to-Audio Generation (ICLR 2026 · similarity 0.87)
- Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.87)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.87)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.86)
- Animate and Sound an Image (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.