Research map / Video / Video generation

UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities

NeurIPS 2026 · Video › Video generation · approach: Video-to-audio synthesis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniFlow-Audio, a flow-matching non-autoregressive model that unifies time-aligned and non-time-aligned audio generation tasks (TTS, SVS, T2A, T2M, SE, SR, V2A) using dual fusion, block-wise fusion, and task-balanced sampling.

Problem

Audio generation tasks are split into time-aligned (TA) and non-time-aligned (NTA) categories that have historically required separate modeling paradigms; prior unified models are autoregressive or NAR models limited to NTA tasks.

Approach

A flow-matching DiT where TA content is expanded by a duration adapter and added to audio latents, NTA content is injected via cross-attention in every block, and the irrelevant stream is replaced by learnable dummy embeddings per task type, combined with task-balanced sampling.

Key results

• UniFlow-Audio (large) reaches WER 3.09 on TTS versus 9.94 for NaturalSpeech 2 and PESQ 2.91 on SE versus 2.50 for DOSE.
• The small variant (208M trainable parameters) outperforms prior single-task baselines on most tasks, e.g., T2A FD 19.7 vs 21.8 for AudioLDM 2 and SR LSD 1.58 vs 1.75 for AudioSR.
• Ablations show cross-attention-only fusion collapses on TA tasks (TTS WER 27.6, SE PESQ 1.10) and input-only fusion degrades TA tasks substantially (TTS WER 42.0).

Surprising findings

• Cross-attention-only fusion performs worse than dual fusion even on NTA tasks (T2A FD 30.1 vs 19.7), suggesting time-aligned data harms cross-attention models.
• For speech enhancement and super-resolution, increasing CFG scale or inference steps degrades quality, unlike T2A where they help.

Limitations

• Tasks with multiple TA/NTA inputs (e.g., voice conversion) and zero-shot generalization to unseen tasks are not explored.
• Data and model size have not been scaled; most tasks have under 1,000 hours of training data.
• Only single-stream audio generation is supported; multi-source generation is not addressed.

Experimental setup

Datasets: LibriTTS M4Singer AudioCaps VoiceBank+Demand Wham! VCTK LJSpeech Musan HQ-TTS MUSDB MoisesDB FreeSound

Benchmarks: LibriTTS M4Singer AudioCaps MusicCaps VoiceBank+Demand VCTK MUSDB ESC-50 VisualSound

Metrics: WER SIM MOS SMOS F0 RMSE SA FD CLAP OVL REL

Baselines: NaturalSpeech 2 DiffSinger AudioLDM 2 MusicGen DOSE AudioSR DiffFoley

Builds on: F5-TTS AudioLDM 2 FastSpeech2 Flan-T5 Sway sampling

Closest papers by meaning

• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.91)
• UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.89)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching (NeurIPS 2024 · similarity 0.89)
• Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.89)
• Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.88)
• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.88)