Research map / Probabilistic and scientific modeling / Molecules and proteins
BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BinauralFlow, a causal, streamable conditional flow matching U-Net with a continuous inference pipeline (buffer bank, midpoint solver, early skip schedule) for binaural speech rendering from mono audio and poses.
Problem
Binaural speech rendering from mono audio and speaker/listener poses requires high realism (reverb, ambient noise, binaural cues) and causal, low-latency streaming inference, which existing regression and non-causal generative methods do not provide.
Approach
Treats binaural rendering as conditional generation via flow matching with a fully causal U-Net (causal 2D conv, per-frame normalization) and a streaming pipeline that uses per-step buffer banks, streaming STFT/ISTFT, a midpoint ODE solver, and an early-skip time schedule to cut steps to 6.
Key results
- BinauralFlow achieves L2 1.00e-5, Mag 0.0071, Phase 1.33, beating SGMSE (1.55, 0.0076, 1.43) and BinauralGrad (2.93, 0.0143, 1.33) on the in-house test set with 6 NFE.
- In an A-B perceptual test with 23 participants, BinauralFlow reaches a 42% confusion rate versus 21% for SGMSE and 3% for BinauralGrad.
- With 6 steps, BinauralFlow runs at 163 ms per 683 ms clip on a single 4090 (RTF 0.239), the fastest among the generative models compared.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use conditional flow matching instead of simplified flow matching (ablation) | BinauralFlow · Binaural speech test set | L2: 1.86 -> 1.00 | Helped |
| use conditional flow matching instead of simplified flow matching (ablation) | BinauralFlow · Binaural speech test set | Mag: 0.0101 -> 0.0071 | Helped |
| use conditional flow matching instead of simplified flow matching (ablation) | BinauralFlow · Binaural speech test set | Phase: 1.35 -> 1.33 | Helped |
| replace midpoint solver with Euler at NFE 6 (ablation) | BinauralFlow · Binaural speech test set | L2: 1.00 -> 0.90 | Helped |
| replace midpoint solver with Heun at NFE 6 (ablation) | BinauralFlow · Binaural speech test set | L2: 1.00 -> 16.86 | Hurt |
| replace midpoint solver with Heun and use 30 NFE (ablation) | BinauralFlow · Binaural speech test set | L2: 1.00 -> 1.27 | Hurt |
| vary the Sway Sampling coefficient from -1.0 to 1.0 (ablation) | BinauralFlow · Binaural speech test set | L2: 1.06; 1.10; 1.00; 1.02; 1.03; 1.04; 1.02 | No clear effect |
| use positive Sway Sampling coefficients, shifting time steps to the second half (ablation) | BinauralFlow · Binaural speech generation | background noise realism | Helped |
| use an early skip schedule rather than the late skip schedule (ablation) | BinauralFlow · Binaural speech generation | rendering quality | Mixed |
| pretrain on over 7,700 hours of artificial-head binaural audio before fine-tuning on limited real data (ablation) | BinauralFlow · Binaural speech synthesis with limited real training data | performance | Helped |
| train with less than 25% of the real dataset (ablation) | BinauralFlow · Binaural speech test set | performance | Hurt |
| use BinauralFlow rather than SGMSE | Binaural speech rendering · Binaural speech test set | L2: 1.55 -> 1.00 | Helped |
Surprising findings
- The simplified flow matching baseline collapses when mono audio is used as a condition, which the authors cite as a reason for their different formulation.
- Late-skip schedules degrade background-noise modeling, while the early-skip schedule preserves quality.
Limitations
- Main dataset is small (10 hours) and collected in one standard room, so real-data scale is limited.
- Large-scale pretraining relies on artificial heads and loudspeakers, which reduces data authenticity.
- On the public dataset, Wave L2 is worse than BinauralGrad (0.192 vs 0.128).
Experimental setup
Datasets: in-house binaural dataset (10 h, 48 kHz) VCTK (for pretraining loudspeaker recordings) public binaural dataset (Richard et al. 2021 / WarpNet)
Benchmarks: in-house binaural test set public binaural dataset
Metrics: L2 Mag Phase PESQ MRSTFT ABX CR A-B CR MUSHRA environment score MUSHRA spatialization score real-time factor
Baselines: SoundSpaces 2.0 2.5D Visual Sound WaveNet WarpNet BinauralGrad SGMSE Simplified Flow Matching
Builds on: conditional flow matching BinauralGrad SGMSE WarpNet Simplified Flow Matching
Closest papers by meaning
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.88)
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.86)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.86)
- PrismAudio: Decomposed Chain-of-Thought and Multi-dimensional Rewards for Video-to-Audio Generation (ICLR 2026 · similarity 0.85)
- Cyclic Learning for Binaural Audio Generation and Localization (CVPR 2024 · similarity 0.85)
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.