Research map / Label-efficient and robust learning / Self-supervised and contrastive

Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

NeurIPS 2026 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech and brain decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a differentiable, GPU-parallel frequency-domain acoustic tube simulator of the vocal tract and uses gradient descent (with neural-field parameterizations, a self-supervised speech autoencoder, and a StyleGAN2 MRI generator) to reconstruct vocal tract geometry and MRI video from speech alone.

Problem

Inferring vocal tract geometry from speech is a non-convex inverse problem; existing physics simulators are CPU-bound and non-differentiable, and learned approaches need paired articulatory or MRI data.

Approach

Solve the linearized Euler/Webster equations in the frequency domain via transmission-line chain matrices (parallel over frequencies, avoiding time-stepping noise), add a smooth differentiable turbulence gate for consonants, and parameterize the area function with a neural field so plain gradient descent escapes local minima.

Key results

• Frequency-domain synthesis is about 70x faster than time-domain synthesis on an RTX 4090 (14.2 ms vs 1.08 s per 1 s utterance) and yields SI-SDR roughly 10 dB higher than TDS.
• Self-supervised autoencoder reduces WER versus TensorTract2 on English (5.30 vs 13.00) and across the 11 evaluated languages, with higher speaker ID similarity.
• MRI-from-speech reconstruction achieves FVD 623 versus 2949 for Speech2rtMRI, and better SSIM and LPIPS.

Surprising findings

• Discrete area-function optimization stalls near initialization, while neural-field parameterizations succeed in fitting vowels.
• Area functions from a self-supervised encoder, with no phonetic labels, form a continuous phoneme spectrum ordered by place of articulation.

Limitations

• Inverse problem is ill-posed: different geometries can produce the same sound, so MRI frames sometimes differ from ground truth.
• MRI dataset does not image the nasal tract, so nasal formants and anti-formants are not reconstructed.

Experimental setup

Datasets: LibriTTS-R X-ray vowel data MRI scan dataset (unnamed in text)

Benchmarks: LibriTTS-R MRI video reconstruction vs Speech2rtMRI

Metrics: WER CER speaker ID cosine similarity SI-SDR STOI PESQ MSE in cm^4 FVD SSIM LPIPS

Baselines: TensorTract2 Speech2rtMRI discrete area optimization time domain synthesis

Builds on: Kelly-Lochbaum method VocalTractLab TensorTract2 StyleGAN2 Wav2Vec 2.0

Closest papers by meaning

• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (AAAI 2022 · similarity 0.85)
• ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.84)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.84)
• Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation (NeurIPS 2026 · similarity 0.84)
• DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis (ICML 2025 · similarity 0.84)
• Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.84)