Research map / Vision-language and multimodal / Cross-modal learning
An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits
Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan, Xiaolin Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual subnetwork (AV instead of audio-only) (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 10.1 -> 13.5 | Improved |
| add thalamic subnetwork fusion instead of direct AV fusion (CCNet) (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 10.9 -> 13.5 | Improved |
| fuse AV at all layers instead of top layer only (CACNet) (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 10.2 -> 13.5 | Improved |
| add fusion across layers within unimodal subnetworks (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 13.1 -> 13.5 | Improved |
| scale DFTNet to match CTCNet parameters (ablation) | DFTNet · LRS2-2Mix | SDRi (dB): 13.1 -> 13.1 | No clear effect |
| add lip-reading pre-training (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 4.5 -> 13.5 | Improved |
| increase auditory cycles m from 1 to 21 (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 11.8 -> 14.9 | Improved |
| use sum instead of concat for AV fusion (ablation) | CTCNet n=3 · LRS2-2Mix | SDRi (dB): 13.6 -> 13.8 | Improved |
| increase AV fusion cycles n from 1 to 3 (sum) (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 13.4 -> 13.8 | Improved |
| share weights across time steps in auditory subnetwork (ablation) | CTCNet · LRS2-2Mix | SDRi (dB): 13.3 -> 13.6 | No clear effect |
| use CTCNet-Lip rather than MS-TCN lip model | CTCNet · LRS2-2Mix | SDRi (dB): 13.9 -> 14.3 | Improved |
| apply CTCNet separation before ASR | Google ASR · LRS2 ASR after separation | WER (%): 84.91 -> 24.82 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- VisualVoice — VisualVoice: Audio-Visual Speech Separation With Cross-Modal Consistency (CVPR 2021) · better on 4 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- AND: Audio Network Dissection for Interpreting Deep Acoustic Models (ICML 2024 · similarity 0.87)
- RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR 2024 · similarity 0.87)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.86)
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.85)
- Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.85)
- Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
