Research map / Vision-language and multimodal / Cross-modal learning
Multi-Modality Associative Bridging Through Memory: Speech Sound Recollected From Face Video
Minsu Kim, Joanna Hong, Se Jin Park, Yong Man Ro
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add recalled audio features via associative bridge to visual lip reading | visual lip reading baseline · LRW | word accuracy: 84.14 -> 85.41 | Improved |
| add associative bridge memory with best slot count N=112 | visual lip reading baseline · LRW-1000 | word accuracy: +5.89%; 50.82 | Improved |
| vary number of memory slots (N=88 best) (ablation) | proposed bridging · LRW | word accuracy: 85.41; +1.27% | Improved |
| use any number of memory slots (ablation) | proposed bridging · LRW and LRW-1000 | word accuracy | Improved |
| use modality-specific memory bridge instead of cross-modal adaptation | lip reading · LRW | word accuracy: 84.20 -> 85.41 | Improved |
| use bridge instead of knowledge distillation from ASR features | lip reading · LRW | word accuracy: 84.50 -> 85.41 | Improved |
| add recalled audio context to speech reconstruction (speaker-dependent) | Lip2Wav · GRID speaker-dependent | ESTOI: 0.535 -> 0.579 | Improved |
| add recalled audio context to speech reconstruction (speaker-independent) | Lip2Wav · GRID speaker-independent | STOI: 0.565 -> 0.600 | Improved |
| vary memory slots in speech reconstruction (N=150 best) (ablation) | proposed bridging · GRID speaker-dependent | STOI/ESTOI/PESQ: 0.738, 0.579, 1.984 | Improved |
| swap Griffin-Lim for WaveNet vocoder | proposed speech reconstruction · GRID human evaluation | intelligibility MOS: 3.56 -> 4.27 | Improved |
| proposed vs Lip2Wav on naturalness | speech reconstruction · GRID human evaluation | naturalness MOS: 2.83 -> 2.93 | No clear effect |
Closest papers by meaning
- Distilling Audio-Visual Knowledge by Compositional Contrastive Learning (CVPR 2021 · similarity 0.89)
- Weakly Paired Associative Learning for Sound and Image Representations via Bimodal Associative Memory (CVPR 2022 · similarity 0.88)
- Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.86)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.86)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
