Research map / Vision-language and multimodal / Cross-modal learning
Disjoint Mapping Network for Cross-modal Matching of Voices and Faces
Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, Rita Singh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| map each modality separately to shared covariates rather than train on paired voice-face examples | DIMNet · 1:2 matching, unstratified | accuracy (%): improving the accuracies by 2.45%-4.02% for the U group | Improved |
| add ID supervision (ablation) | DIMNet with gender-only supervision · 1:2 matching, voice→face, U | accuracy (%): 72.90 -> 83.45 | Improved |
| add ID supervision (ablation) | DIMNet with nationality-only supervision · 1:2 matching, voice→face, U | accuracy (%): 57.53 -> 83.45 | Improved |
| add gender supervision (ablation) | DIMNet-I · 1:2 matching | accuracy (%) | Improved |
| add nationality supervision (ablation) | DIMNet-I · 1:2 matching, voice→face, U | accuracy (%): 83.45 -> 82.95 | Hurt |
| add nationality supervision (ablation) | DIMNet-IG · 1:2 matching, voice→face, U | accuracy (%): 84.12 -> 83.73 | Hurt |
| use gender supervision only (ablation) | DIMNet · 1:2 matching, gender-stratified | accuracy (%) | No clear effect |
| use nationality supervision only (ablation) | DIMNet · 1:2 matching | accuracy (%) | Hurt |
| use ID and gender supervision (ablation) | DIMNet · Verification, U | EER (%): 24.95 -> 24.56 | Improved |
| use ID and gender supervision (ablation) | DIMNet · Retrieval, voice→face, ID | mAP (%): 4.25 -> 4.42 | Improved |
| use gender supervision only (ablation) | DIMNet · Retrieval, voice→face, gender | mAP (%): 97.84 | Improved |
| use gender supervision only instead of ID supervision (ablation) | DIMNet · Retrieval, voice→face, ID | mAP (%): 4.25 -> 1.07 | Hurt |
Closest papers by meaning
- Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching (CVPR 2018 · similarity 0.89)
- Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association (CVPR 2021 · similarity 0.86)
- Cross-Modal Perceptionist: Can Face Geometry Be Gleaned From Voices? (CVPR 2022 · similarity 0.86)
- Learnable PINs: Cross-Modal Embeddings for Person Identity (ECCV 2018 · similarity 0.85)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.83)
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
