Research map / Vision-language and multimodal / Cross-modal learning
Learnable PINs: Cross-Modal Embeddings for Person Identity
Arsha Nagrani, Samuel Albanie, Andrew Zisserman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialise subnetworks with pretrained single-modality identity weights | Joint embedding · seen-heard verification | AUC (%): 73.8 -> 87.0 | Improved |
| initialise subnetworks with pretrained single-modality identity weights | Joint embedding · unseen-unheard verification | AUC (%): 63.5 -> 78.5 | Improved |
| initialise subnetworks with pretrained single-modality identity weights | Joint embedding · unseen-unheard verification | EER (%): 39.2 -> 29.6 | Improved |
| train from scratch with no identity labels versus random weights | Joint embedding · unseen-unheard verification | AUC (%): 50.1 -> 63.5 | Improved |
| choose negative pairs with same gender | Joint embedding (pretrained) · unseen-unheard verification | AUC (%): 78.5 -> 61.1 | Hurt |
| choose negative pairs with same nationality | Joint embedding (pretrained) · unseen-unheard verification | AUC (%): 78.5 -> 77.2 | Hurt |
| choose negative pairs with same age | Joint embedding (pretrained) · unseen-unheard verification | AUC (%): 78.5 -> 74.9 | Hurt |
| sample positive pairs from the same video instead of across videos | Joint embedding (scratch) · seen-heard verification | AUC (%): 73.8 -> 74.5 | Improved |
| sample positive pairs from the same face-track | Joint embedding (scratch) · seen-heard verification | AUC (%): 74.5 -> 74.6 | No clear effect |
| add voice embeddings via max fusion to face-only retrieval | Face-only retrieval · Sherlock one-shot retrieval (Sherlock) | AUC (%): 35.0 -> 37.5 | Improved |
| add voice embeddings via max fusion to face-only retrieval | Face-only retrieval · Sherlock one-shot retrieval (John) | AUC (%): 44.6 -> 45.4 | Improved |
| freeze pretrained face subnetwork as teacher and train voice from scratch (ablation) | Joint embedding · verification | accuracy | Hurt |
Closest papers by meaning
- Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching (CVPR 2018 · similarity 0.89)
- Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association (CVPR 2021 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- Illuminating Visual Identity in Universal Multimodal Embeddings (CVPR 2026 · similarity 0.86)
- Learning Shared Representations from Unpaired Data (NeurIPS 2025 · similarity 0.85)
- Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
