Research map / Vision-language and multimodal / Cross-modal learning
Cross-Modal Scene Networks
Yusuf Aytar, Lluís Castrejón, Carl Vondrick, Hamed Pirsiavash, Antonio B. Torralba
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add modality tuning with fixed shared layers (method A) | BL-Shared-Upper · CMPlaces retrieval (fc7) | mean mAP: 6.5 -> 8.5 | Improved |
| add statistical regularization with GMM prior (method B) | BL-Shared-Upper · CMPlaces retrieval (fc7) | mean mAP: 6.5 -> 6.4 | No clear effect |
| combine tuning and StatReg (method C) | A: Tune · CMPlaces retrieval (fc7) | mean mAP: 8.5 -> 11.9 | Improved |
| use GMM instead of single Gaussian for regularizer (ablation) | StatReg · CMPlaces retrieval (fc6) | mean mAP: 4.2 -> 5.5 | Improved |
| add method C alignment, retrieval from fc7 to fc6 layer | C: Tune + StatReg (GMM) · CMPlaces retrieval (fc6 vs fc7) | mean mAP: BL-Shared-Upper 6.5 -> 3.2 versus C 11.9 -> 11.1 | Improved |
| add Gaussian StatReg regularizer to shared-upper network | BL-Shared-Upper · Zero-shot classification, clip art | accuracy: 22.1 -> 50.5 | Improved |
| add Gaussian StatReg regularizer | BL-Shared-Upper · Zero-shot classification, line drawings | accuracy: 14.9 -> 24.8 | Improved |
| add Gaussian StatReg regularizer | BL-Shared-Upper · Zero-shot classification, text descriptions | accuracy: 43.4 -> 4.2 | Hurt |
| add modality tuning | BL-Shared-Upper · Zero-shot classification, text descriptions | accuracy: 43.4 -> 51.0 | Improved |
| add StatReg to tuning (method C) | A: Modality Tuning · Zero-shot classification, clip art | accuracy: 18.6 -> 16.3 | Hurt |
| add tuning and StatReg together (method C) | BL-Shared-Upper · Zero-shot retrieval, 55 held-out categories | mean mAP: 7.3 -> BL; A 7.2, B 6.8 | Improved |
| add modality tuning (Free) to shared-upper | BL-Shared-Upper · Zero-shot retrieval, 55 held-out categories | mean mAP: 7.3 -> 7.2 | No clear effect |
Closest papers by meaning
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.89)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.87)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.86)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
