Research map / Graph learning / Graph algorithms and clustering
GeoMM: On Geodesic Perspective for Multi-modal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a hierarchical geodesic distance (shortest paths over a clustered kNN graph of a momentum feature queue) as the contrastive distance for vision-language pretraining, plugged into ALBEF/TCL/MAFA and other methods.
Problem
Cosine/Euclidean distances in contrastive multimodal pretraining are one-to-one and may fail to separate samples that look similar but differ semantically, since data lies on nonlinear manifolds.
Approach
Replace trivial distances in image-text contrastive loss with geodesic distances computed by shortest paths (Floyd) on a kNN graph over the sample pool, made scalable via a hierarchical k-means graph, periodic rebuilds, and incremental queue updates, with angle normalization to form the loss.
Key results
- Geo-MAFA reaches 74.7 COCO zero-shot text retrieval R@1 vs 72.6 for reproduced MAFA, and 79.3 vs 78.0 under fine-tuning.
- Geo-ALBEF improves VQA test-dev from 74.54 (ALBEF) to 75.13 and visual grounding TestA from 65.89 to 67.93 on RefCOCO+.
- Geodesic distance improves MoCo v2 from 69.9 to 70.6 top-1 at 200 epochs on ImageNet linear probing, with little added training time (268s vs 255s per 100 iterations for ALBEF).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use geodesic distance in MAFA | MAFA · MSCOCO 5K test, zero-shot text retrieval | R@1: 72.6 -> 74.7 | Helped |
| Use geodesic distance in MAFA | MAFA · MSCOCO 5K test, fine-tuned image retrieval | R@1: 61.2 -> 62.5 | Helped |
| Use geodesic distance in MAFA | MAFA · Flickr30K 1K test, fine-tuned image retrieval | R@1: 84.9 -> 85.6 | Helped |
| Use geodesic distance in MAFA | MAFA · VQA test-dev | accuracy: 75.55 -> 76.04 | Helped |
| Use geodesic distance in MAFA | MAFA · NLVR2 dev | accuracy: 82.52 -> 83.12 | Helped |
| Use geodesic distance in MAFA | MAFA · SNLI-VE test | accuracy: 80.96 -> 81.63 | Helped |
| Use geodesic distance in ALBEF | ALBEF · RefCOCO+ TestA | localization accuracy: 65.89 -> 67.93 | Helped |
| Use geodesic distance in MOCOv2 | MOCOv2 · ImageNet linear classification, 100epoch | Top1 accuracy: 67.4 -> 67.8 | Helped |
| Use geodesic distance in SwAV | SwAV · ImageNet linear classification, 200epoch | Top1 accuracy: 69.1 -> 69.9 | Helped |
| Use geodesic distance in CLIP fine-tuning | CLIPF T · COCO zero-shot text retrieval | R@1: 58.5 -> 59.6 | Helped |
| Use geodesic distance in FLIP fine-tuning | FLIPF T · COCO zero-shot image retrieval | R@1: 44.2 -> 45.1 | Helped |
| Use 2 graph layers instead of 1 (ablation) | ALBEF · COCO retrieval | Text retrieval R@1: 75.1 -> 76.2 | Helped |
Surprising findings
- Geo-MAFA's gains on VQA-style tasks come with lower absolute performance than MAFA on SNLI-VE dev (not reported as worse), so gains are not uniform across tasks.
- Hyperparameters (graph layers, cluster count, neighbors, update frequency) are reported as having little effect on performance.
Limitations
- Geodesic distance relies on a kNN graph and clustering that is rebuilt only periodically, so distances can become stale between rebuilds.
- Theoretical results (connected components, component size) are stated as bounds and do not directly guarantee geodesic accuracy.
- Gains on several tasks are modest (e.g., about 0.4 to 1 point on VQA and NLVR2).
Experimental setup
Datasets: COCO Visual Genome SBU CC3M Flickr30K VQA NLVR2 SNLI-VE RefCOCO+ ImageNet
Benchmarks: MSCOCO Flickr30K VQA NLVR2 SNLI-VE RefCOCO+ ImageNet
Metrics: R@1 R@5 R@10 accuracy top-1 accuracy
Baselines: ALBEF TCL MAFA CLIP FLIP ALIGN UNITER ViLT OSCAR VILLA VLMO ARN
Builds on: ALBEF TCL MAFA CLIP MoCo v2
Closest papers by meaning
- Accept the Modality Gap: An Exploration in the Hyperbolic Space (CVPR 2024 · similarity 0.87)
- Graph4MM: Weaving Multimodal Learning with Structural Information (ICML 2025 · similarity 0.85)
- Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning (CVPR 2025 · similarity 0.84)
- Learning Distances from Data with Normalizing Flows and Score Matching (ICML 2025 · similarity 0.84)
- Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations (ICLR 2025 · similarity 0.83)
- GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.