Research map / Vision-language and multimodal / Cross-modal learning
Accept the Modality Gap: An Exploration in the Hyperbolic Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an angle-based contrastive loss in hyperbolic space that accepts the text-image modality gap instead of enforcing geodesic spatial proximity, improving hierarchy preservation and cross-modal retrieval over MERU and CLIP.
Problem
Hyperbolic multimodal (text-image) contrastive learning with geodesic-distance losses disrupts hierarchical structure in both modalities and conflicts with entailment losses, as in MERU.
Approach
Replace the geodesic-distance contrastive similarity with hyperbolic exterior/deviation angles so matching text-image pairs align along a geodesic from the origin, letting image embeddings sit further out than text; add a centroid regularizer placing text centroids closer to the origin than image centroids.
Key results
- Zero-shot classification: the proposed model beats MERU and CLIP on 13 of 19 datasets.
- Car Parts depth-1 hierarchy accuracy of 94.0-96.8% versus 19.2% for MERU; OpenImages depth-1 69.1-71.4% versus 31.1% for MERU.
- Only the proposed method stays stable across curvatures; MERU fails to converge for curvatures >=0.5 (ImageNet zero-shot accuracy drops to 28.4% and 20.9% at 0.1 and 0.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · COCO text→image retrieval | R5: 23.4 -> 23.7 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · COCO text→image retrieval | R10: 32.1 -> 32.7 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · Flickr text→image retrieval | R5: 33.1 -> 33.2 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · Flickr text→image retrieval | R10: 42.3 -> 42.7 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · COCO image→text retrieval | R5: 26.5 -> 32.6 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · COCO image→text retrieval | R10: 35.3 -> 42.5 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · Flickr image→text retrieval | R5: 31.4 -> 30.7 | Hurt |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · Flickr image→text retrieval | R10: 42.3 -> 39.7 | Hurt |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · ImageNet zero-shot | accuracy: 28.6 -> 29.7 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · Car Parts and OpenImages hierarchies | hierarchy accuracy: 19.2 -> 94.0; 0.6 -> 31.7; 31.1 -> 69.1; 10.5 -> 33.2 | Helped |
| Use angle-based contrastive loss and centroid regularization instead of geodesic proximity-based alignment | Hyperbolic multimodal model · Text hierarchy | hierarchy accuracy: 88.1 -> 93.4; 58.1 -> 73.5 | Helped |
| Fix curvature at 0.5 or higher rather than allowing MERU’s learned curvature (ablation) | MERU · ImageNet zero-shot | accuracy | Hurt |
Surprising findings
- Entailment loss alone did not yield meaningful results in the authors' experiments.
- MERU's learned curvature is pushed down to the clipping value of 0.1 under geodesic contrastive loss, whereas the proposed loss increases curvature.
Limitations
- Zero-shot performance on datasets with classes underrepresented in RedCaps is poor for all models; large-scale training would be needed.
- Experiments use a single small setup (ViT-S, ~8M RedCaps subset) trained from scratch.
Experimental setup
Datasets: RedCaps Car Parts Segmentation OpenImages Celeb-HQ COCO Flickr30K ImageNet Food-101 CIFAR-10 CIFAR-100 CUB SUN397
Benchmarks: ImageNet Food-101 CIFAR-10 CIFAR-100 CUB SUN397 Aircraft DTD Pets Caltech-101 Flowers STL-10
Metrics: top-1 accuracy recall@5 recall@10 hierarchy accuracy ImageNet accuracy vs curvature
Baselines: CLIP MERU
Builds on: MERU CLIP entailment cone loss Lorentz model
Closest papers by meaning
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.89)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.89)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.88)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.88)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.