Research map / Vision-language and multimodal / CLIP and zero-shot
Compositional Entailment Learning for Hyperbolic Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyCoCLIP, a hyperbolic CLIP variant that adds grounded image boxes and noun-phrase text boxes with contrastive and entailment-cone losses to learn compositional hierarchies of images and text.
Problem
Hyperbolic vision-language models such as MERU only model image-text pair hierarchies and ignore intra-modal compositional structure (objects within scenes, phrases within captions).
Approach
Extracts image boxes and text boxes from grounded image-caption pairs via an offline GLIP-based pipeline, then jointly optimizes hyperbolic contrastive loss (hCC) and entailment-cone loss (hCE) with whole-image/caption entailing box-level concepts, including a relaxed aperture threshold eta.
Key results
- On GRIT pretraining with ViT-B/16, HyCoCLIP reaches 45.8% ImageNet zero-shot accuracy vs 40.6% CLIP and 40.1% MERU, best on 15 of 16 classification datasets.
- Object detection on COCO novel categories: HyCoCLIP 68.5 AP vs RegionCLIP 65.2, MERU 55.8, CLIP 51.2.
- VL-Checklist-Object accuracy 59.8 for HyCoCLIP vs 49.3 CLIP and 50.5 MERU; VG-Attribution 68.4 vs 63.3 CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyCoCLIP instead of MERU | ViT-B/16 on GRIT · ImageNet | accuracy: 40.1 -> 45.8 | Helped |
| Use HyCoCLIP instead of CLIP | ViT-B/16 on GRIT · ImageNet | accuracy: 40.6 -> 45.8 | Helped |
| Use compositional entailment learning | ViT-B/16 on GRIT · WordNet | TIE: 3.60 -> 3.17 | Helped |
| Use HyCoCLIP instead of RegionCLIP | COCO 17 novel categories · zero-shot object detection | AP: 65.2 -> 68.5 | Helped |
| Use HyCoCLIP instead of MERU | scene understanding · VL-CO | accuracy: 50.5 -> 59.8 | Helped |
| Remove L*cont(I,T) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 16.0 -> 16.7 | Helped |
| Remove L*cont(T,I) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 16.1 -> 16.7 | Helped |
| Remove L*cont(Ibox,T) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 13.8 -> 16.7 | Helped |
| Remove L*cont(Tbox,I) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 15.2 -> 16.7 | Helped |
| Remove L*ent(I,T) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 14.9 -> 16.7 | Helped |
| Remove L*ent(Ibox,Tbox) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 16.1 -> 16.7 | Helped |
| Remove L*ent(I,Ibox) (equivalent to adding this loss) (ablation) | HyCoCLIP on grounded CC3M · ImageNet | accuracy: 16.1 -> 16.7 | Helped |
Surprising findings
- Naively adding grounded boxes as extra training samples degrades CLIP and MERU retrieval and classification performance.
- Performance saturates with batch sizes beyond 768, showing no benefit from larger batches.
Limitations
- Requires offline grounding to generate boxes, doubling training data volume and training time.
- Hierarchical training separates image and text regions, which may hurt large-scale retrieval; Flickr text retrieval is slightly worse than CLIP.
Experimental setup
Datasets: GRIT RedCaps CC3M Flickr30K COCO ImageNet WordNet VL-Checklist Visual Genome Attribution HierarCaps
Benchmarks: ImageNet CIFAR-10 CIFAR-100 SUN397 Caltech-101 STL-10 Food-101 CUB Cars Aircraft Pets Flowers
Metrics: zero-shot accuracy recall@k average precision TIE LCA Jaccard hierarchical precision hierarchical recall
Baselines: CLIP MERU RegionCLIP
Builds on: MERU CLIP hyperbolic entailment cones GLIP GRIT
Closest papers by meaning
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.90)
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.89)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.89)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.89)
- Hyperbolic Safety-Aware Vision-Language Models (CVPR 2025 · similarity 0.89)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.