Research map / Vision-language and multimodal / CLIP and zero-shot
Contrastive Language-Image Pre-Training with Knowledge Graphs
Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song, Gao Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add knowledge-graph pretraining and multimodal fusion to CLIP | CLIP ViT-L/14 · VQA test-dev | accuracy: 74.10 -> 76.11 | Improved |
| Add knowledge-graph pretraining and multimodal fusion to CLIP | CLIP ViT-L/14 · Flickr30K text retrieval | R@1: 88.6 -> 89.2 | Improved |
| Add knowledge-graph pretraining and multimodal fusion to CLIP | CLIP ViT-L/14 · ImageNet | accuracy (%): 84.2 -> 84.4 | Improved |
| Add knowledge-graph pretraining and multimodal fusion to CLIP | CLIP ViT-L/14 · GLUE MNLI | accuracy: 80.9 -> 83.6 | Improved |
| Train CLIP on knowledge data without graph-based objectives (ablation) | CLIP ViT-B/32 · Flickr30K retrieval | Text R@1: 84.2 -> 84.5 | No clear effect |
| Add the E2E entity-entity objective (ablation) | Knowledge-CLIP · Flickr30K retrieval | Text R@1: 85.0 -> 86.3 | Improved |
| Add the E2R entity-relation objective (ablation) | Knowledge-CLIP · Flickr30K retrieval | Text R@1: 85.7 -> 86.3 | Improved |
| Add the G2E graph-entity objective (ablation) | Knowledge-CLIP · Flickr30K retrieval | Text R@1: 84.9 -> 86.3 | Improved |
| Add the knowledge-distillation loss (ablation) | Knowledge-CLIP · Flickr30K retrieval | Text R@1: 82.4 -> 86.3 | Improved |
| Add the knowledge-distillation loss (ablation) | Knowledge-CLIP · Flickr30K retrieval | Image R@1: 62.5 -> 67.2 | Improved |
| Use Knowledge-CLIP instead of CLIP on negation questions | CLIP · VQA Negation | accuracy: 64.7 -> 66.8 (+2.1) | Improved |
| Use Knowledge-CLIP instead of CLIP on color questions | CLIP · VQA Color | accuracy: 54.2 -> 59.9 (+5.7) | Improved |
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations (AAAI 2024 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
