Research map / Vision-language and multimodal / Human and embodied modeling
Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Structure-CLIP fine-tunes CLIP with scene-graph-guided semantic negative captions and a knowledge-enhanced triple encoder to improve structured (attribute/relation) image-text matching while keeping general retrieval ability.
Problem
CLIP-style vision-language models fail to distinguish captions with identical words but different structure (objects, attributes, relations), e.g., "astronaut rides horse" vs "horse rides astronaut", and random word-swap negatives are often semantically invalid.
Approach
Uses scene graphs parsed from captions to build semantic negatives by swapping objects in relation triples and attributes across objects, and adds a Knowledge-Enhanced Encoder that embeds (head, relation, tail) triples with a head+relation-tail composition and fuses them into the text embedding via a weighted sum.
Key results
- Structure-CLIP reaches 82.3% on VG-Attribution and 84.7% on VG-Relation versus Neg-CLIP's 71.0% and 81.0%.
- Structure-CLIP-Large obtains 83.5% on VG-Relation and 85.1% on VG-Attribution per the main table, and 48.9 MSCOCO text retrieval R@1 versus 41.0 for Neg-CLIP.
- Structure-BLIP reaches 90.3% on VG-Attribution and 88.2% on VG-Relation, surpassing NegCLIP by 19.3 and 7.2 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random word swaps with scene-graph-guided semantic negatives (ablation) | CLIP fine-tuned on MSCOCO · VG-Attribution | accuracy: 73.9 -> 77.8 | Helped |
| replace random word swaps with scene-graph-guided semantic negatives (ablation) | CLIP fine-tuned on MSCOCO · VG-Relation | accuracy: 77.7 -> 79.0 | Helped |
| add KEE without negative sampling (ablation) | CLIP fine-tuned on MSCOCO · VG-Attribution | accuracy: 64.0 -> 65.7 | Helped |
| add KEE without negative sampling (ablation) | CLIP fine-tuned on MSCOCO · VG-Relation | accuracy: 66.5 -> 68.8 | Helped |
| combine KEE with semantic negative sampling (ablation) | CLIP fine-tuned on MSCOCO · VG-Attribution | accuracy: 77.8 -> 82.3 | Helped |
| combine KEE with semantic negative sampling (ablation) | CLIP fine-tuned on MSCOCO · VG-Relation | accuracy: 79.0 -> 84.7 | Helped |
| use 6 KEE Transformer layers instead of 1, 2, or 12 (ablation) | Structure-CLIP · VG-Attribution | accuracy: 82.1; 82.2; 82.3; 81.9 | Mixed |
| use 6 KEE Transformer layers instead of 1, 2, or 12 (ablation) | Structure-CLIP · VG-Relation | accuracy: 82.9; 83.3; 84.7; 83.2 | Mixed |
| integrate structured knowledge with λ=0.01 instead of λ=0.0 (ablation) | Structure-CLIP · VG-Attribution | accuracy: 77.8 -> 82.7 | Helped |
| replace head + relation - tail with concatenated triple embeddings (ablation) | Structure-CLIP · VG-Attribution | accuracy: 82.3 -> 81.1 | Hurt |
| replace head + relation - tail with concatenated triple embeddings (ablation) | Structure-CLIP · VG-Relation | accuracy: 84.7 -> 83.3 | Hurt |
| replace subtraction with addition for the tail entity embedding (ablation) | Structure-CLIP · VG-Attribution | accuracy: 82.3 -> 81.9 | Hurt |
Surprising findings
- Plain fine-tuning on MSCOCO without negatives barely improves structured representations compared with the CLIP baseline.
- Structure-CLIP is reported to outperform OPT-175B on VG benchmarks with under 1% of its parameters.
Limitations
- Hyperparameter results (e.g., Table 3) show sensitivity to KEE depth and fusion weight; fixed sentence-level scene graphs limit knowledge.
- Evaluation focuses on VG-Relation/Attribution and MSCOCO; the Structure-CLIP-Large row is inconsistent with some text claims about parameter counts and comparisons.
Experimental setup
Datasets: MSCOCO Visual Genome Relation Visual Genome Attribution
Benchmarks: VG-Relation VG-Attribution MSCOCO
Metrics: accuracy Recall@1 Recall@5 Recall@10
Baselines: CLIP-Base CLIP-Large Neg-CLIP VILT FLAVA BART FLAN-T5 OPT
Builds on: CLIP NegCLIP BERT Winoground
Closest papers by meaning
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.89)
- Compositional Chain-of-Thought Prompting for Large Multimodal Models (CVPR 2024 · similarity 0.88)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.88)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
- GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.