Research map / Detection and segmentation / Open-set and few-shot detection
Composing Object Relations and Attributes for Image-Text Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CORA, a dual-encoder image-text matching model that parses captions into scene graphs and encodes them with two graph attention networks, trained with triplet and contrastive losses for global and object-entity alignment.
Problem
Image-text matching with cross-attention models is computationally expensive for retrieval, while dual encoders with sequence text models struggle to capture object-attribute and object-object relations.
Approach
Represent the caption as a scene graph of object, attribute and relation nodes/edges, encode it with a two-step graph attention network (object-attribute GAT then object-object GAT) within a dual-encoder, and align image with both the whole caption and individual object entities via contrastive loss plus a specificity loss.
Key results
- On Flickr30K with Bi-GRU, CORA reaches 515.8 RSUM, surpassing HREM (514.2) and NAAF (513.2) without cross-attention.
- On MS-COCO 5K with Bi-GRU, CORA achieves 438.9 RSUM, above the cross-attention CHAN (433.4) and the ensemble SDE (437.8).
- Ablation shows the 2-step object-attribute then object-object encoding (515.8) beats a joint single-GAT graph encoder (504.8) on Flickr30K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the two-step Obj-Att & Obj-Obj GAT encoding instead of one joint GAT (ablation) | CORA · Flickr30K test | RSUM: 504.8 -> 515.8 | Helped |
| Use the parsed scene-graph structure instead of a fully connected graph (ablation) | CORA · Flickr30K test | RSUM: 501.6 -> 515.8 | Helped |
| Add contrastive loss LCON (ablation) | CORA with hardest-negative triplet loss · Flickr30K test | RSUM: 486.9 -> 507.3 | Helped |
| Add hardest-negative triplet loss LHARD (ablation) | CORA with contrastive loss LCON · Flickr30K test | RSUM: 497.8 -> 507.3 | Helped |
| Add specificity loss LSPEC (ablation) | CORA with LHARD and LCON · Flickr30K test | RSUM: 507.3 -> 515.8 | Helped |
| Use 2 Obj-Obj GAT layers instead of 1 (ablation) | CORA with 1 Obj-Att GAT layer · Flickr30K test | RSUM: 509.1 -> 515.8 | Helped |
| Use 3 Obj-Obj GAT layers instead of 2 (ablation) | CORA with 1 Obj-Att GAT layer · Flickr30K test | RSUM: 515.8 -> 514.4 | Hurt |
| Use 2 Obj-Att GAT layers instead of 1 (ablation) | CORA with 1 Obj-Obj GAT layer · Flickr30K test | RSUM: 509.1 -> 510.1 | Hurt |
| Use 2 Obj-Att GAT layers instead of 1 (ablation) | CORA with 2 Obj-Obj GAT layers · Flickr30K test | RSUM: 515.8 -> 514.5 | Hurt |
| Use 2 Obj-Att and 3 Obj-Obj GAT layers instead of 1 and 2 (ablation) | CORA · Flickr30K test | RSUM: 515.8 -> 510.6 | Hurt |
| Rerank with the image-entity score | CORA with Faster R-CNN + Bi-GRU · MS-COCO 5K test | RSUM: 444.6 -> 445.1 | Helped |
| Rerank with the image-entity score | CORA with Faster R-CNN + BERT · MS-COCO 5K test | RSUM: 450.1 -> 450.6 | Helped |
Surprising findings
- Using BERT instead of Bi-GRU gave CORA a smaller gain over prior work, which the authors attribute to BERT suiting long text while CORA encodes short phrases.
Limitations
- Performance depends heavily on the scene graph parser; failed parses mean the text cannot be encoded (e.g., exclamatory captions).
- BERT-based encoding of short phrases gives smaller gains than other methods; text encoder suited to short phrases left as future work.
- Relies on a rule-based parser built on spaCy rather than learned parsing.
Experimental setup
Datasets: Flickr30K MS-COCO Visual Genome
Benchmarks: Flickr30K MS-COCO 1K MS-COCO 5K
Metrics: Recall@1 Recall@5 Recall@10 RSUM
Baselines: VSE∞ VSRN SGM GCN+DIST GSMN CHAN NAAF HREM CODER SDE MV-VSE GraDual
Builds on: GPO GATv2 VSE++ BUTD
Closest papers by meaning
- Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching (AAAI 2020 · similarity 0.92)
- Visual-Semantic Matching by Exploring High-Order Attention and Distraction (CVPR 2020 · similarity 0.89)
- GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.87)
- Relational Distant Supervision for Image Captioning without Image-Text Pairs (AAAI 2024 · similarity 0.87)
- Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps (CVPR 2025 · similarity 0.87)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.