Research map / Vision-language and multimodal / Human and embodied modeling
Relational Distant Supervision for Image Captioning without Image-Text Pairs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes relational distant supervision that infers visual object relationships from images using relationships parsed from a sentence corpus, then builds pseudo image-sentence pairs to train an unsupervised image captioner.
Problem
Unsupervised image captioning without image-sentence pairs, where existing methods bridge images and text only through detected objects or concepts, ignoring object relationships that carry richer context.
Approach
Impart object-relationship knowledge from an external sentence corpus to images via scene-level alignment (distant supervision), so a relationship learner infers relationships in images without an off-the-shelf relationship detector; a relationship-to-sentence module converts them into pseudo captions used to train a BUTD captioner.
Key results
- On COCO-Shutterstock the method reaches CIDEr 38.5 vs 29.0 for R2M (best prior), a 32.8% relative improvement.
- On COCO-GCC it reaches CIDEr 36.3 and BLEU4 10.1 vs 29.3 and 8.3 for R2M, outperforming all listed unsupervised baselines.
- On Flickr30K-COCO results are comparable rather than best (CIDEr 19.2 vs R2M 18.1; BLEU4 10.5 vs 11.7 for R2M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add relationship learning and relationship-to-sentence modules (ablation) | Relational Distant Supervision · COCO-Shutterstock | CIDEr: 27.2 -> 38.5 | Helped |
| add scene-level knowledge (ablation) | Relational Distant Supervision · COCO-Shutterstock | CIDEr: 32.0 -> 38.5 | Helped |
| add entity-level knowledge (ablation) | Relational Distant Supervision · COCO-Shutterstock | CIDEr: 37.3 -> 38.5 | Helped |
| add reconstruction loss (ablation) | Relational Distant Supervision · COCO-Shutterstock | CIDEr: 37.7 -> 38.5 | Helped |
| add relationship learning and relationship-to-sentence modules (ablation) | Relational Distant Supervision · Flickr30K-COCO | CIDEr: 14.9 -> 19.2 | Helped |
| add scene-level knowledge (ablation) | Relational Distant Supervision · Flickr30K-COCO | CIDEr: 15.7 -> 19.2 | Helped |
| add entity-level knowledge (ablation) | Relational Distant Supervision · Flickr30K-COCO | CIDEr: 18.2 -> 19.2 | Helped |
| add reconstruction loss (ablation) | Relational Distant Supervision · Flickr30K-COCO | CIDEr: 18.6 -> 19.2 | Helped |
| add relationship learning and relationship-to-sentence modules (ablation) | Relational Distant Supervision · COCO-GCC | CIDEr: 27.0 -> 36.3 | Helped |
| add scene-level knowledge (ablation) | Relational Distant Supervision · COCO-GCC | CIDEr: 31.4 -> 36.3 | Helped |
| add entity-level knowledge (ablation) | Relational Distant Supervision · COCO-GCC | CIDEr: 35.9 -> 36.3 | Helped |
| add reconstruction loss (ablation) | Relational Distant Supervision · COCO-GCC | CIDEr: 35.6 -> 36.3 | Helped |
Surprising findings
- Directly using relationships from an off-the-shelf scene graph generator (unbiased Neural motifs) as the bridge performs worse than the learned relational distant supervision, e.g. CIDEr 24.1 vs 38.5 on COCO-Shutterstock.
Limitations
- Performance on Flickr30K-COCO is comparable to, not better than, the best baseline (R2M on BLEU4).
- Relies on an external sentence corpus and an off-the-shelf object detector (Faster R-CNN trained on Visual Genome) and object label vocabulary.
- Pipeline depends on NLP parsers (Stanford CoreNLP, NLTK) whose noise can propagate into relationship labels.
Experimental setup
Datasets: COCO Shutterstock sentence corpus Flickr30K GCC Visual Genome
Benchmarks: COCO-Shutterstock Flickr30K-COCO COCO-GCC
Metrics: BLEU1 BLEU2 BLEU3 BLEU4 METEOR ROUGE CIDEr
Baselines: UIC SME R2M OCM Fast RF-UIC relationship detector (unbiased scene graph generator)
Builds on: distant supervision (Mintz et al. 2009) Bottom-up and top-down attention (BUTD) Visual distant supervision (Yao et al. 2021) GCN Transformer
Closest papers by meaning
- Joint Commonsense and Relation Reasoning for Image and Video Captioning (AAAI 2020 · similarity 0.89)
- Unsupervised Image Captioning (CVPR 2019 · similarity 0.89)
- Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning (CVPR 2019 · similarity 0.89)
- Captioning Images With Diverse Objects (CVPR 2017 · similarity 0.88)
- Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning (CVPR 2022 · similarity 0.88)
- Composing Object Relations and Attributes for Image-Text Matching (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.