Research map / Graph learning / Knowledge and temporal graphs
Exploring Visual Relationship for Image Captioning
Ting Yao, Yingwei Pan, Yehao Li, Tao Mei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic and spatial relation graphs with GCN (full GCN-LSTM) over Up-Down | Up-Down region captioner · COCO Karpathy test (cross-entropy) | CIDEr-D: 113.5 -> 117.1 | Improved |
| add semantic and spatial relation graphs with GCN (full GCN-LSTM) over Up-Down | Up-Down region captioner · COCO Karpathy test (CIDEr-D optimization) | CIDEr-D: 120.1 -> 128.7 | Improved |
| add semantic-graph GCN only over Up-Down | Up-Down region captioner · COCO Karpathy test (cross-entropy) | CIDEr-D: 113.5 -> 116.3 | Improved |
| add spatial-graph GCN only over Up-Down | Up-Down region captioner · COCO Karpathy test (cross-entropy) | CIDEr-D: 113.5 -> 115.6 | Improved |
| add spatial graph to semantic-graph model (fuse both) (ablation) | GCN-LSTM_sem · COCO Karpathy test (cross-entropy) | CIDEr-D: 116.3 -> 117.1 | Improved |
| add semantic graph to spatial-graph model (fuse both) (ablation) | GCN-LSTM_spa · COCO Karpathy test (cross-entropy) | CIDEr-D: 115.6 -> 117.1 | Improved |
| use late fusion of decoders instead of early fusion before attention (ablation) | GCN-LSTM · COCO Karpathy test (cross-entropy) | CIDEr-D: 116.4 -> 117.1 | Improved |
| use late fusion of decoders instead of early fusion after attention (ablation) | GCN-LSTM · COCO Karpathy test (cross-entropy) | CIDEr-D: 116.6 -> 117.1 | Improved |
| tune tradeoff alpha between semantic and spatial decoders (0 to 1) (ablation) | GCN-LSTM · COCO Karpathy test (cross-entropy) | BLEU-4, METEOR, CIDEr-D: best at alpha about 0.7 | Improved |
| replace Up-Down with GCN-LSTM on online server c5 | Up-Down region captioner · COCO online test server (c5) | CIDEr-D: 117.9 -> 125.3 | Improved |
| replace Up-Down with GCN-LSTM in human Turing test | Up-Down region captioner · COCO human study | M2 Turing test pass rate: 78.5 -> 82.1 | Improved |
Closest papers by meaning
- Improving OCR-Based Image Captioning by Incorporating Geometrical Relationship (CVPR 2021 · similarity 0.89)
- Reflective Decoding Network for Image Captioning (ICCV 2019 · similarity 0.88)
- Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects (CVPR 2017 · similarity 0.88)
- Image Captioning: Transforming Objects into Words (NeurIPS 2019 · similarity 0.88)
- Relational Distant Supervision for Image Captioning without Image-Text Pairs (AAAI 2024 · similarity 0.87)
- Composing Object Relations and Attributes for Image-Text Matching (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
