Research map / Vision-language and multimodal / Human and embodied modeling
Transform and Tell: Entity-Aware News Image Captioning
Alasdair Tran, Alexander Mathews, Lexing Xie
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LSTM with Transformer (ablation) | RoBERTa captioning model with image attention · NYTimes800k | CIDEr: 24.9 -> 40.3 | Improved |
| Replace GloVe with RoBERTa (ablation) | LSTM captioning model with image attention · NYTimes800k | rare proper noun precision: 0 -> 3.28 | Improved |
| Add image attention (ablation) | Transformer + RoBERTa · GoodNews | CIDEr: 40.9 -> 48.5 | Improved |
| Add learnable weighted sum of RoBERTa layers (ablation) | Transformer + RoBERTa + image attention · GoodNews | CIDEr: 48.5 -> 53.1 | Improved |
| Use text surrounding the image instead of the first article tokens (ablation) | Transformer + weighted RoBERTa · NYTimes800k | CIDEr: 45.1 -> 52.8 | Improved |
| Add face attention (ablation) | Location-aware Transformer + weighted RoBERTa · NYTimes800k | CIDEr: 52.8 -> 53.9 | Improved |
| Add face attention (ablation) | Location-aware Transformer + weighted RoBERTa · NYTimes800k | people's name recall: 30.2 -> 30.8 | Improved |
| Add object attention (ablation) | Location-aware Transformer + face attention · NYTimes800k | CIDEr: 53.9 -> 54.4 | Improved |
| Add object attention (ablation) | Transformer + weighted RoBERTa + face attention · GoodNews | CIDEr: 54.3 -> 53.8 | Hurt |
Closest papers by meaning
- Good News, Everyone! Context Driven Entity-Aware Captioning for News Images (CVPR 2019 · similarity 0.93)
- Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning (AAAI 2026 · similarity 0.90)
- Show, Interpret and Tell: Entity-Aware Contextualised Image Captioning in Wikipedia (AAAI 2023 · similarity 0.88)
- Switchable Novel Object Captioner (TPAMI 2022 · similarity 0.87)
- Panoptic Captioning: An Equivalence Bridge for Image and Text (NeurIPS 2025 · similarity 0.86)
- Noise-Aware Image Captioning with Progressively Exploring Mismatched Words (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
