AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

NewsStories: Illustrating Articles with Visual Summaries

Reuben Tan, Bryan A. Plummer, Kate Saenko, JP Lewis, Avneesh Sud, Thomas Leung

ECCV 2022 · Vision-language and multimodal › Human and embodied modeling · approach: Neural image captioning · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune CLIP on NewsStories with single image-text InfoNCECLIP · GoodNews zero-shot (5 images)R@1: 20.90 -> 15.61Hurt
fine-tune CLIP on NewsStories with MIL-NCECLIP · GoodNews zero-shot (5 images)R@1: 20.90 -> 13.30Hurt
aggregate multiple images (mean) instead of single-image alignmentCLIP fine-tuning · NewsStories test (5 images)R@1: 35.88 -> 49.12Improved
replace mean aggregation with an image transformerMean baseline · NewsStories test (5 images)R@1: 49.12 -> 50.08Improved
add sentence-to-image MIL alignment (MIL-SIM) on top of mean aggregation (ablation)Mean · NewsStories test (5 images)R@1: 49.12 -> 54.24Improved
add MIL-SIM objective (ablation)Mean · GoodNews zero-shot (5 images per set)R@1: 28.77 -> 30.23Improved
add MIL-SIM objective (ablation)Mean · GoodNews zero-shot (variable set size)R@1: 21.30 -> 25.12Improved
train on NewsStories with MIL-SIM versus pretrained CLIPCLIP · NewsStories test (5 images)R@1: 31.03 -> 54.24Improved
increase MIL-SIM input text length from 32 to 256 tokens (ablation)MIL-SIM · NewsStories val (5 images)R@1: 38.08 -> 49.23Improved
increase MIL-SIM input text length from 256 to 512 tokens (ablation)MIL-SIM · NewsStories val (5 images)R@1: 49.23 -> 48.73Hurt
increase MIL-SIM input text length from 256 to 512 tokens (ablation)MIL-SIM · GoodNews zero-shot (5 images)R@1: 30.23 -> 27.06Hurt
use single-image one-to-one alignment as the number of images per set growsSingle Image · GoodNews zero-shotR@1: 17.27 -> 15.61Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy