AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang, Jianfeng Gao, Lijuan Wang, Zicheng Liu

AAAI 2021 · Vision-language and multimodal › Human and embodied modeling · approach: Concept-guided image captioning · 53 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Pre-train on image-tag pairs with VIVO (ablation)COCO-trained captioning model · nocaps validation entireCIDEr: 69.6 -> 77.8Improved
Expand VIVO pre-training tags from 500 to 6.4K classes (ablation)VIVO captioning model · nocaps validation entireCIDEr: 76.5 -> 77.8Improved
Use Hungarian matching for masked tag prediction (ablation)VIVO pre-training · nocaps validation entireCIDEr: 75.8 -> 77.8Improved
Mask multiple tokens rather than only one token (ablation)VIVO pre-training · nocaps validation entireCIDEr: 74.9 -> 77.8Improved
Add VIVO pre-training on Open ImagesOSCAR fine-tuned on Conceptual Captions · COCO test (Karpathy split)CIDEr: 118.2 -> 119.8Improved
Use VIVO rather than OSCARNovel-object caption generation · nocaps validation out-of-domain objectsF1-score: 15.7 -> 30.6Improved
Add constrained beam searchUpDown · nocaps validationCIDEr: +17.8%Improved
Add constrained beam searchOSCAR · nocaps validationCIDEr: +15.5%Improved
Add constrained beam searchVIVO · nocaps validationCIDEr: +3.8%Improved
Add constrained beam searchVIVO · nocaps validation in-domainCIDEr: +1.6%Improved
Add constrained beam searchVIVO · nocaps validation near-domainCIDEr: +1.7%Improved
Add constrained beam searchVIVO · nocaps validation out-of-domainCIDEr: +11.9%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy