Research map / Vision-language and multimodal / Human and embodied modeling
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang, Jianfeng Gao, Lijuan Wang, Zicheng Liu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on image-tag pairs with VIVO (ablation) | COCO-trained captioning model · nocaps validation entire | CIDEr: 69.6 -> 77.8 | Improved |
| Expand VIVO pre-training tags from 500 to 6.4K classes (ablation) | VIVO captioning model · nocaps validation entire | CIDEr: 76.5 -> 77.8 | Improved |
| Use Hungarian matching for masked tag prediction (ablation) | VIVO pre-training · nocaps validation entire | CIDEr: 75.8 -> 77.8 | Improved |
| Mask multiple tokens rather than only one token (ablation) | VIVO pre-training · nocaps validation entire | CIDEr: 74.9 -> 77.8 | Improved |
| Add VIVO pre-training on Open Images | OSCAR fine-tuned on Conceptual Captions · COCO test (Karpathy split) | CIDEr: 118.2 -> 119.8 | Improved |
| Use VIVO rather than OSCAR | Novel-object caption generation · nocaps validation out-of-domain objects | F1-score: 15.7 -> 30.6 | Improved |
| Add constrained beam search | UpDown · nocaps validation | CIDEr: +17.8% | Improved |
| Add constrained beam search | OSCAR · nocaps validation | CIDEr: +15.5% | Improved |
| Add constrained beam search | VIVO · nocaps validation | CIDEr: +3.8% | Improved |
| Add constrained beam search | VIVO · nocaps validation in-domain | CIDEr: +1.6% | Improved |
| Add constrained beam search | VIVO · nocaps validation near-domain | CIDEr: +1.7% | Improved |
| Add constrained beam search | VIVO · nocaps validation out-of-domain | CIDEr: +11.9% | Improved |
Closest papers by meaning
- Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning (CVPR 2022 · similarity 0.90)
- nocaps: novel object captioning at scale (ICCV 2019 · similarity 0.90)
- Scaling Up Vision-Language Pre-Training for Image Captioning (CVPR 2022 · similarity 0.90)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.88)
- EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension (CVPR 2024 · similarity 0.88)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
