Research map / Vision-language and multimodal / CLIP and zero-shot
Tag2Text: Guiding Vision-Language Model via Image Tagging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tag2Text is a vision-language pre-training framework that learns an image tagger from tags parsed from paired text and uses the recognized tags to guide captioning, retrieval, and tagging.
Problem
Object tags used in VLP are either manually annotated (limited, poor generalization) or come from frozen detectors (limited performance, many parameters); detector-free VLP models discard tags entirely.
Approach
Parse tags (objects, scenes, attributes, actions) from paired captions with a text semantic parser to supervise a learned image tagging head, then feed recognized tags into an image-tag-text generation encoder-decoder and use tags as visible alignment indicators for retrieval.
Key results
- Tag2Text (14M, Swin) zero-shot OpenImages tagging mAP 83.4, close to fully supervised ML-Decoder's 85.8.
- Tag2Text-Swin achieves 131.8 COCO caption CIDEr (fine-tuned, 14M) versus BLIP's 127.8.
- Tag guidance improves NoCaps-ZS CIDEr from 88.8 to 89.5 at 4M and OPPO-ZS recall from 45.3 to 52.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tag guidance | Tag2Text · COCO captioning fine-tuned | CIDEr: 123.3 -> 124.6 (+1.3) | Helped |
| Add tag guidance | Tag2Text · NoCaps zero-shot | CIDEr: 88.8 -> 89.5 (+0.7) | Helped |
| Add tag guidance | Tag2Text · OpenImages zero-shot captioning | Recall: 44.6 -> 52.0 (+7.4) | Helped |
| Pre-train on image-text pairs before COCO fine-tuning (ablation) | Tag2Text image tagger · OpenImages zero-shot tagging | mAP: 57.5 -> 83.4 | Helped |
| Expand vocabulary set from 3,429 to 4,585 categories (ablation) | Tag2Text · OPPO | mAP: 81.7 -> 80.3 | Hurt |
| Expand vocabulary set from 3,429 to 4,585 categories (ablation) | Tag2Text · OpenImages | mAP: 84.1 -> 83.1 | Hurt |
| Replace full connection head with recognition decoder (ablation) | Tag2Text · OpenImages zero-shot tagging | mAP: 79.8 -> 81.2 | Helped |
| Share recognition decoder layers with interaction encoder (ablation) | Tag2Text · OpenImages zero-shot tagging | mAP: 81.2 -> 81.9 | Helped |
| Use a recognition decoder instead of a full connection head (ablation) | Tag2Text · COCO captioning fine-tuned | CIDEr: 120.6 -> 121.2 | Helped |
| Share recognition decoder layers with interaction encoder (ablation) | Tag2Text · COCO captioning fine-tuned | CIDEr: 121.2 -> 121.6 | Helped |
Surprising findings
- Tagging performance decreased when the vocabulary was expanded from 3,429 to 4,585 categories.
- Tag2Text trained only on COCO generalizes poorly to OpenImages (mAP 57.5), but pre-training on large data recovers 83.4 mAP.
Limitations
- Incorrect user-provided or recognized tags can cause hallucinated captions.
- Struggles with small objects such as spoon or baseball at standard resolution.
Experimental setup
Datasets: COCO Visual Genome SBU Captions CC-3M CC-12M OpenImages OPPO NUS-WIDE NoCaps Flickr30K VQA v2 ImageNet
Benchmarks: OPPO OpenImages COCO NUS-WIDE COCO Caption NoCaps COCO retrieval Flickr30K VQA v2
Metrics: mAP F1 precision recall BLEU-4 CIDEr SPICE R@1 R@5 R@10
Baselines: CLIP BLIP BLIP-2 ML-Decoder MKT DiHT ALBEF VLMO OSCAR VIVO UFO ViTCap
Builds on: BLIP ALBEF ML-Decoder BERT
Closest papers by meaning
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.88)
- Enhancing Vision-Language Pre-training with Rich Supervisions (CVPR 2024 · similarity 0.87)
- DS-VLM: Diffusion Supervision Vision Language Model (ICML 2025 · similarity 0.87)
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.87)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.87)
- Anchor-based Robust Finetuning of Vision-Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.