Research map / Vision-language and multimodal / Human and embodied modeling
Injecting Semantic Concepts Into End-to-End Image Captioning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CTN-predicted concept tokens and concept-classification initialization (ablation) | ViT/B captioner · COCO captioning | CIDEr: 114.8 -> 121.8 | Helped |
| Add masked-token knowledge distillation (ablation) | ViT/B captioner · COCO captioning | CIDEr: 114.8 -> 120.0 | Helped |
| Use all caption tokens instead of parsed nouns and adjectives as concept targets (ablation) | ViTCAP · COCO captioning | CIDEr: 120.9 -> 121.8 | Helped |
| Use caption-extracted concepts instead of detector-produced VinVL tags (ablation) | ViTCAP · COCO captioning | CIDEr: 119.7 -> 120.9 | Helped |
| Replace binary cross-entropy with focal loss for concept classification (ablation) | ViTCAP · COCO captioning | CIDEr: 115.0 -> 117.1 | Helped |
| Add focal-loss concept initialization to captioning with predicted concept tags (ablation) | ViTCAP · COCO captioning | CIDEr: 117.1 -> 120.5 | Helped |
| Use predicted concepts rather than ground-truth concepts during training (ablation) | ViTCAP · COCO captioning | CIDEr: 119.1 -> 120.6 | Helped |
| Use an independent tokenizer instead of the caption tokenizer for concept tokens (ablation) | ViTCAP · COCO captioning | CIDEr: 119.7 -> 120.1 | No clear effect |
| Increase concept-classification training from 10 to 40 epochs (ablation) | ViTCAP · COCO captioning | CIDEr: 120.5 -> 121.1 | No clear effect |
| Add large-scale pre-training, concept tokens, and knowledge distillation (ablation) | ViTCAP · COCO Karpathy split | CIDEr: 121.8 -> 125.2 | Helped |
| Apply constrained beam search (ablation) | ViTCAP · nocaps validation overall | CIDEr: 89.2 -> 93.8 | Helped |
| Use a detector-free ViTCAP with large-scale pre-training instead of detector-based VinVL | COCO Karpathy split · COCO Karpathy split | CIDEr: 140.4 -> 138.1 | Hurt |
Closest papers by meaning
- Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning (CVPR 2022 · similarity 0.89)
- End-to-End Transformer Based Model for Image Captioning (AAAI 2022 · similarity 0.89)
- Convolutional Image Captioning (CVPR 2018 · similarity 0.88)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.87)
- Top-Down Semantic Refinement for Image Captioning (AAAI 2026 · similarity 0.87)
- FlexCap: Describe Anything in Images in Controllable Detail (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.