Research map / Vision-language and multimodal / Human and embodied modeling
Context-Aware Captions From Context-Agnostic Supervision
Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, Gal Chechik
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use introspective speaker inference instead of literal image-and-class captioning | CUB captioning model · CUB-Justify test | CIDEr-D: 12.3 ± 0.1 -> 18.4 ± 0.2 | Improved |
| Use introspective speaker inference instead of sample-and-rerank RS | CUB captioning model · CUB-Justify test | CIDEr-D: 15.8 ± 0.2 -> 18.4 ± 0.2 | Improved |
| Remove image information from the introspector (ablation) | IS(λ) on CUB-Justify · CUB-Justify test | CIDEr-D: 18.4 ± 0.2 -> 18.5 ± 0.2 | No clear effect |
| Remove image information from both speaker and introspector (ablation) | IS(λ) on CUB-Justify · CUB-Justify test | CIDEr-D: 18.4 ± 0.2 -> 16.1 ± 0.2 | Hurt |
| Add distractor-image context through introspective inference | COCO image captioning model · COCO easy confusion | human target-image selection (%): 74.6 -> 89.0 | Improved |
| Add distractor-image context through introspective inference | COCO image captioning model · COCO hard confusion | human target-image selection (%): 52.5 -> 74.1 | Improved |
| Add emitter-suppressor inference to vis-exp (ablation) | vis-exp on CUB-Justify · CUB-Justify test | CIDEr-D: 20.36 ± 0.16 -> 21.52 ± 0.17 | Improved |
| Lower λ from 1 toward an intermediate value to include distractor context (ablation) | Introspective speaker · CUB-Justify validation | CIDEr-D | Mixed |
| Use a trained listener for RS sample reranking instead of the introspector (ablation) | RS(λ) on CUB-Justify · CUB-Justify validation | CIDEr-D: 13.9 ± 0.2 -> 14.7 ± 0.2 | No clear effect |
| Train the inference objective rather than using it only at inference (ablation) | Introspective speaker · Not specified | performance | No clear effect |
Closest papers by meaning
- Context-Aware Visual Policy Network for Fine-Grained Image Captioning (TPAMI 2019 · similarity 0.91)
- Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning (CVPR 2022 · similarity 0.89)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.88)
- CALVIN: Improved Contextual Video Captioning via Instruction Tuning (NeurIPS 2024 · similarity 0.88)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.88)
- Panoptic Captioning: An Equivalence Bridge for Image and Text (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
