Research map / Vision-language and multimodal / Human and embodied modeling
BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BACON, a prompting method that decomposes VLM captions into structured bag-of-concept graphs (overall description, object list with descriptions, relationships), builds the 103k-pair ECO dataset with GPT-4V, and fine-tunes LLaVA-13B into a captioner that helps grounding and generation models without training.
Problem
Long, entangled image captions are hard to parse for models without LLM-based text encoders (e.g., GroundingDINO, SDXL), and existing captions often omit key objects and relations.
Approach
Prompt VLMs (via in-context learning) to output captions as element-wise, structured graphs of objects, descriptions, styles, themes and relationships, with bounding boxes added via Grounding DINO plus LLaVA and CLIP filtering, so that caption elements can be consumed directly by downstream models.
Key results
- BACON-style captions help Grounding DINO reach 37.7 AP50 and 35.9 recall on the ECO test benchmark, versus 33.1 AP50 and 20.2 recall for plain Grounding DINO.
- LLaVA(BACON)-Captioner achieves 59.1 on NLVR2 CQA vs 57.5 for ShareGPT-4V-13B, and 52.6 on CQAv1 vs 50.7.
- In a human study, BACON with GPT-4V reaches 61.9% precision and 92.4% recall, and the fine-tuned captioner reaches 56.2% precision and 82.8% recall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLAVA(BACON)-CAPTIONER captions instead of ShareGPT-4V-13B captions | Caption Question Answering · NLVR2 CQA | accuracy: 57.5 -> 59.1 | Helped |
| Train the captioner on ECO annotations instead of DCI annotations | LLaVA-13B captioner · NLVR2 | accuracy: 56.16 -> 56.74 | Helped |
| Train the captioner on ECO annotations instead of DOCCI annotations | LLaVA-13B captioner · OK-CQA | accuracy: 46.14 -> 48.49 | Helped |
| Train the captioner on ECO annotations instead of DCI annotations | LLaVA-13B captioner · MMVP | accuracy: 58.33 -> 59.67 | Helped |
| Use BACON-style captions rather than standard captions | Grounding DINO · ECO test benchmark | Recall: 20.2±0.1 -> 35.9 ± 0.7 | Helped |
| Add CLIP matching to Grounding DINO (ablation) | Grounding DINO · BACON benchmark | mIOU: 68.42 -> 80.34 | Helped |
| Add LLaVA filtering after CLIP matching (ablation) | Grounding DINO + CLIP · BACON benchmark | mIOU: 80.34 -> 82.21 | Helped |
| Use BACON-style captions to guide SDXL generation instead of using SDXL directly | SDXL · Text-to-image generation | Object recall (Ro): 59.2±4.0% -> 95.2 ± 1.1% | Helped |
| Use BACON-style captions to guide SDXL generation instead of using SDXL directly | SDXL · Text-to-image generation | Relationship recall (Rr): 41.5±3.5% -> 76.7 ± 0.9% | Helped |
Surprising findings
- The fine-tuned 13B captioner reaches about 91% of GPT-4V-based BACON precision and 90% of its recall.
- BACON-style captions let SDXL surpass DALL-E 3 in human-rated object and relationship recall (76.7% vs 71.6%).
Limitations
- Data collection relies heavily on manual correction since no fully automated method ensures reliability.
- The captioner's grounding ability is insufficient, so Grounding DINO is still needed for accurate bounding boxes.
- Due to resource constraints, larger architectures and datasets were not explored.
Experimental setup
Datasets: MS COCO NLVR2 VQAv1 VQAv2 OK-VQA MMVP Visual Genome LookTwice-QA Visual-7W
Benchmarks: CQA (NLVR2, OK-CQA, CQAv1, CQAv2) MMVP ECO test Visual Genome OV-SGG LookTwice-QA Visual-7W MSCOCO user study
Metrics: answer accuracy AP50 recall mIoU Recall@100 precision object recall relationship recall
Baselines: LLaVA-13B Qwen-VL-max ShareGPT-4V GPT-4V DOCCI IIW DenseFusion ReCap DCI Grounding DINO OV-DQUO DE-VIT
Builds on: LLaVA In-context learning Grounding DINO CLIP
Closest papers by meaning
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.89)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.88)
- Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception (CVPR 2026 · similarity 0.88)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.88)
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.