Research map / Vision-language and multimodal / Human and embodied modeling
COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COCONut-PanCap, a human-edited, panoptic-mask-grounded detailed caption dataset on COCO images, plus a PGC baseline (PanCaper) and experiments showing gains in captioning, VQA, referring segmentation, and text-to-image generation.
Problem
Existing image-text datasets lack detailed, scene-comprehensive captions, and few datasets tie detailed captions to dense panoptic segmentation masks; synthetic VLM captions remain inferior to human-verified ones.
Approach
Annotate COCONut-S panoptic regions with set-of-marks overlays, have GPT-4V draft region captions, human-correct them, then merge into a single detailed caption whose noun phrases are grounded to panoptic masks; a LISA-based PanCaper baseline performs joint panoptic segmentation and grounded captioning.
Key results
- PanCaper-Pro reaches PQ 0.61 and CAPTURE 64.3 on the COCONut-PanCap val set, beating LISA+ (PQ 0.50, CAPTURE 57.9) and GLaMM GCG (PQ 0.55, CAPTURE 56.8).
- SD3-medium fine-tuned on COCONut-PanCap reaches GenEval overall 0.68 versus 0.65 for DOCCI and 0.62 for COCO-caption.
- LLaVA-NeXT with full 118K COCONut-PanCap captions improves MM-Vet from 43.5 to 45.5 and MMBench-en from 71.4 to 75.1 over the original baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaVA-NeXT on COCONut-PanCap captions instead of the original caption data | LLaVA-NeXT · COCONut-PanCap val | CAPTURE: 55.4 -> 58.7 | Helped |
| add panoptic mask-pooled features (ablation) | LLaVA-NeXT · COCONut-PanCap val | CAPTURE: 58.7 -> 61.4 | Helped |
| use human-annotated COCONut-PanCap captions instead of GPT-4V-generated captions | LLaVA-NeXT · COCONut-PanCap val | CAPTURE: 56.2 -> 58.7 | Helped |
| add mask-pooled features (ablation) | PanCaper · COCONut-PanCap val | CAPTURE: 62.6 -> 64.3 | Helped |
| add mask-pooled features (ablation) | PanCaper · COCONut-PanCap val | PQ: 0.56 -> 0.61 | Helped |
| train on COCONut-PanCap instead of GranDf | LISA+ · COCONut-PanCap val | PQ: 0.43 -> 0.50 | Helped |
| train on COCONut-PanCap instead of GranDf | GLaMM GCG · COCONut-PanCap val | PQ: 0.27 -> 0.55 | Helped |
| fine-tune SD3 medium on COCONut-PanCap instead of DOCCI | Stable Diffusion 3 medium · DOCCI test | FID: 22.1 -> 21.4 | Helped |
| fine-tune SD3 medium on COCONut-PanCap instead of DOCCI | Stable Diffusion 3 medium · COCONut-PanCap val | overall score: 0.65 -> 0.68 | Helped |
| add 118K COCONut-PanCap captions for instruction tuning | LLaVA-NeXT · MM-Vet | score: 43.5 -> 45.5 | Helped |
| add COCONut-PanCap training data | PanCaper+ · refCOCO val | score: 74.5 -> 76.2 | Helped |
| add 25% human-annotated captions to synthetic-caption training data (ablation) | SD3 medium · COCONut-PanCap val | FID: 31 -> 26 | Helped |
Surprising findings
- Only 20K human-labeled captions in LLaVA-1.5 instruction tuning match or slightly exceed 100K synthetic ShareGPT4V captions on VQA benchmarks.
- Fine-tuning on GPT-4V-only synthetic captions gives notably worse results than adding just 25% human-annotated data, with FID dropping from 31 to 26.
Limitations
- Dataset is smaller than large-scale synthetic detailed-caption datasets, limiting its use for pretraining.
- Comparisons to GLaMM are uneven because GLaMM uses extra GranD and SA-1B data.
Experimental setup
Datasets: COCO 2017 COCONut-S Objects365 LLaVA 665K DOCCI GranD/GranDf
Benchmarks: COCONut-PanCap val MM-Vet SEED-IMG MMBench-en TextVQA POPE MME GenEval DOCCI test refCOCO refCOCO+ refCOCOg
Metrics: CAPTURE CIDEr BLEU@4 METEOR ROUGE-L PQ FID CLIPScore cIoU
Baselines: LISA+ GLaMM GCG PixelLM LISA-7B LLaVA-NeXT LLaVA-1.5
Builds on: LISA kMaX-DeepLab LLaVA-NeXT GLaMM set-of-marks
Closest papers by meaning
- COCONut: Modernizing COCO Segmentation (CVPR 2024 · similarity 0.89)
- Panoptic Captioning: An Equivalence Bridge for Image and Text (NeurIPS 2025 · similarity 0.89)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.86)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.85)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.85)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.