Research map / Vision-language and multimodal / Multimodal LLMs
Panoptic Captioning: An Equivalence Bridge for Image and Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces panoptic captioning (detailed captions with bounding-boxed entities, attributes, relations, global state), a data engine PancapEngine, the SA-Pancap benchmark, the PancapScore metric, and PancapChain, a stage-wise LLaVA-based captioner that beats larger open and proprietary MLLMs.
Problem
Existing MLLMs and captioners describe images vaguely, without precise entity localization, and existing caption metrics cannot evaluate comprehensive, instance-level captions that include bounding boxes.
Approach
Formulate captioning as generating a comprehensive text description with bounding-box-grounded entity instances across five dimensions, and decompose generation into stages (localize, tag, discover missing instances, caption) trained on an automatically built detect-then-caption dataset, evaluated with an instance-matching and QA-based metric.
Key results
- PancapChain-13B reaches 175.75 overall PancapScore on validation and 173.19 on test, above Gemini-2.0-Pro (150.75 / 157.88) and InternVL-2.5-78B (147.79 / 154.66).
- On DOCCI text-to-text retrieval, PancapChain reaches 61.9 R@1 versus 59.6 for ShareGPT4V and 59.9 for ALIGN.
- PancapScore correlates with human ratings at PCC 0.60 versus 0.17 for the best baseline metric (Capture+BoxScore).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an instance discovery stage to the one-stage baseline (ablation) | PancapChain · SA-Pancap validation | Attribute: 43.06 -> 43.71 | Helped |
| Add an instance discovery stage to the one-stage baseline (ablation) | PancapChain · SA-Pancap validation | Relation: 31.64 -> 32.62 | Helped |
| Split instance discovery into localization and semantic tag assignment stages (ablation) | PancapChain · SA-Pancap validation | Tagging: 56.47 -> 57.04 | Helped |
| Add extra instance discovery after localization and semantic tag assignment (ablation) | PancapChain · SA-Pancap validation | Attribute: 43.76 -> 44.78 | Helped |
| Add extra instance discovery after localization and semantic tag assignment (ablation) | PancapChain · SA-Pancap validation | Relation: 33.69 -> 34.61 | Helped |
| Use PancapChain captions for text-text retrieval instead of ShareGPT4V captions | DOCCI retrieval · DOCCI | Recall@1: 59.6 -> 61.9 | Helped |
Surprising findings
- Current large MLLMs (up to 90B, and GPT-4o / Gemini-2.0-Pro) underperform a 13B model fine-tuned on the auto-generated data.
- LLaVA-1.5-13B fine-tuned directly on SA-Pancap already scores 161.84 validation overall, indicating high data quality.
Limitations
- Task formulation is an approximation of the conceptual minimum text equivalence.
- Test set is small (130 images) and validation/test images were selected by PancapScore and DINOv2 diversity filters.
- Evaluation relies on an LLM (Qwen2.5-14B) for semantic extraction and QA, and training data is produced by MLLMs.
Experimental setup
Datasets: SA-1B SA-Pancap
Benchmarks: SA-Pancap DOCCI
Metrics: PancapScore F-score (tag, location, attribute, relation, global) Recall@1 Capture PCC Kendall tau 1-R^2
Baselines: Molmo-72B LLaVA-OneVision-72B Qwen2-VL-72B Qwen2.5-VL-72B NVLM-72B InternVL-2.5-78B Llama-3.2-90B GPT-4o Gemini-2.0-Pro LLaVA-1.5-13B ShareGPT4V-13B
Builds on: LLaVA ASMv2 RAM OLN Grounding-DINO
Closest papers by meaning
- COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation (NeurIPS 2025 · similarity 0.89)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.87)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.87)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.87)
- Noise-Aware Image Captioning with Progressively Exploring Mismatched Words (AAAI 2024 · similarity 0.87)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.