Research map / Vision-language and multimodal / Multimodal LLMs
FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FineCaption, a mask-referring VLM with a mask-aware CLIP encoder plus ConvNeXt and SAM high-resolution encoders, and the human-annotated CompositionCap dataset for multi-grained region compositional captioning.
Problem
VLMs struggle to describe compositional attributes of arbitrary mask-referred image regions at fine granularity, and bounding boxes or overlaid visual prompts are imprecise references.
Approach
Combine a mask-aware low-resolution CLIP encoder (alpha-channel mask conv, as in Alpha-CLIP) with high-resolution ConvNeXt and SAM encoders at 1024px, fused channel-wise and fed to an LLaMA-3.1-8B decoder, trained on a new 18-aspect region caption dataset.
Key results
- FineCaption-8B reaches CIDEr 127.95 and METEOR 22.01 on CompositionCap, above LLaMA-3.2-11B-Vision (99.11 CIDEr) in the supervised setting.
- On AARC, FineCaption-8B scores 56.84 GPT-4-as-a-Judge accuracy versus 50.26 for LLaMA-3.2-11B-Vision.
- On RefCOCOg region captioning, FineCaption-8B achieves METEOR 17.5 and CIDEr 118.2, exceeding ControlCap (17.0, 111.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove high-resolution encoders, keep only mask-aware low-resolution encoding (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 97.62 (-30.33) | Helped |
| use SAM alone as high-resolution encoder instead of ConvNeXt+SAM (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 106.74 | Helped |
| use ConvNeXt alone as high-resolution encoder instead of ConvNeXt+SAM (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 109.25 | Helped |
| channel-wise concatenation fusion vs self-attention fusion (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 104.85 | Helped |
| channel-wise concatenation fusion vs sequence append fusion (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 93.26 | Helped |
| replace mask referral with bounding box referral (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 96.85 | Helped |
| replace mask referral with mask contour referral (ablation) | FineCaption-8B · CompositionCap | CIDEr: 127.95 vs 95.97 | Helped |
| full FineCaption-8B (mask-aware + high-res encoders) vs supervised baselines | CompositionCap supervised · CompositionCap | CIDEr: 127.95 vs 99.11 (LLaMA-3.2-11B-Vision) | Helped |
| zero-shot evaluation of large VLMs (GPT-4o, LLaMA-3.2-90B, InternVL-2-40B) on region compositional captioning | zero-shot VLMs · CompositionCap | CIDEr: GPT-4o 6.49; LLaMA-3.2-90B 4.53 vs supervised 127.95 | Hurt |
| FineCaption-8B vs region-captioning baselines on RefCOCOg | FineCaption-8B · RefCOCOg test | CIDEr: 118.2 vs 111.4 (ControlCap) | Helped |
Surprising findings
- Zero-shot GPT-4o scores only 6.49 CIDEr on CompositionCap, far below supervised fine-tuned models.
- Freeform overlaid visual prompts (contours or boxes drawn on image) confuse VLMs, which often treat the prompt as image content.
Limitations
- Mask-aware encoder pre-training and three-stage pipeline are costly, with 38 hours on 8 A100 GPUs.
- Evaluation of AARC relies on GPT-4 as judge, and RDC grounding uses a LLaVA-1.6-13B model fine-tuned on the test set to predict boxes.
- Dataset is annotated only on stock-photo and Open Images sources with a limited number of images (5,392 training images).
Experimental setup
Datasets: CompositionCap DOCCI GranD RefCOCO RefCOCO+ RefCOCOg LLaVA-Pretrain Open Images
Benchmarks: CompositionCap RefCOCOg
Metrics: ROUGE-L BLEU-4 METEOR CIDEr BERT Score GPT-4-as-a-Judge accuracy Acc@0.5 grounding score
Baselines: Kosmos-2 Alpha-CLIP Qwen2-VL-7B Ferret ViP-LLaVA LLaMA-3.2-11B-Vision LLaMA-3.2-90B-Vision GPT-4o LLaVA-HR-X GLaMM
Builds on: Alpha-CLIP LLaVA SAM ConvNeXt
Closest papers by meaning
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.90)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.89)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.89)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.89)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.89)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.