Research map / Vision-language and multimodal / Multimodal LLMs
Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free divide-then-aggregate pipeline that splits an image into spatial and semantic patches, generates detailed local captions, and hierarchically aggregates them with semantic-level filtering to reduce hallucinations and add detail.
Problem
MLLM-generated long-form image captions lack fine-grained details and suffer from hallucinations, in both open- and closed-source models; existing fixes often need retraining or only address global/object-level information.
Approach
Divide the image into four relation-based spatial patches plus one semantic patch, caption each patch with multiple samples, then use an LLM to classify candidate sentences as same/contradictory/unique (with BLIP-2 Blip2Score filtering) and hierarchically merge them into a global caption, without any model training.
Key results
- On DID-Bench with LLaVA-1.5-7B (GT-LLaVA), the method reaches CIDEr 5.17 vs 0.47 for the global baseline, with METEOR 17.73 vs 10.62.
- On GPT-4o, CHAIRs hallucination drops from 11.18 (global) to 8.42 and CAPTURE improves from 57.40 to 63.28 on DID-Bench.
- Ablation shows removing semantic filtering or semantic patch lowers scores, and replacing hierarchical aggregation with PoCa-style fusion causes a large drop (e.g., CIDEr 1.00 on LLaVA GT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply divide-then-aggregate caption enhancement | LLaVA-1.5-7B · DID-Bench, GT-{LLaVA} | CIDEr: 0.96 -> 5.17 | Helped |
| apply divide-then-aggregate caption enhancement | LLaVA-1.6-7B · DID-Bench, GT-{LLaVA} | CIDEr: 2.31 -> 5.50 | Helped |
| apply divide-then-aggregate caption enhancement | Mini-Gemini-7B · DID-Bench, GT-{LLaVA} | CIDEr: 6.82 -> 13.43 | Helped |
| apply divide-then-aggregate caption enhancement | GPT-4o · DID-Bench, GT-{LLaVA} | CIDEr: 0.75 -> 2.12 | Helped |
| apply divide-then-aggregate caption enhancement | GLM-4V-Plus · DID-Bench, GT-{LLaVA} | CIDEr: 1.28 -> 1.36 | Helped |
| apply divide-then-aggregate caption enhancement | LLaVA1.5 · D2I-Bench | DINO-score: 63.80 -> 66.32 | Helped |
| apply divide-then-aggregate caption enhancement | LLaVA1.5 · DetailCaps | CAPTURE: 57.51 -> 58.05 | Helped |
| apply divide-then-aggregate caption enhancement | GPT-4o · DID-Bench | CHAIRs: 11.18 -> 8.42 | Helped |
| add a semantic patch (ablation) | LLaVA-1.6-7B · DID-Bench, GT-{LLaVA} | CIDEr: 4.47 -> 5.50 | Helped |
| use semantic and spatial patches instead of four equal patches (ablation) | LLaVA-1.6-7B · DID-Bench, GT-{LLaVA} | CIDEr: 4.76 -> 5.50 | Helped |
| add semantic filtering (ablation) | LLaVA-1.6-7B · DID-Bench, GT-{LLaVA} | CIDEr: 2.93 -> 5.50 | Helped |
| use hierarchical aggregation instead of direct fusion (ablation) | LLaVA-1.6-7B · DID-Bench, GT-{LLaVA} | CIDEr: 1.00 -> 5.50 | Helped |
Surprising findings
- Equal four-patch division and the object-level variant performed in ways that show spatial patch design matters; two-patch division hindered perception.
- Existing training-free methods such as PoCa and IT often degrade CIDEr or hallucination metrics relative to the base model, e.g., PoCa's CHAIRs rises for LLaVA-1.5 when semantic filtering is removed.
Limitations
- Inference is much slower than base MLLM (7.04s vs 0.55s per image) due to multiple MLLM and LLM calls.
- Relies on the quality of an object detector (OVDet), a small VLM (BLIP-2) and an LLM for semantic analysis, so errors can propagate through the pipeline.
- Evaluation is mostly reference-based metrics and CHAIR on a 200-image benchmark, with a small DetailCaps subset of 200 images.
Experimental setup
Datasets: DID-Bench D2I-Bench DetailCaps benchmark POPE VQA-V2 LIN-Bench COCO VG SAM
Benchmarks: DID-Bench D2I-Bench DetailCaps benchmark POPE VQA-V2 LIN-Bench
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 CIDEr METEOR ROUGE-L SPICE WMD CLIP-S
Baselines: IT PoCa Synthesized ShareCaptioner ALLaVA Global (base MLLM)
Builds on: PoCa Synthesized caption enhancement IT (Image-Text enhancement) BLIP-2 OVDet
Closest papers by meaning
- Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception (CVPR 2026 · similarity 0.91)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
- Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.89)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.89)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.