Research map / Vision-language and multimodal / Multimodal LLMs
Image Textualization: An Automatic Framework for Generating Rich and Detailed Image Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Image Textualization (IT), a three-phase pipeline combining an MLLM reference description, vision expert outputs (dense captioner, SAM, depth estimator, open-set detector) and an LLM recaptioner to produce detailed, less-hallucinated image descriptions, released as IT-170K, and shows LLaVA-7B fine-tuned on them gains detail and reduces hallucination.
Problem
Existing image description datasets are either noisy web-scraped pairs or short, costly human annotations, and MLLM-generated descriptions suffer from hallucination and lack fine-grained detail.
Approach
Combine MLLM holistic descriptions with vision expert outputs (object detection for hallucination checking, dense captioning, SAM masks and monocular depth for object size/depth) textualized into text, then have an LLM recaption from text alone without seeing the image.
Key results
- On DID-Bench, IT-LLaVA raises BLEU-4 from 4.09 to 7.21 on GT-LLaVA versus the base LLaVA-7B (tuning with IT data vs. MLLM-generated data as well).
- LLaVA tuned on IT-GPT4-V data improves POPE average from 84.59 to 87.36 and LIN-Bench average from 9.40 to 10.94 (10k samples).
- D2I-Bench CLIP-score rises from 76.49 (GPT-4V descriptions) to 77.10 with IT-GPT-4V descriptions; IT-GPT-4V descriptions average 193 words vs 160 for GPT-4V.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Image Textualization to LLaVA descriptions | LLaVA image descriptions · DID-Bench, GT-{LLaVA} | BLEU-1: 12.90 -> 25.71 | Helped |
| Apply Image Textualization to GPT4-V descriptions | GPT4-V image descriptions · DID-Bench, GT-{GPT4-V} | BLEU-1: 45.26 -> 57.38 | Helped |
| Fine-tune LLaVA-7B on IT-generated descriptions | LLaVA-7B · POPE, Average | accuracy: 84.59 -> 86.47 | Helped |
| Fine-tune LLaVA-7B on IT-generated descriptions | LLaVA-7B · LIN-Bench, Average | average: 9.40 -> 10.26 | Helped |
| Use IT-generated descriptions instead of MLLM-generated descriptions for fine-tuning | LLaVA-7B · DID-Bench, GT-{LLaVA} & GT-{GPT4-V} | BLEU-1: 10.92 -> 23.78 | Helped |
| Use IT-generated descriptions instead of GPT4-V-generated descriptions for fine-tuning | LLaVA-7B · DID-Bench, GT-{LLaVA} & GT-{GPT4-V} | BLEU-1: 28.93 -> 46.79 | Helped |
| Add fine-grained object annotations during recaptioning (ablation) | Image Textualization recaptioning · Image description quality | detail and accuracy | Helped |
| Add in-context examples to recaptioning (ablation) | Image Textualization recaptioning · Image description quality | use of exact spatial values | Helped |
Surprising findings
- Reference-based metrics such as BLEU and CIDEr depend on the MLLM used for the holistic phase, reflecting style bias rather than only visual correctness.
Limitations
- Vision expert bottlenecks cause some inaccuracies remaining in descriptions.
- Reliance on model-generated data gives positivity bias (e.g., 'cozy', 'beautiful') from GPT-4V.
- Relies on BLEU/ROUGE-style metrics that are noisy and may not capture qualitative quality.
Experimental setup
Datasets: COCO IT-170K
Benchmarks: DID-Bench D2I-Bench LIN-Bench POPE
Metrics: BLEU-1 BLEU-4 CIDEr METEOR ROUGE-L SPICE WMD CLIP-score DINO-score ARI
Baselines: LLaVA-7B GPT-4V COCO captions MLLM-generated descriptions
Builds on: LLaVA Grounding DINO SAM dense captioning (DC) POPE
Closest papers by meaning
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.88)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.87)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.87)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.87)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.87)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.