Research map / Vision-language and multimodal / Multimodal LLMs

Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EDC, a captioning pipeline that uses off-the-shelf visual specialists (detection, depth, emotion, OCR, fine-grained recognition, HOI) and LLMs to enrich image captions, producing EDC-1M and EDC-118K, which improve LLaVA-v1.5 and LLaVA-NeXT pretraining.

Problem

Existing descriptive image captions, from human annotation or LMM distillation, lack precision and granularity (missing objects, fine-grained attributes, spatial and interaction relations), limiting LMM training for perception and reasoning.

Approach

Instead of relying on LMM or human captioners alone, ground captions in structured attributes from visual specialist models (fine-grained categories, depth, emotion, OCR, 2D/3D spatial relations, HOI, counts), then use an LLM to fuse these attributes into region captions and then full image captions.

Key results

• LLaVA-v1.5-7B pretrained with EDC captions reaches 64.3 on SEED-Bench and 71.0 on ScienceQA, improving over ShareGPT4V (61.9 and 68.4).
• LLaVA-NeXT-7B with EDC-118K captions achieves 40.1 on MM-Vet and 72.2 on SEED-Bench, the best among compared caption methods.
• Human evaluation on 100 images shows EDC captions cover more attributes than InternVL2 and LLaVA-NeXT, e.g. emotion 0.47 vs 0.23 and 0.14, and OCR 0.48 vs 0.26 and 0.33.

Surprising findings

• The EDC-1M-trained LLaVA-v1.5 performs poorly on MMBench-CN due to lack of Chinese data, and the detection noise lowers POPE performance.
• Captions from a strong LMM (InternVL2-26B) give only marginal gains over human COCO captions on several benchmarks, while EDC gives larger gains.

Limitations

• Detection model noise hurts object-hallucination benchmarks such as POPE.
• Lacks Chinese/multilingual captions, weak on MMBench-CN.
• OCR specialist limits and threshold settings hinder TextVQA gains.

Experimental setup

Datasets: DenseFusion COCO LLaVA-mix-665K LLaVA-NeXT-data

Benchmarks: VQAv2 DocVQA OKVQA GQA TextVQA ScienceQA AI2D MMBench MM-Vet SEED-Bench MMMU POPE

Metrics: accuracy human-rated attribute richness

Baselines: ShareGPT4V DenseFusion ReCap DCI DOCCI IT InternVL2-26B captions LLaVA-NeXT-34B captions human COCO captions

Builds on: LLaVA-v1.5 LLaVA-NeXT ShareGPT4V DenseFusion

Closest papers by meaning

• Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.91)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.90)
• VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.89)
• MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning (CVPR 2026 · similarity 0.88)
• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.88)
• MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suites (ICLR 2026 · similarity 0.88)