Research map / Vision-language and multimodal / Multimodal LLMs
DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds DenseFusion-1M, a 1M-image hyper-detailed caption dataset from LAION using a perceptual fusion caption engine (LLaVA-1.6-7B fine-tuned on GPT-4V captions guided by vision-expert outputs), and shows it improves LLaVA-1.5 and LLaVA-S² across VQA/perception benchmarks.
Problem
Scarcity of hyper-detailed, accurate image-text datasets for training MLLMs on comprehensive visual perception, since existing caption engines miss visual details and human or GPT-4V annotation is costly and not scalable.
Approach
Fuse outputs of multiple vision experts (RAM++ tags, EVA02 and OWL-ViTv2 detection, OCR, LAION short captions as world knowledge) as image priors fed into a low-cost MLLM caption engine, which is distilled from GPT-4V captions to produce dense descriptions at scale.
Key results
- LLaVA-1.5 (Vicuna-7B) with DenseFusion-1M pre-training reaches 37.8 on MM-Vet vs 30.5 for the original LLaVA-1.5 and 62.0 vs 58.2 on TextVQA.
- LLaVA-S² trained on DenseFusion-1M reaches 67.4 TextVQA and 72.1 SQA-IMG, outperforming LLaVA-1.6 (64.9 TextVQA, 70.1 SQA-IMG) with about one fifth of visual tokens.
- Ablation on 100K subset: perceptual fusion raises high-res S² TextVQA from 61.7 to 63.0 and MMBench from 66.9 to 68.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with DenseFusion-1M instead of the baseline data | LLaVA-S², Vicuna-7B · TextVQA | score: 60.8 -> 67.4 | Helped |
| Add perceptual fusion inputs from vision experts (ablation) | LLaVA-1.5 · TextVQA | score: 59.9 -> 60.8 | Helped |
| Add perceptual fusion inputs from vision experts (ablation) | LLaVA-S² · TextVQA | score: 61.7 -> 63.0 | Helped |
| Fine-tune the last 12 vision encoder layers rather than freeze the encoder (ablation) | LLaVA-1.5 · TextVQA | score: 59.6 -> 60.8 | Helped |
| Fully fine-tune rather than fine-tune the last 12 vision encoder layers (ablation) | LLaVA-1.5 · ScienceQA-IMG | score: 68.9 -> 67.2 | Hurt |
| Fully fine-tune rather than fine-tune the last 12 vision encoder layers (ablation) | LLaVA-1.5 · MMBench | score: 67.0 -> 67.3 | Helped |
Surprising findings
- Despite drawing from generic LAION rather than COCO/VG domains, the dataset outperforms ShareGPT4V on benchmarks such as VQAv2 and GQA.
Limitations
- Captions are generated by a model distilled from GPT-4V, so errors and hallucinations from the teacher and the caption engine propagate.
- Dataset relies on LAION short captions that may misalign with image content, which the prompt tells the engine to disregard when inaccurate.
Experimental setup
Datasets: LAION-2B LLaVA-mix-665K ShareGPT4V pre-training data
Benchmarks: ScienceQA-IMG VQAv2 GQA MME POPE TextVQA MMBench SEED-Bench MM-Vet
Metrics: accuracy
Baselines: ShareGPT4V LVIS-4V LLaVA-1.6 Monkey InstructBLIP Qwen-VL mPLUG-Owl2 InternVL-Chat
Builds on: LLaVA-1.5 LLaVA-1.6 Scaling on Scales (S2) ShareGPT4V SemDeDup
Closest papers by meaning
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.92)
- DenseMLLM: Standard Multimodal LLMs for Dense Prediction (ICML 2026 · similarity 0.90)
- Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion (CVPR 2025 · similarity 0.90)
- VCoder: Versatile Vision Encoders for Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.90)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.