Research map / Vision-language and multimodal / Multimodal LLMs
Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Florence-VL uses the generative vision encoder Florence-2, fusing multi-depth and multi-prompt visual features via channel concatenation (DBFusion), and trains MLLMs on Phi-3.5-mini and Llama-3 8B to outperform CLIP-based LLaVA-style models and Cambrian-8B across 25 benchmarks.
Problem
CLIP/SigLIP-style vision encoders give a single high-level, image-level feature that often misses region-level and text details, and multi-encoder mixtures add cost. The paper asks how to extract and align diverse visual features from one vision model with a pretrained LLM.
Approach
Use Florence-2 (a generative vision foundation model) as the MLLM vision encoder, extracting features from multiple depths (DaViT features) and multiple task prompts (detailed caption, OCR, dense region caption), then fuse them by channel concatenation (DBFusion) and project into the LLM, with full-model pretraining followed by projector+LLM finetuning.
Key results
- Florence-VL 8B reports 84.7 on VQAv2 and 89.9 on POPE, outperforming Cambrian 8B (which uses multiple vision encoders) on most reported general benchmarks.
- Under identical LLaVA-1.5 data and backbones, Florence-VL 7B averages 58.3 on general benchmarks versus LLaVA 1.5 7B's baseline, with large OCR gains (e.g., DocVQA 44.5 vs 28.1).
- Channel integration of four features matches average pooling in token count while outperforming token integration (1728 tokens) in average score (50.8 vs 50.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Florence-2 encoder with DBFusion (channel concatenation of multi-prompt features) instead of CLIP, same LLaVA 1.5 data | Vicuna 1.5 7B · General benchmarks (LLaVA 1.5 data) | MME-P: 1510.7 -> 1543.5 | Helped |
| Use Florence-2 encoder with DBFusion instead of CLIP, same LLaVA 1.5 data | Vicuna 1.5 7B · OCR & Chart (LLaVA 1.5 data) | DocVQA: 28.1 -> 44.5 | Helped |
| Channel-concatenate multi-feature fusion instead of token concatenation or average pooling (ablation) | Florence-VL fusion · LLaVA 1.5 data, 8-benchmark average | Average: 50.3 (token) / 50.4 (avg pool) -> 50.8 | Helped |
| Keep higher-level prompt features (caption, OCR, grounding) in addition to DaViT lower-level feature (ablation) | Florence-VL 7B · LLaVA 1.5 data, general benchmarks | MMBench (EN): 64.3 -> 66.1 | Helped |
| Remove caption prompt feature from the breadth set (ablation) | Florence-VL 7B · General/Vision/Knowledge/OCR average | Average: 58.3 -> 57.6 | Hurt |
| Remove OCR prompt feature from the breadth set (ablation) | Florence-VL 7B · General/Vision/Knowledge/OCR average | Average: 58.3 -> 57.3 | Hurt |
| Remove grounding prompt feature from the breadth set (ablation) | Florence-VL 7B · General/Vision/Knowledge/OCR average | Average: 58.3 -> 58.2 | Hurt |
| Add OCR prompt feature to caption and grounding features (ablation) | Florence-VL 7B · OCR & Chart benchmarks | Average: 34.2 -> 34.9 | Helped |
| Add OCR prompt feature to caption and grounding features (ablation) | Florence-VL 7B · Knowledge-based benchmarks | Average: 46.0 -> 46.8 | Helped |
| Remove caption or grounding prompt feature from the knowledge-based subset (ablation) | Florence-VL 7B · Knowledge-based benchmarks | Average: 46.8 -> 46.7 (caption), 46.8 -> 47.0 (grounding) | No clear effect |
| Scale LLM backbone from Llama 3 8B to Phi 3.5 or other stronger LLM with Florence-VL features | Florence-VL · Knowledge-based benchmarks | MathVista: 28.0 (Vicuna 7B) -> 31.8 (Llama 3 8B) | Helped |
| Florence-VL 8B (Florence-2 + DBFusion, Llama 3 8B) vs Cambrian 8B (multi-encoder) | Llama 3 8B Instruct · General, Vision-centric, OCR benchmarks | MMVP and DocVQA: MMVP 51.3 -> 73.3; DocVQA 77.8 -> 84.9 | Helped |
Surprising findings
- Token-level concatenation of all four features (1728 visual tokens) performs slightly worse than channel concatenation and is more expensive to train.
- Removing the OCR feature barely changes TextVQA-type results but hurts most OCR and chart benchmarks less than expected, while knowledge-based benchmarks show little dependence on visual features.
Limitations
- Fusion strategy is simple channel concatenation; more sophisticated adaptive fusion is not explored.
- Florence-2 features are fixed to three prompts, and high-resolution adaptation is not incorporated.
- Main results use a large proprietary-scale compute recipe (64 H100 GPUs) and a 16.9M caption dataset, making direct comparison difficult.
Experimental setup
Datasets: CC12M Redcaps Commonpool PixelProse Cambrian-7M Vision Flan ShareGPT4V Docmatix LLaVA-1.5 data
Benchmarks: VQAv2 GQA MMBench POPE MM-Vet MME SEED-Bench HallusionBench LLaVA-Bench MMStar TextVQA OCRBench
Metrics: accuracy score
Baselines: LLaVA 1.5 LLaVA-NeXT 8B VILA 3B VILA 8B Phi 3.5-Vision Mini-Gemini-HD 8B Cambrian 8B
Builds on: Florence-2 LLaVA LLaVA-1.5
Closest papers by meaning
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.90)
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.89)
- VCoder: Versatile Vision Encoders for Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.