Research map / Vision-language and multimodal / CLIP and zero-shot
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Florence-2 is a prompt-based sequence-to-sequence vision foundation model (232M and 771M params) pre-trained on FLD-5B, a 126M-image dataset with 5.4B annotations produced by an automated data engine, and then fine-tuned for generalist vision tasks.
Problem
Vision models are typically specialized per task and lack a unified architecture and comprehensive, large-scale annotations covering both spatial hierarchy and semantic granularity.
Approach
Cast all vision tasks (captioning, detection, grounding, segmentation) as text-generation conditioned on task prompts, with location tokens for regions, trained with a single cross-entropy objective on a large automatically annotated multitask dataset (FLD-5B) built by an iterative specialist-plus-refinement data engine.
Key results
- Zero-shot Florence-2-L reaches 135.6 CIDEr on COCO caption (Karpathy test) with under 1% of Flamingo-80B's parameters, and 35.8 mIoU on RefCOCO referring segmentation.
- Fine-tuned Florence-2-L reaches 91.7 / 80.5 on RefCOCO / RefCOCO-g style region tasks, establishing new SOTA on RefCOCO/+/g per the paper, and outperforms PolyFormer by 3.0 Acc@0.5 on RefCOCO REC.
- Fine-tuned Florence-2-L achieves 81.5 TextVQA accuracy without external OCR tokens, per the text (Table 4 lists 73.5 for the generalist Florence-2-L, so the text and table disagree).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Florence-2-L instead of Kosmos-2 for zero-shot evaluation | Vision foundation model · Flickr30k test | Recall@1: 78.7 -> 84.4 | Helped |
| Use Florence-2-L instead of Kosmos-2 for zero-shot evaluation | Vision foundation model · RefCOCO val | Accuracy: 52.3 -> 56.3 | Helped |
| Use Florence-2-L instead of Kosmos-2 for zero-shot evaluation | Vision foundation model · RefCOCO+ val | Accuracy: 45.5 -> 53.6 | Helped |
| Use Florence-2-L instead of Kosmos-2 for zero-shot evaluation | Vision foundation model · RefCOCOg val | Accuracy: 60.6 -> 68.0 | Helped |
| Use Florence-2-L instead of Flamingo for zero-shot captioning | Vision foundation model · COCO caption test | CIDEr: 84.3 -> 135.6 | Helped |
| Fine-tune Florence-2-L with public supervised data | Florence-2-L · TextVQA test-dev | Accuracy: 73.5 | Helped |
Surprising findings
- A 0.77B model without an LLM is competitive with much larger specialist and generalist models such as Flamingo-80B and PaLI-X.
Limitations
- Relies on the quality of automated specialist and LLM/LMM annotations, which are noisy; detailed captions were generated only for a small subset due to cost.
- Zero-shot and fine-tuned results on several tasks depend on external OCR or public supervised data.
Experimental setup
Datasets: ImageNet-22k Objects365 Open Images Conceptual Captions LAION FLD-5B COCO Flickr30k Entities RefCOCO RefCOCO+ RefCOCOg TextVQA
Benchmarks: COCO Caption NoCaps TextCaps COCO Object Detection Flickr30k RefCOCO RefCOCO+ RefCOCOg RefCOCO RES VQAv2 TextVQA VizWiz VQA
Metrics: CIDEr mAP Recall@1 Accuracy@0.5 mIoU
Baselines: Flamingo Kosmos-2 PolyFormer UNINEXT Unified-IO PaLI-X GIT2 BLIP-2 CoCa UniTAB SeqTR Ferret
Builds on: Florence Unified-IO OFA PaLI DaViT
Closest papers by meaning
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.88)
- Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024 · similarity 0.88)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.88)
- Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion (CVPR 2025 · similarity 0.88)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.