Research map / Vision-language and multimodal / Multimodal LLMs
Ferret: Refer and Ground Anything Anywhere at Any Granularity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ferret is an MLLM that unifies referring and grounding via a hybrid region representation (discrete box/point coordinates plus continuous features from a spatial-aware visual sampler), trained on a 1.1M-sample GRIT dataset and evaluated on a new Ferret-Bench.
Problem
Existing MLLMs handle spatial referring and open-vocabulary grounding separately and only accept points or boxes as region inputs, not free-form shapes, and suffer from object hallucination.
Approach
A hybrid region representation combining discrete coordinates (as plain text tokens) with continuous region features extracted by a spatial-aware visual sampler (point-cloud-style FPS and kNN grouping over sampled mask points), letting one LLM accept points, boxes and free-form shapes as input and output boxes inline as text.
Key results
- Ferret-13B reaches 89.48% on RefCOCO val (Acc@0.5), above Shikra-13B's 87.83%.
- Ferret-13B scores 66.3 average on Ferret-Bench versus 45.9 for Shikra-7B and 34.0 for LLaVA.
- On POPE random split, Ferret reaches 90.24% accuracy, comparable to Shikra's 86.90%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add grounding data to referring training (ablation) | Ferret · Referring object classification | performance | Helped |
| Add referring data to grounding training (ablation) | Ferret · Grounding tasks | performance | Helped |
| Use Ferret's spatial-aware visual sampler instead of the SEEM visual sampler (ablation) | Ferret-7B · Referring tasks | performance | Helped |
| Increase the LLM backbone size (ablation) | Ferret · Referring and grounding tasks | performance | Helped |
| Train on Ferret's refer-and-ground data and method rather than LLaVA | LLaVA · LLaVA-Bench Detail Description | score: 68.3 to 80.9 | Helped |
Surprising findings
- Ferret outperforms GPT-4V on small-region referring and grounding in qualitative comparisons, while GPT-4V remains stronger on general question answering.
Limitations
- Can produce harmful or counterfactual responses, like most MLLMs.
- Weaker on small regions in some referring cases, and relies on GPT-4 judging for Ferret-Bench.
Experimental setup
Datasets: GRIT Visual Genome Objects365 RefCOCO Flickr30k Entities LLaVA-158k LVIS
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Flickr30k Entities LVIS LLaVA-Bench Ferret-Bench POPE
Metrics: Acc@0.5 relative GPT-4 score accuracy F1
Baselines: Shikra Kosmos-2 LLaVA GPT4ROI MiniGPT-4 InstructBLIP mPLUG-Owl MM-GPT MDETR UniTAB
Builds on: Shikra LLaVA PointMLP Vicuna SAM
Closest papers by meaning
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.89)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.88)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.88)
- See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.88)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.88)
- Pointing at Parts: Training-Free Few-Shot Grounding in Multimodal LLMs (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.