Research map / Vision-language and multimodal / Multimodal LLMs
Osprey: Pixel Understanding with Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Osprey extends MLLMs with mask-text instruction tuning for pixel-level region understanding, using a ConvNeXt-L CLIP encoder, a mask-aware visual extractor, and a new 724K-sample Osprey-724K dataset.
Problem
Existing multimodal LLMs understand images or bounding-box regions at low resolution, which yields imprecise region-text alignment and limits fine-grained pixel-level understanding.
Approach
Inject precise mask-pooled region features (multi-level, from a high-resolution convolutional CLIP encoder) interleaved with text tokens into the LLM, trained on a mask-text instruction dataset with object-level, part-level, negative and short-form samples.
Key results
- On LVIS referring classification Osprey-7B reaches 65.24% SS and 38.19% S-IoU, and on PACO 73.06% SS and 52.72% S-IoU, outperforming Ferret-7B by 14.38 SS points on PACO.
- On Cityscapes open-vocabulary segmentation Osprey-7B reaches 50.64 PQ, versus 35.57 for Ferret-7B and 34.70 for GPT4RoI-7B.
- Region captioning on RefCOCOg gives 16.6 METEOR and 108.3 CIDEr, surpassing GLaMM by 0.4 and 3.3 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mask-based region input (mask-aware visual extractor) instead of box input | Osprey-7B vs box-level MLLMs · Cityscapes, ADE20K-150, LVIS, PACO | PQ / SS / S-IoU: +15.94 PQ over GPT4RoI on Cityscapes; +15.07 PQ over Ferret on Cityscapes | Helped |
| Add short-form response prompts and data to training (ablation) | Osprey-724K · LVIS and PACO | SS and S-IoU: +8.83 SS and +12.54 S-IoU on LVIS; +22.80 SS on PACO | Helped |
| Add positive/negative (spatial- and class-aware) samples to training (ablation) | Osprey-724K · LVIS and PACO | SS and S-IoU: +1.69 SS and +1.49 S-IoU on LVIS; +1.47 SS and +2.33 S-IoU on PACO | Helped |
| Use multi-level mask features (summed across 4 stages) instead of single-level res4 features (ablation) | Mask-Aware Visual Extractor · Cityscapes PQ and LVIS SS | PQ / SS: 46.28 to 50.64 PQ on Cityscapes; 62.46 to 65.24 SS on LVIS | Helped |
| Replace ConvNeXt-L CLIP with ViT-L CLIP at 224 input (ablation) | Osprey vision encoder · Cityscapes PQ and LVIS SS | PQ / SS: 50.64 to 38.58 PQ on Cityscapes; 65.24 to 60.89 SS on LVIS | Hurt |
| Increase input resolution from 512 to 800 with ConvNeXt-L (ablation) | Osprey vision encoder · LVIS | SS and S-IoU: 65.24 to 68.29 SS; 38.19 to 42.66 S-IoU, but speed drops from 3.5 to 1.9 pairs/s | Helped |
| Reduce input resolution from 512 to 224 (ablation) | Osprey vision encoder · LVIS | SS and S-IoU: 65.24 to 53.20 SS; 38.19 to 26.12 S-IoU, speed 6.0 vs 3.5 | Hurt |
| Retrain GPT4RoI on Osprey-724K instead of original data (ablation) | GPT4RoI-7B · PACO and Detailed Description | SS / Detailed Description score: +20.89 SS on PACO; Detailed Description 49.97 to 65.30 | Helped |
| Add 665K LLaVA-1.5 data mixture in Stage 3 (ablation) | Osprey-7B · RefCOCOs detailed description | score ratio (%): 77.54 to 83.78 | Helped |
| Use mask-based input with Osprey vs Ferret on POPE popular/adversarial | Osprey-7B* · POPE Popular and Adversarial | accuracy: 87.83 vs 84.90 (Ferret); 85.33 vs 82.36 (Ferret) | Helped |
| Osprey on POPE Random setting vs Ferret | Osprey-7B* · POPE Random | accuracy: 89.47 vs 90.24 (Ferret) | No clear effect |
Surprising findings
- Box-cropped LLaVA-1.5 is competitive on detailed description (71.11%) yet Osprey without extra data gets 77.54%; image-level LLaVA-1.5 is much weaker on part-level PACO classification (42.20 SS vs 73.06 for Osprey).
- Increasing input resolution to 800x800 gave the best accuracy but processed only 1.9 pairs per second, so 512x512 was chosen as a tradeoff.
Limitations
- Higher input resolution improves accuracy but raises token count and inference cost sharply.
- Relies on GPT-4-generated instruction data and GPT-4 as judge for detailed description evaluation.
- Ferret-Bench evaluation uses boxes as input because mask annotations are unavailable.
Experimental setup
Datasets: COCO RefCOCO RefCOCO+ RefCOCOg LLaVA-115K PACO-LVIS LVIS CC3M Pascal Part PartImageNet Visual Genome VCR
Benchmarks: Cityscapes ADE20K-150 LVIS PACO RefCOCO validation (detailed description) Ferret-Bench POPE RefCOCOg (region captioning)
Metrics: PQ AP mIoU semantic similarity S-IoU accuracy METEOR CIDEr precision recall
Baselines: LLaVA-1.5 Kosmos-2 Shikra-7B GPT4RoI-7B Ferret-7B GLaMM GRIT InstructBLIP MiniGPT4 MM-GPT mPLUG-Owl CLIP-ConvNeXt-L
Builds on: LLaVA LLaVA-1.5 SAM CLIP Vicuna GPT4RoI
Closest papers by meaning
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.88)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.88)
- Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model (CVPR 2024 · similarity 0.88)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.