Research map / Vision-language and multimodal / Multimodal LLMs
UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UFO, which unifies detection, segmentation, and vision-language tasks in one open-ended text interface by writing boxes as textual numbers and retrieving masks via dot-product between a predicted <MASK> token embedding and image features, applied to a ViT generalist and to InternVL2.5-8B and LLaVA-1.5-7B.
Problem
Integrating fine-grained perception (detection, segmentation) into generalist models and MLLMs without task-specific decoders or lossy polygon/textual-class mask representations.
Approach
Reformulate segmentation as embedding retrieval: the model autoregressively emits <MASK> tokens whose embeddings dot-product with image features to produce masks; multiple mask tokens (N^2=16) upsample the output resolution, and parallel decoding of per-grid-point subtasks handles multi-object outputs.
Key results
- UFO-ViT-H multi-task reaches 48.1 AP on COCO instance segmentation, 12.3 above GiT-H (35.8), and 55.7 mIoU on ADE20K vs GiT-H 52.4.
- UFO-InternVL2.5-8B fine-tuned reaches 67.0 gIoU on ReasonSeg, 6.2 above HiMTok-8B (60.8) without a CoT strategy.
- UFO-InternVL2.5-8B achieves 89.6 average REC accuracy (P@0.5) after specific finetuning, versus 85.6 for VisionLLM v2 (no-finetune UFO-InternVL2.5-8B averages 88.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UFO rather than GiT | ViT-H multi-task model · COCO instance segmentation | AP: 35.8 -> 48.1 | Helped |
| use UFO rather than GiT | ViT-H multi-task model · ADE20K semantic segmentation | mIoU: 52.4 -> 55.7 | Helped |
| fine-tune on reasoning segmentation data (ablation) | UFO-InternVL2.5-8B · ReasonSeg test | gIoU: 60.0 -> 67.0 | Helped |
| increase mask-token count from 1 to 16 (ablation) | UFO-ViT-B single-task · COCO instance segmentation | mAP: 38.9 -> 42.6 | Helped |
| increase mask-token count from 16 to 25 (ablation) | UFO-ViT-B single-task · COCO instance segmentation | mAP: 42.6 -> 42.9 | Helped |
| increase mask-token count from 16 to 25 (ablation) | UFO-ViT-B single-task · COCO instance segmentation | FPS: 3.6 -> 2.8 | Hurt |
| use open-ended decoding instead of the default restricted decoding (ablation) | UFO-ViT-B single-task · COCO detection | mAP: 45.1 -> 43.0 | Hurt |
| add beam search to open-ended decoding (ablation) | UFO-ViT-B single-task · COCO detection | mAP: 43.0 -> 45.1 | Helped |
| increase beam number from 1 to 3 (ablation) | UFO-ViT-B single-task · COCO detection | mAP: 45.6 -> 47.8 | Helped |
| increase beam number from 2 to 3 (ablation) | UFO-ViT-B single-task · COCO Caption | BLEU-4: 34.2 -> 34.0 | Hurt |
| add embedding retrieval for masks (ablation) | UFO-ViT-B single-task · COCO instance segmentation | AP: 31.3 -> 39.2 | Helped |
| add copy-paste augmentation (ablation) | UFO-ViT-B · COCO detection | AP: +1.1 | Helped |
Surprising findings
- Embedding retrieval reaches strong mask quality with only 16 mask tokens, with gains plateauing beyond 16 in the ablation.
- Open-ended detection suffers from severe positive/negative class imbalance, so the paper uses beam search rather than a restricted vocabulary.
Limitations
- Mask token count trades off accuracy against speed (FPS drops with more tokens).
- Open-ended detection needs beam search to counter class imbalance, adding inference cost.
- Performance on MLLMs partly depends on pretraining; UFO-ViT aligns better with detection/segmentation while InternVL2.5 suits captioning/REC.
Experimental setup
Datasets: COCO 2017 COCO Caption RefCOCO RefCOCO+ RefCOCOg ADE20K LLaVA-v1.5-mix665k LLaVA-OneVision Objects365 LVIS OpenImages nuImages
Benchmarks: COCO detection COCO instance segmentation ADE20K semantic segmentation COCO Caption RefCOCO/RefCOCO+/RefCOCOg REC and RES ReasonSeg GQA MMBench MMVP HallusionBench DRIVE NYUv2 depth
Metrics: AP mIoU BLEU-4 CIDEr Acc@0.5 P@0.5 cIoU gIoU F1
Baselines: GiT VisionLLM v2 HiMTok Cores LISA Ferret-v2 VistaLLM SAM4MLLM Mask2Former Deformable-DETR Mask R-CNN
Builds on: GiT LISA LLaVA-1.5 InternVL2.5 SAM
Closest papers by meaning
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.89)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- PerceptionGPT: Effectively Fusing Visual Perception into LLM (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.