Research map / Vision-language and multimodal / CLIP and zero-shot
General Object Foundation Model for Images and Videos at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GLEE, a unified object-level foundation model that detects, segments, tracks, grounds and identifies arbitrary objects in images and videos, trained jointly on ~5M (later 10M) images with mixed supervision.
Problem
Existing visual foundation models focus on image-level features or specialized object tasks; there is no general object-level model that unifies detection, segmentation, tracking, grounding and identification across images and videos with zero-shot transfer.
Approach
A MaskDINO-based object decoder with text encoder (CLIP-distilled) and visual prompter, using dynamic text-embedding classification so categories, names, captions and referring expressions share one interface, trained jointly across multi-granularity datasets and scaled with automatically labeled SA-1B and GRIT data.
Key results
- Zero-shot on TAO, GLEE surpasses previous best OVTrack by 36.0%; on LV-VIS it outperforms OV2Seg by 43.6%.
- GLEE-Pro reaches 62.0 APbox on COCO val and 54.2 APmask, competitive with specialist models.
- On OVIS, GLEE-Pro reaches SOTA only after fine-tuning (38.7 zero-shot vs 39.7 after fine-tuning reported as 55.5 AP75).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 2 million SA1B images and 5 million GRIT examples to GLEE-Lite training (ablation) | GLEE-Lite · OmniLabel | AP: 20.3 -> 22.7 | Helped |
| Add 2 million SA1B images and 5 million GRIT examples to GLEE-Lite training (ablation) | GLEE-Lite · OmniLabel | AP-descr: 14.0 -> 16.7 | Helped |
| Add 2 million SA1B images and 5 million GRIT examples to GLEE-Plus training (ablation) | GLEE-Plus · OmniLabel | AP: 25.4 -> 27.0 | Helped |
| Add 2 million SA1B images and 5 million GRIT examples to GLEE-Plus training (ablation) | GLEE-Plus · OmniLabel | AP-descr: 17.5 -> 19.4 | Helped |
| Use GLEE-Pro instead of GLEE-Plus, with larger training and inference resolutions | GLEE video models · LV-VIS | AP: 30.3 -> 23.9 | Hurt |
Surprising findings
- On OVIS, GLEE does not reach SOTA zero-shot, requiring a few hours of fine-tuning to surpass specialists.
- Replacing SAM with GLEE in LISA yields comparable referring segmentation results rather than improvement.
Limitations
- Weaker on OVIS occlusion-heavy videos without fine-tuning.
- Lower AP-descr on OmniLabel due to limited caption data in training.
- Image-level training alone cannot guarantee temporal consistency under severe occlusion.
Experimental setup
Datasets: Objects365 OpenImages COCO LVIS BDD YTVIS19 YTVIS21 OVIS RefCOCO RefCOCO+ RefCOCOg Visual Genome
Benchmarks: COCO LVIS RefCOCO RefCOCO+ RefCOCOg UVO TAO BURST LV-VIS YTVIS2019 OVIS OmniLabel
Metrics: AP AP50 Precision@0.5 oIoU AR@100 HOTA TETA mAP
Baselines: UNINEXT OV2Seg Florence-2 ODISE PolyFormer OVTrack GLIP LISA GenVIS DVIS
Builds on: MaskDINO DetCLIP UNINEXT IDOL MinVIS
Closest papers by meaning
- Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.88)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.86)
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos (NeurIPS 2025 · similarity 0.86)
- Improving Single Domain-Generalized Object Detection: A Focus on Diversification and Alignment (CVPR 2024 · similarity 0.86)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.86)
- GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.