Research map / Detection and segmentation / Open-set and few-shot detection
Aligning and Prompting Everything All at Once for Universal Visual Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APE, a DETR-based universal perception model that reformulates grounding as open-vocabulary detection with gated region-sentence fusion and treats stuff as proxy instances, trained once on 10 datasets and evaluated across 160+ datasets.
Problem
Existing vision foundation models either use heavy object-word fusion that cannot scale to thousands of prompts for detection and grounding, or suffer from the thing/stuff granularity gap in segmentation.
Approach
Reformulate visual grounding as open-vocabulary detection with independent, sentence-level prompt embeddings and gated cross-modality fusion (static zero-token for vocabularies), and equalize thing and stuff by converting connected components of stuff masks into proxy instances so all categories are learned at instance level in one model.
Key results
- APE-L (D) reaches 59.6 LVIS-1203 APb on val and 64.7 on RF100 APb, outperforming listed baselines such as GLIP (26.9) and UNINEXT (14.0) on LVIS.
- On the D3 grounding benchmark, APE-L (D) achieves 82.7 full-set ARb in inter-scenario evaluation versus 45.3 for UNINEXT, using a single forward pass for all references.
- Gated region-sentence fusion uses about 0.6x-1.4x less additional memory overhead than GLIP for large prompt sets, per Table 5 (APE-L shows 12% memory increase vs GLIP-L's 60% for detection).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| equalize thing and stuff learning as instance-level objectives (ablation) | APE segmentation model trained on COCO-Panoptic · COCO-Stuff | mIoU: 33.9 -> 37.6 | Helped |
| equalize thing and stuff learning as instance-level objectives (ablation) | APE segmentation model trained on COCO-Panoptic · COCO panoptic | PQ: 47.4 -> 48.2 | Helped |
| add SA-1B class-agnostic segmentation data (ablation) | APE trained on COCO · COCO val | APb: 50.0 -> 50.1 | Helped |
| add SA-1B class-agnostic segmentation data (ablation) | APE trained on COCO · COCO val | APm: 42.6 -> 43.4 | Helped |
| add SA-1B class-agnostic segmentation data (ablation) | APE trained on LVIS · LVIS val | APb: 37.3 -> 39.1 | Helped |
| add SA-1B class-agnostic segmentation data (ablation) | APE trained on LVIS · LVIS val | APm: 33.4 -> 35.3 | Helped |
| add a history embedding bank to region-sentence fusion (ablation) | APE with CLIP text embeddings · RefCOCO val | P@1: 80.7 -> 82.7 | Helped |
| use Llama2 instead of CLIP as the text model (ablation) | APE region-sentence fusion · RefCOCO val | P@1: 80.7 -> 85.3 | Helped |
| use region-sentence instead of object-word fusion (ablation) | APE trained with RefCOCO/+/g · RefCOCO val | P@1: 85.5 -> 80.7 | No clear effect |
Surprising findings
- Object-word fusion can deteriorate LVIS detection performance, whereas the proposed region-sentence formulation improves detection results.
Limitations
- Produces overlapping segments, so panoptic quality (PQ) is disadvantaged since only instance-level annotations are used for training.
- Performance on some datasets lags (e.g., APE-L (A) has low ADE-847 mIoU of 4.1 and OpenImages-based results vary across variants).
Experimental setup
Datasets: MS COCO Objects365 OpenImages LVIS SA-1B Visual Genome RefCOCO RefCOCO+ RefCOCOg GQA Flickr30K PhraseCut
Benchmarks: LVIS RF100 OID Objects365 ODinW COCO ADE-847 PC-459 ADE-150 SegInW PC-59 BDD-40
Metrics: AP mIoU PQ oIoU P@1 AR FPS GPU memory
Baselines: GLIP GLIPv2 UNINEXT GroundingDINO OpenSeeD X-Decoder OVSeg OWL-ViT OFA-DOD MDETR
Builds on: DETA DETR GLIP Mask2Former MaskDINO
Closest papers by meaning
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.88)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.87)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.87)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.87)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.87)
- Re-Aligning Language to Visual Objects with an Agentic Workflow (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.