Research map / Vision-language and multimodal / Visual question answering
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the AS-1B dataset of 1.2B region annotations built with a human-in-the-loop data engine, and the All-Seeing Model (ASM), a location-aware LLM-based model for region recognition, captioning, and VQA.
Problem
Open-world instance-level (region-level) visual recognition and understanding is limited by the lack of large-scale region-text data with open-world semantics, and existing VLLMs only understand whole images.
Approach
A semi-automatic data engine in which off-the-shelf LLMs/VLLMs, detectors and CLIP annotate regions, with iterative human verification and model fine-tuning; plus a location-aware query tokenizer feeding an LLM decoder (Husky-7B) trained jointly with next-token prediction and CLIP-style region-text contrastive loss.
Key results
- ASM improves zero-shot COCO region recognition mAP over CLIP by 10.4 (58.9 to 69.3) and LVIS by 14.3 (47.1 to 61.4).
- ASM-FT reaches 145.1 CIDEr on Visual Genome and 103.0 CIDEr on RefCOCOg region captioning, above GRiT's 142.0 and 71.6.
- Zero-shot ASM achieves 77.9 CIDEr on Flickr30K, above Kosmos-2 (66.7) and InstructBLIP (82.8 is higher, so ASM is below InstructBLIP on that metric).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| second-stage fine-tune ASM on additional high-quality image- and region-level data | ASM · RefCOCOg | Meteor: 13.6 -> 20.8 | Helped |
| second-stage fine-tune ASM on additional high-quality image- and region-level data | ASM · Flickr30K | CIDEr: 77.9 -> 87.7 | Helped |
| add region-aware recognition using ASM instead of R-CLIP | ASM · LVIS | mAP: 54.8 -> 61.4 | Helped |
| increase semantic-tag training data from 1M to 5M images (ablation) | R-CLIP · COCO | mAP: 67.8 -> 68.6 | Helped |
| increase semantic-tag training data from 1M to 5M images (ablation) | R-CLIP · LVIS | mAP: 54.0 -> 54.8 | Mixed |
| add data cleaning and re-rank semantic candidates with CLIPSeg (ablation) | R-CLIP · COCO | mAP: 61.8 -> 67.8 | Helped |
| add data cleaning and re-rank semantic candidates with CLIPSeg (ablation) | R-CLIP · LVIS | mAP: 46.5 -> 54.0 | Helped |
| fine-tune on human-verified region annotations (ablation) | R-CLIP · COCO at input scale 224 | mAP: 67.8 -> 70.2 | Helped |
| fine-tune on human-verified region annotations (ablation) | R-CLIP · COCO at input scale 896 | mAP: 76.7 -> 80.0 | Helped |
| fine-tune on human-verified region annotations (ablation) | R-CLIP · LVIS at input scale 224 | mAP: 54.8 -> 55.0 | Helped |
| fine-tune on human-verified region annotations (ablation) | R-CLIP · LVIS at input scale 896 | mAP: 65.7 -> 68.4 | Helped |
Surprising findings
- Only 47.1% of generated VQA attribute answers were judged correct by human verifiers, and 54.8% top-1 semantic tag accuracy, indicating substantial annotation noise.
- Zero-shot ASM scores lower than GRiT on Visual Genome Meteor (12.6 vs 17.1) before second-stage fine-tuning.
Limitations
- Generated semantic tags and VQA answers are noisy; only 47.1% of attribute QA pairs were correct.
- Zero-shot region captioning on Visual Genome lags GRiT in Meteor before fine-tuning.
- Human evaluation is small (5 participants, 20 samples per dataset).
Experimental setup
Datasets: AS-1B LaionCOCO MiniGPT-4 data LLaVA-150k COCO caption Visual Genome RefCOCOg SA-1B BigDetection LVIS
Benchmarks: COCO LVIS Visual Genome RefCOCOg Flickr30K NoCaps
Metrics: mAP CIDEr SPICE Meteor human preference rate
Baselines: CLIP OpenCLIP GRiT Kosmos-2 SLR InstructBLIP Shikra-13B BLIP-2 MiniGPT4 LLaVA
Builds on: Husky CLIP SAM RoIAlign RegionCLIP
Closest papers by meaning
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos (NeurIPS 2025 · similarity 0.89)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.89)
- VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion (NeurIPS 2025 · similarity 0.87)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.86)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.