Research map / Vision-language and multimodal / Human and embodied modeling
Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends SAM 2 with a Semantic Perceiver and a Qwen2.5 LLM decoder so one model segments a region while producing categories, definitions, functions, and captions for images and videos, trained on a refined 1.5M image / 0.6M video region-semantic dataset.
Problem
Segmentation models like SAM and SAM 2 segment regions but lack semantic understanding of them, while existing region-level VLMs produce limited outputs, are modality-specific, and rely on external mask generators.
Approach
Treat the SAM 2 segmentation backbone itself as the vision encoder: a Semantic Perceiver converts SAM 2's mask tokens and image embeddings into visual and semantic tokens for an LLM, with a parallel mask decoder, enabling simultaneous segmentation and multi-granularity region semantics; a storyboard-driven GPT-4o pipeline builds the training data, including region-level streaming video captions.
Key results
- PAM-3B achieves 74.9 semantic IoU on PACO and 52.3% accuracy on Total-Text, outperforming prior methods on those benchmarks.
- PAM-3B reaches 23.3 METEOR on HC-STVG, surpassing DAM-8B's 21.0 METEOR; it also achieves SOTA on Elysium (24.3 METEOR) and BensMOT (21.6 METEOR).
- PAM runs 1.2-2.4x faster than prior approaches and uses less GPU memory, per the paper's efficiency comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase semantic tokens from 16 to 64 (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 80.0 | Helped |
| reduce semantic tokens from 16 to 4 (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 78.9 | Hurt |
| remove semantic tokens (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 77.6 | Hurt |
| train all stages in one stage rather than the three-stage curriculum (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 78.7 | Hurt |
| skip Stage 1.5 video-enhanced pretraining (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 79.7 | No clear effect |
| use image embeddings before S2-FFM instead of after it (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 78.4 | Hurt |
| use all tokens plus semantic tokens instead of mask tokens plus semantic tokens (ablation) | PAM-3B · LVIS | Semantic IoU: 79.6 -> 79.9 | No clear effect |
| increase semantic tokens from 16 to 64 (ablation) | PAM-3B · RefCOCOg | METEOR: 26.9 -> 27.0 | No clear effect |
| increase semantic tokens from 16 to 64 (ablation) | PAM-3B · HC-STVG | METEOR: 23.3 -> 23.5 | Helped |
| use Stage 1 → 1.5 → 2 rather than all-in-one training (ablation) | PAM-3B · RefCOCOg | METEOR: 25.8 -> 26.9 | Helped |
| use Stage 1 → 1.5 → 2 rather than Stage 1 → 2 (ablation) | PAM-3B · HC-STVG | METEOR: 22.4 -> 23.3 | Helped |
| use image embeddings after S2-FFM instead of before it (ablation) | PAM-3B · RefCOCOg | METEOR: 25.0 -> 26.9 | Helped |
Surprising findings
- Initializing from a Stage 1.5 image-video checkpoint outperforms all-in-one training, suggesting staged curriculum matters more than joint training.
- On VideoRefer-Bench-D, PAM-3B trails VideoRefer-7B and DAM-8B, and on MDVP-Bench it is only comparable to VP-SPHINX-13B rather than clearly better.
Limitations
- Lacks general vision-language capabilities such as VQA; trained only on four region-level tasks.
- Real-time streaming video region captioning is hindered by the large number of visual tokens processed by the LLM.
- Errors include hallucinated elements, misread text, and defaulting to salient objects when the target is small or disappears; annotation bias from GPT-4o may contribute.
Experimental setup
Datasets: LVIS PACO COCO-Text Total-Text RefCOCOg Visual Genome Ref-L4 Ferret-Bench MDVP-Bench Elysium BensMOT HC-STVG
Benchmarks: LVIS PACO COCO-Text Total-Text VG RefCOCOg Ref-L4 Ferret-Bench MDVP-Bench Elysium BensMOT HC-STVG
Metrics: Semantic Similarity Semantic IoU accuracy ROUGE-L METEOR CIDEr G-STDC inference time GPU memory
Baselines: Shikra GPT4RoI Osprey Ferret VP-LLaVA VP-SPHINX DAM GLaMM Omni-RGPT RegionGPT VideoRefer Artemis
Builds on: SAM 2 Qwen2.5 Set-of-Mark prompting DAM TRACE-Uni
Closest papers by meaning
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
- The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World (ICLR 2024 · similarity 0.89)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.89)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.89)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
- Segment and Caption Anything (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.