Research map / Detection and segmentation / Open-set and few-shot detection
Segment and Caption Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Equips SAM with regional captioning by training only a lightweight query-based text feature mixer over a frozen SAM encoder and frozen language model, with weak-supervision pretraining on detection/segmentation data before fine-tuning on Visual Genome.
Problem
SAM segments anything class-agnostically but lacks semantic understanding, and region captioning data (e.g., Visual Genome, ~100K images) is scarce, limiting regional captioning performance.
Approach
Stack an additional lightweight query-based feature mixer (text query tokens reusing SAM tokens) on SAM's frozen features and align region features with a frozen pretrained language model's embedding space, training only the mixer (tens of millions of parameters); pretrain with weak supervision using category names from Objects365 and COCO-Panoptic before fine-tuning on VG.
Key results
- SCA (GPT2-large, Pretrain+VG) reaches 149.8 CIDEr-D, 17.5 METEOR, and 31.4 SPICE on Visual Genome, the best reported in Table 1, surpassing GRiT at 142.2 CIDEr-D.
- Query-based text mixer with SAM tokens reaches 137.4 CIDEr-D versus 45.2 for ROI Align, showing query-based mixing is much stronger.
- Weak-supervision pretraining on Objects365 raises VG CIDEr-D from 127.9 (no pretrain) to 134.5 under the ablation setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weak-supervision pretraining on Objects365 before VG fine-tuning (ablation) | SCA · Visual Genome test split | CIDEr-D: 127.9 -> 134.5 | Helped |
| Use Objects365 pretraining rather than COCO pretraining (ablation) | SCA · Visual Genome test split | CIDEr-D: 130.2 -> 134.5 | Helped |
| Use Objects365 pretraining rather than V3Det pretraining (ablation) | SCA · Visual Genome test split | CIDEr-D: 130.4 -> 134.5 | Helped |
| Freeze the larger GPT2-large text decoder (ablation) | SCA · Visual Genome test split | CIDEr-D: 135.6 -> 136.0 | Helped |
| Use a 12-layer rather than 24-layer feature mixer (ablation) | SCA · Visual Genome test split | CIDEr-D: 124.5 -> 127.7 | Helped |
| Replace ROI Align with a query-based feature mixer (ablation) | SCA · Visual Genome test split | CIDEr-D: 45.2 -> 137.4 | Helped |
| Add an MLP to ROI Align (ablation) | SCA · Visual Genome test split | CIDEr-D: 45.2 -> 82.5 | Helped |
| Add dedicated text query tokens rather than directly decoding SAM query tokens (ablation) | SCA · Visual Genome test split | CIDEr-D: 130.6 -> 137.4 | Helped |
| Reuse SAM tokens in the text query mixer (ablation) | SCA · Visual Genome test split | CIDEr-D: 136.6 -> 137.4 | Helped |
| Increase the SAM image encoder from ViT-large to ViT-huge (ablation) | SCA · Visual Genome test split | CIDEr-D: 129.6 -> 130.9 | No clear effect |
| Apply large-scale jittering data augmentation (ablation) | SCA · Visual Genome test split | CIDEr-D: 137.6 -> 140.2 | Helped |
| Use stronger LSJ (0.1, 2.0) instead of LSJ (1.0, 2.0) (ablation) | SCA · Visual Genome test split | CIDEr-D: 140.2 -> 140.8 | Helped |
Surprising findings
- Different SAM image encoder sizes (ViT-base, large, huge) yield similar captioning performance.
- Fixing a large GPT2-large text decoder without finetuning performs better than finetuning it.
Limitations
- Wrong attribute prediction such as incorrect colors or textures.
- Confuses visually similar concepts (e.g., lemon vs. orange).
- No supervision aligns mask predictions with captions, so foreground and background may be predicted separately.
Experimental setup
Datasets: Visual Genome Objects365 COCO-Panoptic
Benchmarks: Visual Genome
Metrics: CIDEr-D METEOR SPICE BLEU ROUGE phrase coverage
Baselines: SAM+BLIP-base SAM+BLIP-large SAM+GIT-large SAM+BLIP2-OPT-6.7B GRiT
Builds on: Segment Anything Model GIT GRiT prompt tuning
Closest papers by meaning
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.89)
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos (NeurIPS 2025 · similarity 0.88)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.88)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.88)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.87)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.