Research map / Detection and segmentation / CNN features
A Simple Image Segmentation Framework via In-Context Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SINE, a Transformer encoder-decoder for in-context image segmentation that uses a frozen DINOv2 encoder, an In-Context Interaction module and a Matching Transformer decoder to output multiple task-specific masks (identical object, instance, semantic), resolving task ambiguity.
Problem
In-context segmentation models such as SegGPT struggle with task ambiguity: an in-context example may not specify whether the target should be segmented as an identical object, an instance, or a semantic category.
Approach
Predict multiple granularity-specific output masks from one in-context example, using a dual-path Matching Transformer (M-Former) with fixed matching for ID queries and Hungarian matching for instance/semantic outputs, on top of a frozen DINOv2 encoder with only ~19M trainable parameters.
Key results
- SINE reaches 64.5 one-shot mIoU on COCO-20i and 31.2 one-shot on LVIS-92i, versus SegGPT's 56.1 and 18.6.
- On DAVIS 2017 video object segmentation without video data, SINE gets 77.0 J&F versus SegGPT's 75.6.
- SINE trains only 19M parameters with a frozen DINOv2-L encoder (304M), compared with SegGPT's roughly 300M trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SINE rather than SegGPT | few-shot semantic segmentation · COCO-20^i one-shot | mIoU: 56.1 -> 64.5 | Helped |
| Use SINE rather than SegGPT | few-shot semantic segmentation · COCO-20^i few-shot | mIoU: 67.9 -> 66.1 | Hurt |
| Use SINE rather than SegGPT | few-shot semantic segmentation · PASCAL-5^i | mIoU: 83.2 -> 85.4; 89.8 -> 86.2 | Mixed |
| Use SINE rather than Matcher | few-shot semantic segmentation · LVIS-92^i | mIoU: 33.0 -> 31.2; 40.0 -> 35.5 | Hurt |
| Add ADE20K training data to COCO (ablation) | SINE · LVIS-92^i | mean mIoU: 24.4 -> 25.5 | Helped |
| Add Objects365 training data (ablation) | SINE · LVIS-92^i | mean mIoU: 25.5 -> 28.3 | Helped |
| Add In-Context Fusion (ablation) | SINE with Mask2Former decoder · LVIS-92^i | mean mIoU: 22.2 -> 23.5 | Helped |
| Replace Mask2Former with M-Former (ablation) | SINE with In-Context Fusion · LVIS-92^i | mean mIoU: 23.5 -> 24.4 | Helped |
| Increase VOS frame count from 1 to 6 (ablation) | SINE · DAVIS 2017 | J&F: 70.9 -> 77.0 | Helped |
| Increase VOS frame count from 8 to 10 (ablation) | SINE · DAVIS 2017 | J&F: 77.0 -> 76.0 | Hurt |
| Replace DINOv2-L with a smaller DINOv2 backbone (ablation) | SINE · COCO-20^i | mIoU: 56.8; 61.7; 64.5 | Hurt |
| Replace DINOv2-L with CLIP-L (ablation) | SINE · COCO-20^i | mIoU: 64.5 -> 34.8 | Hurt |
Surprising findings
- SINE trained without Objects365 still beats SegGPT on COCO-20i, PASCAL-5i and LVIS-92i, so the gains are not solely from extra data.
- CLIP-L backbone performs far worse than DINOv2-L (34.8 vs 64.5 on COCO-20i one-shot), suggesting image-to-image patch matching favors DINOv2.
Limitations
- Performance lags SegGPT on complex inter-frame video relations because of the lightweight in-context fusion module.
- Only ID, instance and semantic ambiguities are addressed; more complex ambiguities (parts, color, position) need multimodal examples.
- No error bars are reported due to compute cost.
Experimental setup
Datasets: ADE20K COCO Objects365 COCO-NOVEL COCO2VOC LVIS-92i PASCAL-5i COCO-20i DAVIS 2017 DAVIS 2016 YouTube-VOS 2018 PASCAL-Part
Benchmarks: COCO-20i PASCAL-5i LVIS-92i COCO-NOVEL COCO2VOC DAVIS 2017 DAVIS 2016 YouTube-VOS 2018 PASCAL-Part ADE20K COCO panoptic
Metrics: mIoU AP J&F J F G
Baselines: SegGPT Painter Matcher PerSAM-F HSNet VAT FPTrans SEEM DINOv Mask2Former
Builds on: SegGPT Mask2Former DETR Segment Anything
Closest papers by meaning
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.88)
- Visual In-Context Prompting (CVPR 2024 · similarity 0.87)
- UNICL-SAM: Uncertainty-Driven In-Context Segmentation with Part Prototype Discovery (CVPR 2025 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.86)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.