Research map / Detection and segmentation / Open-set and few-shot detection
Region-Based Representations Revisited
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that mask-pooled DINOv2 features over class-agnostic SAM+SLIC regions give compact region representations that compete on semantic segmentation, one-shot object retrieval, multi-view segmentation, and video activity classification.
Problem
Whether region-based image representations, largely abandoned for pixel and patch features, remain effective for recognition and multi-image applications when built from modern segmenters and self-supervised features.
Approach
Generate a few dozen regions with SAM augmented by SLIC superpixels for coverage, average-pool upsampled frozen DINOv2 features within each mask, and use the resulting compact region vectors with simple linear, MLP, or transformer decoders across several tasks.
Key results
- SAM+SLIC regions with a linear decoder on DINOv2 ViT-L/14 reach 86.9 mIoU on PASCAL VOC and 52.9 on ADE20K, versus 81.2 and 47.7 for the patch-based DINOv2 linear baseline.
- On COCO one-shot object retrieval, region representations reach 0.45 mAP versus 0.27 for CLIP-Crop and 0.13 for DINOv2.
- On Kinetics-400 activity classification, region features with a transformer reach 79.5% versus 76.3% for a DINOv2 linear patch baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SLIC regions to SAM regions (ablation) | Region representations with DINOv2 features · Pascal VOC 2012 val | mIoU: 83.6 -> 87.2 | Helped |
| Add SLIC regions to SAM regions (ablation) | Region representations with DINOv2 features · ADE20K val | mIoU: 50.2 -> 52.8 | Helped |
| Upsample features and average-pool within masks (ablation) | DINOv2 region representations · Pascal VOC 2012 val | mIoU: 76.3 -> 83.6 | Helped |
| Upsample features and average-pool within masks (ablation) | DINOv2 region representations · ADE20K val | mIoU: 44.5 -> 50.2 | Helped |
| Replace patch features with SAM region features (ablation) | DINOv2 ViT-L/14 semantic segmentation · Pascal VOC 2012 val | mIoU: 79.9 -> 83.6 | Helped |
| Replace patch features with SAM region features (ablation) | DINOv2 ViT-L/14 semantic segmentation · ADE20K val | mIoU: 43.4 -> 50.2 | Helped |
| Use DINOv2 ViT-L/14 instead of DINOv1, CLIP, MaskCLIP, or ImageNet features (ablation) | Region-based semantic segmentation · Pascal VOC 2012 val | mIoU: DINOv2 ViT-L/14: 83.6; DINOv1 ViT-B/16: 66.2; CLIP ViT-B/32: 65.7; MaskCLIP ViT-L: 76.7; ImageNet ViT-L: 54.6 | Helped |
| Increase the DINOv2 backbone from ViT-S to ViT-L (ablation) | Region-based semantic segmentation · Pascal VOC 2012 val | mIoU: 75.1 -> 83.6 | Helped |
| Increase the DINOv2 backbone from ViT-L to ViT-G (ablation) | Region-based semantic segmentation · ADE20K val | mIoU: 50.2 -> 49.7 | Hurt |
| Use an MLP decoder instead of a linear decoder (ablation) | SAM+SLIC semantic segmentation · Pascal VOC 2012 val | mIoU: 86.9 -> 88.4 | Helped |
| Use a transformer decoder instead of a linear decoder (ablation) | SAM+SLIC semantic segmentation · ADE20K val | mIoU: 52.9 -> 53.5 | Helped |
| Add 2D and 3D positional embeddings (ablation) | Per-region linear probe on ScanNet · ScanNet val | mIoU: 66.0 -> 66.1 | No clear effect |
Surprising findings
- Unsupervised SLIC superpixels outperformed MobileSAMv1 as region generator for semantic segmentation because MobileSAMv1 left large image areas unsegmented.
- Region features outperformed patch features across all tested feature backbones, with the largest gain for MaskCLIP.
Limitations
- SAM generation is slow (about 4.6 s per image), the main bottleneck.
- Semantic segmentation remains below state-of-the-art decoders (52.9 vs 62.5 mIoU on ADE20K), and multi-view results trail SOTA on ScanNet.
- Ground truth labels in ScanNet are noisy, so numerical evaluation may understate performance.
Experimental setup
Datasets: PASCAL VOC 2012 ADE20K ScanNet COCO Kinetics-400
Benchmarks: PASCAL VOC 2012 ADE20K ScanNet 2D semantic label benchmark COCO object retrieval Kinetics-400
Metrics: mIoU mAP precision@50 time per image regions per image
Baselines: DINOv2 patch-based linear decoder MaskCLIP CLIP-Crop Virtual Multi-view Fusion BPNet ATM InternImage Mask2Former
Builds on: SAM DINOv2 SLIC MaskCLIP
Closest papers by meaning
- REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders (NeurIPS 2025 · similarity 0.87)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.86)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.85)
- Recognize Any Regions (NeurIPS 2024 · similarity 0.85)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.85)
- Learning Spatial-Semantic Features for Robust Video Object Segmentation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.