Research map / Detection and segmentation / CNN features
REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains REN, a lightweight point-prompted cross-attention network that turns frozen patch-encoder features (DINO, DINOv2, OpenCLIP) into region tokens without running SAM at inference, achieving ~60x faster token generation and 35x less memory than SAM-based region methods.
Problem
Region-based image representations built from SAM masks plus patch-encoder pooling are computationally expensive because of the segmentation step, and linear mask pooling discards fine-grained and contextual detail while producing overlapping or incomplete coverage.
Approach
A point-promptable cross-attention decoder (four blocks, point prompts as queries, frozen patch features as keys/values) is trained with a SAM-derived contrastive loss plus a feature-similarity loss to produce region tokens directly, then grid or SLIC prompts are aggregated by cosine-similarity connected components; it can also be transferred to other encoders by masking the final-layer CLS attent
Key results
- Generates region tokens 61.7x faster than EfficientViT-SAM and uses 35.2x less GPU memory for a 32x32 grid on a single A40.
- Achieves state-of-the-art on Ego4D VQ2D, outperforming RELOCATE, with VQ2D score 63.7 vs 49.7 for the feature-similarity-only variant in the ablation.
- On Visual Haystacks single-needle, REN with SigLIP 2 improves average performance by 8.7% over SigLIP 2 and outperforms listed proprietary LMMs at larger N.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pool SigLIP 2 features into region tokens using pretrained REN_DINO | SigLIP 2 · Visual Haystacks single-needle, N=1K | accuracy: 54.9 -> 59.2 | Helped |
| Replace SAM-based region token generation with REN | Region token generation · 32×32 grid | generation speed: 61.7× faster than EfficientViT-SAM | Helped |
| Replace SAM-based region token generation with REN | Region token generation · 32×32 grid | GPU memory: 35.2× less GPU memory | Helped |
| Use SLIC prompting instead of grid prompting (ablation) | REN · ADE20K | mAP: 50.4 -> 50.9 | Helped |
| Use SLIC prompting instead of grid prompting (ablation) | REN · COCO | mRP: 0.66 -> 0.66 | No clear effect |
| Add token aggregation to grid prompting (ablation) | REN · ADE20K | mAP: 50.4 -> 50.4 | No clear effect |
| Add token aggregation to SLIC prompting (ablation) | REN · ADE20K | mAP: 50.9 -> 51.0 | No clear effect |
| Add token aggregation to grid prompting (ablation) | REN · COCO | mRP: 0.66 -> 0.65 | No clear effect |
| Add token aggregation to SLIC prompting (ablation) | REN · COCO | mRP: 0.66 -> 0.65 | No clear effect |
| Add feature similarity loss to contrastive loss (ablation) | REN · VOC2012 | score: 77.7 -> 86.5 | Helped |
| Add feature similarity loss to contrastive loss (ablation) | REN · VQ2D validation, 500 videos | score: 46.9 -> 63.7 | Helped |
| Add contrastive loss to feature similarity loss (ablation) | REN · VOC2012 | score: 86.0 -> 86.5 | Helped |
Surprising findings
- Adding direct SAM-based attention supervision degraded VQ2D performance (63.7 to 60.3) and did not improve VOC2012.
- Token aggregation reduces tokens per image by 19.5x (to about 72 tokens) without lowering ADE20K mAP.
Limitations
- Point prompts introduce ambiguity between whole objects and parts, handled only by tuning the aggregation threshold.
- REN region segmentations are less precise than SAM, limiting use for interactive segmentation.
- Transfer to other encoders relies on a pretrained DINO-based REN and task-specific aggregation thresholds.
Experimental setup
Datasets: Ego4D ADE20K VOC2012 COCO
Benchmarks: ADE20K VOC2012 Ego4D VQ2D Visual Haystacks COCO
Metrics: mAP mRP@50 mRP accuracy processing time peak GPU memory parameter count
Baselines: EfficientViT-SAM Batched-SAM SAM 2 RELOCATE SigLIP 2 Gemini 1.5 Pro GPT-4o MIRAGE
Builds on: SAM DINO DINOv2 OpenCLIP InfoNCE
Closest papers by meaning
- Recognize Any Regions (NeurIPS 2024 · similarity 0.88)
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.87)
- Region-Based Representations Revisited (CVPR 2024 · similarity 0.87)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.87)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.86)
- Segment and Caption Anything (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.