Research map / Detection and segmentation / CNN features
Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Segment This Thing (STT), a point-prompted segmentation model that uses foveated variable-resolution patch tokenization centered on the prompt, trained from scratch with MAE pretraining and SA-1B, to get large efficiency gains over SAM and its distilled variants.
Problem
Prompted image segmentation with SAM-style models is computationally expensive, and prior efficiency work mostly shrinks model size; the paper targets low-latency, low-bandwidth single-point segmentation for streaming AR/robotics use cases.
Approach
Crop a region around the point prompt and tokenize it with a foveated pattern where patch size grows with distance from the prompt, so all patches are resampled to a common size; this cuts token count (172 vs 4096) while keeping model capacity, and the foveation acts as an inductive bias toward the prompted region.
Key results
- STT uses about 172 tokens versus 4096 for SAM, reducing inference latency roughly 42x versus SAM-H (26.2 ms vs 572.7 ms on RTX 3080) and 5.6x versus EfficientSAM.
- STT-L is competitive with EfficientSAM-Ti in mIoU while running nearly 3x faster, and all STT variants significantly outperform MobileSAM at similar latency.
- STT-H shows diminishing returns over STT-L, while STT-L significantly outperforms STT-B on the nine evaluation datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use foveated tokenization instead of uniform patch tokenization | STT compared with SAM · Model input | token count: roughly 24x | Helped |
| Use foveated tokenization | STT compared with SAM-H · Single image, single point prompt | inference latency: roughly 42x | Helped |
| Use foveated tokenization | STT compared with EfficientSAM · Single image, single point prompt | inference latency: 5.6x | Helped |
| Increase model capacity from STT-B to STT-L (ablation) | STT · Segmentation benchmarks | mIoU | Helped |
| Increase model capacity from STT-L to STT-H (ablation) | STT · Segmentation benchmarks | mIoU | No clear effect |
| Use STT instead of MobileSAM | Point-prompted segmentation · Segmentation benchmarks | mIoU | Helped |
| Use STT-L instead of EfficientSAM-Ti | Point-prompted segmentation · Segmentation benchmarks | mIoU | No clear effect |
Surprising findings
- Foveated tokenization lets STT use larger models than other efficient SAM variants while still needing fewer FLOPs and lower latency.
- STT can segment extremely small targets that SAM misses, because its central patches yield full-resolution output masks.
Limitations
- Coarser segment boundaries for large objects due to peripheral downsampling, lowering mIoU versus larger SAM models.
- Each prompt requires rerunning the encoder, so no amortization across prompts on the same image.
- Only evaluated for single-point image segmentation; extension to video left to future work.
Experimental setup
Datasets: SA-1B
Benchmarks: ADE20K Cityscapes EgoHOS NDD20 PPDLS TimberSeg VISOR ZeroWaste WoodScape
Metrics: mIoU latency (ms) GFLOPs
Baselines: SAM EfficientSAM MobileSAM
Builds on: Segment Anything Masked Autoencoders Vision Transformer Focal loss Dice loss
Closest papers by meaning
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
- Foveated Instance Segmentation (CVPR 2025 · similarity 0.85)
- Subobject-level Image Tokenization (ICML 2025 · similarity 0.85)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.85)
- FocSAM: Delving Deeply into Focused Objects in Segmenting Anything (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.