Research map / Detection and segmentation / Open-set and few-shot detection
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMI, a masked image pretraining method that reconstructs SAM ViT-H encoder features, and uses the resulting lightweight ViT encoders to build EfficientSAM models that are finetuned on SA-1B.
Problem
SAM's huge ViT-H image encoder (632M parameters) makes it too computationally expensive for real-time and wider practical deployment.
Approach
Use SAM's ViT-H image encoder features as reconstruction targets for an MAE-style masked image pretraining (with a cross-attention decoder and linear projection head) on ImageNet, then plug the pretrained lightweight ViT encoders into SAM's prompt decoder and finetune on SA-1B.
Key results
- SAMI-S (ViT-Small) reaches 82.7% top-1 on ImageNet-1K, above MAE-S (81.5%) and iBOT-S (82.3%).
- EfficientSAM-S achieves 44.4 AP on COCO zero-shot instance segmentation vs 46.5 for SAM and 37.9 for FastSAM, with about 20x fewer parameters and runtime than SAM.
- On zero-shot instance segmentation, EfficientSAM-Ti beats FastSAM by 4.1 AP on COCO and 5.3 AP on LVIS with 9.8M parameters vs 68M for FastSAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain ViT-Tiny with SAMI instead of MAE | ViT-Tiny · ImageNet-1K | top-1 accuracy: 75.2 -> 76.8 | Helped |
| pretrain ViT-Small with SAMI instead of iBOT | ViT-Small · ImageNet-1K | top-1 accuracy: 82.3 -> 82.7 | Helped |
| use SAMI pretraining instead of MAE pretraining | ViT backbones with Mask2former · ADE20K | mIOU: 39.0 -> 42.7; 44.1 -> 48.8; 49.3 -> 51.8 | Helped |
| use EfficientSAM-S instead of SAM | SAM · COCO box prompt | mIoU: 78.4 -> 76.9 | Hurt |
| use EfficientSAM-S instead of FastSAM | FastSAM · COCO zero-shot instance segmentation | AP: 37.9 -> 44.4 | Helped |
| replace MSE reconstruction loss with cosine similarity loss (ablation) | SAMI-Ti · ImageNet-1K | top-1 accuracy: 76.8 -> 76.1 | Hurt |
| replace MSE reconstruction loss with cosine similarity loss (ablation) | SAMI-S · ImageNet-1K | top-1 accuracy: 82.7 -> 82.3 | Hurt |
| replace the MAE decoder with the SAMI cross-attention decoder (ablation) | SAMI-B · ImageNet-1K | top-1 accuracy: 84.4% to 84.8% | Helped |
| use a high mask ratio of 75% (ablation) | SAMI · ImageNet-1K | top-1 accuracy | Helped |
| use CLIP features instead of SAM features as the reconstruction target (ablation) | SAMI with ViT-Tiny · ImageNet-1K | top-1 accuracy: +0.8% | Helped |
| increase EfficientSAM finetuning from 0.1 epoch to 1 epoch (ablation) | EfficientSAM-Ti and EfficientSAM-S · COCO | mIoU: larger than 2.5 mIoU | Helped |
Surprising findings
- MSE reconstruction loss outperformed cosine similarity loss for reconstructing SAM features, which the authors interpret as favoring direct feature regression over angular alignment.
Limitations
- Gap to SAM remains (about 2 AP on COCO zero-shot instance segmentation for EfficientSAM-S).
- Pretraining evaluated at 224x224 resolution on ImageNet only; SAM finetuning requires SA-1B and 64 A100 GPUs.
Experimental setup
Datasets: ImageNet-1K SA-1B COCO LVIS ADE20K
Benchmarks: ImageNet-1K COCO LVIS ADE20K
Metrics: top-1 accuracy AP APbox APmask mIoU throughput parameters
Baselines: SAM MobileSAM FastSAM MAE DeiT SSTA DMAE BEiT CAE DINO iBOT BootMAE
Builds on: Masked Autoencoders (MAE) Segment Anything Model (SAM) ViTDet Mask2Former
Closest papers by meaning
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.90)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.88)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.87)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.