Research map / Architectures and efficiency / Pruning and efficient tuning
TinySAM: Pushing the Envelope for Efficient Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a tiny SAM (TinyViT-5M student from SAM-H teacher) using full-stage distillation with hard prompt sampling and hard mask weighting, adds W8A8 post-training quantization for prompt-based segmentation, and a hierarchical everything-mode inference strategy.
Problem
Segment Anything Model (SAM) is too computationally heavy for deployment on edge devices, and prior efficient variants (MobileSAM, FastSAM) lose zero-shot segmentation accuracy.
Approach
Multi-stage distillation from SAM-H to a TinyViT student (embedding, decoder-token, and output losses) with online hard prompt sampling and hard mask weighting, combined with PTQ adapted to prompt-based segmentation via Hessian-guided scale search and a coarse-to-fine everything-mode grid.
Key results
- TinySAM reaches 42.3 AP on COCO zero-shot instance segmentation with 42.0G MACs and 38.4 ms latency, versus MobileSAM's 41.0 AP at the same cost.
- Q-TinySAM (W8A8) reaches 41.4 AP on COCO with 20.3G MACs and 24.0 ms latency on a T4 GPU.
- Hierarchical everything-mode inference reduces mIoU-evaluation time from 1.7790 s to 0.9303 s for TinySAM with mIoU dropping only from 0.6137 to 0.6061.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add full-stage knowledge distillation loss (ablation) | TinySAM student · COCO val zero-shot instance segmentation | AP (%): 40.7 -> 41.4 | Helped |
| Add hard prompt sampling (ablation) | TinySAM student with full-stage distillation · COCO val zero-shot instance segmentation | AP (%): 41.4 -> 41.9 | Helped |
| Add hard mask weighting (ablation) | TinySAM student with full-stage distillation and hard prompt sampling · COCO val zero-shot instance segmentation | AP (%): 41.9 -> 42.3 | Helped |
| Add post-training quantization (ablation) | TinySAM · COCO val zero-shot instance segmentation | AP (%): 42.3 -> 41.4 | Hurt |
| Add embedding distillation loss to output loss (ablation) | TinySAM student · COCO val zero-shot instance segmentation | AP (%): 41.6 -> 41.7 | Helped |
| Add token distillation loss to embedding and output losses (ablation) | TinySAM student · COCO val zero-shot instance segmentation | AP (%): 41.7 -> 41.9 | Helped |
| Add hard mask weighting to output distillation loss (ablation) | TinySAM student with three distillation losses · COCO val zero-shot instance segmentation | AP (%): 41.9 -> 42.3 | Helped |
| Apply W8A8 quantization (ablation) | TinySAM · COCO val zero-shot instance segmentation | AP (%): 42.3 -> 41.4 | Hurt |
| Apply W6A6 quantization (ablation) | TinySAM · COCO val zero-shot instance segmentation | AP (%): 42.3 -> 39.0 | Hurt |
| Use hierarchical point grid strategy | TinySAM · COCO val2017 first 100 images | Time (s): 1.7790 -> 0.9303 | Helped |
| Use hierarchical point grid strategy | TinySAM · COCO val2017 first 100 images | mIoU: 0.6137 -> 0.6061 | Hurt |
| Increase first/second point density from 8/32 to 10/32 (ablation) | TinySAM hierarchical strategy · COCO val | mIoU: 0.6061 -> 0.6078 | Helped |
Surprising findings
- Hard mask weighting and hard prompt sampling are reported as the largest gains, but the paper notes the full-stage distillation alone only lifts AP from 40.7 to 41.4 over baseline.
- W6A6 quantization drops COCO AP to 39.0 for TinySAM, showing a steep drop below 8 bits.
Limitations
- Quantization causes a 0.9 AP drop (8-bit) and larger drops at 6-bit.
- Distillation uses only 1% of SA-1B with 8 epochs, and results depend on the hierarchical threshold and point-density hyperparameters.
Experimental setup
Datasets: SA-1B
Benchmarks: COCO LVIS v1 BBBC038v1 DOORS TimberSeg
Metrics: AP mIoU MACs latency
Baselines: SAM-H SAM-L SAM-B FastSAM EfficientSAM-Ti SlimSAM-77 MobileSAM
Builds on: Segment Anything Model MobileSAM TinyViT ViTDet
Closest papers by meaning
- EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (CVPR 2024 · similarity 0.90)
- PTQ4SAM: Post-Training Quantization for Segment Anything (CVPR 2024 · similarity 0.90)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.89)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective (ICML 2025 · similarity 0.89)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.