Research map / Image generation / Image restoration
RobustSAM: Segment Anything Robustly on Degraded Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds lightweight anti-degradation modules to a frozen SAM, trained on the 688K-pair Robust-Seg dataset with synthetic degradations, to segment degraded images while keeping zero-shot promptability.
Problem
SAM's segmentation quality drops on degraded images (low light, blur, noise, weather), and existing remedies (restoration preprocessing, full fine-tuning) either fail to help segmentation or harm SAM's zero-shot generalization.
Approach
Keep SAM frozen and add small Anti-Degradation Mask Feature Generation (instance/batch norm, Fourier amplitude-phase suppression) and Output Token Generation modules plus a fine-tuned Robust Output Token, trained with feature consistency losses that align degraded-image features to those SAM extracts from the clear image.
Key results
- On the BDD-100k and LIS real-world degradation sets with point prompts, RobustSAM reaches IoU 0.3717 versus 0.3056 for SAM.
- On zero-shot NDD20/STREETS/FSS-1000 with point prompts, RobustSAM averages IoU 0.8616 versus 0.8207 for SAM.
- Using RobustSAM as the prior improves single-image dehazing PSNR from 21.677 to 23.159 and deblurring PSNR from 27.491 to 29.351.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RobustSAM modules (AMFG, AOTG, ROT) on frozen SAM with degradation augmentation training | SAM · MSRA10k seen degradations, point prompts | IoU (Average): 0.8207 -> 0.8616 | Helped |
| Fine-tune the entire SAM model on degraded images (ablation) | SAM · BDD-100k+LIS, point prompts | IoU: 0.3056 -> 0.1871 | Hurt |
| Fine-tune only the SAM mask decoder (ablation) | SAM · BDD-100k+LIS, point prompts | IoU: 0.3056 -> 0.2476 | Hurt |
| Fine-tune only the output token (ablation) | SAM · BDD-100k+LIS, point prompts | IoU: 0.3056 -> 0.3194 | Helped |
| Add AMFG module (ablation) | RobustSAM · BDD-100k+LIS, point prompts | IoU: 0.3056 -> 0.3455 | Helped |
| Add Fourier Degradation Suppression to AMFG (ablation) | RobustSAM · BDD-100k+LIS, point prompts | IoU: 0.3455 -> 0.3535 | Helped |
| Add AOTG module (ablation) | RobustSAM · BDD-100k+LIS, point prompts | IoU: 0.3535 -> 0.3651 | Helped |
| Add Robust Output Token (ROT) fine-tuning (ablation) | RobustSAM · BDD-100k+LIS, point prompts | IoU: 0.3651 -> 0.3717 | Helped |
| Replace SAM with RobustSAM as prior for dehazing and deblurring | SAM-based dehazing and deblurring · SOTS dehazing and GoPro deblurring | PSNR: dehazing 21.677 -> 23.159; deblurring 27.491 -> 29.351 | Helped |
| Use AirNet or URIE image restoration before SAM | SAM · MSRA10k seen degradations, point prompts | IoU (Average): AirNet+SAM 0.8162, URIE+SAM 0.8231 vs SAM 0.8207 | No clear effect |
| Use AirNet preprocessing before SAM | SAM · BDD-100k+LIS, point prompts | IoU: 0.3056 -> 0.3245 (improves here); 0.8162 vs 0.8207 on MSRA10k (hurts there) | Mixed |
Surprising findings
- Fully fine-tuning SAM or its decoder sharply reduces zero-shot IoU (e.g., to 0.1871 for full fine-tuning on BDD-100k/LIS), showing that naive adaptation harms generalization.
Limitations
- Relies on synthetic degradations for training, with real-world generalization shown only through evaluation on BDD-100k and LIS.
- Trained with point prompts only in the main setup.
Experimental setup
Datasets: LVIS ThinObject-5k MSRA10K NDD20 STREETS FSS-1000 COCO BDD-100k LIS SOTS GoPro
Benchmarks: MSRA10K LVIS NDD20 STREETS FSS-1000 COCO BDD-100k LIS SOTS GoPro
Metrics: IoU Dice pixel accuracy average precision PSNR SSIM
Baselines: SAM HQ-SAM AirNet+SAM URIE+SAM
Builds on: Segment Anything Model HQ-SAM URIE AirNet
Closest papers by meaning
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.89)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.88)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.88)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.88)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.87)
- DarkSAM: Fooling Segment Anything Model to Segment Nothing (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.