Research map / Architectures and efficiency / Pruning and efficient tuning
GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaRA, a gated rank-1-component LoRA adapter inserted into a frozen SAM that selects a per-input subset of rank components, yielding GaRA-SAM, a robust segmentation model trainable on unpaired real corrupted images.
Problem
SAM's segmentation performance degrades under input corruptions (noise, blur, low light, adverse weather), and prior robustification methods either need paired clean/degraded images, full fine-tuning, or fixed-rank adapters that cannot adapt to per-input corruption.
Approach
Reinterpret LoRA's update as a sum of rank-1 components and use hierarchical Gumbel-Sigmoid gating (rank-space selection, then component selection) to activate an input-dependent subset of components, so the effective rank adapts per image without test-time optimization.
Key results
- On LVIS with ViT-B and point prompts, GaRA-SAM reaches 77.0 degraded IoU vs. 72.7 for RobustSAM, a gain of 4.3 points.
- Training GaRA-SAM on real BDD+LIS corrupted images yields 88.8 IoU on ACDC vs. 67.5 for RobustSAM, the reported gain of up to 21.3 points.
- GaRA-SAM uses 343M learnable parameters versus 403M for RobustSAM and runs at 2.6 FPS vs. 2.8 for RobustSAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed-rank LoRA with input-adaptive GaRA (ablation) | SAM ViT-B · LVIS degraded, point prompts | IoU: 75.6 -> 77.0 | Helped |
| replace RobustSAM with GaRA-SAM | SAM ViT-B · COCO degraded, point prompts | IoU: 72.7 -> 77.2 | Helped |
| train GaRA-SAM on real BDD+LIS images rather than synthetic Robust-Seg data | GaRA-SAM, SAM ViT-L · ACDC, box prompts | IoU: 71.6 -> 88.8 | Helped |
| remove rank-space separation (Gating 1) and use a unified fixed maximum rank of 16 (ablation) | GaRA-SAM · LVIS degraded, point prompts | IoU: 77.0 -> 76.2 | Hurt |
| remove rank-space separation (Gating 1) and use a unified fixed maximum rank of 256 (ablation) | GaRA-SAM · LVIS degraded, point prompts | IoU: 77.0 -> 73.3 | Hurt |
| replace GaRA with MoE LoRA (2, 16, 128, 256) (ablation) | SAM ViT-B · LVIS degraded, point prompts | IoU: 77.0 -> 76.4 | Hurt |
| replace GaRA with MoE LoRA (2, 16, 128, 1024) (ablation) | SAM ViT-B · LVIS degraded, point prompts | IoU: 77.0 -> 75.4 | Hurt |
| vary the Gumbel-Sigmoid temperature (ablation) | GaRA-SAM · LVIS | IoU | No clear effect |
Surprising findings
- The best LoRA rank varies across corruption types and individual images; an oracle per-instance rank selection substantially outperforms any fixed rank.
- Fixed-rank LoRA on SAM already outperforms RobustSAM on LVIS without clean reference images or auxiliary losses.
Limitations
- Real-world results on ACDC are reported with GaRA-SAM trained on real BDD+LIS; the synthetic-trained variant gains less.
- Inference speed is slightly lower than SAM and RobustSAM (2.6 vs 2.9 and 2.8 FPS).
- Evaluation relies on IoU/Dice with point and box prompts only; the paper notes no official evaluation code exists for prior work so baselines were reproduced by the authors.
Experimental setup
Datasets: Robust-Seg LVIS MSRA-10K ThinObjects-5K BDD-100K LIS
Benchmarks: LVIS MSRA-10K COCO STREETS NDD20 BDD-100K LIS ACDC
Metrics: IoU Dice
Baselines: SAM HQ-SAM RobustSAM AirNet+SAM URIE+SAM LoRA MoE LoRA
Builds on: Segment Anything Model LoRA Gumbel-Sigmoid RobustSAM mixture of experts
Closest papers by meaning
- RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.87)
- Robust SAM: On the Adversarial Robustness of Vision Foundation Models (AAAI 2025 · similarity 0.86)
- Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model (ICLR 2024 · similarity 0.86)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.85)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.85)
- Segment Any-Quality Images with Generative Latent Space Enhancement (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.