Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoSAM, an information-theoretic distillation framework for parameter-efficient fine-tuning of SAM that compresses domain-invariant intra-SAM relations and maximizes teacher-student relational mutual information.
Problem
Existing PEFT methods for SAM adapt specialized domains but disrupt the domain-invariant relations between SAM's image encoder and mask decoder learned during pre-training.
Approach
Extract a compressed relation from the frozen teacher SAM via an attention-based relation module trained with a matrix-based Renyi mutual information bottleneck, then distill it into the fine-tuned student by maximizing matrix-based Renyi mutual information between teacher and student relations (alpha=2, Frobenius-norm surrogate).
Key results
- On SAM ViT-B with Adapter, InfoSAM reaches 75.6 IoU on Leaf vs 74.4 for Adapter baseline and 72.2 for LoRA+Ours style variants; Adapter+Ours reaches 61.4 IoU on Road vs 60.5 for Adapter.
- InfoSAM outperforms distillation baselines (e.g., 94.4 S-alpha on Kvasir vs 92.4-93.7 for PKT/VID/SemCKD), while most distillation methods such as TinySAM (28.7 to 23.9 IoU on Road) degrade below the student.
- On SAM2 Hiera-B+, Adapter+Ours obtains 94.5 S-alpha on Kvasir, 77.3 IoU on Leaf and 61.3 IoU on Road, improving over the Adapter baseline (94.4, 76.8, 60.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add InfoSAM relation compression and distillation to Adapter-based SAM | SAM ViT-B with Adapter · Leaf | IoU: 74.4 -> 75.6 | Helped |
| Add InfoSAM relation compression and distillation to Adapter-based SAM | SAM ViT-B with Adapter · Kvasir | Sα: 93.4 -> 94.4 | Helped |
| Add InfoSAM relation compression and distillation to Adapter-based SAM | SAM ViT-B with Adapter · Road | IoU: 60.5 -> 61.4 | Helped |
| Add relation distillation loss L_d without relation compression loss L_r (ablation) | SAM ViT-B with Adapter · Leaf | IoU: 74.4 -> 75.2 (+0.8) | Helped |
| Add relation compression loss L_r alongside relation distillation loss L_d (ablation) | SAM ViT-B with Adapter and relation distillation · Leaf | IoU: 75.2 -> 75.6 | Helped |
| Add relation module (ablation) | TinySAM · Leaf | IoU: 48.6 ± 1.14 -> 50.3 ± 0.76 | Helped |
| Add relation module (ablation) | TinySAM · Road | IoU: 28.7 ± 1.69 -> 33.9 ± 0.32 | Helped |
| Add relation module (ablation) | MobileSAM · Leaf | IoU: 71.9 ± 0.30 -> 73.8 ± 0.22 | Helped |
| Add relation module (ablation) | MobileSAM · Road | IoU: 59.2 ± 0.09 -> 61.3 ± 0.35 | Helped |
| Use a frozen relation module pretrained on Leaf (ablation) | InfoSAM on Kvasir · Kvasir | Sα: 94.4 ± 0.12 -> 93.7 ± 0.24 | Hurt |
| Use a frozen relation module pretrained on Kvasir (ablation) | InfoSAM on Leaf · Leaf | IoU: 75.6 ± 0.30 -> 75.4 ± 0.45 | Hurt |
Surprising findings
- Most distillation methods (e.g., PKD, IBD, TinySAM, ReviewKD) underperform the non-distilled student during PEFT, with TinySAM collapsing to 7.2% IoU-level teacher performance on Road.
- The relation module trained on one domain (Leaf) transfers to another (Kvasir) with only small loss in performance, suggesting domain-invariant relations.
Limitations
- Gains over the strongest PEFT baselines are often small (e.g., within about 1 point) and with overlapping standard errors.
- Requires a frozen teacher SAM forward pass during fine-tuning, adding training cost.
- Evaluated only on eight segmentation datasets from four domains with ViT-B and Hiera-B+ backbones.
Experimental setup
Datasets: COD10K CAMO CHAMELEON Kvasir CVC-ClinicDB ISIC 2017 Leaf Disease Segmentation Massachusetts Roads Dataset
Benchmarks: CAMO CHAMELEON COD10K Kvasir CVC-612 ISIC 2017 Leaf Disease Segmentation Massachusetts Roads
Metrics: S-alpha E-phi F-beta weighted Jaccard Dice IoU Boundary F1
Baselines: BitFit AdaptFormer LoRA Adapter HQ-SAM SU-SAM ConvLoRA-SAM Logit distillation PKD PKT IBD VID
Builds on: Segment Anything Model SAM2 Adapter LoRA matrix-based Renyi entropy
Closest papers by meaning
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.89)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.89)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.87)
- Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model (CVPR 2024 · similarity 0.87)
- UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model (ICML 2025 · similarity 0.87)
- BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.