Research map / Vision-language and multimodal / Cross-modal learning
Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAGE, which uses SAM-derived semantic patches in a Semantic Persistent Attention main network and distills it into a lightweight sub-network via bi-level triplet-loss distillation for infrared-visible image fusion.
Problem
Infrared and visible image fusion must balance visual quality with downstream segmentation/detection performance, but task-coupled fusion methods have conflicting objectives and full SAM is too expensive at inference.
Approach
A Semantic Persistent Attention module with a persistent source repository injects SAM semantic priors into fusion, and a DARTS-style bi-level distillation with feature, context, and contrastive semantic (triplet) losses transfers this knowledge into a small sub-network that runs without SAM at inference.
Key results
- On FMB, SAGE reaches 51.1 mIoU with the open-vocabulary segmentation setting and about 3.0 mIoU over the second-best method in the retrained SegFormer setting.
- On MFNet, SAGE achieves 54.0 mIoU versus 52.7 for the next best (FILM).
- The distilled sub-network has 0.136M parameters and 10.47 ms runtime on M3FD, with lower FLOPs than most compared fusion methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SAM semantic patches with randomly cropped source patches (ablation) | SAGE · FMB | mIoU: 61.2 -> 56.5 | Helped |
| generate semantic patches with segmentation labels (ablation) | SAGE · FMB | mIoU: 56.4 -> 61.2 | Helped |
| replace open-vocabulary model SO with a conventional segmentation network (ablation) | SAGE · FMB | mIoU: 55.5 -> 61.2 | Helped |
| add latent representation Z to SPA (ablation) | SAGE · FMB | mIoU: 57.2 -> 61.2 | Helped |
| add key-value pairs to the persistent repository (ablation) | SAGE · FMB | mIoU: 56.1 -> 61.2 | Helped |
| add the persistent repository to SPA (ablation) | SAGE · FMB | mIoU: 57.2 -> 61.2 | Helped |
| add the feature alignment loss Lfea (ablation) | SAGE · FMB | mIoU: 51.6 -> 61.2 | Helped |
| add the context loss Lcont (ablation) | SAGE · FMB | mIoU: 56.2 -> 61.2 | Helped |
| add the contrastive semantic loss Lcs (ablation) | SAGE · FMB | mIoU: 54.3 -> 61.2 | Helped |
| use bi-level optimization distillation instead of offline distillation (ablation) | SAGE · FMB | mIoU: 50.4 -> 61.2 | Helped |
Surprising findings
- Removing SAM from the main network (replacing semantic patches with random source patches) gave the best SCD and MS-SSIM fusion metrics, though with lower segmentation mIoU on MFNet.
Limitations
- Relies on SAM-derived semantic patches and an open-vocabulary segmentation model whose quality limits the priors.
- Mixed results on background categories in MFNet segmentation according to the authors.
Experimental setup
Datasets: TNO RoadScene MFNet FMB M3FD
Benchmarks: TNO RoadScene M3FD FMB MFNet
Metrics: EN SD SCD MS-SSIM BRISQUE NIQE MUSIQ PaQ-2-PiQ mIoU runtime (ms)
Baselines: DDFM U2Fusion TarDAL SegMiF FILM SHIP MRFS EMMA TIMFusion
Builds on: Segment Anything Model DARTS SegFormer TarDAL
Closest papers by meaning
- MRFS: Mutually Reinforcing Image Fusion and Segmentation (CVPR 2024 · similarity 0.88)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.87)
- Image Fusion via Vision-Language Model (ICML 2024 · similarity 0.87)
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.86)
- Revisiting Generative Infrared and Visible Image Fusion Based on Human Cognitive Laws (NeurIPS 2025 · similarity 0.86)
- Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.