Research map / Vision-language and multimodal / Cross-modal learning
MRFS: Mutually Reinforcing Image Fusion and Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MRFS, a coupled CNN/transformer framework that jointly trains infrared-visible image fusion and semantic segmentation so the two tasks mutually reinforce each other, using IGM-Att and PC-Att modules.
Problem
Infrared-visible image fusion and semantic segmentation are usually treated as isolated or sequential tasks, causing feature neutralization, low-light information loss, mismatched feature validity scores, and insufficient feature aggregation.
Approach
Couples fusion and segmentation via two attention modules: IGM-Att (CNN, pooling attention plus a learned gating correction) for visual completion and PC-Att (transformer, closed-loop self and cross-modal attention) for semantic completion; fusion head adds salient information integration (max over gradient and intensity) and weakened information recovery (augmented positive samples) losses.
Key results
- Achieves highest mIoU on all three segmentation datasets: 59.1 on MFNet, 86.9 on PST900, 61.2 on FMB.
- On FMB, adding IGM-Att and PC-Att raises LASNet mIoU from 55.7 to 58.1 and SGFNet from 56.0 to 57.7.
- On LLVIP detection retraining, mAP@[0.5:0.95] reaches 74.2 versus 73.4 to 74.2 for most fusion baselines and 72.8 for infrared alone; mAP@0.5 is 99.3 for MRFS, slightly above DATFuse's 99.3 tie level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add salient information integration instead of using a proportional strategy (ablation) | MRFS · FMB test | mIoU: 60.5 -> 61.2 | Helped |
| add weakened information recovery (ablation) | MRFS · FMB test | mIoU: 60.9 -> 61.2 | Helped |
| use IGM-Att instead of conventional pooling-based attention (ablation) | MRFS · FMB test | mIoU: 60.2 -> 61.2 | Helped |
| use PC-Att instead of cross-attention-based feature integration (ablation) | MRFS · FMB test | mIoU: 59.5 -> 61.2 | Helped |
| add the image fusion head (ablation) | MRFS · FMB test | mIoU: 59.8 -> 61.2 | Helped |
| remove the image fusion head (ablation) | MRFS · image fusion ablation | visual quality | Hurt |
| remove the semantic segmentation head (ablation) | MRFS · image fusion ablation | visibility of ground traffic markings | Hurt |
Surprising findings
- Removing the image fusion head from the joint model reduces segmentation mIoU to 59.8 on FMB (vs 61.2 full), and the paper reports that a more complex fusion branch is not strictly necessary for every metric.
Limitations
- Parameter count (134.97M) and FLOPs are higher than several dedicated methods, which the authors acknowledge.
- Fusion comparisons on PST900 are omitted due to low image quality; EGFNet and MMSMCNet results are taken from original papers rather than retrained on FMB.
Experimental setup
Datasets: MFNet PST900 FMB LLVIP
Benchmarks: MFNet PST900 FMB LLVIP
Metrics: mIoU EN SD precision recall mAP@0.5 mAP@0.75 mAP@[0.5:0.95] FLOPs parameters
Baselines: SeAFusion SegFormer EGFNet LASNet SegMiF MDRNet+ SGFNet MMSMCNet EAEFNet SDNet U2Fusion DetFusion
Builds on: SegFormer SeAFusion SegMiF CBAM-style channel and spatial attention
Closest papers by meaning
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.88)
- Image Fusion via Vision-Language Model (ICML 2024 · similarity 0.86)
- Revisiting Generative Infrared and Visible Image Fusion Based on Human Cognitive Laws (NeurIPS 2025 · similarity 0.86)
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.85)
- One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.85)
- SMR-Net: Semantic-Guided Mutually Reinforcing Network for Cross-Modal Image Fusion and Salient Object Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.