Research map / Vision-language and multimodal / Cross-modal learning
A Robust Mutual-Reinforcing Framework for 3D Multi-Modal Medical Image Fusion Based on Visual-Semantic Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a 3D multi-modal medical image fusion framework where visual fusion and lesion segmentation share a fusion module and are optimized jointly, with a degradation-robust Swin Transformer autoencoder trained with two-level refinement constraints.
Problem
Multi-modal 3D medical image fusion treats visual fusion and lesion segmentation separately, with performance bottlenecks and poor robustness to common image degradations (noise, blur, structure loss).
Approach
Couple visual fusion loss and lesion segmentation loss through a shared fusion module on frozen features from a degradation-robust autoencoder, so visual and semantic features are pulled into a unified domain (mutual reinforcement); the autoencoder uses image-level and feature-level refinement losses on clean and degraded inputs.
Key results
- Lesion segmentation average Dice on clean MRBrainS reaches 0.7848 versus 0.7646 for Swin UNETR, with WML Dice 0.6994.
- On degraded images the proposed method drops only 0.0275 in average Dice from clean, while other methods drop by generally above 0.04.
- In a psychophysical study with 35 researchers, 98.33% voted the proposed fusion result best or second best on degraded data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the mutual-reinforcing framework instead of separate visual fusion methods | Visual fusion · MRBrainS MRI, clean images | AG: 4.8879 -> 5.8144 | Helped |
| Use the mutual-reinforcing framework instead of separate visual fusion methods | Visual fusion · MRBrainS MRI, clean images | SCD: 1.2182 -> 1.3529 | Helped |
| Use the mutual-reinforcing framework instead of Swin Unetr alone | Lesion segmentation · MRBrainS MRI, clean images | Dice similarity coefficient (average): 0.7646 -> 0.7848 | Helped |
| Use the mutual-reinforcing framework instead of 3D-UX-Net | Lesion segmentation · MRBrainS MRI, degraded images | Dice similarity coefficient (average): 0.7273 -> 0.7573 | Helped |
| Add lesion segmentation loss LLS (ablation) | Visual fusion · MRBrainS MRI, clean images | EN: 4.0446 -> 4.3779 | Helped |
| Add lesion segmentation loss LLS (ablation) | Visual fusion · MRBrainS MRI, clean images | FMI: 0.9019 -> 0.8994 | Hurt |
| Add lesion segmentation loss LLS (ablation) | Visual fusion · MRBrainS MRI, clean images | AG: 4.1453 -> 5.8144 | Helped |
| Add lesion segmentation loss LLS (ablation) | Visual fusion · MRBrainS MRI, clean images | SCD: 0.4616 -> 1.3529 | Helped |
| Add visual fusion loss LVF (ablation) | Lesion segmentation · MRBrainS MRI, clean images | Dice similarity coefficient (average): 0.7801 -> 0.7848 | Helped |
Surprising findings
- Removing the visual fusion loss lowers segmentation performance, indicating visual fusion supervision helps segmentation.
- Proposed method has lower EN and AG on degraded images than baselines, which the authors attribute to these metrics correlating with high-frequency noise.
Limitations
- Model has 63.09M parameters and training is costly in 3D.
- Not compatible with segmentation methods like HyperDenseNet that rely on their own multi-modal interaction structure.
- Visual fusion quality cannot be fully evaluated without ground truth, relying on non-reference metrics and a small human study.
Experimental setup
Datasets: MRBrainS MRI Prostate158
Benchmarks: MRBrainS MRI Prostate158
Metrics: EN FMI AG SCD Dice similarity coefficient
Baselines: 3D-DTCWT SF-Net IFCNN SwinFusion U2Fusion UNet VNet HyperDenseNet 3D-UX-Net Swin UNETR
Builds on: Swin Transformer Swin UNETR denoising autoencoder Dice loss
Closest papers by meaning
- S2S2: Semantic Stacking for Robust Semantic Segmentation in Medical Imaging (AAAI 2025 · similarity 0.84)
- MRFS: Mutually Reinforcing Image Fusion and Segmentation (CVPR 2024 · similarity 0.84)
- Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities (AAAI 2025 · similarity 0.84)
- FuseUNet: A Multi-Scale Feature Fusion Method for U-like Networks (ICML 2025 · similarity 0.84)
- ConDSeg: A General Medical Image Segmentation Framework via Contrast-Driven Feature Enhancement (AAAI 2025 · similarity 0.84)
- DyCON: Dynamic Uncertainty-aware Consistency and Contrastive Learning for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.