Research map / Image generation / Image restoration
Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text-DiFuse embeds information fusion inside a conditional diffusion process (with a fusion control module) and uses text-guided OWL-ViT and SAM localization to re-modulate fusion toward user-specified objects, evaluated on infrared-visible and medical image fusion.
Problem
Multi-modal image fusion under compound degradations (noise, color casts, improper exposure) tends to ignore restoration, and fusion rules treat foreground objects of interest the same as background.
Approach
Couples degradation removal and multi-modal feature fusion within a single diffusion sampling loop via a fusion control module that generates fusion weights per step, and adds a text-driven (OWL-ViT + SAM) re-modulation that boosts salience of user-specified objects.
Key results
- On MSRS, Text-DiFuse reports EN 7.08, AG 3.31, SD 47.44, SCD 1.44, VIF 0.76, the best among 9 fusion baselines.
- Text-controlled re-modulation raises SegNext mIoU on MFNet from 57.53 (basic) to 58.63 (modulatable), the best among compared RGB-T segmentation methods.
- On detection with YOLO-v5 on MSRS, mAP@0.5 rises from 87.3 (basic) to 89.7 (modulatable).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Text-DiFuse instead of the strongest listed baseline | Multi-modal image fusion · MSRS | EN: 6.91 -> 7.08 | Helped |
| Use Text-DiFuse instead of the strongest listed baseline | Multi-modal image fusion · Harvard Medicine | EN: 7.24 -> 6.44 | Hurt |
| Add text-controlled fusion re-modulation | Text-DiFuse · MFNet semantic segmentation | mIoU: 57.53 -> 58.63 | Helped |
| Add text-controlled fusion re-modulation | Text-DiFuse · MSRS object detection | mAP@0.5: 87.3 -> 89.7 | Helped |
| Add text-controlled fusion re-modulation | Text-DiFuse · MSRS object detection | mAP@[0.5:0.95]: 56.3 -> 60.9 | Helped |
| Use learned FCM rather than a maximum rule (ablation) | Text-DiFuse · MSRS ablation | EN: 5.71 -> 7.08 | Helped |
| Use learned FCM rather than an addition rule (ablation) | Text-DiFuse · MSRS ablation | EN: 6.08 -> 7.08 | Helped |
| Use learned FCM rather than a mean rule (ablation) | Text-DiFuse · MSRS ablation | EN: 5.60 -> 7.08 | Helped |
| Use learned FCM rather than a variance-based rule (ablation) | Text-DiFuse · MSRS ablation | EN: 5.91 -> 7.08 | Helped |
| Add gradient loss (ablation) | Text-DiFuse · MSRS ablation | EN: 6.20 -> 7.08 | Helped |
| Add intensity loss (ablation) | Text-DiFuse · MSRS ablation | EN: 6.67 -> 7.08 | Helped |
| Use diffusion rather than the AE route (ablation) | Text-DiFuse · MSRS ablation | EN: 6.37 -> 7.08 | Helped |
Surprising findings
- Concatenating a separate enhancement or denoising preprocessing step before fusion baselines does not close the gap to the coupled approach, per the authors' comparisons.
- On RoadScene, Text-DiFuse does not have the top AG (2.96 vs TarDAL 11.84) and on Harvard dataset EN is below MRFS (6.44 vs 7.24).
Limitations
- Relies on diffusion T-step sampling, which is slow, though iDDPM is used to mitigate cost.
- Re-modulation depends on the accuracy of the zero-shot localization pipeline (OWL-ViT and SAM) for identifying the text-specified objects.
Experimental setup
Datasets: MSRS Harvard medicine dataset LLVIP RoadScene MFNet
Benchmarks: MSRS Harvard medicine dataset LLVIP RoadScene MFNet
Metrics: EN AG SD SCD VIF mIoU mAP@0.5 mAP@[0.5:0.95]
Baselines: RFN-Nest GANMcC SDNet U2Fusion TarDAL DeFusion LRRNet DDFM MRFS CLIP-LIT SDAP AWB
Builds on: iDDPM DDIM CBAM DDFM
Closest papers by meaning
- Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion (CVPR 2024 · similarity 0.91)
- Image Fusion via Vision-Language Model (ICML 2024 · similarity 0.87)
- Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning (NeurIPS 2025 · similarity 0.86)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.86)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.85)
- Revisiting Generative Infrared and Visible Image Fusion Based on Human Cognitive Laws (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.