Research map / Image generation / Image restoration
Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Text-IF, a text-guided infrared-visible image fusion network that uses frozen CLIP text features and a semantic interaction guidance module to handle degradations and give user-controllable fusion outputs.
Problem
Existing image fusion methods cannot adapt to degraded low-quality source images (low light, overexposure, low contrast, noise) and produce fixed, non-interactive fusion results that cannot follow user requirements.
Approach
Injects frozen CLIP text embeddings into a Transformer-based fusion decoder via an MLP-generated scale and bias (feature modulation) in a Semantic Interaction Guidance Module, so one model performs degradation-aware, text-controlled fusion; text also modulates the loss weights per task.
Key results
- Without text guidance, Text-IF achieves best SCD and VIF on MSRS and LLVIP (e.g., VIF 1.046 on MSRS vs 0.726 for MetaFusion).
- With text-guided restoration-plus-fusion comparisons, Text-IF achieves overall best metrics across MSRS, LLVIP, MFNet, DN-MSRS, and RoadScene using one set of parameters.
- On LLVIP object detection with YOLOv8, Text-IF reaches mAP@0.50 of 0.941 versus 0.926 for the best competitor MURF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text-guided Text-IF instead of the strongest listed baseline | Infrared-visible fusion · MSRS, LLVIP, and RoadScene | VIF: 0.726 -> 1.046; 0.833 -> 1.011; 0.554 -> 0.739 | Helped |
| Use Text-IF for fusion under degradation with text guidance | Infrared-visible fusion · MSRS, LLVIP, MFNet, DN-MSRS, and RoadScene | selected quality metrics | Mixed |
| Use Text-IF fusion results for object detection | YOLOv8 on LLVIP · LLVIP | mAP@0.50: 0.926 -> 0.941 | Helped |
| Add intensity loss (ablation) | Text-IF · LLVIP | VIF: 0.794 -> 1.011 | Mixed |
| Add structural similarity loss (ablation) | Text-IF · LLVIP | VIF | Helped |
| Add maximum gradient loss (ablation) | Text-IF · LLVIP | VIF: 0.831 -> 1.011 | Mixed |
| Add color consistency loss (ablation) | Text-IF · LLVIP | VIF: 0.987 -> 1.011 | Mixed |
Limitations
- Training data requires manually restored high-quality source images and hand-written text descriptions, which is labor-intensive.
- Hyper-parameters for loss weighting are described as a discrete complex map and deferred to supplementary material, making the text-to-loss mapping hard to inspect.
- Degradation-handling comparisons rely on separate restoration models that were combined with fusion baselines, not a unified benchmark.
Experimental setup
Datasets: MSRS MFNet RoadScene LLVIP
Benchmarks: MSRS LLVIP RoadScene MFNet DN-MSRS
Metrics: SCD SD EN VIF QAB/F CLIP-IQA NIQE MUSIQ BRISQUE SF
Baselines: UMF-CMGR TarDAL ReCoNet MURF U2Fusion MetaFusion DDFM URetinex AirNet GDID LMPEC
Builds on: CLIP Restormer PromptIR SSIM loss
Closest papers by meaning
- Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model (NeurIPS 2024 · similarity 0.91)
- Image Fusion via Vision-Language Model (ICML 2024 · similarity 0.87)
- Robust Fusion Controller: Degradation-Aware Image Fusion with Fine-Grained Language Instructions (AAAI 2026 · similarity 0.85)
- ControlFusion: A Controllable Image Fusion Network with Language-Vision Degradation Prompts (NeurIPS 2025 · similarity 0.84)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.84)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.