Research map / Vision-language and multimodal / Human and embodied modeling
ControlFusion: A Controllable Image Fusion Network with Language-Vision Degradation Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ControlFusion, a controllable infrared-visible image fusion network that couples restoration and fusion under text and visual degradation prompts, trained on a physics-driven composite degradation dataset (DDL-12).
Problem
Infrared-visible image fusion struggles with real-world composite degradations, lacks degradation-level modeling, and offers limited user control over fusion output.
Approach
A physics-driven degradation imaging model (Retinex, atmospheric scattering, sensor noise/blur) builds a 12-type, 4-level dataset; a CLIP text encoder embeds user degradation prompts, a spatial-frequency visual adapter (SFVA) infers equivalent visual prompts from inputs, and a prompt-modulated module (FiLM-style scaling/shifting) modulates a joint restoration-fusion transformer.
Key results
- On LLVIP, ControlFusion reports EN 7.354, SD 56.631, VIF 0.968 and Qabf 0.738 (best SD/VIF/Qabf on MSRS and LLVIP).
- In the pre-enhancement comparison on visible low-light degradation, ControlFusion gets CLIP-IQA 0.196 and MUSIQ 52.319 on visible rain, outperforming baselines in most scenarios.
- On object detection on LLVIP, ControlFusion reaches mAP@0.5:0.95 of 0.609, the highest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the color consistency loss (ablation) | ControlFusion · VI low light and noise | CLIP-IQA: 0.132 -> 0.225 | Helped |
| remove the maximum gradient loss (ablation) | ControlFusion · VI OE and IR low-contrast | CLIP-IQA: 0.151 -> 0.187 | Helped |
| remove the intensity loss (ablation) | ControlFusion · VI rain and haze and IR noise | CLIP-IQA: 0.156 -> 0.179 | Helped |
| remove PMM (ablation) | ControlFusion · VI low light and noise | CLIP-IQA: 0.129 -> 0.225 | Helped |
| remove the SFVA frequency branch (ablation) | ControlFusion · VI low light and noise | CLIP-IQA: 0.173 -> 0.225 | Helped |
| train using four severity levels (1, 4, 7, 10) rather than two extreme levels (1, 10) (ablation) | ControlFusion · LLVIP | CLIP-IQA: 0.320 -> 0.347 | Helped |
| train using four severity levels (1, 4, 7, 10) rather than two moderate levels (4, 7) (ablation) | ControlFusion · LLVIP | CLIP-IQA: 0.332 -> 0.347 | Helped |
| train using four severity levels (1, 4, 7, 10) rather than two extreme levels (1, 10) (ablation) | ControlFusion · FMB | CLIP-IQA: 0.192 -> 0.208 | Helped |
Surprising findings
- Training on sparser degradation levels (only 1 and 10, or only 4 and 7) noticeably reduces real-world generalization compared with four anchor levels.
Limitations
- The degradation imaging model is specific to infrared-visible fusion and hard to transfer to medical or multi-focus fusion.
- Relies on synthetic degradations, with real-world gap only partially addressed.
- Pre-enhancement baselines require chaining separate restoration models, making comparison dependent on choice of preprocessors.
Experimental setup
Datasets: RoadScene LLVIP MSRS FMB AWMM-100k DDL-12
Benchmarks: MSRS LLVIP RoadScene FMB AWMM-100k LLVIP object detection
Metrics: EN SD VIF Qabf CLIP-IQA MUSIQ TReS mAP@0.5:0.95 precision recall
Baselines: DDFM DRMF EMMA LRRNet SegMiF Text-IF Text-DiFuse OneRestore InstructIR DA-CLIP WDNN
Builds on: CLIP PromptIR DA-CLIP Retinex theory LIME
Closest papers by meaning
- Robust Fusion Controller: Degradation-Aware Image Fusion with Fine-Grained Language Instructions (AAAI 2026 · similarity 0.92)
- Task-Gated Multi-Expert Collaboration Network for Degraded Multi-Modal Image Fusion (ICML 2025 · similarity 0.86)
- Enhancing Infrared Vision: Progressive Prompt Fusion Network and Benchmark (NeurIPS 2025 · similarity 0.85)
- Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR 2024 · similarity 0.85)
- Task-driven Image Fusion with Learnable Fusion Loss (CVPR 2025 · similarity 0.85)
- Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.