Research map / Vision-language and multimodal / Cross-modal learning
Task-Gated Multi-Expert Collaboration Network for Degraded Multi-Modal Image Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TG-ECNet, a unified all-in-one network with degradation-aware and fusion-aware gated multi-expert blocks and two-stage training that jointly restores and fuses degraded infrared-visible images, and releases the DeMMI-RF benchmark.
Problem
Infrared-visible image fusion degrades when inputs suffer from mixed degradations (noise, blur, haze, stripe noise); cascaded restore-then-fuse pipelines need many restoration models and propagate errors, and existing unified methods handle complex multi-degradation cases poorly.
Approach
Two task-aware gates route inputs to expert groups: a degradation-aware gate in the encoder selects restoration experts, and a fusion-aware gate in the decoder weights cross-modal features for fusion. A two-stage schedule (restoration first, then fusion with partly frozen parameters) balances the two tasks.
Key results
- On DeMMI-RF single-task averages, TG-ECNet reaches PSNR 30.58 and MS-SSIM 0.297 vs. best baseline MGDN PSNR 30.22 and Text-IF MS-SSIM 0.238.
- On the multi-degradation setting (noise+haze+defocus+stripe), TG-ECNet reaches PSNR 30.22 and Nabf 0.011, versus DRMF Nabf 0.239.
- Detection on noise (sigma=50) with YOLOv5 gives mAP50 0.969 and AP(0.5:0.95) 0.537, the best among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TG-ECNet instead of the strongest listed baseline, MGDN | Multi-modal fusion · DeMMI-RF and EMS average | PSNR: 30.2216 -> 30.5822 | Helped |
| Remove task-aware gating (adding task-aware gating) (ablation) | TG-ECNet · DeMMI-RF noise average (σ = 15, 25, 50) | PSNR: 31.136 -> 31.388 | Helped |
| Remove task-aware gating (adding task-aware gating) (ablation) | TG-ECNet · DeMMI-RF stripe noise | PSNR: 30.577 -> 30.932 | Helped |
| Remove all multi-expert blocks (adding all multi-expert blocks) (ablation) | TG-ECNet · DeMMI-RF noise average (σ = 15, 25, 50) | PSNR: 31.314 -> 31.388 | Helped |
| Remove all multi-expert blocks (adding all multi-expert blocks) (ablation) | TG-ECNet · DeMMI-RF stripe noise | PSNR: 30.844 -> 30.932 | Helped |
| Remove the restoration-stage multi-expert block (adding it) (ablation) | TG-ECNet · DeMMI-RF noise average (σ = 15, 25, 50) | PSNR: 30.014 -> 31.388 | Helped |
| Remove the restoration-stage multi-expert block (adding it) (ablation) | TG-ECNet · DeMMI-RF stripe noise | PSNR: 29.897 -> 30.932 | Helped |
| Remove the fusion-stage multi-expert block (adding it) (ablation) | TG-ECNet · DeMMI-RF noise average (σ = 15, 25, 50) | PSNR: 31.343 -> 31.388 | No clear effect |
| Remove the fusion-stage multi-expert block (adding it) (ablation) | TG-ECNet · DeMMI-RF stripe noise | PSNR: 30.894 -> 30.932 | Helped |
| Remove two-stage training (adding two-stage training) (ablation) | TG-ECNet · DeMMI-RF noise average (σ = 15, 25, 50) | PSNR: 31.010 -> 31.388 | Helped |
| Remove two-stage training (adding two-stage training) (ablation) | TG-ECNet · DeMMI-RF stripe noise | PSNR: 30.310 -> 30.932 | Helped |
Limitations
- Baselines are run with AdaIR preprocessing for cascaded fusion models, so comparisons depend on that pipeline choice.
- Degradations are synthetically generated, and real-world evaluation is limited to qualitative haze results on AWMM.
Experimental setup
Datasets: DeMMI-RF EMS dataset
Benchmarks: DeMMI-RF EMS dataset AWMM
Metrics: CC MSE PSNR Nabf MS-SSIM mAP50 AP(0.5:0.95)
Baselines: DRMF Text-IF AWFusion DenseFuse SwinFuse CDDFuse SeAFusion MGDN EMMA AdaIR
Builds on: Restormer AdaIR CDDFuse
Closest papers by meaning
- Task-driven Image Fusion with Learnable Fusion Loss (CVPR 2025 · similarity 0.86)
- ControlFusion: A Controllable Image Fusion Network with Language-Vision Degradation Prompts (NeurIPS 2025 · similarity 0.86)
- Robust Fusion Controller: Degradation-Aware Image Fusion with Fine-Grained Language Instructions (AAAI 2026 · similarity 0.85)
- One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.84)
- Task-Customized Mixture of Adapters for General Image Fusion (CVPR 2024 · similarity 0.84)
- Multi-axis Prompt and Multi-dimension Fusion Network for All-in-one Weather-degraded Image Restoration (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.