Research map / Vision-language and multimodal / Cross-modal learning
One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIFNet, a three-branch network that trains a single model jointly on low-level multi-modal (IVIF) and multi-focus (MFIF) fusion tasks with a shared RGB reconstruction branch, enabling task-agnostic image fusion and single-modality enhancement.
Problem
Existing image fusion methods rely on high-level semantic tasks (detection, segmentation, captioning) for supervision, which introduces semantic gaps, high computational cost, and need for task-specific models, limiting generalisation across fusion tasks.
Approach
Use low-level digital photography fusion (multi-focus) as pixel-level auxiliary supervision for unsupervised infrared-visible fusion, coupled via alternating main/auxiliary training, a shared RGB reconstruction branch, and a learnable cross-fusion gating mechanism built on Swin Transformer blocks.
Key results
- GIFNet achieves the best VIF on Lytro (1.74) and the best results on most IVIF metrics on LLVIP (VIF 0.73, SCD 1.61, AG 6.70), reported as the best in most settings.
- On CIFAR100 classification with enhanced training data, GIFNet reaches 56.18% top-1 accuracy, above the original training set (54.11%) and all compared fusion methods.
- GIFNet has 3.30 MB model size and 9.96 GFLOPs, about 98.7% fewer GFLOPs than Text-IF (82.85) and 99.1% smaller than the next compared models in the efficiency table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train GIFNet jointly on IVIF and MFIF with CFGM and REC (ablation) | GIFNet · LLVIP IVIF | EI: 49.84 -> 62.46 | Helped |
| Add the reconstruction branch (ablation) | GIFNet trained on IVIF and MFIF · LLVIP IVIF | EI: 49.84 -> 62.46 | Helped |
| Add the cross-fusion gating mechanism (ablation) | GIFNet trained on IVIF and MFIF · LLVIP IVIF | EI: 52.56 -> 62.46 | Helped |
| Replace MFIF auxiliary training with MEIF (ablation) | GIFNet · LLVIP IVIF | EI: 62.46 -> 47.69 | Hurt |
| Replace adaptive CFGM with conventional fusion operations (ablation) | GIFNet · IVIF ablation | VIF | Hurt |
| Use MFIF as an auxiliary task for IVIF (ablation) | GIFNet · LLVIP IVIF | EI: 28.71 -> 62.46 | Helped |
| Train with IVIF and MFIF but remove both CFGM and REC (ablation) | GIFNet · LLVIP IVIF | EI: - | Hurt |
| Use GIFNet-enhanced images instead of original CIFAR100 images for classifier training | CIFAR-ResNet56 · CIFAR100 classification | Top-1 accuracy: 54.11% -> 56.18% | Helped |
Surprising findings
- Adding a second fusion task without the proposed CFGM or REC components prevents the network from converging, producing non-functional outputs.
- Multi-focus (MFIF) supervision was more beneficial as an auxiliary task than multi-exposure (MEIF), which the authors attribute to pixel-level clarity compatibility.
Limitations
- Only MFIF is used as the auxiliary task in the final model; the paper does not explore many task combinations.
- Visual information fidelity on LLVIP is weaker than choose-max baselines, which the authors attribute to bias in those methods rather than a GIFNet weakness.
- Augmented joint dataset details are deferred to supplement, and training-set size is not stated in the main text.
Experimental setup
Datasets: LLVIP TNO Lytro MFI-WHU Harvard medical VIS-NIR Scene SCIE Quickbird CIFAR100
Benchmarks: LLVIP TNO Lytro MFI-WHU SCIE VIS-NIR Scene Quickbird Harvard CIFAR100
Metrics: EI VIF SCD AG top-1 accuracy top-5 accuracy
Baselines: Text-IF CDDFuse DDFM LRRNet ZMFF UNIFusion MURF MUFusion U2Fusion SDNet
Builds on: FusionBooster SwinTransformer U2Fusion CDDFuse
Closest papers by meaning
- Image Fusion via Vision-Language Model (ICML 2024 · similarity 0.87)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.86)
- Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.85)
- Task-Customized Mixture of Adapters for General Image Fusion (CVPR 2024 · similarity 0.85)
- Task-driven Image Fusion with Learnable Fusion Loss (CVPR 2025 · similarity 0.85)
- MRFS: Mutually Reinforcing Image Fusion and Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.