Research map / Vision-language and multimodal / Cross-modal learning
Task-Customized Mixture of Adapters for General Image Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TC-MoA, a parameter-efficient mixture-of-adapters prompting of a frozen MAE-large ViT that unifies visible-infrared, multi-exposure, and multi-focus image fusion in a single model.
Problem
General image fusion across multi-modal, multi-exposure, and multi-focus tasks suffers from cross-task gaps, so single unified models trade off task-specific performance against commonality.
Approach
Shared adapters act as MoE experts with task-specific routers that generate task-customized complementary prompts over multi-source ViT features, constrained by a mutual information regularization (prompt_x + prompt_y ≈ 1) so the dominant intensity bias adapts per task.
Key results
- On LLVIP VIF, TC-MoA reports VIF 0.726 and Qcv 423.773, outperforming all listed general fusion methods (best other general method CDDFuse/IFCNN at 0.694/0.679 VIF).
- On MEFB multi-exposure fusion, TC-MoA reaches the highest MEF-SSIM (0.964) and PSNR (57.213) among compared methods.
- Adding only 2.8% learnable parameters (3.87M trainable for ViT-base) with a frozen backbone, the method runs at 3.33 FPS on VIF and is about 178% faster than the unoptimized base model after shifted-window optimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TC-MoA rather than CDDFuse | General image fusion · LLVIP test | VIF: 0.694 -> 0.726 | Helped |
| Use TC-MoA rather than MEFNet | General image fusion · MEFB | MEF-SSIM: 0.914 -> 0.964 | Helped |
| Use TC-MoA rather than IFCNN | General image fusion · MFF test | NMI: 0.847 -> 0.875 | Helped |
| Use TC-MoA rather than IFCNN | General image fusion · MFF test | MS-SSIM: 0.991 -> 0.990 | Hurt |
| Use multiple adapters instead of a single adapter (ablation) | TC-MoA multi-task VIF · VIF | Qabf: 0.6007 -> 0.6385 | Helped |
| Use multiple adapters instead of a single adapter (ablation) | TC-MoA multi-task MEF · MEF | Qabf: 0.6449 -> 0.6362 | Hurt |
Surprising findings
- Naive multi-task training degrades MEF performance relative to single-task training, indicating inter-task competition that the adapter mixture mitigates.
- The learned prompt can be linearly rescaled to zero-shot transfer to medical image fusion and pan-sharpening with reasonable results.
Limitations
- Evaluation uses limited test subsets (e.g., 70 randomly selected VIF test samples), and the unsupervised losses are task-customized, requiring per-task engineering.
- Inference speed is lower than some lightweight methods (e.g., U2Fusion at 4.72 FPS vs 3.33 FPS for TC-MoA base).
Experimental setup
Datasets: LLVIP SCIE RealMFF MFI-WHU MEFB MFIF
Benchmarks: LLVIP MEFB MFIF
Metrics: VIF Qc EN SD Qcv MS-SSIM FMI Qw PSNR Qp
Baselines: DenseFuse AUIF DIDFuse TarDAL YDTR RFN-Nest SwinFuse UMF-CMGR DDFM CDDFuse SDNet IFCNN
Builds on: Mixture of Experts AdaptFormer MAE Sparsely-gated MoE (Shazeer et al.)
Closest papers by meaning
- One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.85)
- Task-driven Image Fusion with Learnable Fusion Loss (CVPR 2025 · similarity 0.85)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.84)
- Task-Gated Multi-Expert Collaboration Network for Degraded Multi-Modal Image Fusion (ICML 2025 · similarity 0.84)
- Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner (NeurIPS 2024 · similarity 0.84)
- Task-Conditioned Adaptation of Visual Features in Multi-Task Policy Learning (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.