Research map / Vision-language and multimodal / Cross-modal learning

Equivariant Multi-Modality Image Fusion

CVPR 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 233 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EMMA, a self-supervised multi-modality image fusion paradigm that learns frozen pseudo-sensing networks and trains a U-Net-like fuser with an equivariant imaging loss under shifts, rotations and reflections.

Problem

Multi-modality image fusion (infrared-visible and medical) lacks ground-truth fused images, so supervised training is infeasible and hand-crafted fusion losses and priors are domain-dependent.

Approach

Learn frozen pseudo-sensing U-Nets that map a fused image back to each source modality, and enforce equivariance of the fuser-sensor composition under geometric transforms (equivariant imaging prior), so the fuser is trained without fusion ground truth.

Key results

• On MSRS IVF, EMMA reaches EN 6.71, SD 44.13, SF 11.56, VIF 0.97, among the best on most metrics versus 9 SOTA methods.
• On M3FD detection with YOLOv5, EMMA achieves 82.9 mAP@0.5, above the best baseline Meta/CDDFuse at 81.5/81.6.
• On MSRS segmentation with DeepLabV3+, EMMA achieves 63.7 mIoU, the highest among compared fusion methods (SegMIF 63.2).

Measured effects reported in the paper

Surprising findings

• Simply applying the same geometric transformations as data augmentation with the traditional fusion loss yields only slight improvement, suggesting the equivariant loss rather than augmentation drives gains.

Limitations

• Pseudo-sensing modules depend on fused outputs of 15 existing SOTA methods as pseudo ground truth, so quality is bounded by them.
• Results on Harvard medical fusion are mixed on some metrics (e.g., SCD and VIF not uniformly best).

Experimental setup

Datasets: MSRS RoadScene M3FD Harvard Medical dataset

Benchmarks: MSRS RoadScene M3FD Harvard Medical dataset

Metrics: EN SD SF AG SCD VIF mAP@0.5 IoU

Baselines: SDNet TarDAL DeFusion MetaFusion CDDFuse LRRNet MURF DDFM SegMIF

Builds on: equivariant imaging Restormer U-Net DeepLabV3+ YOLOv5

Closest papers by meaning

• Equivariant Self-Supervised Learning: Encouraging Equivariance in Representations (ICLR 2022 · similarity 0.85)
• Self-supervised Transformation Learning for Equivariant Representations (NeurIPS 2024 · similarity 0.84)
• Multi-modal Vision Pre-training for Medical Image Analysis (CVPR 2025 · similarity 0.83)
• Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.83)
• Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.83)
• When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.82)