Research map / Vision-language and multimodal / Cross-modal learning

Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes relation-constrained supervision (RCS) for multi-modal image fusion that replaces spatial-domain source supervision with losses in a learned relation space built by a learnable feature adapter over frozen DINO and CLIP features, trained with contrastive ranking and alternating optimization with the fusion network.

Problem

Multi-modal image fusion lacks ground-truth fused images, so existing methods use source images or gradient variants as surrogate targets, which is misaligned with fusion goals and causes pixel-level compromise or modality bias.

Approach

Move fusion supervision from the spatial domain into a learned relation space: an adapter aligns frozen DINO and CLIP features and predicts sharedness, dominance, and coordination radius, which define shared, complementary, and coordination losses; the adapter is trained with self-supervised source-pair and source-fused ranking and alternated with the fusion network.

Key results

• On RoadScene IVF, Ours(Transformer) reports EN 61.069 and SD 50.581, the best among compared methods (e.g., ReFusion EN 59.336).
• On MRI-PET MIF, Ours(Transformer) reports EN 99.031 and SD 78.072, best in the table.
• On M3FD object detection with YOLOv12, Ours(Transformer) reaches mAP 0.730 versus 0.686 for ISFusion.

Limitations

• Relies on warm-up with conventional spatial-domain losses before alternating optimization.
• Evaluation relies on fused-image metrics and downstream detectors/segmenters rather than a ground truth fused image.

Experimental setup

Datasets: MSRS Harvard Medical Dataset

Benchmarks: RoadScene M3FD MSRS CT-MRI MRI-PET MRI-SPECT BraTS2018

Metrics: AG EN SD MI Q_AB/F Q_M Q_P IoU Dice mAP

Baselines: EMMA Text-DiFuse Tc-MoA ReFusion C2RF GIF-Net SAGE Omni-Fuse CCF Mask-Difuser CLDyN ISFusion

Builds on: DINO CLIP

Closest papers by meaning

• Multi-Modal Image Fusion via Intervention-Stable Feature Learning (CVPR 2026 · similarity 0.88)
• When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.85)
• ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.85)
• One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.85)
• Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.84)
• Hybrid-Supervised Dual-Search: Leveraging Automatic Learning for Loss-Free Multi-Exposure Image Fusion (AAAI 2024 · similarity 0.84)