Research map / Detection and segmentation / CNN features
E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E2E-MFD, a single-stage end-to-end framework that jointly trains visible-infrared image fusion and object detection (ORPPT fusion net + CFDP diffusion detection head) with a gradient-matrix task-alignment (GMTA) step on shared parameters.
Problem
Visible-infrared multimodal fusion and object detection are usually trained as cascaded, multi-step pipelines, which is inefficient and suffers from task conflicts and local optima when sharing parameters.
Approach
Synchronously and jointly optimize a fusion network (Object-Region-Pixel Phylogenetic Tree) and a diffusion-based detection head (Coarse-to-Fine Diffusion Process) sharing a backbone, and align shared-parameter gradients via a condition-number-based orthogonalization (GMTA, derived from Aligned-MTL).
Key results
- On M3FD, E2E-MFD reaches 91.8 mAP50 and 63.8 mAP50:95 with YOLOv5s-style detection, a 3.9% mAP50 gain over the best prior method per the abstract.
- On DroneVehicle, E2E-MFD achieves 77.4 mAP50, a 2.0% gain over the best prior comparison (CALNet, 75.4).
- Ablation on M3FD: removing GMTA lowers E2E-MFD from 91.80 to 90.05 mAP50 and reduces fusion MI from 15.47 to 14.70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GMTA instead of joint optimization without GMTA (ablation) | E2E-MFD · M3FD validation | mAP50: 90.05 -> 91.80 | Helped |
| Use GMTA instead of separate OD-only training (ablation) | E2E-MFD · M3FD validation | mAP50: 90.28 -> 91.80 | Helped |
| Replace GMTA with PCGrad, CAGrad, or Nash-MTL (ablation) | E2E-MFD · M3FD validation | mAP50: 90.59; 90.71; 90.91 vs 91.80 | Hurt |
| Set GMTA interval to n=500, 1500, or 2000 instead of 1000 (ablation) | E2E-MFD · M3FD validation | mAP50: 90.93; 91.10; 90.35 vs 91.80 | Mixed |
| Use branch configurations 0 only, 0,1, or 0,1,2 instead of 0,1,2,3 (ablation) | ORPPT in E2E-MFD · M3FD validation | mAP50: 91.51; 91.65; 91.73 vs 91.80 | Hurt |
| Add ORPPT branch 4 (ablation) | ORPPT in E2E-MFD · M3FD validation | mAP50: 91.80 -> 91.73 | Hurt |
| Replace CFDP with RPN (ablation) | E2E-MFD · M3FD validation | mAP50: 90.13 vs 91.80 | Hurt |
| Change CFDP proposal count to 300 or 1000 instead of 500 (ablation) | E2E-MFD · M3FD validation | mAP50: 90.89; 92.05 vs 91.80 | Mixed |
| Use E2E-MFD rather than CFT with YOLOv5s | M3FD object detection · M3FD | mAP50: 89.8 -> 91.8 | Helped |
| Use E2E-MFD rather than CALNet | DroneVehicle oriented detection · DroneVehicle test set | mAP50: 75.4 -> 77.4 | Helped |
Surprising findings
- Adding higher-level region branches (0-4) degrades fusion quality, which the authors attribute to deeper shared backbone features losing detail.
Limitations
- Validation is limited to visible-infrared modalities.
- Relies on a pretrained DiffusionDet / LSKNet initialization for the detector.
- Lacks validation on datasets with new modalities due to scarce data.
Experimental setup
Datasets: M3FD DroneVehicle
Benchmarks: TNO RoadScene M3FD DroneVehicle
Metrics: EN MI VIF mAP50 mAP50:95
Baselines: DIDFuse U2Fusion PIAFusion SwinFusion CDDFuse Tardal MetaFusion CFT ICAFusion PCGrad CAGrad Nash-MTL
Builds on: DiffusionDet MetaFusion Aligned-MTL LSKNet
Closest papers by meaning
- IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection (CVPR 2024 · similarity 0.86)
- DiMSOD: A Diffusion-Based Framework for Multi-Modal Salient Object Detection (AAAI 2025 · similarity 0.85)
- CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection (NeurIPS 2024 · similarity 0.84)
- SSLFusion: Scale and Space Aligned Latent Fusion Model for Multimodal 3D Object Detection (AAAI 2025 · similarity 0.84)
- GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer (ICML 2024 · similarity 0.84)
- When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.