Research map / Vision-language and multimodal / Cross-modal learning
Event-based Visible and Infrared Fusion via Multi-task Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVIF, an event-camera-based visible-infrared fusion system whose three tasks (event-based visible reconstruction, event-guided IR deblurring, and fusion) are trained collaboratively via cross-task event enhancement and bi-level min-max mutual information optimization.
Problem
Conventional visible-infrared image fusion degrades under extreme lighting (over/under-exposure) and high-dynamic motion (motion blur) because frame-based cameras have limited dynamic range and frame rate.
Approach
Pair an infrared camera with a visible event camera and jointly learn event-based visible texture reconstruction, event-guided infrared deblurring, and fusion, with a bi-directional ConvGRU cross-task channel attention that injects reconstruction texture into the deblurring network and a bi-level MI objective (minimize MI between modality features, maximize MI between fused and original features)
Key results
- On synthetic KAIST data, EVIF reaches EN 7.326 and MI 1.978, best among compared VIF methods, with SSIM 1.285 and QCB 0.377 (per Table 1).
- Infrared deblurring PSNR rises from 31.23 (NAFNet) to 33.94 with EVIF, and 33.32 without cross-task enhancement.
- Qualitative comparison on a real hybrid DAVIS346 plus IR camera system shows reduced blur and exposure issues versus nine VIF methods; no ground truth is available so no quantitative real results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EVIF instead of GFCE | Visible-infrared fusion on KAIST · KAIST test | EN: 7.314 -> 7.326 | Helped |
| Use EVIF instead of CDDFuse | Visible-infrared fusion on KAIST · KAIST test | MI: 1.939 -> 1.978 | Helped |
| Use EVIF instead of DenseFuse | Visible-infrared fusion on KAIST · KAIST test | SSIM: 1.283 -> 1.285 | Helped |
| Add cross-task event enhancement (ablation) | EVIF infrared image deblurring · KAIST synthetic | PSNR: 33.320 -> 33.940 | Helped |
| Use bi-level mutual information optimization instead of no MI optimization (ablation) | EVIF fusion · KAIST synthetic | fusion quality | Helped |
| Use bi-level mutual information optimization instead of MI minimization only (ablation) | EVIF fusion · KAIST synthetic | fusion quality | Helped |
| Use bi-level mutual information optimization instead of MI maximization only (ablation) | EVIF fusion · KAIST synthetic | fusion quality | Helped |
Limitations
- Real-data evaluation is qualitative only because no ground truth is available.
- Evaluated on a single synthetic dataset (KAIST) plus a small prototype hardware setup.
- Requires a hybrid event and infrared camera rig with manual spatial alignment and custom synchronization.
Experimental setup
Datasets: KAIST multispectral dataset
Benchmarks: KAIST
Metrics: CE EN MI AG SSIM QCB PSNR
Baselines: SwinFuse DenseFuse YDTR U2Fusion DIDFuse LRRNet CDDFuse HMSD_GF GFCE NAFNet
Builds on: E2VID EFNet NAFNet InfoNCE ConvGRU
Closest papers by meaning
- Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow (CVPR 2024 · similarity 0.80)
- Probing Synergistic High-Order Interaction in Infrared and Visible Image Fusion (CVPR 2024 · similarity 0.79)
- Event Fields: Capturing Light Fields at High Speed, Resolution, and Dynamic Range (CVPR 2025 · similarity 0.79)
- One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.78)
- A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking (NeurIPS 2025 · similarity 0.78)
- FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies (NeurIPS 2025 · similarity 0.77)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.