Research map / Video / Video segmentation and tracking
A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniVF, a multi-frame Restormer-based video fusion network with optical-flow feature warping and a temporal consistency loss, and VF-Bench, a four-task video fusion benchmark with new temporal metrics (BiSWE, MS2R).
Problem
Image fusion methods process frames independently, causing flicker and temporal inconsistency in fused videos; there is also no video fusion benchmark or temporal evaluation protocol.
Approach
Fuse video by jointly encoding three consecutive frames per source, warping neighboring-frame features to the current frame with SEA-RAFT optical flow, decoding with a Restormer decoder, and training with an added flow-based temporal consistency loss; the benchmark is built from HDR synthesis, depth-based blur simulation, and curated RGBT/medical datasets.
Key results
- On the VF-Bench multi-exposure branch, UniVF reaches VIF 0.82 and SSIM 0.99 and the lowest MS2R 0.33, beating the strongest baselines on most metrics.
- On the infrared-visible branch, UniVF reports BiSWE 3.94 versus 4.35 for the next best (TDFusion), with the best VIF-tier metrics except VIF (0.44 vs 0.45).
- Ablation on IVF: removing the temporal loss raises BiSWE from 3.94 to 5.79 and removing feature warping raises it to 4.18, with the full model best on MI, Qabf, BiSWE and MS2R.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniVF instead of frame-based fusion baselines | MEF 2K · MEF 2K | VIF: 0.78 -> 0.82 | Helped |
| Use UniVF instead of frame-based fusion baselines | MFF 2K · MFF 2K | MI: 6.31 -> 6.32 | Helped |
| Use UniVF instead of frame-based fusion baselines | IVF · IVF | BiSWE: 4.35 -> 3.94 | Helped |
| Use UniVF instead of frame-based fusion baselines | MVF · MVF | VIF: 0.33 -> 0.35 | Helped |
| Add optical-flow feature warping (ablation) | UniVF on IVF · IVF | VIF: 0.40 -> 0.44 | Helped |
| Add multi-frame inputs with feature warping rather than frame-by-frame fusion (ablation) | UniVF on IVF · IVF | VIF: 0.38 -> 0.44 | Helped |
| Add temporal consistency loss (ablation) | UniVF on IVF · IVF | BiSWE: 5.79 -> 3.94 | Helped |
Surprising findings
- Several per-frame baselines (e.g., CDDFuse on medical) achieve competitive or better spatial metrics than UniVF, so temporal gains are not uniform across metrics.
Limitations
- Assumes well-aligned modality inputs; rare misaligned frames cause ghosting or artifacts.
- Multi-focus data is synthesized via depth-based blur with an approximate CoC model and a fixed blur-strength constant.
- Training and evaluation on 2K/low-resolution subsets with a single RTX 4090 GPU; limited scale of training scenes per task.
Experimental setup
Datasets: YouTube-HDR DAVIS VTMOT Harvard Medical
Benchmarks: VF-Bench multi-exposure branch VF-Bench multi-focus branch VF-Bench infrared-visible branch VF-Bench medical branch
Metrics: VIF SSIM MI Qabf BiSWE MS2R
Baselines: CUNet DDMEF HoLoCo CRMEF TC-MoA FILM ReFusion IFCNN RFL EPT CDDFuse EMMA
Builds on: Restormer SEA-RAFT CDDFuse EMMA
Closest papers by meaning
- Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow (CVPR 2025 · similarity 0.87)
- Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes (NeurIPS 2025 · similarity 0.86)
- Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow (CVPR 2024 · similarity 0.85)
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.84)
- One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion (CVPR 2025 · similarity 0.84)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.