Research map / Vision-language and multimodal / Cross-modal learning
Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisMoFlow, a hierarchical RGB-event-LiDAR fusion framework that uses event data as a bridge to fuse modalities in luminance, structure, and motion-correlation homogeneous spaces for all-day scene flow estimation.
Problem
Scene flow estimation from RGB or LiDAR relies on visual features, and direct RGB-LiDAR fusion suffers a modality gap due to their heterogeneous visual nature, degrading motion features, especially at night.
Approach
Uses event camera as a bridge between RGB and LiDAR, fusing them hierarchically in homogeneous spaces: luminance (RGB-event HDR fusion with spatiotemporal gradient consistency), structure (event-LiDAR self-similarity clustering), and motion correlation (KL-divergence alignment of RGB, event and LiDAR correlation distributions).
Key results
- On synthetic Event-KITTI daytime, VisMoFlow achieves EPE 0.198 and ACC 97.11% versus RPEFlow's 0.556 / 88.98%.
- On real DSEC nighttime, VisMoFlow reaches EPE 0.090 and ACC 68.31%, compared with CamLiFlow's 0.094 / 66.49% (RPEFlow-fused variant reported in the table).
- Ablation on DSEC: hierarchical visual-motion fusion reaches EPE 0.084 and ACC 70.34%, versus 0.151 / 49.20% without any fusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical visual-motion fusion (ablation) | VisMoFlow · DSEC | EPE: 0.151 -> 0.084 | Helped |
| Add visual fusion to motion fusion (ablation) | VisMoFlow · DSEC | ACC: 62.87% -> 70.34% | Helped |
| Add visual fusion without motion fusion (ablation) | VisMoFlow · DSEC | EPE: 0.151 -> 0.118 | Helped |
| Add motion fusion without visual fusion (ablation) | VisMoFlow · DSEC | EPE: 0.151 -> 0.098 | Helped |
| Add spatiotemporal gradient consistency loss Lconsis (ablation) | VisMoFlow · DSEC | EPE: 0.122 -> 0.112 | Helped |
| Add pseudo-label loss Lpse (ablation) | VisMoFlow · DSEC | EPE: 0.122 -> 0.107 | Helped |
| Add motion distribution alignment loss Lkl_corr (ablation) | VisMoFlow · DSEC | EPE: 0.122 -> 0.092 | Helped |
| Add all three fusion losses (ablation) | VisMoFlow · DSEC | EPE: 0.122 -> 0.084 | Helped |
| Add a structure neighbor (ablation) | VisMoFlow · DSEC | EPE: 0.095 -> 0.084 | Helped |
| Use clustering-based rather than grid-based structure neighbors (ablation) | VisMoFlow · DSEC | EPE: 0.088 -> 0.084 | Helped |
| Use VisMoFlow rather than CamLiFlow | scene flow estimation · Event-KITTI Day | EPE: 0.033 -> 0.012 | Helped |
| Use VisMoFlow rather than CamLiFlow | scene flow estimation · Event-KITTI Night | EPE: 0.047 -> 0.027 | Helped |
Surprising findings
- Motion fusion is the key driver of scene flow gains while visual fusion mainly raises the upper bound, per the ablation.
- Image enhancement (KinD++) before RGB-based motion estimation gives only limited improvement at night.
Limitations
- Fails in adverse weather such as rain and fog, which add streak noise and attenuate light.
- Requires a multi-step training pipeline with pretraining of motion estimators before fusion.
- Relies on synthetic Event-KITTI for much of the evaluation.
Experimental setup
Datasets: Event-KITTI DSEC KITTI
Benchmarks: Event-KITTI DSEC
Metrics: EPE ACC (within 5cm for scene flow, 1px for optical flow)
Baselines: RAFT-3D PV-RAFT CamLiFlow RPEFlow SMURF FlowFormer KinD++ image enhancement
Builds on: RAFT-3D CamLiFlow RPEFlow PV-RAFT V2E
Closest papers by meaning
- Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow (CVPR 2025 · similarity 0.91)
- Event-Driven Dynamic Scene Depth Completion (NeurIPS 2025 · similarity 0.87)
- Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes (NeurIPS 2025 · similarity 0.86)
- Motion Deblurring via Spatial-Temporal Collaboration of Frames and Events (AAAI 2024 · similarity 0.85)
- RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method (CVPR 2024 · similarity 0.85)
- STGC-NeRF: Spatial-Temporal Geometric Consistency for LiDAR Neural Radiance Fields in Dynamic Scenes (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.