Research map / 3D vision / Depth and stereo
EAG3R: Event-Augmented 3D Geometry Estimation for Dynamic and Extreme-Lighting Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments the MonST3R pointmap reconstruction pipeline with event streams via a Retinex-style enhancer, a Swin-based event adapter with SNR-aware fusion, and an event photometric consistency loss, evaluated zero-shot on MVSEC nighttime sequences.
Problem
Pose-free 3D geometry and dynamic reconstruction from video fails under dynamic objects and extreme low light because RGB frames are degraded by blur, over/underexposure, and noise.
Approach
Fuses RGB and asynchronous event features through an SNR-weighted, Retinex-guided gating scheme inside a pointmap regressor, and adds an event-based photometric consistency loss (brightness increments from events vs. predicted from image gradients and motion) to global optimization.
Key results
- Depth on MVSEC Night3: EAG3R Abs Rel 0.288 and delta<1.25 0.533 vs. MonST3R (Finetune) 0.302 and 0.509, zero-shot without nighttime training.
- Pose on MVSEC Night1: EAG3R ATE 0.482 vs. best baseline Easi3R (Finetune) 0.540; RPE rot 0.143 vs. 0.214.
- Ablation on Night3: adding events gives Abs Rel 0.297, LightUp 0.291, and full SNR fusion 0.288; event adapter adds about 0.4 GB VRAM over MonST3R.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add event inputs (ablation) | MonST3R (Finetune) · MVSEC Night3 depth estimation | Abs Rel: 0.302 -> 0.297 | Helped |
| add LightUp to event inputs (ablation) | MonST3R with event inputs · MVSEC Night3 depth estimation | Abs Rel: 0.297 -> 0.291 | Helped |
| add SNR Fusion to event inputs and LightUp (ablation) | MonST3R with event inputs and LightUp · MVSEC Night3 depth estimation | Abs Rel: 0.291 -> 0.288 | Helped |
| apply LightUp preprocessing without event-guided fusion (ablation) | MonST3R · MVSEC Night1–3 depth estimation | Abs Rel: 0.370 -> 0.370; 0.309 -> 0.316; 0.317 -> 0.329 | Hurt |
| replace the lightweight Swin Transformer event adapter with Zero_Conv (ablation) | EAG3R · MVSEC Night1–3 depth estimation | Abs Rel: 0.353 / 0.307 / 0.288 vs 0.377 / 0.323 / 0.302 | Hurt |
| replace multi-stage adaptive fusion with simple feature addition (ablation) | EAG3R · MVSEC Night1–3 depth estimation | Abs Rel: 0.357 vs 0.353; 0.312 vs 0.307; 0.295 vs 0.288 | Mixed |
| replace multi-stage adaptive fusion with last-layer attention (ablation) | EAG3R · MVSEC Night1–3 depth estimation | Abs Rel: 0.361 vs 0.353; 0.322 vs 0.307; 0.296 vs 0.288 | Hurt |
| select Harris corners rather than random samples for global optimization (ablation) | EAG3R · MVSEC Night1–3 sequences subsets | ATE: 0.687 -> 0.655 | Helped |
| select Harris corners rather than SuperPoint features for global optimization (ablation) | EAG3R · MVSEC Night1–3 sequences subsets | ATE: 0.685 -> 0.655 | Helped |
| use EAG3R instead of the strongest baseline | MonST3R-based depth estimation · MVSEC Night3 | Abs Rel: 0.302 -> 0.288 | Helped |
| use EAG3R instead of the strongest baseline | MonST3R-based depth estimation · MVSEC Night3 | δ < 1.25: 0.509 -> 0.533 | Helped |
| use EAG3R instead of the strongest baseline | MonST3R-based depth estimation · MVSEC Night3 | RMSE log: 0.401 -> 0.371 | Helped |
Surprising findings
- Applying RetinexFormer image enhancement as a preprocessing step gives no significant gain and sometimes degrades MonST3R depth, which the authors attribute to missing joint optimization.
- V2E-simulated events caused gradient explosion during fine-tuning, forcing the use of real event data from MVSEC.
Limitations
- Training and evaluation are limited to MVSEC, a vehicle-centric dataset with few sequences, and the authors note scarcity of real event-RGB-GT data.
- Gains in some metrics are marginal or absent; e.g., on Night3 pose ATE EAG3R (0.409) is worse than Easi3R (Finetune) (0.394), and on Night2 RPE trans is second best.
- Event loss increases global-optimization VRAM from 12.08 to 14.02 GB at sequence length 20.
Experimental setup
Datasets: MVSEC outdoor_day2 ETartanAir
Benchmarks: MVSEC outdoor_night1 MVSEC outdoor_night2 MVSEC outdoor_night3 M3ED robot dog penno_plaza_lights M3ED high-speed drone MVSEC indoor drone
Metrics: Abs Rel delta<1.25 RMSE log ATE RPE trans RPE rot
Baselines: DUSt3R MonST3R Easi3R RetinexFormer (LightUp) MonST3R (Finetune)
Builds on: DUSt3R MonST3R RetinexFormer Easi3R EvLight
Closest papers by meaning
- EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025 · similarity 0.87)
- SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos (CVPR 2025 · similarity 0.87)
- Event-3DGS: Event-based 3D Reconstruction Using 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.87)
- EvGGS: A Collaborative Learning Framework for Event-based Generalizable Gaussian Splatting (ICML 2024 · similarity 0.86)
- DarkVGGT: Seeing Through Darkness Using Thermal Geometry without Daylight Tax (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.