Research map / 3D vision / Depth and stereo
Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvRGBHand, a transformer-based framework that fuses event streams and RGB frames for 3D hand mesh reconstruction, trained on normal scenes with an EvRGBDegrader augmentation to generalize to challenging conditions.
Problem
3D hand mesh reconstruction degrades under strong light, fast motion, and flash conditions, where RGB frames suffer overexposure and motion blur and event streams suffer foreground scarcity and background overflow.
Approach
Combines RGB and event modalities via deformable spatial alignment (with ConvLSTM-estimated offsets), complementary fusion, and temporal attention in a FastMETRO-style transformer, plus a degradation augmentation (color jitter, motion-blur interpolation, salt-and-pepper event noise) so that training on normal scenes generalizes to challenging ones.
Key results
- EvRGBHand achieves 22.34 mm MPJPE in strong light versus 35.02 mm for FastMETRO-RGB and 40.59 mm for Mesh Graphormer.
- EvRGBHand reaches 20.44 mm MPJPE under flash versus 53.69 mm for EventHands.
- EvRGBHand has 55.92M params and 8.15 GFLOPs, 60.5% fewer params than FastMETRO-Event, with average MPJPE 16.66 mm vs 23.59 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse event streams and RGB frames with EvImHandNet | EvRGBHand · EVREALHANDS strong light | MPJPE (mm): 35.02 -> 22.34 | Helped |
| Fuse event streams and RGB frames with EvImHandNet | EvRGBHand · EVREALHANDS flash | MPJPE (mm): 23.13 -> 20.44 | Helped |
| Use EvRGBHand rather than the strongest listed single-event baseline | EVREALHANDS normal · EVREALHANDS normal | MPJPE (mm): 18.36 -> 11.47 | Helped |
| Use EvRGBHand rather than the strongest listed RGB baseline | EVREALHANDS normal · EVREALHANDS normal | MPJPE (mm): 11.57 -> 11.47 | Helped |
| Add EvRGBDegrader simulations of overexposure and background overflow (ablation) | EvRGBHand · Challenging scenes | MPJPE | Helped |
| Add spatial alignment, complementary fusion, and temporal attention (ablation) | EvRGBHand · Challenging scenes | MPJPE: 2.5 ∼3 mm | Helped |
| Use event and image features with direct transformer fusion instead of EvImHandNet (ablation) | EvRGBHand-vanilla · EVREALHANDS normal | MPJPE (mm): 11.47 -> 11.84 | Hurt |
| Use event and image features with direct transformer fusion instead of EvImHandNet (ablation) | EvRGBHand-vanilla · EVREALHANDS strong light | MPJPE (mm): 22.34 -> 25.26 | Hurt |
| Use event and image features with direct transformer fusion instead of EvImHandNet (ablation) | EvRGBHand-vanilla · EVREALHANDS flash | MPJPE (mm): 20.44 -> 23.13 | Hurt |
Surprising findings
- The vanilla transformer fusion baseline is far larger (277M params) yet only modestly better than single-sensor methods in some scenes.
Limitations
- Performance degrades when overexposure and motion blur co-occur with challenging hand poses.
- Performance of each modality is affected by mismatched pixel resolutions between RGB and event cameras.
- Evaluation of fast-motion and outdoor sequences is qualitative only due to lack of 3D annotations.
Experimental setup
Datasets: EVREALHANDS InterHand2.6M
Benchmarks: EVREALHANDS
Metrics: MPJPE MPVPE PA-MPJPE FLOPs Params
Baselines: Mesh Graphormer FastMETRO-RGB EventHands FastMETRO-Event EvRGBHand-vanilla
Builds on: FastMETRO EventHands Deformable Convolution MANO
Closest papers by meaning
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.86)
- Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera (CVPR 2025 · similarity 0.86)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.85)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.85)
- Pose-Guided Temporal Enhancement for Robust Low-Resolution Hand Reconstruction (CVPR 2025 · similarity 0.85)
- Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.