Research map / 3D vision / Human pose estimation
Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OphNet-3D, a 7.1M-frame multi-view RGB-D dataset of cataract surgery hand-instrument interactions with an automatic optimization-based annotation pipeline, and baseline models H-Net and OH-Net for bimanual hand and two-hand-two-instrument reconstruction.
Problem
Accurate 3D reconstruction of both hands and surgical instruments in ophthalmic microsurgery is hampered by the lack of realistic, large-scale annotated datasets and reliable annotation tools.
Approach
A multi-stage optimization pipeline fuses multi-view RGB-D observations, a data-driven hand motion prior, biomechanical constraints, and collision-aware interaction losses to auto-label MANO hand meshes and 6-DoF articulated instrument poses; baselines use a collision-aware center-based representation with weak-perspective camera modeling.
Key results
- OH-Net-D reaches 15.94 mm MPJPE and 76.31% ADD-S on the test split of the hand-instrument interaction benchmark, versus 17.36 mm MPJPE and 61.32% ADD-S for the best prior baseline (HOISDF).
- H-Net-D attains 15.97 mm MPJPE on the bimanual hand test split, improving over ACR (18.86 mm) and other hand baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RGB-D input instead of RGB input | H-Net · Bimanual hand pose estimation Val and Test | MPJPE (mm): 17.39 -> 15.28; 17.66 -> 15.97 | Helped |
| Use RGB-D input instead of RGB input | H-Net · Bimanual hand pose estimation Val and Test | MPVPE (mm): 18.72 -> 16.37; 18.66 -> 16.18 | Helped |
| Add T-Net (ablation) | H-Net · Bimanual hand pose estimation Val and Test | MPJPE (mm): 18.57 -> 17.39; 18.98 -> 17.66 | Helped |
| Add T-Net (ablation) | H-Net · Bimanual hand pose estimation Test | MPVPE (mm): 19.14 -> 18.66 | No clear effect |
| Add T-Net (ablation) | OH-Net · Hand-instrument interaction Val and Test | ADD-S (%): 66.87 -> 71.52; 65.94 -> 70.79 | Helped |
| Add T-Net (ablation) | OH-Net · Hand-instrument interaction Val and Test | MPJPE (mm): 17.48 -> 17.12; 17.89 -> 17.34 | Helped |
| Use RGB-D input instead of RGB input | OH-Net · Hand-instrument interaction Val and Test | ADD-S (%): 71.52 -> 76.68; 70.79 -> 76.31 | Helped |
| Use RGB-D input instead of RGB input | OH-Net · Hand-instrument interaction Val and Test | MPJPE (mm): 17.12 -> 15.23; 17.34 -> 15.94 | Helped |
| Use RGB-D input instead of RGB input | OH-Net · Hand-instrument interaction Val and Test | Pen (mm): 5.87 -> 5.14; 5.91 -> 5.18 | Mixed |
Limitations
- Relies on data from a single center
- Microscope illumination causes instrument-tip overexposure
- Does not yet integrate the microscope view for joint reconstruction
Experimental setup
Datasets: OphNet-3D
Benchmarks: OphNet-3D bimanual hand pose estimation OphNet-3D two-hand-instrument interaction
Metrics: MPJPE MPVPE MRRTE ADD-S MAE Pen MRRTE(h,o)
Baselines: DIR InterWild IntagHand ACR Hasson et al. HFL-Net HOISDF
Builds on: MANO SAM2 DynHaMR ICP
Closest papers by meaning
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.84)
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting (CVPR 2025 · similarity 0.84)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.84)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.84)
- A Simple Baseline for Efficient Hand Mesh Reconstruction (CVPR 2024 · similarity 0.84)
- HandOS: 3D Hand Reconstruction in One Stage (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.