Research map / 3D vision / Face and shape
Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffPose, a self-supervised patient-specific 2D/3D X-ray-to-CT registration method that pretrains a CNN pose regressor on synthetic X-rays and refines poses with differentiable rendering and sparse multiscale NCC in se(3).
Problem
Registering intraoperative 2D X-rays to preoperative 3D CT accurately and quickly, without manually annotated landmarks or large labeled datasets, since conventional optimization is slow and prone to local minima and learned methods generalize poorly.
Approach
Train a patient-specific ResNet18 pose regressor on unlimited synthetic X-rays rendered from the patient's own CT with a differentiable renderer, using a composite multiscale NCC plus SE(3) geodesic loss; then refine at test time by optimizing the pose in se(3) with sparse multiscale NCC through the differentiable renderer.
Key results
- On DeepFluoro, DiffPose reaches 87% sub-millimeter success rate (mTRE 0.9 mm) versus 69% for PnP-Regularizer and 37% for DiffDRR, with 2.2 s average runtime.
- Without test-time optimization, the encoder reaches sub-millimeter mTRE on only 1% of test cases.
- On the Ljubljana neurovascular dataset, DiffPose achieves 85% SMSR (median mTRE 0.2 mm) versus 25% for DiffDRR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add test-time optimization to the pose-regression initialization (ablation) | DiffPose · DeepFluoro | SMSR: 1% -> 87% | Helped |
| add multiscale NCC to the pretraining loss (ablation) | DiffPose without test-time optimization · DeepFluoro | SMSR: 0% -> 1% | Helped |
| add geodesic losses Lgeo and Llog to pose regression (ablation) | DiffPose without test-time optimization · DeepFluoro | SMSR: 0% -> 0% | No clear effect |
| use se(3) instead of axis-angle × R3 pose parameterization (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 75% -> 87% | Helped |
| use se(3) instead of Euler angles × R3 pose parameterization (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 83% -> 87% | Helped |
| use se(3) instead of quaternion × R3 pose parameterization (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 83% -> 87% | Helped |
| use se(3) instead of Rotation6D × R3 pose parameterization (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 83% -> 87% | Helped |
| use se(3) instead of Rotation10D × R3 pose parameterization (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 81% -> 87% | Helped |
| use se(3) instead of Quaternion Adjugate × R3 pose parameterization (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 77% -> 87% | Helped |
| use sparse multiscale NCC instead of global NCC (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 27% -> 87% | Helped |
| use sparse multiscale NCC instead of local NCC (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 81% -> 87% | Helped |
| use sparse multiscale NCC instead of gradient NCC (ablation) | DiffPose test-time optimization · DeepFluoro | SMSR: 09% -> 87% | Helped |
Surprising findings
- Pose regression alone reaches sub-millimeter accuracy on only 1% of test cases, so test-time refinement is essential.
- Euler angles and quaternions in a standard parameterization perform close to the proposed se(3) for test-time optimization, despite known deficiencies.
Limitations
- Estimates a single rigid transform, so it cannot directly handle deformable structures.
- Per-subject training takes hours, which may be too slow for emergency surgery.
- Evaluated on only six cadaver pelvis CTs plus ten neurovascular patients.
Experimental setup
Datasets: DeepFluoro Ljubljana neurovascular X-ray angiography dataset
Benchmarks: DeepFluoro Ljubljana
Metrics: mTRE sub-millimeter success rate runtime
Baselines: PoseNet xReg DiffDRR PnP PnP-Regularizer
Builds on: DiffDRR PoseNet PnP-Regularizer xReg
Closest papers by meaning
- PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations (NeurIPS 2025 · similarity 0.90)
- CARL: A Framework for Equivariant Image Registration (CVPR 2025 · similarity 0.84)
- Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration (CVPR 2024 · similarity 0.84)
- Deep Learning in Medical Image Registration: Magic or Mirage? (NeurIPS 2024 · similarity 0.84)
- A Plug-and-Play Image Registration Network (ICLR 2024 · similarity 0.83)
- LiFT: Lifted Inter-slice Feature Trajectories for 3D Image Generation from 2D Generators (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.