Research map / 3D vision / Human pose estimation
Matrix3D: Large Photogrammetry Model All-in-One
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Matrix3D is a unified multi-modal diffusion transformer that performs pose estimation, depth prediction, and novel view synthesis (plus 3DGS-based sparse reconstruction) with one model, trained via masked multi-modal learning on incomplete data.
Problem
Photogrammetry pipelines need many dense images and chain separate task-specific modules (SfM, MVS, novel view synthesis) that are not jointly optimized, causing accumulated errors and requiring dense inputs.
Approach
Represent images as VAE latents, cameras as Plücker ray maps, and depth as disparity maps in a shared 2D space; extend a DiT to multi-view, multi-modal generation with masked learning over views/modalities so one model handles any input/output combination, and trains on partially labeled data (image-pose, image-depth pairs).
Key results
- Pose estimation on CO3D: relative rotation accuracy @15° of 95.6% at 2 images for RGB-only, vs 90.4% for RayDiffusion and 85.6% for DUSt3R.
- Novel view synthesis on GSO (view setting θ=30°): PSNR 20.45 vs SyncDreamer 19.22; Zero123-XL 18.75 vs Matrix3D 18.87 on the 360°/elevation setting.
- Monocular metric depth on DTU: AbsRel 0.036 (CFG=1.5) vs 0.064 for Metric3D v2 and 0.077 for Depth Anything v2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth input (ablation) | Matrix3D pose estimation · CO3D, 8 images | camera center accuracy @ 0.1: 87.8 -> 89.5 | Helped |
| add depth input (ablation) | Matrix3D pose estimation · CO3D, 8 images | relative rotation accuracy @ 15°: 96.1 -> 96.3 | Helped |
| add depth input (ablation) | Matrix3D novel view synthesis · GSO, azimuth {0°, 360°}, elevations {−90°, 90°} | PSNR: 18.87 -> 19.16 | Helped |
| disable classifier-free guidance (ablation) | Matrix3D monocular depth prediction · DTU | RMS: 47.806 -> 40.214 | Helped |
| disable classifier-free guidance (ablation) | Matrix3D monocular depth prediction · DTU | δ1: 0.985 -> 0.992 | Helped |
| use ground-truth cameras (ablation) | Matrix3D point-cloud reconstruction · DTU | Overall Chamfer distance: 2.266 -> 2.098 | Helped |
| use Matrix3D instead of the strongest listed pose baseline | CO3D pose estimation · 8 images | relative rotation accuracy @ 15°: 92.4 -> 96.1 | Helped |
| use Matrix3D instead of the strongest listed pose baseline | CO3D pose estimation · 8 images | camera center accuracy @ 0.1: 81.9 -> 87.8 | Helped |
| use Matrix3D instead of SyncDreamer | novel view synthesis · GSO, azimuth {0°, 360°}, elevation 30° | PSNR: 19.22 -> 20.45 | Helped |
| use Matrix3D instead of Wonder3D | novel view synthesis · GSO, azimuth {0°, 360°}, elevation 0° | PSNR: 13.28 -> 18.97 | Helped |
| use Matrix3D instead of Robust MVD Baseline | multi-view depth prediction · DTU | rel: 2.7 -> 1.83 | Helped |
| use Matrix3D instead of DUSt3R | back-projected point-cloud reconstruction without ground-truth cameras · DTU | Overall Chamfer distance: 1.741 -> 2.266 | Hurt |
Surprising findings
- Matrix3D's point cloud accuracy is worse than DUSt3R's on DTU even though its depth and pose are better, which the authors attribute to DUSt3R being supervised directly on point positions.
- Sparse-view 3-view 3DGS reconstruction on CO3D is slightly worse than CAT3D (PSNR 20.02 vs 20.57), attributed to fewer novel views used for 3DGS training.
Limitations
- Point cloud quality lags DUSt3R on DTU.
- Sparse-view 3DGS reconstruction slightly underperforms CAT3D on CO3D.
Experimental setup
Datasets: Objaverse MVImgNet CO3D-v2 RealEstate10k Hypersim ARKitScenes
Benchmarks: CO3D GSO DTU ARKitScenes
Metrics: relative rotation accuracy camera center accuracy PSNR SSIM LPIPS AbsRel delta1 accuracy Chamfer distance CLIP score
Baselines: COLMAP PoseDiffusion RelPose++ DUSt3R RayDiffusion SyncDreamer Wonder3D InstantMesh Zero123-XL CAT3D Metric3D v2 Depth Anything v2
Builds on: Hunyuan-DiT RayDiffusion 4M v-prediction diffusion 3D Gaussian Splatting
Closest papers by meaning
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis (ICML 2025 · similarity 0.84)
- Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer (NeurIPS 2026 · similarity 0.84)
- Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention (NeurIPS 2025 · similarity 0.84)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.84)
- Continuous 3D Perception Model with Persistent State (CVPR 2025 · similarity 0.84)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.