Research map / 3D vision / Depth and stereo
VGGT: Visual Geometry Grounded Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents VGGT, a large feed-forward transformer that jointly predicts camera parameters, depth maps, point maps, and point tracks from one to hundreds of images in under a second.
Problem
Estimating all key 3D attributes of a scene (cameras, depth, point maps, tracks) typically requires specialized single-task networks and slow geometry post-optimization such as bundle adjustment.
Approach
A standard large transformer with minimal 3D inductive biases (DINO patch tokens, alternating frame-wise and global attention, camera and register tokens, DPT and CoTracker-style heads) is trained jointly on many 3D-annotated datasets to predict all 3D attributes in one forward pass, with depth+camera-derived point maps preferred at inference.
Key results
- Camera pose estimation on CO3Dv2 reaches 88.2 AUC@30 feed-forward (85.3 on RealEstate10K unseen) versus 83.4 for VGGSfM v2, in about 0.2 s; with BA reaches 91.8 on CO3Dv2 and 93.5 on RealEstate10K.
- Multi-view depth on DTU reduces Overall error from 1.741 (DUSt3R) to 0.382 without known cameras.
- Fine-tuning CoTracker with VGGT backbone raises TAP-Vid RGB-S delta_vis_avg from 78.9 to 84.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bundle adjustment to VGGT predictions | VGGT · RealEstate10K | AUC@30: 85.3 -> 93.5 | Helped |
| add bundle adjustment to VGGT predictions | VGGT · CO3Dv2 | AUC@30: 88.2 -> 91.8 | Helped |
| replace alternating attention with cross-attention (ablation) | VGGT · ETH3D | Overall: 0.709 -> 1.061 | Hurt |
| replace alternating attention with global self-attention only (ablation) | VGGT · ETH3D | Overall: 0.709 -> 0.827 | Hurt |
| add camera estimation loss (ablation) | VGGT multi-task training · ETH3D | Overall: 0.834 -> 0.709 | Helped |
| add depth estimation loss (ablation) | VGGT multi-task training · ETH3D | Overall: 0.727 -> 0.709 | Helped |
| add track estimation loss (ablation) | VGGT multi-task training · ETH3D | Overall: 0.790 -> 0.709 | Helped |
| construct point cloud from predicted depth and camera instead of using the point map head (ablation) | VGGT · ETH3D | Overall: 0.709 -> 0.677 | Helped |
| fine-tune CoTracker using VGGT pretrained weights | CoTracker · TAP-Vid Kinetics | AJ: 49.6 -> 57.2 | Helped |
| fine-tune CoTracker using VGGT pretrained weights | CoTracker · TAP-Vid RGB-S | δvis avg: 78.9 -> 84.0 | Helped |
Surprising findings
- Combining independently predicted depth and camera parameters yields more accurate point clouds than the dedicated point map head (ETH3D Overall 0.677 vs 0.709).
- Predicting over-complete outputs (point maps, depth, cameras together) during training improves accuracy even though they are related by closed-form relations.
Limitations
- Feed-forward output still benefits from bundle adjustment post-processing, so accuracy is not fully achieved by the network alone.
- Novel view synthesis uses a smaller internal training set (about 20% of Objaverse) and is only competitive, not state-of-the-art.
- Dynamic tracking relies on fine-tuning CoTracker on Kubric, since VGGT is trained on unordered static image collections.
Experimental setup
Datasets: CO3Dv2 RealEstate10K DTU ETH3D ScanNet Kubric Objaverse GSO TAP-Vid
Benchmarks: CO3Dv2 RealEstate10K DTU ETH3D ScanNet-1500 GSO TAP-Vid
Metrics: AUC@30 Accuracy Completeness Overall AUC@5/10/20 PSNR SSIM LPIPS AJ delta_vis_avg
Baselines: DUSt3R MASt3R VGGSfM v2 PixSfM PoseDiff MV-DUSt3R CUT3R FLARE Fast3R LGM GS-LRM LVSM
Builds on: DUSt3R VGGSfM DINO CoTracker2 LVSM
Closest papers by meaning
- QVGGT: Post-Training Quantized Visual Geometry Grounded Transformer (CVPR 2026 · similarity 0.92)
- RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer (NeurIPS 2026 · similarity 0.91)
- TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (NeurIPS 2026 · similarity 0.90)
- LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging (CVPR 2026 · similarity 0.89)
- VGGSfM: Visual Geometry Grounded Deep Structure From Motion (CVPR 2024 · similarity 0.89)
- VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.