Research map / 3D vision / 3D generation
MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MegaSynth, a 700K-scene procedurally generated non-semantic 3D dataset, and shows that training feed-forward scene LRMs (GS-LRM, Long-LRM) jointly with or pre-trained on it improves reconstruction by 1.2-1.8 dB PSNR.
Problem
Feed-forward large reconstruction models for wide-coverage 3D scenes are limited by scarce, low-diversity real scene data (DL3DV has ~10K scenes), while realistic scene generation requires costly semantic modeling.
Approach
Generate scenes procedurally without semantic content (floor plans of object boxes filled with random non-semantic primitives, random textures and lighting, controlled camera sampling), arguing that scene semantics matter little for multi-view reconstruction, so the data can scale cheaply and provides exact geometry metadata for a depth-based loss.
Key results
- Joint training or pre-training with MegaSynth improves PSNR by 1.2 to 1.8 dB over DL3DV-only training across in-domain and out-of-domain test sets (Table 1).
- Long-LRM trained with DL3DV+MegaSynth reaches 25.44 PSNR on DL3DV at resolution 128 vs 24.18 without MegaSynth; GS-LRM reaches 25.75 vs 24.60.
- MegaSynth-only training yields 21.50 PSNR on Hypersim vs 23.89 for real-only, showing comparable-order performance without semantics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MegaSynth pre-training before DL3DV training | GS-LRM · DL3DV, Hypersim, and MipNeRF360 & TT (resolution 128) | PSNR: 24.60 -> 25.75; 23.89 -> 25.46; 19.93 -> 21.19 | Helped |
| Add MegaSynth pre-training before DL3DV training | Long-LRM · DL3DV, Hypersim, and MipNeRF360 & TT (resolution 256) | PSNR: 23.71 -> 25.14; 22.51 -> 24.26; 18.61 -> 19.84 | Helped |
| Add data-distribution control to MegaSynth generation (ablation) | GS-LRM trained with MegaSynth, then fine-tuned on DL3DV · Hypersim, resolution 128 | PSNR: 17.18 -> 18.71 | Helped |
| Add geometry supervision loss (ablation) | GS-LRM trained with controlled MegaSynth · Hypersim, resolution 128 | PSNR: 18.71 -> 20.72 | Helped |
| Scale controlled MegaSynth training data from 100K to 700K examples (ablation) | GS-LRM trained with geometry supervision · Hypersim, resolution 128 | PSNR: 20.72 -> 21.07 | Helped |
| Train on MegaSynth only instead of DL3DV real data (ablation) | GS-LRM · Hypersim, resolution 128 | PSNR: 23.89 -> 21.50 | Hurt |
| Jointly train on MegaSynth and DL3DV instead of training on DL3DV only (ablation) | GS-LRM · Hypersim, resolution 128 | PSNR: 23.89 -> 25.33 | Helped |
| Pre-train on MegaSynth, then fine-tune on DL3DV instead of training on DL3DV only (ablation) | GS-LRM · Hypersim, resolution 128 | PSNR: 23.89 -> 25.46 | Helped |
| Use joint training instead of MegaSynth pre-training followed by DL3DV fine-tuning (ablation) | GS-LRM trained on MegaSynth and DL3DV · Hypersim, resolution 128 | PSNR: 25.33 vs. 25.46 | No clear effect |
| Add Kubric training data | GS-LRM trained on DL3DV · DL3DV, 8 input views, resolution 128 | PSNR: 18.31 -> 18.28 | No clear effect |
| Add Front3D training data | GS-LRM trained on DL3DV · DL3DV, 8 input views, resolution 128 | PSNR: 18.31 -> 18.40 | No clear effect |
| Add MegaSynth training data | GS-LRM trained on DL3DV · DL3DV, 8 input views, resolution 128 | PSNR: 18.31 -> 19.58 | Helped |
Surprising findings
- Unregulated synthetic data (without data control) lowers performance below real-data-only training and causes training instability.
- Realistic synthetic datasets Kubric and Front3D do not improve LRM performance, while MegaSynth does, suggesting realism or scene composition is not what drives reconstruction gains.
Limitations
- MegaSynth-only models underperform on complicated geometry patterns and large scene scales.
- Performance gap to real data persists; MegaSynth is primarily indoor and lacks outdoor characteristics.
- Training instability arises from high data randomness unless geometry supervision and data control are used.
Experimental setup
Datasets: MegaSynth DL3DV Hypersim MipNeRF360 Tanks and Temples Kubric Front3D
Benchmarks: DL3DV Hypersim MipNeRF360 Tanks and Temples
Metrics: PSNR SSIM LPIPS AbsRel delta1
Baselines: 3DGS Long-LRM GS-LRM Kubric Front3D Depth Anything V2
Builds on: GS-LRM Long-LRM 3D Gaussian Splatting LRM-Zero
Closest papers by meaning
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.89)
- LRM: Large Reconstruction Model for Single Image to 3D (ICLR 2024 · similarity 0.88)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.88)
- Large Spatial Model: End-to-end Unposed Images to Semantic 3D (NeurIPS 2024 · similarity 0.87)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.87)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.