Research map / 3D vision / 3D generation

A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

NeurIPS 2026 · 3D vision › 3D generation · approach: Score distillation for text-to-3D

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 3DRAE, which repurposes frozen 2D encoders (DINOv2, DA3, SigLIP2) into a fixed-length 3D latent tokenizer for scenes, and 3DDiT, a flow-matching diffusion transformer that generates scenes in this view-decoupled 3D latent space.

Problem

2D multi-view and video diffusion approaches to scene-level 3D generation carry representation redundancy across views and limited 3D spatial consistency, since they generate in view-coupled 2D latent spaces.

Approach

Lift frozen 2D representation features into a fixed-length set of implicit 3D latent tokens via a latent fuse neck and ray-map-queried decoder trained with novel-view and point-map supervision, then run flow-matching diffusion directly in that 3D latent space with flexible conditioning via masked views.

Key results

• On 1-view DL3DV, 3DDiT reaches FID 40.93 vs. 51.68 for the image-only variant and SEVA's 45.96, with ATE_t 0.025 vs. SEVA's 0.029.
• 3DDiT renders 16 views in about 1.98s versus roughly 85s for SEVA and 200s for ViewCrafter, while 3DRAE runs in about 0.23s.
• 3DRAE with DA3 achieves the best 6-view reconstruction (DL3DV PSNR 21.62), while DINOv2/SigLIP2 variants generate better 3D scenes; 3DRAE represents 128-view scenes with 1024 tokens without reported degradation.

Surprising findings

• Semantic encoders (DINOv2, SigLIP2) yield worse reconstruction than even the image-only baseline but substantially better generation quality.
• The 3DRAE with DA3 gives the best reconstruction, but its generation performance is worse than DINOv2 for diffusion.

Limitations

• 3DDiT is not designed for faithful reconstruction and can deviate from sparse conditioning views due to denoising stochasticity.
• Single-view 1-view generation makes 3DRAE alone fail to extrapolate to unobserved regions; generation relies on the diffusion model.
• Benchmark is relatively small (358 scenes) and the training data is proprietary-curated across 17 datasets with no release statement.

Experimental setup

Datasets: DL3DV RE10K CO3D ScanNet WildRGBD ARKitScenes BlendedMVG Waymo ImageNet ScanNet++ Taskonomy HM3D

Benchmarks: DL3DV RE10K CO3D ScanNet Tanks and Temples Mip-NeRF 360 DTU

Metrics: PSNR LPIPS FID ATE_r ATE_t runtime

Baselines: DepthSplat LVSM MotionCtrl ViewCrafter SEVA

Builds on: Representation Autoencoder (RAE) LightningDiT DDT head VGGT DINOv2

Closest papers by meaning

• LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.94)
• Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation (CVPR 2025 · similarity 0.91)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.90)
• Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.90)
• Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.90)
• MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)