Research map / 3D vision / Gaussian splatting
ERUPT: Efficient Rendering with Unposed Patch Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ERUPT, a latent novel-view-synthesis transformer that renders patch-based rays from unposed images with a learned latent camera pose, and releases the MSVS-1M real-world street-view dataset.
Problem
Novel view synthesis from small sets of unposed RGB images is expensive (pixel-wise decoding) and needs precise camera poses for training targets, which existing latent scene models (SRT, RUST) struggle with.
Approach
Replace pixel-based ray queries with patch-based ray queries in the decoder (64 pixels per query), combine a learnable latent camera token with sparse ground-truth pose during training, and use a frozen DINOv2 feature extractor, optionally followed by GAN or Stable Diffusion rendering.
Key results
- ERUPT B+LORA reaches 24.69 PSNR / 0.749 SSIM on MSN with unposed inputs, outperforming UpSRT (23.03) and RUST (23.49); ERUPT L+LORA reaches 25.26 PSNR.
- ERUPT B+SD achieves FID 6.89 on MSN, better than DORSal's 9.00, and ERUPT B+GAN achieves FID 7.45.
- Patch-based decoding renders 224x224 images at over 600 fps; RUST runs out of memory at 224-pixel resolution on a 48GB A6000, while ERUPT-B uses roughly 64x less VRAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use patch-based decoding instead of pixel-based decoding | ERUPT · MSN, computational performance | decoder speed: an order of magnitude faster | Helped |
| Train with one target image instead of five (ablation) | ERUPT B · MSN | PSNR: 23.85 to 23.43 | Hurt |
| Replace the proposed scene transformer with a separate pose-estimation branch and simple token mixing (ablation) | ERUPT B · MSN | PSNR: 23.85 to 23.20 | Hurt |
| Fine-tune the DINOv2 feature extractor with LoRA (ablation) | ERUPT B · MSN | PSNR: 23.85 to 24.69 | Helped |
| Increase model size from B to L (ablation) | ERUPT · MSN | PSNR: 23.85 to 24.29 | Helped |
| Use ground-truth target poses for only 5% of training examples (ablation) | ERUPT B · MSN | PSNR: 23.85 to 23.55 | Hurt |
| Use Stable Diffusion for final rendering instead of the image decoder | ERUPT B · MSN | FID: 99.7 to 6.89 | Helped |
| Use Stable Diffusion for final rendering instead of LoRA rendering | ERUPT B · MSVS-1M | LPIPS: 0.703 to 0.371 | Helped |
| Use Stable Diffusion for final rendering instead of LoRA rendering | ERUPT B · MSVS-1M | PSNR: 20.64 to 17.10 | Hurt |
| Use GAN and perception losses for fine-tuning | ERUPT B · MSN | LPIPS: 0.360 to 0.204 | Helped |
| Use GAN and perception losses for fine-tuning | ERUPT B · MSVS-1M | LPIPS: 0.703 to 0.727 | Hurt |
Surprising findings
- Training with only 5% ground-truth target poses causes only a marginal PSNR drop (23.85 to 23.55) relative to the baseline.
Limitations
- Reliance on L2 objective yields blurry outputs under out-painting or low overlap.
- Diffusion-based rendering renders each view independently, causing cross-view inconsistency under limited input.
- GAN and LoRA results degrade notably on the real-world MSVS-1M data.
Experimental setup
Datasets: MSN MSVS-1M
Benchmarks: MSN MSVS-1M
Metrics: PSNR SSIM LPIPS FID
Baselines: SRT UpSRT RUST DORSal Patch RUST
Builds on: Scene Representation Transformer (SRT) RUST DINOv2 Stable Diffusion SwinIR
Closest papers by meaning
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
- Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections (NeurIPS 2024 · similarity 0.87)
- EVPGS: Enhanced View Prior Guidance for Splatting-based Extrapolated View Synthesis (CVPR 2025 · similarity 0.87)
- pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (CVPR 2024 · similarity 0.87)
- SplatFormer: Point Transformer for Robust 3D Gaussian Splatting (ICLR 2025 · similarity 0.87)
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.