Research map / 3D vision / NeRF and radiance fields
Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decouples semantic (RGB) and spatial (Plücker ray) tokens in feedforward NVS transformers with Independent-V attention (shared Q/K, branch-specific values), plus optional iREPA/DA3 supervision and bidirectional modulation, improving LVSM on RealEstate10K and Objaverse.
Problem
Feedforward novel view synthesis transformers such as GS-LRM and LVSM mix RGB appearance and Plücker-ray spatial features in a shared token space, so spatial bias can interfere with appearance representation and degrade rendering fidelity.
Approach
Split each token into semantic (I) and spatial (P) halves with shared query-key attention routing but branch-specific value, output projections, layer norms and FFNs (Independent-V attention), then add optional branch-specific supervision (iREPA with DINOv3 for I, DA3-derived geometric correspondence consistency for P) and bidirectional FiLM-style modulation between branches.
Key results
- On RealEstate10K, the full decoupled LVSM variant reaches 27.21 PSNR vs 26.10 for the LVSM baseline (+1.1 dB) at 57.2M vs 85.8M params.
- Decoupling alone improves decoder-only LVSM from 26.10 to 26.70 PSNR with virtually zero added inference latency, and full model adds about 8% inference time.
- On Objaverse, the full variant reaches 26.46 PSNR vs 23.75 for LVSM.
Surprising findings
- Categorized supervision alone hurts the entangled baseline (26.10 to 26.02 PSNR) but helps the decoupled model, and modulation degrades the entangled model (25.93) while helping the decoupled one.
- Adding register tokens to the baseline does not remove grid-like artifacts or improve rendering quality.
Limitations
- Baseline numbers are controlled reimplementations under a reduced 50K-iteration budget and may differ from originally reported numbers.
- Wide-angle gains are inconsistent: SSIM and LPIPS improvements are less consistent under extreme viewpoints, and PSNR gain at 270° is small (+0.21 dB).
- Generative comparisons (GEN3C, Lyra) use adapted variants and are not directly comparable to published full-scale results.
Experimental setup
Datasets: RealEstate10K Objaverse DL3DV
Benchmarks: RealEstate10K Objaverse DL3DV
Metrics: PSNR SSIM LPIPS
Baselines: LVSM GS-LRM LagerNVS MVSplat PRoPE DepthSplat pixelSplat LGM
Builds on: LVSM iREPA Plücker ray maps register tokens
Closest papers by meaning
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.88)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.87)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.86)
- DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.86)
- Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis (NeurIPS 2026 · similarity 0.86)
- Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modeling (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.