Research map / Vision-language and multimodal / Multimodal LLMs
Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GST, an autoregressive transformer that tokenizes both images and Plücker-ray camera maps into one sequence, modeling their joint distribution to perform novel view synthesis and relative camera pose estimation in a unified model.
Problem
Spatial intelligence requires answering "Where am I?" (camera localization) and "What will I see?" (novel view prediction), which are usually treated as separate tasks with separate models.
Approach
Tokenize camera poses as Plücker-ray camera maps via a dedicated camera VQVAE, concatenate observation, image and camera tokens, and train one decoder-only transformer on the joint distribution p(i,c|o) with attention restricted to the observation, instead of alternating two conditional objectives.
Key results
- Novel view synthesis on Objaverse: GST achieves LPIPS 0.085 vs 0.135 for Zero-1-to-3, and SSIM 0.871 vs 0.845, though PSNR is lower (13.95 vs 14.77).
- CO3D relative rotation accuracy @15° on unseen categories: GST 85.1% vs Ray Diffusion 83.5%, but on seen categories GST 86.6% trails Ray Diffusion 91.8%.
- On DTU, joint training improves over alternating training: @15° 85.7 vs 84.6 and LPIPS 0.500 vs 0.554.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GST for novel view synthesis instead of Zero-1-to-3 | GST · Objaverse | LPIPS: 0.135 -> 0.085 | Helped |
| use GST for novel view synthesis instead of Zero-1-to-3 | GST · Objaverse | PSNR: 14.77 -> 13.95 | Hurt |
| use GST for novel view synthesis instead of Zero-1-to-3 | GST · Objaverse | SSIM: 0.845 -> 0.871 | Helped |
| use GST for relative pose estimation instead of Ray Diffusion | GST · CO3D, seen categories | Rotation accuracy @ 15°: 91.8 -> 86.6 | Hurt |
| use GST for relative pose estimation instead of Ray Diffusion | GST · CO3D, unseen categories | Rotation accuracy @ 15°: 83.5 -> 85.1 | Helped |
| train the joint distribution instead of alternating between two target distributions (ablation) | GST · DTU | Rotation accuracy @ 15°: 84.6 -> 85.7 | Helped |
| train the joint distribution instead of alternating between two target distributions (ablation) | GST · DTU | Rotation accuracy @ 30°: 92.8 -> 93.5 | Helped |
| train the joint distribution instead of alternating between two target distributions (ablation) | GST · DTU | LPIPS: 0.554 -> 0.500 | Helped |
| train the joint distribution instead of alternating between two target distributions (ablation) | GST · DTU | PSNR: 10.97 -> 11.90 | Helped |
| train the joint distribution instead of alternating between two target distributions (ablation) | GST · DTU | SSIM: 0.328 -> 0.358 | Helped |
| use a camera codebook of size 2048 and dimension 4 (ablation) | Camera tokenizer · Camera tokenizer | Codebook usage: 77.0% -> 90.4% | Helped |
| replace Plücker camera tokens with a 3-token Euler-angle representation (ablation) | GST · Objaverse subset | Novel view synthesis quality | Hurt |
Surprising findings
- Zero-1-to-3 achieves higher PSNR than GST despite GST's better LPIPS and SSIM, which the authors attribute to Zero-1-to-3 training on the full Objaverse including the test set.
Limitations
- Only single observation and single novel view scenario explored; multi-view sampling may have consistency issues.
- Trained on datasets without real-world scale; extension to metric-scale scenes is untested.
- Scarcity of multi-view datasets with precise camera annotations limits scaling.
Experimental setup
Datasets: Objaverse CO3D RealEstate10k MVImgNet DTU
Benchmarks: Objaverse test subset CO3D DTU
Metrics: LPIPS PSNR SSIM rotation accuracy @15° rotation accuracy @30°
Baselines: Zero-1-to-3 Zero-1-to-3 XL RelPose PoseDiffusion RelPose++ Ray Diffusion
Builds on: LlamaGen VQGAN Zero-1-to-3 Ray Diffusion CAT3D
Closest papers by meaning
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.89)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.88)
- EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligence (CVPR 2026 · similarity 0.88)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.88)
- SpaceVista: All-Scale Visual Spatial Reasoning from mm to km (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.