Research map / Image generation / Text-to-image diffusion
TOSS: High-quality Text-guided Novel View Synthesis from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TOSS, a text-guided single-image novel view synthesis model that fine-tunes Stable Diffusion with dense cross-attention image conditioning, frequency-encoded pose tokens, and expert denoisers, outperforming Zero123 on GSO/RTMV.
Problem
Single-view novel view synthesis is severely under-constrained, so generated views are often semantically implausible and offer users no explicit control over occluded parts.
Approach
Use text as high-level semantic constraint on NVS by initializing from text-to-image Stable Diffusion rather than an image-to-image model; condition on the input view via dense VAE-feature cross-attention, encode relative camera pose as an extra frequency-encoded key-value pair, and train two timestep-specialized expert denoisers.
Key results
- On GSO with 250M training images, TOSS with text and expert denoisers reaches PSNR 20.16 vs. 18.67 for Zero1-to-3 and LPIPS 0.1109 vs. 0.1257.
- On RTMV, TOSS (w/ expert denoisers) achieves PSNR 11.62 and LPIPS 0.3115 vs. Zero1-to-3's 10.28 and 0.3592.
- For 3D reconstruction on GSO, Chamfer Distance drops from 0.0757 (Zero123) to 0.0655 (TOSS), and IoU rises from 0.5151 to 0.5205.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TOSS with text guidance instead of Zero123 | single-view NVS · GSO | PSNR: 18.67 -> 20.16 | Helped |
| use TOSS with text guidance instead of Zero123 | single-view NVS · RTMV | PSNR: 10.28 -> 11.62 | Helped |
| use TOSS with text guidance instead of Zero123 | 3D consistency evaluation · GSO | PSNR: 21.05 -> 21.54 | Helped |
| use TOSS with text guidance instead of Zero123 | 3D consistency evaluation · RTMV | PSNR: 11.38 -> 12.36 | Helped |
| use TOSS with text guidance instead of Zero123 | 3D reconstruction · GSO | CD: 0.0757 -> 0.0655 | Helped |
| use TOSS with text guidance instead of Zero123 | 3D reconstruction · RTMV | CD: 0.1546 -> 0.1236 | Helped |
| replace the original image conditioning mechanisms with dense cross attention (ablation) | Zero123 · Objaverse vehicles | PSNR: 12.73 -> 15.43 (+2.70) | Helped |
| add text conditioning (ablation) | Zero123 + token-level attention · Objaverse vehicles | PSNR: 15.43 -> 16.84 (+1.41) | Helped |
| add a camera pose token (ablation) | Zero123 + token-level attention + text prompt · Objaverse vehicles | PSNR: 16.84 -> 16.95 (+0.11) | Helped |
| add a camera pose token (ablation) | Zero123 + token-level attention + text prompt · Objaverse vehicles | SSIM: 0.8275 -> 0.8369 (+0.0094) | Helped |
| refine text prompts with more detailed descriptions (ablation) | TOSS · GSO subset | PSNR: +0.1 | Helped |
Surprising findings
- Training from scratch rather than from Stable Diffusion weights produces significant shape and texture distortions, underscoring the value of text-to-image pretraining.
Limitations
- Relies on automatically generated captions and optional text descriptions; caption models are not tailored to object details for NVS.
- Fine-tuned on synthetic 3D renderings, which can cause distribution shift to natural images.
Experimental setup
Datasets: Objaverse Cap3D captions GSO RTMV
Benchmarks: GSO RTMV Objaverse vehicles
Metrics: PSNR SSIM LPIPS KID Chamfer Distance Volume IoU
Baselines: Zero123
Builds on: Zero123 Stable Diffusion Cap3D threestudio
Closest papers by meaning
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.88)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.87)
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.86)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.86)
- CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning (ICML 2025 · similarity 0.86)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.