Research map / 3D vision / 3D generation
L-MAGIC: Language Model Assisted Generation of Images with Coherence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates coherent 360° panoramas from a single image by using BLIP-2 and ChatGPT-4 to plan per-view layout prompts and avoid duplicate objects during zero-shot warping-and-inpainting with Stable Diffusion v2.
Problem
Generating coherent 360° panoramic scenes from a single perspective image, where multi-view diffusion inpainting produces duplicated objects and incoherent global layouts due to lack of scene layout priors.
Approach
Use pre-trained vision-language (BLIP-2) and language (ChatGPT-4) models without fine-tuning to generate per-view layout descriptions, scene-level prompts, and repeated-object lists that steer iterative warping-and-inpainting, with VLM checks to re-run inpainting when banned objects reappear; add super-resolution and weighted multi-view fusion.
Key results
- Human evaluators (15 voters) preferred L-MAGIC over baselines with >70% preference in the reported comparisons on quality and layout.
- Ablation: removing ChatGPT layout prior (No GPT) and removing all prompts (No prompt) degrade scene layout, with no prompt giving the largest degradation in both quality and layout.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L-MAGIC's language-model guidance for multi-view inpainting | L-MAGIC · Image-to-panorama and text-to-panorama | human-rated quality and scene layout | Helped |
| Add ChatGPT global layout prior (ablation) | L-MAGIC · Image-to-panorama | scene layout | Helped |
| Add text guidance (ablation) | L-MAGIC · Image-to-panorama | scene layout and perspective-view rendering quality | Helped |
| Use Stable Diffusion v2 inpainting instead of Blended Latent Diffusion inpainting (ablation) | L-MAGIC · Image-to-panorama | human-rated scene layout and rendering quality | Helped |
| Use Stable Diffusion v2 inpainting instead of Deep Floyd inpainting (ablation) | L-MAGIC · Image-to-panorama | scene layout and rendering quality | Helped |
| Use Stable Diffusion v2 inpainting instead of Stable Diffusion XL inpainting (ablation) | L-MAGIC · Image-to-panorama | scene layout and rendering quality | Helped |
| Add super-resolution (ablation) | L-MAGIC · Image-to-panorama | perspective-view rendering quality | Helped |
| Add smoothing during multi-view fusion (ablation) | L-MAGIC · Image-to-panorama | perspective-view rendering quality | Helped |
Surprising findings
- Inception Score can be misleading: some inpainting baselines (e.g., BLD inpaint) get higher IS than L-MAGIC despite worse human-rated quality, due to adversarial-like samples.
Limitations
- Generation time of 2-5 minutes per panorama due to repeated inpainting and VLM checks.
- Reliance on monocular depth estimators introduces artifacts in 3D point clouds.
- Inception Score does not capture human-perceived quality.
Experimental setup
Datasets: Tanks and Temples RealEstate10K
Benchmarks: Tanks and Temples RealEstate10K
Metrics: Inception Score human preference score
Baselines: Stable Diffusion v2 Text2room MVDiffusion Text2light LDM3D Blended Latent Diffusion DeepFloyd IF inpainting SDXL inpainting
Builds on: Stable Diffusion v2 inpainting Text2room MVDiffusion BLIP-2
Closest papers by meaning
- DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion (NeurIPS 2024 · similarity 0.87)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.86)
- Conditional Panoramic Image Generation via Masked Autoregressive Modeling (NeurIPS 2025 · similarity 0.85)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.85)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.85)
- ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.