Research map / Image generation / Text-to-image diffusion
LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stitches reference and masked target views side by side and fine-tunes only prompt embeddings (plus block-causal attention for NVS) on inpainting Stable Diffusion for reference-guided inpainting and novel view synthesis.
Problem
Reference-guided image synthesis (reference-based inpainting and single/multi-view novel view synthesis) typically requires extra image encoders, adapters, or heavy full fine-tuning of text-to-image diffusion models.
Approach
Reformulate reference-guided synthesis as contextual inpainting by placing the reference on the left and the masked target on the right of one canvas, control the model via learned task and view prompt embeddings in cross-attention, reshape features for multi-view self-attention, and use a block causal mask for autoregressive novel view generation.
Key results
- On MegaDepth Ref-inpainting, LeftRefill (1-view) reaches 20.926 PSNR and 18.680 FID vs. 22.744 PSNR and 26.291 FID for TransFill; 4-view reaches 21.779 PSNR and 16.632 FID.
- On Objaverse 1-view NVS, LeftRefill achieves 21.404 PSNR and 0.1151 LPIPS vs. 19.402 PSNR and 0.1309 LPIPS for Zero123.
- Only 0.05M to 0.065M trainable prompt parameters are added for Ref-inpainting with the frozen LDM backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stitch the reference and target spatially (ablation) | LeftRefill · MegaDepth test, 1-view Ref-inpainting | PSNR: 20.489 -> 20.926 | Helped |
| add reference views, from 1 to 4 (ablation) | LeftRefill · MegaDepth test, Ref-inpainting | PSNR: 20.926 -> 21.779 | Helped |
| add adaptive masking (ablation) | LeftRefill · Objaverse, 1-view NVS | PSNR: 21.675 -> 21.404 | Hurt |
| fine-tune with LoRA instead of the full LeftRefill setup (ablation) | LeftRefill · Objaverse, 1-view NVS | PSNR: 19.514 vs 21.404 | Hurt |
| use autoregressive generation (ablation) | LeftRefill · Objaverse, 4-view sequential NVS | P-CLIP: 0.7709 -> 0.7958 | Helped |
| use deep prompt tuning at 16 cross-attention layers rather than shallow tuning (ablation) | LeftRefill · MegaDepth test, Ref-inpainting | PSNR: 20.49 -> 20.15 | Hurt |
| increase shallow prompt length from 25 to 50 tokens (ablation) | LeftRefill · MegaDepth test, Ref-inpainting | PSNR: 20.35 -> 20.49 | Helped |
| increase shallow prompt length from 50 to 75 tokens (ablation) | LeftRefill · MegaDepth test, Ref-inpainting | PSNR: 20.49 -> 20.38 | Hurt |
| allocate 5 view tokens instead of 0, retaining 45 task tokens (ablation) | LeftRefill · MegaDepth test, 3-view Ref-inpainting | PSNR: 21.224 -> 21.356 | Helped |
| increase the view-token share from 5 to 25 (ablation) | LeftRefill · MegaDepth test, 3-view Ref-inpainting | PSNR: 21.356 -> 21.127 | Hurt |
| add incremental positional encoding (ablation) | LeftRefill-simple · Objaverse, 1-reference NVS | PSNR: 20.352 -> 20.508 | Helped |
| add incremental positional encoding (ablation) | LeftRefill-simple · Objaverse, 4-reference NVS | PSNR: 22.097 -> 22.324 | Helped |
Surprising findings
- Prompt tuning alone on a frozen backbone is sufficient for complex Ref-inpainting, despite only 0.05M-0.065M trainable parameters.
- Deep prompt tuning across 16 cross-attention layers did not outperform shallow prompt tuning and may overfit.
Limitations
- NVS requires full fine-tuning of the LDM to achieve precise pose control, unlike the frozen Ref-inpainting setup.
- Relies on a fixed view count per trained model for multi-view Ref-inpainting (trained separately for 1-4 views).
Experimental setup
Datasets: MegaDepth ETH3D Objaverse Google Scanned Objects
Benchmarks: MegaDepth ETH3D Objaverse Google Scanned Objects
Metrics: PSNR SSIM FID LPIPS CLIP score pairwise CLIP score
Baselines: SD (inpainting) ControlNet Perceiver+ImageCLIP Paint-by-Example TransFill Zero123
Builds on: Stable Diffusion inpainting Zero123 Visual prompt tuning ControlNet TransFill
Closest papers by meaning
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.87)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.86)
- Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects (CVPR 2025 · similarity 0.86)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.85)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.85)
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.