Research map / Image generation / Autoregressive and latent generation
RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
RepLDM reprograms pretrained SDXL for training-free high-resolution generation via a training-free self-attention guidance stage followed by progressive pixel-space upsampling, reporting better FID/IS and about 5x faster inference than DemoFusion and AccDiffusion.
Problem
Latent diffusion models produce structurally distorted images when sampled above their training resolution, and existing training-free high-resolution methods either yield poor quality (sliding window, parameter rectification) or are slow (progressive latent upsampling).
Approach
A two-stage training-free pipeline: first, a parameter-free self-attention (TFSA, with Q=K=V shared and no projections) linearly blended with the latent guides structure at training resolution with a cosine-decayed, delayed guidance scale; second, progressive upsampling is performed in pixel space (decode, upsample, re-encode, partial re-denoise) with few denoising steps, avoiding latent-space ups
Key results
- At 4096x4096, RepLDM reports FID 90.6 versus 101.4 for DemoFusion and 103.2 for AccDiffusion (FID, lower is better).
- On a single 3090 GPU at 4096x4096, RepLDM takes 5.7 minutes versus 25.0 for DemoFusion and 26.0 for AccDiffusion, roughly a 5x speedup.
- In a 16-volunteer user study, RepLDM scored highest on structural consistency (7.42 vs 6.28 for AccDiffusion) and detail richness (7.41 vs 6.18).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention guidance (ablation) | RepLDM · 2048 × 2048; 2048 × 4096; 4096 × 2048; 4096 × 4096 | FID: 66.8 -> 66.0; 91.6 -> 89.0; 95.3 -> 93.2; 92.0 -> 90.6 | Helped |
| use guidance scale γ=0.004 (ablation) | RepLDM · 1024 × 1024, 1600 × 1600, and 2048 × 2048 | FID: 89.40; 89.91; 90.11 | Helped |
| delay attention guidance with η₁=0.06 (ablation) | RepLDM · 1024 × 1024, 1600 × 1600, and 2048 × 2048 | FID: 89.44; 89.91; 90.11 | Mixed |
| use progressive scheduler η₂=[0.2] instead of larger scheduler values (ablation) | RepLDM · 1600 × 1600 and 2048 × 2048 | FID: 93.09; 92.71 | Mixed |
| replace cosine guidance-scale decay with linear or exponential decay (ablation) | RepLDM · 2048 × 2048 | FID: 66.0; 66.2; 66.8 | No clear effect |
| replace TFSA with asymmetric TFSA-A using pooled keys and values (ablation) | RepLDM · 2048 × 2048 | FID: 66.0 -> 67.4 | Hurt |
| apply attention guidance during late denoising steps (16 to 1) rather than early-to-middle steps (ablation) | RepLDM · Image generation | visual clarity and detail | No clear effect |
| use pixel-space rather than latent-space upsampling (ablation) | RepLDM · High-resolution image generation | image quality | Helped |
| reduce high-resolution denoising steps using progressive upsampling and first-stage initialization | RepLDM · 4096 × 4096 | generation time (seconds): 1684 -> 343 | Helped |
| use RepLDM rather than the strongest FID baseline | SDXL-based high-resolution generation · 2048 × 2048; 2048 × 4096; 4096 × 2048; 4096 × 4096 | FID: 71.6 -> 66.0; 95.5 -> 89.0; 99.6 -> 93.2; 101.4 -> 90.6 | Helped |
| use RepLDM rather than DemoFusion or AccDiffusion | SDXL-based high-resolution generation · User study | structural consistency, color abundance, and detail richness scores: 7.42; 7.64; 7.41 | Helped |
| use a larger guidance scale (ablation) | RepLDM · Image generation | detail richness | Helped |
Surprising findings
- Attention guidance changes FID and IS only marginally, while human raters notice clearer improvements, so standard distribution metrics understate the gain.
- HiDiffusion is faster than RepLDM at 2048x4096 and 4096x4096 in the reported timing table (1.9 and 3.4 minutes vs 2.0 and 5.7).
Limitations
- Fails to correct text rendering errors inherited from SDXL.
- Ultra-high resolutions (e.g., 12800x12800) require more sub-stages, increasing inference time.
- Attention guidance has limited effect on standard FID/IS metrics.
Experimental setup
Datasets: ImageNet SA-1B (SAM dataset)
Benchmarks: SAM dataset 33k image prompts subset
Metrics: FID IS FID_c IS_c CLIP score inference time user study scores
Baselines: SDXL MultiDiffusion ScaleCrafter DemoFusion Upsample Guidance AccDiffusion HiDiffusion
Builds on: DemoFusion SDXL classifier-free guidance FreeU
Closest papers by meaning
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.91)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.90)
- ZoomLDM: Latent Diffusion Model for Multi-scale Image Generation (CVPR 2025 · similarity 0.90)
- ResMaster: Mastering High-Resolution Image Generation via Structural and Fine-Grained Guidance (AAAI 2025 · similarity 0.89)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.89)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.