Research map / Image generation / Autoregressive and latent generation

UltraDiff:Transferring High-Fidelity Priors to Compressed Latent Spaces for High-resolution Image Generation

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: High-resolution diffusion synthesis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UltraDiff, which trains a dual-branch f8/f32 MixTeacher DiT with shared attention and distills its f32-branch vector field into a standalone f32 DiT student for efficient 2K–4K text-to-image generation.

Problem

Latent diffusion at high resolution trades compute against fidelity: aggressive f32 latent compression cuts tokens but yields ambiguous targets and weaker detail and prompt adherence than f8 models.

Approach

Model the joint f8–f32 latent distribution with a dual-branch DiT whose branches share self-attention, then distill the teacher's f32-branch velocity field directly into a single-branch f32 student, avoiding cross-resolution feature alignment, and initialize from a pretrained f8 model.

Key results

• The f32 student reaches 0.72 GenEval versus 0.61 for f8-initialized f32 and 0.73 for the f8 anchor at 512 resolution (Table 1).
• At 2K, UltraDiff generates an image in 6.2 s with 20 steps on one A100, a 40.3x speedup over the f8 anchor (249.8 s), while the f8 anchor runs out of memory at 4K.
• On DPG at 1K, UltraDiff scores 85.40 versus 74.02 for the no-distillation control; distillation vs DINOv2/f8-feature alignment gives GenEval 0.72 vs 0.33/0.31 (Table 4).

Surprising findings

• Feature alignment to DINOv2 or to f8 features hurt performance (GenEval 0.33 and 0.31), well below the no-distillation baseline of 0.61.
• Mixed-mode f32 branch outputs of MixTeacher are visually indistinguishable from the f8 branch, suggesting cross-space attention resolves ambiguity.

Limitations

• Evaluated on only one autoencoder-DiT pipeline and one proprietary DiT family; no open-source teacher validation.
• Distillation loss is applied only up to 1K resolution; 2K–4K relies on flow matching alone because high-resolution teachers are unreliable or expensive.
• Training data and f8 anchor are proprietary, limiting reproduction; cross-system comparisons differ in many factors.

Experimental setup

Datasets: proprietary text-image corpus (~0.37B pairs) MJHQ-30k Aesthetic-4K

Benchmarks: GenEval DPG-Bench MJHQ-3K HPSv3 FID on Aesthetic-4K pFID CLIP score ImageReward

Metrics: GenEval DPG CLIP score ImageReward HPSv3 FID pFID aesthetic score latency per step

Baselines: UltraPixel Diffusion-4K Sana HunyuanImage-2.1 SDXL PixArt-Sigma SD3-medium FLUX-dev FLUX-schnell PlayGroundv2.5

Builds on: flow matching REPA latent diffusion models T5-XXL

Closest papers by meaning

• PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (NeurIPS 2026 · similarity 0.89)
• FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
• OSCAR: One-Step Diffusion Codec Across Multiple Bit-rates (NeurIPS 2025 · similarity 0.89)
• Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.89)
• Diffusion Models Without Attention (CVPR 2024 · similarity 0.89)
• UltraPixel: Advancing Ultra High-Resolution Image Synthesis to New Peaks (NeurIPS 2024 · similarity 0.88)