Research map / Image generation / Autoregressive and latent generation
LaVin-DiT: Large Vision Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaVin-DiT, a unified generative vision foundation model that combines a spatial-temporal VAE with a joint diffusion transformer and in-context learning to handle over 20 vision tasks without fine-tuning.
Problem
Large vision models built on autoregressive visual-sentence sequence modeling are computationally inefficient for high-dimensional visual data and disrupt spatial relationships; a unified generalist vision model is needed.
Approach
Replace autoregressive token prediction with a flow-matching diffusion transformer operating on ST-VAE latents, using full-sequence joint attention between task-context (clean) and noisy target latents and 3D RoPE, so that input-target example pairs act as in-context task definitions.
Key results
- On NYU-v2 depth estimation, LaVin-DiT (3.4B) reaches AbsRel 6.2 vs 30.2 for LVM 7B.
- ImageNet inpainting FID 1.65 for LaVin-DiT vs 4.05 for LVM.
- Inference latency at 256 resolution is 4.67s vs 8.1s for LVM (7B), and 20.1s vs 47.2s at 512.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LaVin-DiT instead of LVM | LaVin-DiT · Foreground segmentation and single object detection | mIoU | Helped |
| Use LaVin-DiT instead of LVM | LaVin-DiT · Colorization | MSE: 0.51 -> 0.24 | Helped |
| Use LaVin-DiT instead of LVM | LaVin-DiT · Colorization | LPIPS: 0.46 -> 0.26 | Helped |
| Use LaVin-DiT instead of LVM | LaVin-DiT · ImageNet-1K inpainting | FID: 4.05 -> 1.65 | Helped |
| Use LaVin-DiT instead of Marigold | LaVin-DiT · NYU-v2 depth estimation | AbsRel: 6.0 -> 6.2 | Hurt |
| Use LaVin-DiT instead of Marigold | LaVin-DiT · NYU-v2 depth estimation | δ1: 95.9 -> 96.1 | Helped |
| Use LaVin-DiT instead of StableNormal | LaVin-DiT · NYU-v2 surface normal estimation | MAE: 19.707 -> 15.901 | Helped |
| Use LaVin-DiT instead of StableNormal | LaVin-DiT · NYU-v2 surface normal estimation | < 11.25°: 53.042 -> 58.382 | Helped |
| Increase model size from 0.1B to 3.4B (ablation) | LaVin-DiT · Colorization | MSE: 0.609 -> 0.273 | Helped |
| Increase model size from 0.1B to 3.4B (ablation) | LaVin-DiT · Depth estimation | AbsRel: 7.6 -> 6.2 | Helped |
| Increase task context length (ablation) | LaVin-DiT · Depth-to-image generation | FID | Helped |
| Increase task context length (ablation) | LaVin-DiT · De-motion blur | PSNR | Helped |
Surprising findings
- The paper reports that LVM's official 7B checkpoint performs much worse than LaVin-DiT on depth estimation (AbsRel 30.2 vs 6.2), with the authors noting LVM did not report such results originally.
Limitations
- Struggles to generalize when task definitions deviate significantly from the training distribution.
- Limited by large-scale training data, diverse task annotations, and compute relative to LLMs.
Experimental setup
Datasets: ImageNet-1K multi-task vision dataset (3.2M images, 0.6M videos)
Benchmarks: NYU-v2 ImageNet-1K inpainting colorization foreground segmentation single object detection
Metrics: mIoU LPIPS MSE AbsRel delta1 MAE FID PSNR
Baselines: LVM MAE-VQGAN MAE DPT Marigold StableNormal
Builds on: Diffusion Transformer (DiT) MM-DiT LVM Painter flow matching
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.90)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.90)
- LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer (ICML 2026 · similarity 0.90)
- Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations (NeurIPS 2025 · similarity 0.89)
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.89)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.