Research map / Image generation / Autoregressive and latent generation
EditAR: Unified Conditional Generation with Autoregressive Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EditAR, a single autoregressive transformer (built on LlamaGen) that takes image and text conditions to perform image editing and canny/depth/segmentation-to-image translation, with a DINOv2 feature distillation loss.
Problem
Building a single unified model for diverse conditional image generation and editing tasks, since diffusion models need task-specific designs and autoregressive models had not been shown effective for image editing on large benchmarks.
Approach
Prefill a text-to-image autoregressive model with conditioning image tokens (with separate positional embeddings) and text instructions, train with next-token cross-entropy plus an MSE distillation loss aligning hidden features to DINOv2, and use classifier-free guidance on both image and text conditions.
Key results
- On PIE-Bench, EditAR achieves the best overall results among feed-forward methods, with whole CLIP similarity 24.87 and structure distance 39.43 versus MGIE's 67.41.
- On MultiGen depth-to-image, EditAR reaches FID 15.97, the best FID among compared methods (ControlNet++ 16.66 with RMSE 28.32 best).
- Segmentation-to-image FID 16.13 vs ControlNet++ 19.29; canny-edge FID 13.91 vs T2I-Adapter 15.96.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DINOv2 feature distillation (ablation) | EditAR · Image editing | Text-image alignment | Helped |
| use DINOv2 with distillation coefficient λdistill = 0.5 (ablation) | EditAR · Image editing | Overall performance | Helped |
| remove the distillation loss (ablation) | EditAR · Image editing | Text-image alignment | Hurt |
| use DINOv2 distillation coefficient λdistill = 1.0 instead of 0.5 (ablation) | EditAR · Image editing | Overall performance | Hurt |
| replace DINOv2 with CLIP for distillation (ablation) | EditAR · Image editing | Text-image alignment | Mixed |
| set classifier-free guidance strength η=1.0 (ablation) | EditAR · Image editing | Text-image alignment | Hurt |
| set classifier-free guidance strength η=7.5 (ablation) | EditAR · Image editing | Reconstruction quality | Hurt |
| set classifier-free guidance strength η=3.0 (ablation) | EditAR · Image editing | Balance between alignment and reconstruction | Helped |
Surprising findings
- On depth-to-image, EditAR has a worse RMSE (34.93) than ControlNet++ (28.32) despite its best FID.
Limitations
- Semantics in segmentation-to-image may not align with ground truth
- Inversion-based methods still achieve higher background preservation on PIE-Bench
- Depth-map RMSE trails ControlNet++
Experimental setup
Datasets: SEED-Data-Edit-Unsplash PIPE COCOStuff MultiGen-20M PIE-Bench
Benchmarks: PIE-Bench COCOStuff MultiGen-20M
Metrics: PSNR LPIPS MSE SSIM CLIP similarity structure distance RMSE FID mIOU
Baselines: InstructPix2Pix MGIE Prompt-to-Prompt PnPInversion Null-text Inversion ControlNet ControlNet++ UniControl UniControlNet T2I-Adapter
Builds on: LlamaGen VQVAE DINOv2 ControlNet
Closest papers by meaning
- ControlAR: Controllable Image Generation with Autoregressive Models (ICLR 2025 · similarity 0.92)
- NEP: Autoregressive Image Editing via Next Editing Token Prediction (NeurIPS 2025 · similarity 0.90)
- The Promise of RL for Autoregressive Image Editing (NeurIPS 2025 · similarity 0.89)
- ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR 2024 · similarity 0.89)
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.89)
- OmniGen-AR: AutoRegressive Any-to-Image Generation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.