Research map / Image generation / Text-to-image diffusion
ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACE, a Diffusion Transformer that unifies many visual generation and editing tasks via a Long-context Condition Unit input format, trained on 0.7B synthesized and mined image pairs with MLLM-labeled instructions.
Problem
Foundational diffusion models are mostly text-only and cannot handle multi-modal conditions for many visual editing and creation tasks, and training data for such unified tasks is scarce.
Approach
A unified condition format (Condition Unit, extended to a Long-context Condition Unit with history) combined with a DiT using image indicator embeddings, 3D RoPE, and per-condition-unit cross-attention, plus a scalable data pipeline (synthesis and clustering-based pairing) with a fine-tuned MLLM Instruction Captioner for instruction labeling.
Key results
- On MagicBrush single-turn instruction-guided editing, ACE reaches L1 0.0507 and CLIP-T 0.2841, beating MagicBrush and UltraEdit on most metrics.
- Human study with 5 designers: ACE has highest prompt-following score in 7 of 12 global and 8 of 10 local editing tasks.
- ACE (0.6B parameters, around 512 resolution) is reported to underperform SDXL-based expert methods on image quality for some tasks such as general and style editing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add long-context history (ablation) | ACE · MagicBrush multi-turn, global description-guided | L1: 0.0778 -> 0.0768 | Helped |
| add long-context history (ablation) | ACE · MagicBrush multi-turn, global description-guided | CLIP-T: 0.2843 -> 0.2819 | Hurt |
| add long-context history (ablation) | ACE · MagicBrush multi-turn, instruction-guided | L1: 0.0773 -> 0.0761 | Helped |
| add long-context history (ablation) | ACE · MagicBrush multi-turn, instruction-guided | CLIP-T: 0.2855 -> 0.2809 | Hurt |
Surprising findings
- Multi-turn long-context (LC) conditioning gives only marginal gains over independent multi-turn editing on MagicBrush (e.g., L1 0.0768 vs 0.0778).
Limitations
- Model is small (0.6B, ~512 resolution), which hurts image quality versus larger SDXL-based experts.
- Training dataset is internal and not released, limiting reproducibility.
- Human evaluation uses only 5 designers; general and style editing are weaker.
Experimental setup
Datasets: LAION-5B OpenImages MS-COCO JourneyDB AnyWord3M LaionGlyph-10M MagicBrush Emu Edit
Benchmarks: MagicBrush ACE benchmark Emu Edit
Metrics: L1 L2 CLIP-I DINO CLIP-T human prompt-following score human image-quality score
Baselines: SD-SDEdit Null Text Inversion GLIDE Blended Diffusion HIVE InstructPix2Pix MagicBrush UltraEdit ControlNet CosXL SEED-X StyleBooth
Builds on: Diffusion Transformer MagicBrush InstructPix2Pix StyleBooth
Closest papers by meaning
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.89)
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.89)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.89)
- RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- OmniGen: Unified Image Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.