Research map / Image generation / Editing and style transfer
MagicQuill: An Intelligent Interactive Image Editing System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents MagicQuill, an open-source interactive image editing system combining brushstroke-based diffusion control (edge and color branches), an MLLM that predicts edit prompts from strokes (Draw&Guess), and a brush-based UI.
Problem
Precise, efficient fine-grained image editing is hard because text prompts are imprecise and interfaces lack intuitive region-level controls such as shape and color editing.
Approach
Combines three brushstroke types (add/subtract scribbles on an edge map and color blocks) fed into a plug-in inpainting branch plus ControlNet-style control branch on a frozen diffusion UNet, with a LoRA-fine-tuned LLaVA that guesses the user's intended prompt from strokes so users never type prompts.
Key results
- Editing Processor reaches LPIPS 0.0667, PSNR 27.282, SSIM 0.902 on 490-image test set, beating BrushNet+ControlNet (0.0748, 25.770, 0.894).
- Painting Assistor scores GPT-4 similarity 2.712 vs 1.976 for GPT-4o, BERT 0.749, CLIP 0.824 on the same test set.
- User study (10 users) shows average time savings of 24.92% on iPad and 19.58% on PC per edit with the prompt guesser.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the dual-branch Editing Processor with brush-based edge and color control instead of BrushNet + ControlNet | MagicQuill Editing Processor · MagicQuill test dataset (490 images) | LPIPS: 0.0748 -> 0.0667 | Helped |
| Use the dual-branch Editing Processor with brush-based edge and color control instead of BrushNet + ControlNet | MagicQuill Editing Processor · MagicQuill test dataset (490 images) | PSNR: 25.770 -> 27.282 | Helped |
| Use the dual-branch Editing Processor with brush-based edge and color control instead of BrushNet + ControlNet | MagicQuill Editing Processor · MagicQuill test dataset (490 images) | SSIM: 0.894 -> 0.902 | Helped |
| Fine-tune LLaVA for Draw&Guess | MagicQuill Painting Assistor · Draw&Guess test dataset (490 images) | GPT-4 similarity: 1.976 -> 2.712 | Helped |
| Fine-tune LLaVA for Draw&Guess | MagicQuill Painting Assistor · Draw&Guess test dataset (490 images) | BERT similarity: 0.721 -> 0.749 | Helped |
| Fine-tune LLaVA for Draw&Guess | MagicQuill Painting Assistor · Draw&Guess test dataset (490 images) | CLIP similarity: 0.795 -> 0.824 | Helped |
| Use the Painting Assistor to predict prompts rather than entering prompts manually | MagicQuill editing workflow on iPad · Editing time per edit (iPad, 10 users) | time: 17.70s (+4.41s) -> 13.29s; average time savings of 24.92% | Helped |
| Use the Painting Assistor to predict prompts rather than entering prompts manually | MagicQuill editing workflow on PC · Editing time per edit (PC, 10 users) | time: 15.53s (+3.04s) -> 12.49s; average time savings of 19.58% | Helped |
Limitations
- Add-brush interpretation of abstract sketches remains challenging, motivating the custom fine-tuning dataset.
- Evaluation of Painting Assistor and editing quality relies on a constructed test set simulated from DCI rather than real user edits.
- User studies involve small participant counts (30 and 10).
Experimental setup
Datasets: DCI (Densely Captioned Images)
Benchmarks: 490-image test set constructed from DCI
Metrics: LPIPS PSNR SSIM GPT-4 similarity BERT similarity CLIP similarity SUS-style user ratings editing time
Baselines: SmartEdit SketchEdit BrushNet BrushNet+ControlNet SDEdit UniPaint LLaVA-1.5 LLaVA-Next GPT-4o
Builds on: ControlNet BrushNet LLaVA Stable Diffusion
Closest papers by meaning
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.85)
- SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.85)
- AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea (CVPR 2025 · similarity 0.85)
- ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.84)
- CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions (CVPR 2026 · similarity 0.84)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.