Research map / Image generation / Text-to-image diffusion
Text-Driven Image Editing via Learnable Regions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a bounding-box region generator (DINO anchors + Gumbel-Softmax selection) trained with CLIP-based editing losses so that pre-trained mask-based editors (Stable Diffusion, MaskGIT) can perform mask-free local text-driven image editing.
Problem
Text-driven image editing usually requires user-provided masks or pixel-level masks; mask-free methods rely on fine pixel masks that are hard to get accurate, and box-only models like MaskGIT/Muse cannot use pixel masks.
Approach
A small region generation network selects a bounding box per DINO self-attention anchor point, trained end-to-end through a Gumbel-Softmax straight-through estimator with a CLIP-based composite editing loss (CLIP, directional, structural), so an existing mask-based editor can perform mask-free local edits; inference ranks candidates with a CLIP quality score.
Key results
- User study with 203 participants: proposed method preferred over Plug-and-Play (80.5%), InstructPix2Pix (73.2%), Null-text (88.2%), DiffEdit (91.9%), MasaCtrl (90.8%), average 84.9%.
- Against DINO-anchor-random-size baseline, preferred in 71.0%; against random anchor baseline, 83.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add directional loss LDir (ablation) | Learnable Regions · text-driven image editing | text-prompt alignment | Helped |
| add structural loss LStr (ablation) | Learnable Regions · text-driven image editing | preservation of source-image posture and shape | Helped |
| use learned region generation instead of random anchors and random box sizes (ablation) | Learnable Regions · user study | preference for Ours: 83.9% ±2.6% | Helped |
| learn box sizes instead of choosing random sizes at DINO anchors (ablation) | Learnable Regions · user study | preference for Ours: 71.0% ±3.2% | Helped |
Limitations
- Performance depends on the self-supervised model used for anchor initialization; failures when anchors fall on background.
- Predicted regions may include background, causing unintended changes.
- MaskGIT setting restricted to the class vocabulary it was trained on.
Experimental setup
Datasets: Unsplash images
Metrics: user preference rate (paired comparisons)
Baselines: Plug-and-Play InstructPix2Pix Null-text inversion DiffEdit MasaCtrl
Builds on: DINO Stable Diffusion MaskGIT StyleCLIP
Closest papers by meaning
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.90)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.88)
- Multi-Region Text-Driven Manipulation of Diffusion Imagery (AAAI 2024 · similarity 0.88)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.88)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.88)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.