Research map / Image generation / Editing and style transfer
CAMILA: Context-Aware Masking for Image Editing with Language Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMILA, an MLLM-based context-aware masking framework that classifies instruction tokens as [MASK] or [NEG] to edit only executable instructions in IP2P-style diffusion, and introduces multi-instruction and context-aware editing benchmarks built from MagicBrush.
Problem
Text-guided image editing models try to execute all instructions, including infeasible or contradictory ones, producing nonsensical or over-edited images; existing multi-instruction benchmarks do not test non-executable requests.
Approach
An MLLM (LLaVA-7B, LoRA-tuned) outputs [MASK]/[NEG] tokens per instruction; a Token Broadcaster aligns them to text embeddings via cosine similarity, a two-layer Transformer Token Decoder produces binary masks, and these masks modulate IP2P cross-attention. A surrogate module predicting CLIP-T is used to refine mask generation.
Key results
- On the context-aware multi-instruction task, CAMILA reports L1 0.0661, CLIP-I 0.9296, DINO 0.8932, versus best baseline FoI at 0.0891, 0.8895, 0.8190.
- PickScore improves over the strongest baseline FoI by 18.1% (multi-instruction) and 24.0% (context-aware).
- Token classification accuracy of 90.21% on the context-aware dataset; mask IoU 0.3819 and Dice 0.4986 for applicable instructions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAMILA instead of FoI | image editing models · Multi-instruction Image Editing | L1: 0.1054 -> 0.0945 | Helped |
| Use CAMILA instead of FoI | image editing models · Context-Aware Image Editing | DINO: 0.8190 -> 0.8932 | Helped |
| Use CAMILA instead of FoI | image editing models · Multi-instruction Image Editing | PickScore: 0.2363 -> 0.2790 | Helped |
| Use CAMILA instead of FoI | image editing models · Context-Aware Image Editing | PickScore: 0.2285 -> 0.2834 | Helped |
| Use CAMILA instead of SE | EMU single-instruction editing · EMU single-instruction | CLIP-dir: 0.094 -> 0.092 | Hurt |
| Add surrogate-module training (ablation) | CAMILA · Single-instruction single-turn | CLIP-T: 0.3020 -> 0.3022 | Helped |
| Add surrogate-module training (ablation) | CAMILA · Single-instruction multi-turn | L1: 0.0931 -> 0.0782 | Helped |
| Add surrogate-module training (ablation) | CAMILA · Multi-instruction Image Editing | CLIP-T: 0.2975 -> 0.2984 | Helped |
| Add surrogate-module training (ablation) | CAMILA · Context-Aware Image Editing | L1: 0.0673 -> 0.0661 | Helped |
| Replace the Token Decoder with SAM (ablation) | CAMILA · Multi-instruction Image Editing | CLIP-T: 0.2975 -> 0.2895 | Hurt |
| Replace the Token Decoder with SAM (ablation) | CAMILA · Single-instruction single-turn | L1: 0.0602 -> 0.0585 | Helped |
Surprising findings
- Generated masks have only moderate overlap with ground truth (IoU 0.38), which the authors attribute to high-level guidance rather than strict spatial matching.
- SAM substitution for the Token Decoder yields slightly higher L1/L2 in some settings but lower CLIP and DINO scores.
Limitations
- Failure cases: masks ignore relative object sizes, yielding too-small additions, due to the pretrained diffusion model not being trained for mask-based editing.
- Non-applicable instructions are generated by ChatGPT-4V and the benchmark is derived from MagicBrush, limiting diversity.
- Total inference time (9.2s) is higher than MGIE and SmartEdit.
Experimental setup
Datasets: MagicBrush EMU
Benchmarks: MagicBrush EMU Multi-instruction editing (constructed) Context-aware instruction editing (constructed)
Metrics: L1 L2 CLIP-I DINO CLIP-T CLIP-dir PickScore IoU Dice
Baselines: InstructPix2Pix MGIE SmartEdit FoI EMILIE
Builds on: InstructPix2Pix Stable Diffusion LLaVA FoI MagicBrush
Closest papers by meaning
- Improving Editability in Image Generation with Layer-wise Memory (CVPR 2025 · similarity 0.89)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.88)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.87)
- Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.86)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.86)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.