Research map / Image generation / Editing and style transfer
Multi-Reward as Condition for Instruction-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-reward conditioning for instruction-based image editing: GPT-4o scores and critiques existing InsPix2Pix triplets on three perspectives (RewardEdit-20K), and these rewards are injected as conditions into InsPix2Pix and SmartEdit, evaluated on a new Real-Edit benchmark.
Problem
Training triplets for instruction-based image editing, mainly InsPix2Pix, are noisy: they often fail to follow instructions, change non-edited regions, and contain generation artifacts, which yields inaccurate supervision.
Approach
Instead of cleaning ground-truth edited images, score each triplet on instruction following, detail preserving, and generation quality with GPT-4o (0-5 scores plus text feedback), then encode these multi-reward signals as conditions injected into the latent space and U-Net of the editing model; at inference, set scores to 5 and text to none.
Key results
- Reward-InsPix2Pix improves following accuracy from 52% to 63% and preserving accuracy from 53% to 58% on Real-Edit versus InsPix2Pix.
- Reward-SmartEdit reaches 69% following, 74% preserving accuracy on Real-Edit, versus 64% and 66% for SmartEdit.
- Human evaluation on Real-Edit shows Reward-SmartEdit scoring higher than SmartEdit on all three perspectives (e.g., 3.34 vs 3.09 following).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-reward conditioning | InsPix2Pix · Real-Edit | Following accuracy: 52% -> 63% | Helped |
| add multi-reward conditioning | SmartEdit · Real-Edit | Preserving accuracy: 66% -> 74% | Helped |
| add multi-reward conditioning | SmartEdit · MagicBrush test | CLIP-I: 90.4 -> 91.3 | Helped |
| add multi-reward conditioning | SmartEdit · Real-Edit human evaluation | Following score: 3.09 -> 3.34 | Helped |
| add reward score without reward text (ablation) | InsPix2Pix · Real-Edit ablation | Following accuracy: 49% -> 61% | Helped |
| add reward text without reward score (ablation) | InsPix2Pix · Real-Edit ablation | Following accuracy: 49% -> 60% | Helped |
| add reward score and reward text (ablation) | InsPix2Pix · Real-Edit ablation | Following accuracy: 49% -> 63% | Helped |
| add reward attention integration only (ablation) | InsPix2Pix · Real-Edit ablation | Following accuracy: 49% -> 60% | Helped |
| add reward by U-Net addition only (ablation) | InsPix2Pix · Real-Edit ablation | Preserving accuracy: 38% -> 56% | Helped |
| use reward scores 3, 3, 3 instead of 0, 0, 0 (ablation) | Reward-conditioned InsPix2Pix · Real-Edit ablation | Preserving accuracy: 30% -> 42% | Helped |
| use reward scores 5, 5, 5 instead of 3, 3, 3 (ablation) | Reward-conditioned InsPix2Pix · Real-Edit ablation | Preserving accuracy: 42% -> 58% | Helped |
| increase training resolution from 256 to 512 (ablation) | Reward-InsPix2Pix · Real-Edit ablation | Following score: 3.39 -> 3.28 | Hurt |
Surprising findings
- Reward-SmartEdit does not improve quality accuracy (49% vs 45% is an improvement but its quality score 3.67 is below SmartEdit's, and HQ-Edit has higher quality score than most methods), showing gains are uneven across perspectives.
- Reward-conditioned model with reward scores of 0 performs notably worse, but even with all scores at 5 the generated outputs do not always reach a score of 5.
Limitations
- Reward annotations come from GPT-4o, which the authors acknowledge is less reliable than human annotation.
- Multi-reward module must be trained separately for each base editing model.
- Model struggles with quantifiers and spatial position words in instructions.
Experimental setup
Datasets: InstructPix2Pix dataset MagicBrush test set
Benchmarks: Real-Edit MagicBrush test set
Metrics: GPT-4o yes/no accuracy GPT-4o score 0-5 human score 0-5 CLIP-I CLIP-T CLIP score FID
Baselines: KOSMOS-G MagicBrush MGIE InstructDiffusion HIVE HQ-Edit InsPix2Pix SmartEdit
Builds on: InstructPix2Pix SmartEdit Stable Diffusion
Closest papers by meaning
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.89)
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing (ICLR 2026 · similarity 0.88)
- Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editing (ICML 2025 · similarity 0.88)
- RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward (CVPR 2026 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.88)
- HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.