Research map / Vision-language and multimodal / Multimodal LLMs

Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces hypothetical instruction-reasoning image editing (HI-IE), the Reason50K dataset (~51K triplets over physical, temporal, causal, story reasoning), and ReasonBrain, an MLLM plus diffusion editor with fine-grained reasoning cue extraction and a cross-modal enhancer.

Problem

Existing instruction-based image editing methods handle only explicit, atomic instructions and fail on implicit hypothetical instructions requiring physical, temporal, causal, or story reasoning; existing datasets are small or lack such reasoning scenarios.

Approach

Pairs an MLLM (LLaVA) that generates reasoning-aware visual guidance with a diffusion editor, feeding it fine-grained visual and textual reasoning cues (FRCE) and enriching the MLLM-diffusion interface with a bidirectional Cross-Modal Enhancer (CME), trained on a new hypothetical-instruction dataset.

Key results

• ReasonBrain reaches CLIP 0.259, MLLM 0.877, Ins-Align 0.847 on Reason50K, versus 0.163/0.752/0.388 for the MLLM+diffusion baseline group.
• In human evaluation, 75.71% of participants chose ReasonBrain for instruction alignment versus under 10% for every baseline.
• GPT-4o instruction conversion followed by Qwen-ImageEdit reaches only 0.512 Ins-Align, about 39.5% below ReasonBrain.

Surprising findings

• The lightweight ReasonBrain-3B variant has higher CLIP and MLLM scores in its settings table than the 7B model it is compared with, which conflicts with the text's claim of only a slight drop.
• Multi-target one-to-two and one-to-three editing degrades performance sharply (Ins-Align 0.847 to 0.315 and 0.204).

Limitations

• Dataset is largely synthetic (over 90% generated via diffusion and GPT pipelines), which may miss real-world artifacts and temporal dynamics.
• Performance degrades sharply for multi-target (one-to-many) editing settings.

Experimental setup

Datasets: Reason50K EditWorld PhyBench MagicBrush Emu Edit ReasonEdit Complex-Edit

Benchmarks: Reason50K ReasonEdit EditWorld Complex-Edit MagicBrush Emu Edit

Metrics: CLIP Score MLLM Score Ins-Align Score L1 distance CLIP image similarity DINO similarity

Baselines: InstructPix2Pix MagicBrush MGIE SmartEdit UltraEdit PixWizard Flux-Kontext Qwen-ImageEdit

Builds on: InstructPix2Pix LLaVA Flux SmartEdit MGIE PhyBench

Closest papers by meaning

• ReasonEdit: Towards Reasoning-Enhanced Image Editing Models (CVPR 2026 · similarity 0.94)
• SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models (NeurIPS 2025 · similarity 0.90)
• Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing (NeurIPS 2025 · similarity 0.90)
• Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.90)
• InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.90)