Research map / Large language models / Agents, code and math
JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JarvisArt, an MLLM-based photo retouching agent that drives Lightroom's 200+ tools via CoT SFT followed by GRPO-R with tool-use and perceptual rewards, evaluated on the new MMArt-Bench.
Problem
Professional photo retouching in Lightroom demands expertise and manual effort, while existing AI editing methods have limited adjustability, poor content fidelity, and weak generalization to diverse user intents and high resolutions.
Approach
An MLLM agent that reasons like a professional artist, outputs a Lightroom operation configuration (ROC) file over 200 tools and masks, and is trained with SFT on CoT traces then GRPO-R using rewards for tool/parameter accuracy (ROA) and perceptual quality (PQ), with an Agent-to-Lightroom protocol to execute edits non-destructively.
Key results
- On MMArt-Bench, JarvisArt reports L1 12.44 (x10^-2) vs GPT-4o 22.84, a 45.6% reduction in pixel-level error, with O 8.52 vs GPT-4o 9.18.
- In the ablation, SFT+RL reaches L1 12.44 and O 8.52, versus 14.42/7.94 for SFT only and 17.55/7.38 for RL only.
- A 30-participant usability study found over 90% of participants rated JarvisArt 4 or 5 on willingness to use it in future (vs 43.3% for Lightroom).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JarvisArt instead of GPT-4o | Photo retouching · MMArt-Bench scene-level | L1×10²: 22.84 -> 12.44 | Helped |
| Use JarvisArt instead of GPT-4o | Photo retouching · MMArt-Bench region-level | L1×10²: 15.71 -> 7.63 | Helped |
| Use JarvisArt instead of GPT-4o | Photo retouching · MMArt-Bench scene-level | O: 9.18 -> 8.52 | Hurt |
| Use JarvisArt instead of GPT-4o | Photo retouching · MIT-FiveK | L1×10²: 21.49 -> 12.98 | Helped |
| Use JarvisArt instead of GPT-4o | Photo retouching · MIT-FiveK | O: 9.22 -> 8.48 | Hurt |
| Add GRPO-R after SFT (ablation) | JarvisArt · MMArt-Bench scene-level | O: 7.94 -> 8.52 | Helped |
| Add SFT before GRPO-R instead of training with GRPO-R from scratch (ablation) | JarvisArt · MMArt-Bench scene-level | O: 7.38 -> 8.52 | Helped |
| Add PQ reward (ablation) | JarvisArt reward design · MMArt-Bench scene-level | O: 8.04 -> 8.52 | Helped |
| Add ROA reward (ablation) | JarvisArt reward design · MMArt-Bench scene-level | O: 8.15 -> 8.52 | Helped |
Surprising findings
- Under Qwen2.5-VL-72B as judge, JarvisArt's instruction-following O (6.67) is well below GPT-4o (7.85), so the authors note MLLM-based metrics may be unreliable.
- Unlike DeepSeek-R1, JarvisArt shows no clear 'aha moment' during GRPO-R, which the authors attribute to lacking intermediate visual feedback.
Limitations
- Evaluation uses 512x512 crops for baselines, so high-resolution performance is not directly compared.
- MLLM-based SC/PQ metrics may be unreliable and disagree across judges.
- No step-wise visual reward, so the agent cannot validate intermediate retouching decisions.
Experimental setup
Datasets: MMArt PPR10K MIT-FiveK
Benchmarks: MMArt-Bench MIT-FiveK
Metrics: L1 L2 SC PQ O (geometric mean of SC and PQ) region-level L1/L2/SC/PQ/O
Baselines: GPT-4o Gemini-2-Flash Step1X-Edit InstructPix2Pix MagicBrush OmniGen VARGPT-v1.1 3DLUT RSFNet
Builds on: GRPO DeepSeek-R1 Qwen2.5-VL
Closest papers by meaning
- RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward (CVPR 2026 · similarity 0.90)
- CREA: A Collaborative Multi-Agent Framework for Creative Image Editing and Generation (NeurIPS 2025 · similarity 0.88)
- FaSTA*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editing (ICLR 2026 · similarity 0.86)
- Text2Relight: Creative Portrait Relighting with Text Guidance (AAAI 2025 · similarity 0.86)
- CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions (CVPR 2026 · similarity 0.85)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.