Research map / Image generation / Image restoration
RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RestoreAgent, a LoRA-fine-tuned Llava-Llama3-8b multimodal LLM that plans the task order and model choice for restoring images with multiple degradations, trained on search-derived optimal pipelines.
Problem
Restoring natural images with multiple simultaneous degradations requires choosing tasks, their execution order, and specific models, which is time-consuming for humans and poorly served by all-in-one models that over-smooth outputs.
Approach
An MLLM agent is fine-tuned on a dataset of exhaustively searched optimal (task order, model) pipelines scored by a z-score-summed multi-metric function, so it predicts restoration pipelines directly from the degraded image, with rollback and stop decisions.
Key results
- Average balanced score 4.38 with ranking 12.9% among all candidate pipelines, versus 19.5% for human experts and 34.7% for random order and model.
- On noise+JPEG, RestoreAgent reaches PSNR 25.32 and LPIPS 0.2308, beating MiOIR (23.98 PSNR) and Real-ESRGAN (23.43 PSNR).
- Adapting to desnowing took about thirty minutes of fine-tuning and reached balanced score 4.23 vs 3.91 for human expert.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RestoreAgent for per-image task-order and model selection | multi-degradation image restoration · Average Result Across All Datasets | ranking (%): 19.5 -> 12.9 | Helped |
| use RestoreAgent rather than all-in-one restoration | multi-degradation image restoration · Noise + JPEG | PSNR: 23.98 -> 25.32 | Helped |
| use RestoreAgent rather than all-in-one restoration | multi-degradation image restoration · Haze + Noise | PSNR: 17.56 -> 20.47 | Helped |
| use RestoreAgent rather than all-in-one restoration | multi-degradation image restoration · Rain + Haze + Noise | PSNR: 19.45 -> 19.53 | Helped |
| use RestoreAgent rather than all-in-one restoration | multi-degradation image restoration · Rain + Haze + Noise + JPEG | PSNR: 15.30 -> 19.72 | Helped |
| increase training data from 7k to 14k pairs (ablation) | RestoreAgent · Average Result Across All Datasets | ranking (%): 16.2 -> 13.6 | Helped |
| increase training data from 14k to 23k pairs (ablation) | RestoreAgent · Average Result Across All Datasets | ranking (%): 13.6 -> 12.9 | Helped |
| optimize PSNR instead of balanced metrics (ablation) | RestoreAgent · Average Result Across All Datasets | PSNR: 22.61 -> 22.72 | Helped |
| optimize SSIM instead of balanced metrics (ablation) | RestoreAgent · Average Result Across All Datasets | SSIM: 0.7700 -> 0.7763 | Helped |
| optimize LPIPS instead of balanced metrics (ablation) | RestoreAgent · Average Result Across All Datasets | LPIPS: 0.2513 -> 0.2477 | Helped |
| optimize DISTS instead of balanced metrics (ablation) | RestoreAgent · Average Result Across All Datasets | DISTS: 0.1890 -> 0.1875 | Helped |
| add desnowing task and fine-tune for thirty minutes | RestoreAgent · Desnowing | balanced: 3.91 -> 4.23 | Helped |
Surprising findings
- Human experts with more than five years of low-level vision experience were outperformed by the agent on the balanced score (19.5% vs 12.9% ranking).
- A predefined expert order and model choice ranks in the top 22.1%, showing fixed pipelines beat random ones but still lag behind the learned planner.
Limitations
- Limited scope of restoration models and degradation tasks examined.
- Restoration models used are not the latest and generalize poorly to small degradation variations.
- Step-wise replanning gives only a modest improvement over single-shot planning.
Experimental setup
Datasets: synthetically degraded images with exhaustive pipeline search labels
Benchmarks: 200-image multi-degradation test set (7 degradation combinations)
Metrics: PSNR SSIM LPIPS DISTS balanced z-score sum ranking percentile among all decisions
Baselines: Random order and model Pre-defined order and model Human expert Real-ESRGAN StableSR AirNet PromptIR MiOIR InstructIR DA-CLIP AutoDIR
Builds on: LLaVA LoRA Clarity ChatGPT RL-Restore
Closest papers by meaning
- Hybrid Agents for Image Restoration (CVPR 2026 · similarity 0.92)
- An Intelligent Agentic System for Complex Image Restoration Problems (ICLR 2025 · similarity 0.90)
- OPERA: An Agent for Image Restoration with End-to-End Joint Planning–Execution Optimization (NeurIPS 2026 · similarity 0.88)
- DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (NeurIPS 2024 · similarity 0.87)
- Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (CVPR 2024 · similarity 0.85)
- Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.