Research map / Image generation / Text-to-image diffusion
Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates an unlimited synthetic training set from Flux (FluxGen) and trains a lightweight 0.4B MM-DiT-based adapter with squeeze-and-excitation layers (FluxIR) to control the 12B Flux model for real-world image restoration at roughly 8.5% of prior training cost.
Problem
Controlling large pre-trained text-to-image models for image restoration typically requires massive high-quality real image datasets and very expensive adapter training, which is costly and raises privacy and copyright concerns.
Approach
Use the unconditional generation of Flux with an empty prompt, filter by no-reference IQA scores, and simulate degradations to build paired data; then control Flux with a single initialized MM-DiT block plus low-rank squeeze-and-excitation layers that broadcast control signals to all 57 MM-DiT blocks, trained with a modified timestep sampler and a pixel-space loss.
Key results
- FluxIR achieves the best MANIQA on all synthetic and real-world datasets, e.g. 0.6331 on DIV2K-Val and 0.6314 on RealLQ250, while trailing on full-reference metrics such as PSNR and SSIM.
- Training takes about 14 GPU-days (3.5 days on 4 H800s), cutting training cost by about 93.5% vs SUPIR and 91.5% vs DreamClear (about 8.5% of their cost).
- Ablation shows Flux-generated data with IQA selection outperforms SDXL-generated data (MANIQA 0.6267 vs 0.5630 on RealLQ250), and the timestep sampler plus pixel-space loss raise MANIQA from 0.5720 to 0.6314.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FluxIR instead of the strongest baseline | Image restoration model · DIV2K-Val | MANIQA: 0.6043 -> 0.6331 | Helped |
| Use FluxIR instead of the strongest baseline | Image restoration model · RealSR | MANIQA: 0.6115 -> 0.6334 | Helped |
| Use FluxIR instead of the strongest baseline | Image restoration model · DrealSR | MANIQA: 0.5928 -> 0.6024 | Helped |
| Use FluxIR instead of the strongest baseline | Image restoration model · RealLQ250 | MANIQA: 0.5927 -> 0.6314 | Helped |
| Use SE rank 32 rather than rank 16, 64, or 128 (ablation) | FluxIR · RealLQ250 | CLIPIQA: 0.5389; 0.5639; 0.5529; 0.5606 | Mixed |
| Use SE rank 32 rather than rank 16, 64, or 128 (ablation) | FluxIR · RealLQ250 | MUSIQ: 69.72; 70.78; 70.37; 70.54 | Helped |
| Use SE rank 32 rather than rank 16, 64, or 128 (ablation) | FluxIR · RealLQ250 | MANIQA: 0.6222; 0.6314; 0.6306; 0.6292 | Helped |
| Replace logit-normal timestep sampling with f(u) (ablation) | FluxIR · RealLQ250 | CLIPIQA: 0.4416 -> 0.5082 | Helped |
| Add pixel-space LP loss (ablation) | FluxIR with f(u) timestep sampling · RealLQ250 | CLIPIQA: 0.5082 -> 0.5639 | Helped |
| Replace SDXL-generated training data with Flux-generated data, retaining IQA selection (ablation) | FluxGen training data for FluxIR · RealLQ250 | MANIQA: 0.5630 -> 0.6267 | Helped |
| Add IQA-based image selection to FluxGen (ablation) | FluxIR training data · RealLQ250 | CLIPIQA: 0.5254 -> 0.5487 | Helped |
Surprising findings
- Strong non-reference scores are obtained with a 0.4B adapter, yet full-reference metrics (PSNR, SSIM) are lower than most baselines, indicating a perception-distortion trade-off.
- Restricting the SE layer to rank 32 keeps performance comparable to a full-rank MLP while saving over 72% of its parameters.
Limitations
- Full-reference metrics (PSNR, SSIM, LPIPS) are weaker than several baselines, as is typical of generative IR methods.
- Training data comes only from Flux-generated images, so restoration quality is bounded by the realism and diversity of the generator.
Experimental setup
Datasets: FluxGen synthetic images (350k) DIV2K-Val RealSR DrealSR RealLQ250
Benchmarks: DIV2K-Val RealSR DrealSR RealLQ250
Metrics: PSNR SSIM LPIPS MANIQA CLIPIQA MUSIQ
Baselines: BSRGAN Real-ESRGAN SwinIR DASR StableSR DiffBIR ResShift SinSR SeeSR SUPIR DreamClear
Builds on: Flux ControlNet Stable Diffusion 3 SUPIR DreamClear
Closest papers by meaning
- Adapting Text-to-Image Generation with Feature Difference Instruction for Generic Image Restoration (CVPR 2025 · similarity 0.89)
- DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (NeurIPS 2024 · similarity 0.88)
- Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts (NeurIPS 2024 · similarity 0.87)
- UP-Restorer: When Unrolling Meets Prompts for Unified Image Restoration (AAAI 2025 · similarity 0.86)
- GenDeg: Diffusion-based Degradation Synthesis for Generalizable All-In-One Image Restoration (CVPR 2025 · similarity 0.86)
- Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.