Research map / Image generation / Text-to-image diffusion
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCEdit, an efficient tuning framework that edits U-Net skip connections with lightweight SC-Tuner/CSC-Tuner modules for text-to-image adaptation and controllable generation.
Problem
Fine-tuning large image diffusion models is costly in memory and compute; existing efficient tuning methods such as LoRA backpropagate through the whole U-Net, and controllable generation methods like ControlNet add many parameters.
Approach
Insert lightweight adapter-style tuners directly on the U-Net skip connections, which carry multi-scale structural information, so that backpropagation only passes through the decoder and the encoder is decoupled from gradient computation; a conditional variant (CSC-Tuner) injects control signals through the same path.
Key results
- On COCO2017 text-to-image, SCEdit reaches FID 13.82 vs 14.15 for full fine-tuning and 13.96 for LoRA/r=64, using 19.68M params and 29.02G memory (about 52% less than LoRA/r=64's 60.57G).
- On canny-edge controllable synthesis, SCEdit/k=3 reaches FID 71.78 vs 74.86 for ControlNet, using 99.11M params (vs 364.23M) and 35.28G memory (vs 49.51G).
- The paper reports SCEdit uses 7.9% of ControlNet's parameters and a 30% reduction in memory consumption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full fine-tuning with SCEdit | SD v1.5 · COCO2017 | FID: 14.15 -> 13.82 | Helped |
| replace LoRA rank 64 with SCEdit | SD v1.5 · COCO2017 | FID: 13.96 -> 13.82 | Helped |
| replace LoRA rank 64 with SCEdit | SD v1.5 · COCO2017 | Training memory: 60.57G -> 29.02G | Helped |
| reduce tuner hidden dimensions by 10× (ablation) | SCEdit · COCO2017 | FID: 13.82 -> 13.99 | Hurt |
| reduce tuner hidden dimensions by 10× (ablation) | SCEdit · COCO2017 | Trainable parameters: 19.68M -> 1.98M | Helped |
| use convolution kernel size 3 instead of 1 (ablation) | SCEdit controllable generation · LAION artistic, canny edge | FID: 73.18 -> 71.78 | Helped |
| use convolution kernel size 3 instead of 1 (ablation) | SCEdit controllable generation · LAION artistic, canny edge | Trainable parameters: 28.82M -> 99.11M | Hurt |
| replace ControlNet with SCEdit/k=3 | SD v2.1 · LAION artistic, canny edge | FID: 74.86 -> 71.78 | Helped |
| replace ControlNet with SCEdit/k=3 | SD v2.1 · LAION artistic, canny edge | Training memory: 49.51G -> 35.28G | Helped |
Surprising findings
- Lowering the tuner hidden dimension 10x (SCEdit10, 1.98M params) still beats LoRA/r=6 in FID by 1.13, per the paper's reporting.
Limitations
- Few-shot evaluation is qualitative only (visual comparison in Fig. 6), with no quantitative metrics reported.
- Controllable generalization (sketch-to-image, outpainting) is shown only qualitatively.
Experimental setup
Datasets: COCO2017 LAION artistic (filtered) customized style dataset (ModelScope)
Benchmarks: COCO2017 LAION artistic (filtered)
Metrics: FID trainable parameters training memory training time
Baselines: Full fine-tuning LoRA ControlNet T2I-Adapter ControlNet-XS ControlLoRA
Builds on: LoRA Res-Tuning ControlNet T2I-Adapter Stable Diffusion
Closest papers by meaning
- The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling (ICML 2024 · similarity 0.90)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.88)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.87)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.87)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.87)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.