Research map / Image generation / Text-to-image diffusion
DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEFT, a parameter-efficient fine-tuning method for text-to-image models that decomposes the weight update into a projection onto the complement of a learned low-rank subspace plus a low-rank update, evaluated on personalization and universal in-context generation with SDXL and OmniGen.
Problem
Efficient fine-tuning of text-to-image models for personalization from few images tends to overfit and lose instruction-following ability, and unified multi-task models struggle to be adapted without losing generality.
Approach
Decomposes the weight update into (1) a projection removing a learned low-rank subspace from the pretrained weight, W - PP^T W, and (2) a low-rank update PR^T, with both P and R trainable; extends PaRa and LoRA with an extended column space and optional decompositions (QR, TSVD, LRMF, NMF) of P.
Key results
- On DreamBench Plus with SDXL v1.0, DEFT reaches CLIP-T 0.361, above PaRa (0.354) and DreamBooth LoRA (0.341).
- On VisualCloze Canny edge conditioning, DEFT attains CLIP-Image 95.78 and DINOv2 90.65, versus OmniGen 95.45 and 87.60.
- In the training-steps ablation at 8000 steps, LoRA's CLIP-T drops to 0.219 while DEFT keeps 0.319.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SDXL v1.0 with DEFT rather than PaRa | SDXL v1.0 · DreamBench Plus | CLIP-T: 0.354 -> 0.361 | Helped |
| Fine-tune OmniGen with DEFT rather than its base model | OmniGen · VisualCloze Canny | CLIP-Score Image: 95.45 -> 95.78 | Helped |
| Fine-tune OmniGen with DEFT rather than its base model | OmniGen · VisualCloze Depth | DINOv2: 77.39 -> 85.75 | Helped |
| Use DEFT rather than OmniGen for style transfer | OmniGen · VisualCloze InstantStyle | Image Score: 0.52 -> 0.69 | Helped |
| Use DEFT rather than OmniGen for style transfer | OmniGen · VisualCloze ReduxStyle | Image Score: 0.58 -> 0.69 | Helped |
| Use DEFT rather than OmniGen | OmniGen · VisualCloze Canny Edge | SSIM: 0.47 -> 0.66 | Helped |
| Use DEFT rather than OmniGen | OmniGen · VisualCloze Depth Map | SSIM: 0.69 -> 0.72 | Helped |
| Increase DEFT rank from 4 to 8 (ablation) | DEFT · DreamBooth Dog | CLIP-I: 0.8721 -> 0.8882 | Helped |
| Train DEFT for 8000 rather than 2000 steps (ablation) | DEFT on OmniGen · DreamBooth Dog | CLIP-T: $0.302 \pm 0.026$ -> $0.319 \pm 0.025$ | Helped |
| Train LoRA for 8000 rather than 2000 steps (ablation) | LoRA on OmniGen · DreamBooth Dog | CLIP-T: $0.286 \pm 0.033$ -> $0.219 \pm 0.023$ | Hurt |
| Use learnable Relaxing P rather than LRMF decomposition (ablation) | DEFT · DreamBooth Dog | CLIP-T: $0.220 \pm 0.051$ -> $0.235 \pm 0.041$ | Helped |
Surprising findings
- In the DreamBench DINO comparison appendix, LoRA beats DEFT on DINOv1/DINOv2 for several subjects, which the authors attribute to artistic flexibility.
- The authors state the default DEFT setting uses no decomposition at all, so the decomposition variants are not the main configuration.
Limitations
- Mathematical justification of the column-space claims relies on assumptions (col(Q) subset of col(W0)) that are stated but not enforced, and the proofs are loose.
- Reported gains over LoRA are small on several metrics, and LoRA wins on DINO scores in some appendix comparisons.
- Evaluation largely uses CLIP and DINO metrics without human evaluation or standard benchmark comparisons across many prompts.
Experimental setup
Datasets: DreamBooth DreamBench Plus InsDet VisualCloze
Benchmarks: DreamBench Plus DreamBooth VisualCloze test InsDet Canny/Depth controllability benchmark
Metrics: CLIP-T CLIP-I CLIP-Score DINOv1 DINOv2 F1 RMSE FID SSIM Aesthetic
Baselines: LoRA PaRa DreamBooth Textual Inversion Custom Diffusion SVDiff BLIP-Diffusion IP-Adapter VisualCloze ControlNet OmniControl OneDiffusion
Builds on: LoRA PaRa DreamBooth OmniGen
Closest papers by meaning
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.88)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.88)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.87)
- FineStyle: Fine-grained Controllable Style Personalization for Text-to-image Models (NeurIPS 2024 · similarity 0.87)
- Alchemist: Turning Public Text-to-Image Data into Generative Gold (NeurIPS 2025 · similarity 0.87)
- Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.