Research map / Architectures and efficiency / Pruning and efficient tuning
DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-efficient Fine-Tuning
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, Zhenguo Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learnable scale factors to BitFit (ablation) | BitFit on DiT-XL/2 · 8 downstream datasets | average FID: 16.82 -> 15.39 | Improved |
| DiffFit instead of full fine-tuning | DiT-XL/2 · 8 downstream datasets | average FID: 16.59 -> 15.39 | Improved |
| use LoRA instead of DiffFit | DiT-XL/2 · 8 downstream datasets | average FID: 15.39 -> 81.25 | Hurt |
| add scale factor to QKV-linear layers on BitFit (ablation) | BitFit · Food101 | FID: 9.17 -> 7.37 | Improved |
| add scale factors in patch embedding or time embedding (ablation) | BitFit · Food101 | FID: 9.17 -> 9.05 | No clear effect |
| restrict scale factors to the first 14 blocks instead of all 28 (ablation) | DiffFit · Food101 | FID: 8.19 -> 7.61 | Improved |
| raise learning rate to 10x pre-training for parameter-efficient tuning (ablation) | DiffFit · Food101 | FID: 15.68 -> 8.19 | Improved |
| raise learning rate from 10x to 20x (ablation) | DiffFit · Food101 | FID: 8.19 -> 8.30 | Hurt |
| add positional encoding interpolation trick for 512 resolution (ablation) | DiffFit · Food-101 512x512 | FID: 19.50 -> 19.10 | Improved |
| adapt low-resolution DiT to 512x512 with DiffFit | DiT-XL/2 · ImageNet 512x512 | FID and training cost: 3.04 -> 3.02 FID; 1733 -> 51 GPU days | Improved |
Later papers that compared with this method
1 later accepted paper reports DiffFit as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Efficient Training of Large Vision Models via Advanced Automated Progressive Learning (TPAMI 2026 · better on 4 of 4 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.91)
- Diffusion Models Without Attention (CVPR 2024 · similarity 0.91)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.91)
- Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization (ICLR 2025 · similarity 0.91)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.91)
- ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
