AI Paper Atlas

Research map / Image generation / Text-to-image diffusion

Controlling Text-to-Image Diffusion by Orthogonal Finetuning

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf

NeurIPS 2023 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 21 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use OFT instead of DreamBooth/LoRA finetuningStable Diffusion v1.5 · DreamBooth subject-driven generationDINO: 0.614 -> 0.632Improved
use OFT instead of DreamBoothStable Diffusion v1.5 · DreamBooth subject-driven generationCLIP-T: 0.239 -> 0.237No clear effect
use OFT instead of ControlNetStable Diffusion v1.5 · Segmentation to image (ADE20K)mIoU: 20.88 -> 27.06Improved
add the orthogonal constraint (ablation)OFT · Controllable generation (ControlNet setting)qualitative controlImproved
add the epsilon deviation constraint (COFT vs OFT) (ablation)OFT · Subject-driven generation, long trainingstabilityImproved
COFT vs OFTStable Diffusion v1.5 · Landmark to facelandmark error: 7.07 -> 6.92Improved
increase number of diagonal blocks from 4 to 16 (ablation)OFT · Segmentation to image (ADE20K)mIoU: 27.06 -> 21.0Hurt
decrease number of diagonal blocks from 4 to 2 (ablation)OFT · Segmentation to image (ADE20K)mIoU: 27.06 -> 27.18Mixed
add post-stage magnitude fitting (re-scaled) (ablation)COFT · ADE20K S2IFID: 30.8 -> 30.2Improved
finetune only conv layers instead of attention layers (ablation)COFT · ADE20K S2IFID: 30.8 -> 39.8Hurt
finetune attention plus conv layers (ablation)COFT · ADE20K S2IFID: 30.8 -> 30.4Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy