Research map / Image generation / Text-to-image diffusion
Beyond Textual Constraints: Learning Novel Diffusion Conditions with Fewer Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a few-shot method for learning new spatial diffusion conditions (sketch, segmentation, pose, depth, canny) from ~100 examples by removing text conditioning via a prompt-free encoder and rectifying the CFG negative prompt per condition.
Problem
Learning novel controllable diffusion conditions (e.g., sketch, pose, depth) normally requires tens of thousands of paired examples; with few exemplars, text prompts bias learning and cause structural misalignment.
Approach
A prompt-free encoder initialized from the SD encoder with text cross-attention pruned, trained with null-text fine-tuning to minimize textual influence, plus a lightweight negative rectifier that adapts the CFG negative prompt embedding to each exemplar's structural condition embedding.
Key results
- With 100 training pairs, sketch FID 21.049 / cSSIM 0.692 on COCO, matching or beating ControlNet (22.046 / 0.690) and T2I-Adapter (19.445 / 0.683 full-trained).
- Segmentation on COCO-stuff: FID 20.726 and cSSIM 0.835, beating fully trained ControlNet (27.377 / 0.820) and PromptDiff (35.837 / 0.815).
- Pose on Human-Art: FID 32.968 and AP 23.10, versus ControlNet-100 at AP 19.64 and T2I-Adapter-100 at AP 13.80; HumanSD-100 has better FID (32.339).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prompt-free conditional learning and condition-specific negative rectification | Stable Diffusion 1.5 · COCO sketch | FID: 19.445 -> 21.049 | Hurt |
| Add prompt-free conditional learning and condition-specific negative rectification | Stable Diffusion 1.5 · COCO segmentation | FID: 24.254 -> 20.726 | Helped |
| Add prompt-free conditional learning and condition-specific negative rectification | Stable Diffusion 1.5 · COCO segmentation | cSSIM: 0.820 -> 0.835 | Helped |
| Add prompt-free conditional learning and condition-specific negative rectification | Stable Diffusion 1.5 · InstructPix2Pix depth | FID: 15.856 -> 19.137 | Hurt |
| Add prompt-free conditional learning and condition-specific negative rectification | Stable Diffusion 1.5 · InstructPix2Pix Canny edge | FID: 13.539 -> 16.710 | Hurt |
| Add prompt-free conditional learning and condition-specific negative rectification | Stable Diffusion 1.5 · Human-Art pose | FID: 36.817 -> 32.968 | Helped |
| Add condition-specific negative rectification (ablation) | Prompt-free conditional learning · COCO sketch | FID: 21.245 -> 21.049 | Helped |
| Add condition-specific negative rectification (ablation) | Prompt-free conditional learning · COCO sketch | cSSIM: 0.684 -> 0.692 | Helped |
| Add condition-specific negative rectification (ablation) | Prompt-free conditional learning · Human-Art pose | FID: 36.334 -> 32.968 | Helped |
| Add condition-specific negative rectification (ablation) | Prompt-free conditional learning · Human-Art pose | AP: 23.10 -> 23.10 | No clear effect |
| Add condition-specific negative rectification (ablation) | Prompt-free conditional learning · Human-Art pose | cSSIM: 0.853 -> 0.855 | Helped |
| Add condition-specific negative rectification (ablation) | Prompt-free conditional learning · InstructPix2Pix depth | FID: 19.299 -> 19.137 | Helped |
Surprising findings
- Retrained ControlNet-100 and T2I-Adapter-100 can be worse than their fully trained counterparts in some cases, and removing text constraints helps structural alignment in few-shot settings.
- Ours with 100 samples outperforms fully trained ControlNet and PromptDiff on segmentation FID and cSSIM.
Limitations
- Text-free training can leave some content inconsistent with the text prompt in aligned regions (noted in Fig. 3).
- Fine-grained results depend on the exemplar-based training; FID/cSSIM only, with no human evaluation reported in the main text.
Experimental setup
Datasets: COCO COCO-stuff Human-Art InstructPix2Pix
Benchmarks: COCO Human-Art InstructPix2Pix
Metrics: FID cSSIM AP
Baselines: ControlNet T2I-Adapter PromptDiff HumanSD ControlNet-100 T2I-Adapter-100 PromptDiff-100 HumanSD-100
Builds on: ControlNet T2I-Adapter classifier-free guidance PromptDiff HumanSD
Closest papers by meaning
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.89)
- Zero-Shot Conditioning of Score-Based Diffusion Models by Neuro-Symbolic Constraints (AAAI 2025 · similarity 0.88)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.87)
- Constrained Diffusion Models via Dual Training (NeurIPS 2024 · similarity 0.87)
- CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling (ICLR 2024 · similarity 0.87)
- No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.