Research map / Image generation / Fast diffusion sampling
Readout Guidance: Learning Control from Diffusion Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Readout Guidance trains lightweight readout heads on frozen diffusion U-Net features and uses their gradients to guide sampling for controllable text-to-image generation.
Problem
Enabling arbitrary user control over text-to-image diffusion outputs usually requires substantial training on large annotated datasets; the paper seeks a cheaper way to add diverse controls (spatial and relative) to frozen diffusion models.
Approach
Train small readout heads on frozen diffusion features at every timestep to predict properties (pose, depth, edges, correspondence, appearance similarity), then use the distance between readout and a user target as classifier-style guidance during sampling, unifying different controls under one framework.
Key results
- Readout Guidance with T2I-Adapter improves pose PCK (alpha=0.05) by 30% over T2I-Adapter alone (2.3x in PCK) on SDXL.
- Readout pose head trained on 8.5k PascalVOC images performs comparably to T2I-Adapter trained on 3M images, about 350x less training data.
- Readout heads are small (about 5.9M params, 49MB) and train in at most three hours on a single A100 40G.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Readout Guidance to T2IAdapter | T2IAdapter · MSCOCO validation | PCK (α=0.05): 30% | Helped |
| Add Readout Guidance to T2IAdapter | T2IAdapter · MSCOCO validation | PCK: 2.3x | Helped |
| Train the pose readout head on more PascalVOC images (100, 1k, or 8.5k) (ablation) | Readout Guidance pose control · MSCOCO validation | PCK | Helped |
| Train the pose readout head on 8.5k PascalVOC images rather than fewer examples (ablation) | Readout Guidance pose control · MSCOCO validation | PCK | Helped |
| Increase appearance-similarity guidance weight from 0.0 to 1.0 (ablation) | Readout Guidance appearance preservation · Image variations | appearance similarity | Mixed |
Surprising findings
- Pose readout trained on only 100 examples already yields usable pose control.
Limitations
- Sampling-time guidance requires more memory and runtime due to backpropagation through intermediate features.
- Can produce improbable cartoonish or unrealistic images that satisfy readout constraints.
Experimental setup
Datasets: PascalVOC DAVIS CelebA-HQ MSCOCO
Benchmarks: MSCOCO validation (pose PCK) qualitative drag-based manipulation comparisons
Metrics: PCK at alpha=0.05 qualitative visual comparison
Baselines: DragDiffusion ControlNet T2I-Adapter
Builds on: classifier guidance Diffusion Hyperfeatures DDIM ControlNet T2I-Adapter
Closest papers by meaning
- REG: Rectified Gradient Guidance for Conditional Diffusion Models (ICML 2025 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.89)
- Understanding and Improving Training-free Loss-based Diffusion Guidance (NeurIPS 2024 · similarity 0.89)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.88)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.