Research map / Image generation / Text-to-image diffusion
Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ctrl-X performs training-free, guidance-free zero-shot structure and appearance control for T2I (SDXL) and T2V diffusion models by injecting structure features/attention and applying spatially-aware (attention-weighted) appearance statistic transfer during sampling.
Problem
Existing controllable diffusion methods either need training auxiliary modules on paired data or rely on guidance (backpropagated score functions), which is slow, memory-hungry, and sensitive to guidance strength.
Approach
Replace guidance with feed-forward feature and self-attention injection from a forward-noised structure latent, and transfer appearance via self-attention-weighted mean/std normalization of appearance features, enabling disentangled structure and appearance control with no training or gradients.
Key results
- Ctrl-X achieves DINO-I 0.686/0.698/0.676 across natural, ControlNet-supported, and new conditions, the best appearance alignment among all methods in Table 2.
- Ctrl-X inference takes 10.91 s versus 378.89 s total for FreeControl (about 35x faster) and 42.80 s for Cross-Image Attention, with 11.51 GiB peak memory on a single H100.
- In a 150-comparison-per-baseline user study, Ctrl-X's overall fidelity preference rates were 83% vs Cross-Image Attention, 74% vs FreeControl, and 74% vs Uni-ControlNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ctrl-X instead of training-based or guidance-based baselines | Structure and appearance control · ControlNet-supported and new conditions | DINO-I: 0.686 vs 0.555; 0.698 vs 0.574; 0.676 vs 0.506 | Helped |
| Use Ctrl-X instead of FreeControl | Conditional generation · ControlNet-supported and new conditions | LPIPS: 0.635 vs 0.557; 0.590 vs 0.492 | Helped |
| Use Ctrl-X instead of FreeControl | Conditional generation · ControlNet-supported and new conditions | CLIP score: 0.322 vs 0.340; 0.326 vs 0.339 | Hurt |
| Use Ctrl-X instead of FreeControl | Structure and appearance control · SDXL v1.0 inference | Inference latency (s): 139.53 -> 10.91 | Helped |
| Add structure control to appearance-only control (ablation) | Ctrl-X · Structure and appearance control | Structure preservation | Helped |
| Add appearance transfer to structure-only control (ablation) | Ctrl-X · Structure and appearance control | Image quality | Helped |
| Use attention-weighted appearance statistics instead of global normalization (ablation) | Ctrl-X appearance transfer · Structure and appearance control | Image contrast | Helped |
| Use forward diffusion instead of DDIM inversion (ablation) | Ctrl-X structure control · Challenging structure conditions | Structure-image appearance leakage | Helped |
| Enable self-recurrence steps (ablation) | Ctrl-X inference · Structure and appearance control | Artifacts | Helped |
| Decrease structure control schedule (ablation) | Ctrl-X · Cross-class structure-appearance pairs | Realism | Helped |
| Decrease appearance control schedule (ablation) | Ctrl-X · Challenging structure-appearance pairs | Artifacts | Helped |
Surprising findings
- Splicing ViT Features achieves the best DINO self-similarity and DINO-I scores yet its outputs are often blurry with structure appearance leakage, showing the metrics can be misleading.
- Uni-ControlNet attains strong structure preservation but weak appearance transfer, showing a trade-off between self-similarity and DINO-I.
Limitations
- Fails to localize and transfer appearance when the corresponding subject is very small due to low feature-map resolution.
- Structure and appearance schedules (tau_s, tau_a) require per-pair tuning for challenging pairs; default 0.6 does not work for all.
- Trade-off between structure preservation (self-similarity) and appearance similarity (DINO-I).
Experimental setup
Datasets: ImageNet-style natural images from the web SDXL-generated images ControlNet annotator-converted conditions
Benchmarks: Ctrl-X structure-appearance control dataset (256 pairs) Ctrl-X conditional generation dataset (175 pairs)
Metrics: DINO self-similarity DINO-I CLIP score LPIPS user preference rate inference latency peak GPU memory
Baselines: ControlNet + IP-Adapter T2I-Adapter + IP-Adapter Uni-ControlNet Splicing ViT Features FreeControl Cross-Image Attention SDEdit Prompt-to-Prompt Plug-and-Play InfEdit IP-Adapter
Builds on: FreeControl Cross-Image Attention AdaAttN DDIM Stable Diffusion XL
Closest papers by meaning
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.91)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control (NeurIPS 2024 · similarity 0.88)
- FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition (CVPR 2024 · similarity 0.88)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.88)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.