Research map / Image generation / Text-to-image diffusion
Preventing Shortcuts in Adapter Training via Providing the Shortcuts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Shortcut-Rerouted adapter training, which routes confounding factors (distribution shift, pose, expression, lighting) through frozen auxiliary modules (LoRA, ControlNet, background adapter) during IP-Adapter identity training and removes them at inference, improving prompt adherence and prior preservation on FLUX.1-dev face and body personalization.
Problem
Identity adapters trained with single-image reconstruction entangle the target identity with incidental factors such as pose, expression, lighting, and dataset style, which limits generalization and prompt adherence in personalized text-to-image generation.
Approach
Explicitly provide the shortcuts to be avoided during adapter training: confounders are absorbed by frozen auxiliary modules (a dataset-domain LoRA, a pose/expression ControlNet, or a background adapter) so the adapter has no incentive to encode them, and these modules are discarded at inference.
Key results
- On face adapters, SR-CN IPA reduces head pose error to 12.68 degrees versus 16.12 for IPA and improves prior LPIPS to 0.3937 versus 0.4800.
- On body adapters, SR-CN IPA achieves body pose error 137.69 versus 167.40 for IPA and prior LPIPS 0.4133 versus 0.4566.
- SR-CN IPA gets the highest EMOCA expression score (0.5800) among face adapters, while FaceNet identity is slightly lower than PuLID and InfU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reroute pose and expression through ControlNet during training | IP-Adapter face personalization · Face adapters | Head Pose: 16.1199 -> 12.6755 | Helped |
| Reroute pose and expression through ControlNet during training | IP-Adapter face personalization · Face adapters | Prior (LPIPS): 0.4800 -> 0.3937 | Helped |
| Reroute pose and expression through ControlNet during training | IP-Adapter face personalization · Face adapters | FaceNet Id.: 0.7150 -> 0.7118 | Hurt |
| Reroute pose and expression through ControlNet during training | IP-Adapter face personalization · Face adapters | EMOCA Expr.: 0.3470 -> 0.5800 | Helped |
| Reroute pose and expression through ControlNet during training | IP-Adapter body personalization · Body adapters | Body Pose: 167.4000 -> 137.6888 | Helped |
| Reroute pose and expression through ControlNet during training | IP-Adapter body personalization · Body adapters | Prior (LPIPS): 0.4566 -> 0.4133 | Helped |
| Reroute dataset distribution shift through LoRA during training | IP-Adapter face personalization · Face adapters | Prior (LPIPS): 0.4800 -> 0.4330 | Helped |
Surprising findings
- Face identity distance (FaceNet) fails to capture identity shifts from head shape and hairstyle, motivating an LLM-based identity judge.
- Adding a shortcut module during training can improve prompt-driven expression control even though the module is removed at inference.
Limitations
- Evaluation focuses on encoder-based (IP-Adapter) adapters; applicability to LoRA training is not demonstrated.
- Only the simple IP-Adapter is used as the base; stronger baselines might yield different results.
- Code and internal dataset cannot be released due to legal and compliance constraints.
Experimental setup
Datasets: internal dataset of a few million curated human images
Benchmarks: internal evaluation prompts (accessory, activity, expression, lighting, pose categories)
Metrics: LLM Id. FaceNet Id. LLM Expr. EMOCA Expr. Head Pose Body Pose Prior (LPIPS)
Baselines: InfU PuLID IP-Adapter (trained without shortcut rerouting) InstantX/FLUX.1-dev-IP-Adapter
Builds on: IP-Adapter ControlNet LoRA FLUX.1 [Dev] conditional flow matching
Closest papers by meaning
- Improved Training Technique for Shortcut Models (NeurIPS 2025 · similarity 0.86)
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · similarity 0.84)
- DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation (ICLR 2025 · similarity 0.84)
- UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation (CVPR 2025 · similarity 0.84)
- Adapters Strike Back (CVPR 2024 · similarity 0.84)
- Sparse autoencoders reveal selective remapping of visual concepts during adaptation (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.