Research map / Image generation / Editing and style transfer
Learning Flow Fields in Attention for Controllable Person Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Leffa, a flow-field-based attention regularization loss added to a SD1.5 diffusion baseline to reduce fine-grained texture distortion in virtual try-on and pose transfer.
Problem
Controllable person image generation (virtual try-on and pose transfer) distorts fine-grained textures from reference images because the target query fails to attend to corresponding reference regions in attention.
Approach
Convert the attention map between target queries and reference keys into a flow field that warps the reference image toward the target, and add an L2 warping loss on the masked region so attention is explicitly supervised to focus on correct correspondences during training only, with no extra parameters or inference cost.
Key results
- On VITON-HD paired setting, Leffa reaches FID 4.54 vs 5.42 for CatVTON (previous best), with KID 0.05.
- On DressCode (all categories), Leffa achieves paired FID 2.06 vs 3.99 for CatVTON.
- Adding the Leffa loss to IDM-VTON and CatVTON lowers their paired FID by 0.64 and 0.56 on VITON-HD, showing model-agnostic gains.
- In a 50-participant human study, Leffa received the highest preference share across VITON-HD, DressCode and DeepFashion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Leffa attention-flow regularization (ablation) | SD1.5-based Leffa · VITON-HD paired and unpaired | FID: 5.31 -> 4.54; 9.38 -> 8.52 | Helped |
| Add Leffa loss | IDM-VTON · VITON-HD paired and unpaired | FID: 5.76 -> 5.12; 9.84 -> 9.24 | Helped |
| Add Leffa loss | CatVTON · VITON-HD paired and unpaired | FID: 5.42 -> 4.86; 9.02 -> 8.77 | Helped |
| Add Leffa attention-flow regularization | Leffa · VITON-HD paired and unpaired | FID: 5.42 -> 4.54; 9.02 -> 8.52 | Helped |
| Add Leffa attention-flow regularization | Leffa · DressCode paired and unpaired | FID: 3.99 -> 2.06; 6.14 -> 4.48 | Helped |
| Add Leffa attention-flow regularization | Leffa · DeepFashion 256 × 176 and 512 × 352 | FID: 6.25 -> 5.71; 5.93 -> 4.23 | Helped |
| Add Leffa attention-flow regularization | Leffa · DeepFashion 256 × 176 | SSIM: 0.737 -> 0.732 | Hurt |
| Add Leffa attention-flow regularization | Leffa · DeepFashion 256 × 176 and 512 × 352 | LPIPS: 0.152 -> 0.114; 0.152 -> 0.119 | Helped |
| Increase Leffa loss weight from 0.0 to 10−3 (ablation) | Leffa · VITON-HD | FID | Helped |
| Increase Leffa loss weight beyond 10−3 (ablation) | Leffa · VITON-HD | FID | Hurt |
| Set the attention resolution threshold outside the 1/32–1/16 range (ablation) | Leffa · VITON-HD | FID | Hurt |
| Increase the timestep threshold to 750 or 1000 (ablation) | Leffa · VITON-HD | FID | Hurt |
Limitations
- Leffa loss is applied only to high-resolution attention layers and small timesteps, so its benefit depends on hyperparameter selection (resolution threshold, timestep threshold, temperature).
- Overly strong loss weight degrades generation quality.
- Authors note future work toward a unified model controlling both appearance and pose.
Experimental setup
Datasets: VITON-HD DressCode DeepFashion
Benchmarks: VITON-HD DressCode DeepFashion
Metrics: FID KID SSIM LPIPS human preference
Baselines: IDM-VTON CatVTON OOTDiffusion StableVITON PIDM CFLD NTED X-MDPT-L PoCoLD GP-VTON LADI-VTON DCI-VTON
Builds on: Stable Diffusion progressive training (Zhu et al.) dot-product attention
Closest papers by meaning
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.86)
- IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation (NeurIPS 2024 · similarity 0.85)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.85)
- Attention Distillation: A Unified Approach to Visual Characteristics Transfer (CVPR 2025 · similarity 0.84)
- CapHuman: Capture Your Moments in Parallel Universes (CVPR 2024 · similarity 0.84)
- MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.