Research map / Image generation / Editing and style transfer
Latent Space Editing in Transformer-Based Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explores image editing in transformer-based (U-ViT) flow matching by identifying a semantic latent "u-space" for vector-arithmetic edits, enabling adaptive-step ODE solvers via direction interpolation, and proposing self-attention prompt reweighting for local text-driven edits.
Problem
Latent structure and editing ability of transformer-based flow matching models (U-ViT backbone) are unknown, and prior semantic editing methods rely on fixed-step solvers and UNet-specific cross-attention.
Approach
Defines a semantic editing space (u-space) at the beginning of U-ViT, computes attribute directions from real image latents across time, interpolates these directions to support adaptive step-size ODE solvers, and performs local prompt editing by scaling attention between prompt tokens and image tokens inside the full self-attention of U-ViT.
Key results
- Local prompt reweighting in U-ViT outperforms Prompt-to-Prompt in LPIPS fidelity to the unedited image across reweighting scales on MM-CelebA-HQ (200 prompts).
- Semantic direction interpolation with adaptive solvers (dopri5, bosh3, adaptive_heun) gives edits visually similar to fixed-step euler with 100 steps.
- Injecting semantic guidance in early time steps (about 30-50 of 100 steps) gives effective edits while preserving the source image; injecting over all steps over-constrains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Inject semantic guidance during the first 30–50 of 100 sampling steps instead of only the first 5–10 steps (ablation) | u-space semantic editing · CelebA-HQ 256×256, male attribute | edit effectiveness | Helped |
| Inject guidance through all 100 sampling steps instead of limiting it to early steps (ablation) | u-space semantic editing · CelebA-HQ 256×256, male attribute | source-image preservation | Hurt |
| Distribute guidance evenly throughout 100 steps instead of applying it in early steps (ablation) | u-space semantic editing · CelebA-HQ 256×256, male attribute | semantic edit effectiveness | Hurt |
| Use the intermediate U-ViT layer for semantic manipulation instead of the beginning of U-ViT (ablation) | u-space semantic editing · CelebA-HQ 256×256 | semantic edit quality | Hurt |
| Use interpolated semantic directions with adaptive-step ODE solvers instead of fixed-step Euler (ablation) | u-space semantic editing · CelebA-HQ, male attribute | visual similarity of edited results | No clear effect |
| Use nearest-neighbor direction lookup instead of semantic-direction interpolation (ablation) | u-space semantic editing · CelebA-HQ, male attribute | relative edit error | Mixed |
| Increase the number of direction-collection time steps (ablation) | u-space semantic editing · CelebA-HQ, male attribute | relative edit error | Helped |
| Reweight prompt-token attention in U-ViT instead of using prompt-to-prompt | local-prompt editing · MM-CelebA-HQ | LPIPS perceptual distance to unedited image | Helped |
| Remove an uninformative prompt such as “he has” | local-prompt editing · text-conditioned image editing | image change | No clear effect |
| Increase the attention rescaling factor excessively (ablation) | local-prompt editing · text-conditioned image editing | source-image preservation | Hurt |
| Edit along PCA directions in u-space instead of using supervised semantic directions (ablation) | u-space editing · CelebA-HQ samples | consistency of semantic edits | Hurt |
| Apply semantic edits in different prompt orders (ablation) | sequential u-space editing · CelebA-HQ | final edit result consistency | No clear effect |
Surprising findings
- Splitting the guidance injection evenly across the whole trajectory fails to achieve semantic manipulation, even when injected every 5th step.
- Middle-layer (UNet-style) latent space does not yield a robust semantic space in U-ViT; the beginning of U-ViT works best.
Limitations
- Over-constraining with too large guidance or self-attention rescaling factors degrades images.
- Concept entanglement when adding an attribute already present (e.g., male to male image).
- Background may change slightly due to convolutional encoder/decoder; only images with captions are editable.
Experimental setup
Datasets: CelebA-HQ MultiModal-CelebA-HQ MS-COCO
Benchmarks: CelebA-HQ MultiModal-CelebA-HQ MS-COCO
Metrics: LPIPS relative edit error
Baselines: Prompt-to-Prompt
Builds on: U-ViT Flow Matching Prompt-to-Prompt Diffusion h-space editing
Closest papers by meaning
- Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing (CVPR 2025 · similarity 0.88)
- FluxSpace: Disentangled Semantic Editing in Rectified Flow Models (CVPR 2025 · similarity 0.87)
- SplitFlow: Flow Decomposition for Inversion-Free Text-to-Image Editing (NeurIPS 2025 · similarity 0.87)
- Stable Flow: Vital Layers for Training-Free Image Editing (CVPR 2025 · similarity 0.87)
- Safe Few-Step Generation via Velocity Editing (NeurIPS 2026 · similarity 0.86)
- Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.