Research map / Image generation / Text-to-image diffusion
XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XVerse, a tuning-light multi-subject personalization method for FLUX that injects reference-image features as token-specific offsets to the text-stream modulation, plus VAE features in single blocks, and introduces the XVerseBench benchmark.
Problem
Multi-subject personalization in Diffusion Transformers struggles to preserve individual identities and semantic attributes without causing artifacts, attribute entanglement, or degraded editability when reference images are injected via attention.
Approach
Instead of injecting reference features through attention, a perceiver-resampler adapter maps CLIP image features and text features to shared and per-block offsets added to the text-stream AdaLN modulation vector for each token, with VAE features used only as a supporting signal in single blocks and region-preservation and text-image attention losses for disentanglement.
Key results
- On XVerseBench, XVerse reaches an overall score of 73.40, above DreamO (69.25) and the other baselines.
- Multi-subject AVG of 70.08 with ID-Sim 66.59 and IP-Sim 71.48, the best among compared methods.
- Single-subject AVG of 76.72 with ID-Sim 79.48, the best single-subject average, while DreamO leads on DPG (96.93 vs 93.69).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use XVerse instead of DreamO | single-subject generation · XVerseBench single-subject | AVG: 74.46 -> 76.72 | Helped |
| Use XVerse instead of DreamO | multi-subject generation · XVerseBench multi-subject | AVG: 64.04 -> 70.08 | Helped |
| Use XVerse instead of DreamO | overall controlled generation · XVerseBench overall | Overall: 69.25 -> 73.40 | Helped |
| Add text-image attention loss (ablation) | XVerse · XVerseBench | text-image editability and semantic interaction | Helped |
Surprising findings
- The authors report that a reconstruction-only training paradigm produces copy-paste effects, motivating a third stage with cross-image data.
- CLIP-T is argued to be less informative than DPG for complex prompts, so the authors rely on DPG for editability.
Limitations
- Lack of high-quality, large-scale cross-image multi-subject datasets.
- Focus limited to the text-stream modulation pathway; image-modulation and pixel- or region-level control left unexplored.
- Injection fails when the injected prompt semantically mismatches the reference image (e.g., person reference with a dog prompt).
Experimental setup
Datasets: Subject200K DreamBench++ in-house multi-view human dataset Florence-2 and SAM2 annotated 1M single-image set FLUX-synthesized 1M high-aesthetic images
Benchmarks: XVerseBench
Metrics: DPG ID-Sim IP-Sim AES AVG
Baselines: MS-Diffusion MIP-Adapter OmniGen UNO OmniGen2 DreamO
Builds on: FLUX StyleGAN OmniControl UNO DreamBench++
Closest papers by meaning
- VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis (AAAI 2025 · similarity 0.87)
- Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (CVPR 2025 · similarity 0.87)
- PreciseCam: Precise Camera Control for Text-to-Image Generation (CVPR 2025 · similarity 0.86)
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.86)
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.86)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.