Research map / Image generation / Text-to-image diffusion
TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrioPose, a triple-stream pose-aware DiT built on SD3.5 with zero-initialized dual-residual injection, learnable relational bias masks, and heatmap-error loss weighting for multi-person pose-guided text-to-image generation.
Problem
Pose-guided text-to-image generation in multi-person scenes suffers from limb distortion and feature crosstalk; naive concatenation into MM-DiTs disrupts pre-trained latent distributions.
Approach
Treats pose as an independent third stream in an MM-DiT with layer-wise activation and zero-initialized dual residual injection, maps five topological occlusion states to learnable continuous attention biases, and re-weights the rectified-flow loss with ViTPose heatmap error maps.
Key results
- TrioPose† reaches 64.33 AP on Human-Art vs 49.50 for GRPose (best prior), with FID 1.65 vs 10.03 for HumanSD.
- Ablation: adding TSPA alone raises AP from 8.17 to 63.50; full model with LRBM and PSLW gives 64.33 AP and FID 1.65.
- Mixed-dataset training: TrioPose reaches 58.56 AP on CrowdPose vs 50.25 for Stable-Pose, and 62.59 AP on OCHuman vs 58.84.
Surprising findings
- The SD3.5M baseline without pose control gets only 8.17 AP, and the paper reports that naive concatenation disrupts the pre-trained distribution.
- A binary hard mask in place of the learnable 5-state mask yields a higher CLIP score (34.58) than the full model.
Limitations
- Relies on ViTPose-derived heatmap errors for loss weighting, and only evaluates on three pose benchmarks.
- Lacks reported inference cost; authors note future work on lightweight attention for inference acceleration.
- Authors flag deepfake misuse risk and plan watermarking of released checkpoints.
Experimental setup
Datasets: Human-Art CrowdPose OCHuman COCO-17
Benchmarks: Human-Art CrowdPose OCHuman
Metrics: AP CAP PCE FID KID CLIP-score
Baselines: T2I-Adapter ControlNet Uni-ControlNet GLIGEN HumanSD Stable-Pose GRPose SD3.5M
Builds on: SD3.5 ControlNet HumanSD ViTPose LoRA
Closest papers by meaning
- Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2024 · similarity 0.88)
- Fast Text-to-3D-Aware Face Generation and Manipulation via Direct Cross-modal Mapping and Geometric Regularization (ICML 2024 · similarity 0.87)
- EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing (ICLR 2025 · similarity 0.87)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.87)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.87)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.