Research map / Image generation / Text-to-image diffusion
UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniMC, a PixArt-α DiT-based keypoint-guided generator conditioned on per-instance class, box and keypoint tokens, plus the HAIG-2.9M human-and-animal keypoint image dataset.
Problem
Keypoint-guided diffusion methods rely on skeleton images, which cannot disambiguate class or instance for multi-class, overlapping humans and animals, and existing keypoint datasets cover only humans or only animals.
Approach
Replace skeleton-image conditions with compact per-instance tokens (class embedding, Fourier-encoded keypoints and bounding box) and inject them into a DiT via a timestep-aware keypoint modulator with self-attention and a shared timestep adapter.
Key results
- UniMC reaches 91.71% animal class accuracy and 28.38 animal pose AP, versus 52.03% and 20.29 for ControlNet and 9.90% and 3.96 for GLIGEN.
- Training on HAIG-2.9M gives FID 23.63 versus 33.79 for COCO+APT36K, with human class accuracy 93.55% versus 67.97%.
- Config a (self-attention with shared timestep adapter) is best among four modulator variants; inserting the controller in all 28 layers performs best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniMC rather than PixArt-α | PixArt-α · HAIG-2.9M test | Animal class accuracy (%): 13.79 -> 91.71 | Helped |
| Use UniMC rather than ControlNet | Keypoint-guided generation · HAIG-2.9M test | Animal pose accuracy (AP): 20.29 -> 28.38 | Helped |
| Use UniMC rather than GLIGEN | Keypoint-guided generation · HAIG-2.9M test | Animal pose accuracy (AP): 3.96 -> 28.38 | Helped |
| Replace Config a with GLIGEN-style gated self-attention (ablation) | UniMC timestep-aware keypoint modulator · HAIG-2.9M test | Animal pose accuracy (AP): 28.38 -> 18.05 | Hurt |
| Use pretrained timestep-encoder features instead of the newly introduced timestep adapter (ablation) | UniMC timestep-aware keypoint modulator · HAIG-2.9M test | Animal pose accuracy (AP): 28.38 -> 19.26 | Hurt |
| Replace global-wise modulation with block-wise adaLN-Zero time embedding (ablation) | UniMC timestep-aware keypoint modulator · HAIG-2.9M test | Animal pose accuracy (AP): 28.38 -> 23.55 | Hurt |
| Insert controller only in the first 14 layers instead of all 28 (ablation) | UniMC Config a · HAIG-2.9M test | Animal pose accuracy (AP): 28.38 -> 26.55 | Hurt |
| Insert controller only in the last 14 layers instead of all 28 (ablation) | UniMC Config a · HAIG-2.9M test | Animal pose accuracy (AP): 28.38 -> 27.95 | Hurt |
| Insert controller in the first 7 and last 7 layers instead of all 28 (ablation) | UniMC Config a · HAIG-2.9M test | Animal pose accuracy (AP): 28.38 -> 27.28 | Hurt |
Surprising findings
- GLIGEN-style gated self-attention, which works for UNet models, performs worst when transplanted to the DiT backbone.
- Training only on COCO yields very poor animal generation, and the base PixArt-α scores 0.06 animal pose AP, so it cannot control animal poses without the new training.
Limitations
- Long-tail class distribution in HAIG-2.9M (e.g., rhino, hippo, buffalo have few samples).
- Limited number of categories (31 classes); controllable generation of arbitrary non-rigid objects remains open.
Experimental setup
Datasets: HAIG-2.9M COCO APT36K JourneyDB SA-1B LAION-PoP LAION-Aesthetics-V2 Pexels Pixabay Stocksnap Unsplash
Benchmarks: HAIG-2.9M test split
Metrics: FID KID CLIP score class accuracy (YOLO-World detection, IoU>0.5) pose AP human preference study
Baselines: PixArt-α ControlNet GLIGEN
Builds on: PixArt-α DiT GLIGEN ControlNet
Closest papers by meaning
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.88)
- MEAT: Multiview Diffusion Model for Human Generation on Megapixels with Mesh Attention (CVPR 2025 · similarity 0.87)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.87)
- Cross-view Masked Diffusion Transformers for Person Image Synthesis (ICML 2024 · similarity 0.87)
- UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation (CVPR 2025 · similarity 0.86)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.