Research map / Image generation / Text-to-image diffusion
UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UNIC-Adapter, an MM-DiT-based adapter on frozen SD3-medium that unifies pixel-level, subject-driven, and style-image controllable generation via instruction-guided cross-attention with RoPE.
Problem
Text-only prompts lack precise control over layout, object appearance, and style, and prior conditional-image methods need separate specialized models per control type; existing unified models require full-model multi-stage training.
Approach
Use MM-DiT blocks to jointly process task-instruction tokens and conditional-image latents in an adapter, then inject their keys/values into the main SD3 generation branch via an added cross-attention with RoPE on queries and keys, while freezing the base model.
Key results
- On pixel-level control, UNIC-Adapter reaches 38.94 canny F1 and 0.8369 HED SSIM, beating OmniGen (35.54, 0.8237) and ControlNet++ (37.04, 0.8097) on those metrics.
- On DreamBench subject-driven generation, UNIC-Adapter achieves DINO 0.816 versus OmniGen's 0.801 and SuTI's 0.741, with CLIP-T 0.306.
- Ablations show RoPE beats absolute position embedding and adding the extra query projection L_cross^q further improves canny F1 to 37.95 (from 34.65 with RoPE alone).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Kist without Kcon as the key feature (ablation) | UNIC-Adapter · MultiGen-20M pixel-level control | F1 Score: 31.32 -> 22.38 | Hurt |
| Use Kist without Kcon as the key feature (ablation) | UNIC-Adapter · DreamBench | DINO: 0.769 -> 0.694 | Hurt |
| Use Qimg as query and Kcon plus Kist as keys (ablation) | UNIC-Adapter · MultiGen-20M pixel-level control | F1 Score: 29.98 -> 31.32 | Helped |
| Remove position embedding (ablation) | UNIC-Adapter · MultiGen-20M canny | F1 Score: 37.95 -> 31.32 | Hurt |
| Use RoPE instead of no position embedding (ablation) | UNIC-Adapter · MultiGen-20M canny | F1 Score: 31.32 -> 34.65 | Helped |
| Add Lqcross with RoPE (ablation) | UNIC-Adapter · MultiGen-20M canny | F1 Score: 34.65 -> 37.95 | Helped |
| Add Lqcross with RoPE (ablation) | UNIC-Adapter · DreamBench | DINO: 0.761 -> 0.784 | Helped |
| Remove text features (ablation) | UNIC-Adapter · MultiGen-20M canny | F1 Score: 37.95 -> 36.31 | Hurt |
| Remove text features (ablation) | UNIC-Adapter · DreamBench | DINO: 0.784 -> 0.788 | Helped |
| Remove instruction-feature updates (ablation) | UNIC-Adapter · DreamBench | DINO: 0.788 -> 0.772 | Hurt |
| Replace the feature-injection cross-attention with addition (ablation) | UNIC-Adapter · MultiGen-20M canny | F1 Score: 37.95 -> 38.18 | Mixed |
| Replace the feature-injection cross-attention with addition (ablation) | UNIC-Adapter · DreamBench | DINO: 0.784 -> 0.749 | Hurt |
Surprising findings
- Injecting only the instruction features or only the conditional-image features through cross-attention performs worse than injecting both, and IPAdapter-Instruct* performs poorly on pixel-level control (canny F1 15.81) despite strong subject-driven scores.
Limitations
- Pixel-level control results are only comparable, not superior, on depth and segmentation tasks versus ControlNet++ and OmniGen.
- Style-image evaluation is qualitative only, with no quantitative metrics reported.
Experimental setup
Datasets: MultiGen-20M ADE20k OpenImages v7 GRIT OpenStory++ WikiArt StyleBooth DreamBench
Benchmarks: MultiGen-20M ADE20k DreamBench
Metrics: F1 score SSIM mIoU RMSE DINO CLIP-I CLIP-T
Baselines: T2I-Adapter Gligen ControlNet ControlNet++ UniControl Uni-ControlNet OmniGen Textual Inversion DreamBooth BLIP-Diffusion Re-Imagen SuTI
Builds on: MM-DiT IP-Adapter ControlNet IPAdapter-Instruct
Closest papers by meaning
- Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.88)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.87)
- PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions (ICLR 2025 · similarity 0.87)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.87)
- Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis (CVPR 2024 · similarity 0.86)
- UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.