Research map / Vision-language and multimodal / Human and embodied modeling

WeGen: A Unified Model for Interactive Multimodal Generation as We Chat

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Text-driven human motion generation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WeGen, a unified MLLM-plus-SDXL model for interactive multimodal generation and understanding, using video-derived DIIC data for identity consistency and prompt self-rewriting for diversity.

Problem

Existing multimodal generative models struggle to keep identity consistency with reference images and to produce diverse outputs from underspecified instructions, limiting their use as interactive design copilots.

Approach

Combines an LLM (LLaMA-2-7B-chat) with frozen EVA-CLIP features and an SDXL decoder trained on interleaved text-visual sequences; adds a Dynamic Instance Identity Consistency data pipeline built from video tracking (Grounding DINO, SAM2) and a Prompt Self-Rewriting mechanism that samples detailed prompts before image feature generation.

Key results

• Achieves FID 9.39 and CLIP-T 0.308 on COCO2014 text-to-image, the best among MLLM-based methods reported.
• On DreamBench, reaches DINO 0.823 and CLIP-I 0.882, above OmniGen (0.801 / 0.847).
• Top-P plus temperature prompt sampling raises PSNRd diversity to 9.66 versus 19.88 without sampling, at a small CLIP-T cost.

Measured effects reported in the paper

Surprising findings

• Pure prompt sampling without rewriting yields only modest CLIP-T change, while the combined Top-P and temperature setting gives the best trade-off, which is a tuning-dependent result.
• Authors note that a larger reference-image encoder improves reconstruction, suggesting encoder information loss is a key bottleneck for identity consistency.

Limitations

• Diversity gains come with lower subject fidelity on some metrics, e.g. PSNRd-based diversity measured alongside CLIP-T trade-offs.
• Relies on large-scale CLIP encoder and SDXL decoder, adding parameters (WeGen 7B plus frozen components).
• Dataset and rewriting data release are promised but not yet available.

Experimental setup

Datasets: LAION COCO2014 DreamBench DIIC (internet videos)

Benchmarks: COCO2014 DreamBench

Metrics: FID CLIP-T DINO CLIP-I PSNRd LPIPSd

Baselines: GLIDE LDM DALL-E 2 SDXL Imagen Chameleon GILL Kosmos-G Emu Emu2-Gen SEED-X OmniGen

Builds on: GILL Emu2 Kosmos-G SDXL SEED-X

Closest papers by meaning

• DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.92)
• MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.88)
• Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models (ICLR 2026 · similarity 0.88)
• OmniGen2: Towards Instruction-Aligned Multimodal Generation (CVPR 2026 · similarity 0.88)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.87)
• IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.87)