Research map / Video / Video generation
TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TweedieMix, a tuning-free inference-time method that composes multiple personalized diffusion models by splitting sampling into a multi-object content-aware stage with resampling and a fusion stage that blends concept-specific denoised predictions via Tweedie's formula, plus a training-free feature-injection extension to image-to-video models.
Problem
Generating images and videos that faithfully integrate multiple personalized concepts is hard: joint training and weight merging cause concept blending, concept loss, and poor scaling beyond a few concepts.
Approach
Use a non-fine-tuned base model with CFG++ and a resampling strategy early to secure all objects, extract regional masks with a text-guided segmenter, then fuse per-concept fine-tuned models' denoised predictions (Tweedie space) under those masks, sharing the base unconditional score; extend to I2V by injecting first-frame residual features.
Key results
- On 10 multi-concept combinations from CustomConcept101, TweedieMix reaches CLIP text sim 0.3816 and image sim 0.8311, versus 0.3707 / 0.8095 for ConceptWeaver.
- DINO image similarity 0.5950 for TweedieMix vs 0.5352 for ConceptWeaver and 0.5211 for Mix-of-Show.
- In a 20-participant user study, video results scored 4.5 for the method vs 1.7 for DreamVideo; image user study gave best text-match, concept-match, and realism scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TweedieMix instead of Concept Weaver | Multi-concept image generation · Multi-concept image generation | CLIP text sim: 0.3707 -> 0.3816 | Helped |
| Use TweedieMix instead of Concept Weaver | Multi-concept image generation · Multi-concept image generation | CLIP image sim: 0.8095 -> 0.8311 | Helped |
| Use TweedieMix instead of Concept Weaver | Multi-concept image generation · Multi-concept image generation | DINO image sim: 0.5352 -> 0.5950 | Helped |
| Use TweedieMix for multi-concept video generation instead of DreamVideo | Multi-concept video generation · Multi-concept video generation | User preference: 1.7 -> 4.5 | Helped |
| Add resampling (ablation) | TweedieMix · Multi-concept image generation | CLIP text sim: 0.3551 -> 0.3872 | Helped |
| Add resampling (ablation) | TweedieMix · Multi-concept image generation | CLIP image sim: 0.8078 -> 0.8202 | Helped |
| Use CFG++ and its resampling strategy instead of standard CFG without resampling (ablation) | TweedieMix · Multi-concept image generation | CLIP text sim: 0.3496 -> 0.3872 | Helped |
| Use CFG++ and its resampling strategy instead of standard CFG without resampling (ablation) | TweedieMix · Multi-concept image generation | CLIP image sim: 0.8020 -> 0.8202 | Helped |
| Add initial content-aware sampling instead of starting fusion with manual masks (ablation) | TweedieMix · Multi-concept image generation | CLIP text sim: 0.3427 -> 0.3872 | Helped |
| Add initial content-aware sampling instead of starting fusion with manual masks (ablation) | TweedieMix · Multi-concept image generation | CLIP image sim: 0.8026 -> 0.8202 | Helped |
| Mix concepts in Tweedie's denoised space instead of the noisy latent space (ablation) | TweedieMix · Multi-concept image generation | CLIP image sim: 0.8146 -> 0.8202 | Helped |
| Inject first-frame residual features into later video frames (ablation) | TweedieMix video extension · Multi-concept video generation | Concept appearance preservation | Helped |
Surprising findings
- Using the non-fine-tuned model's unconditional score across all concepts works better than each concept model's own null-text output, which hindered mixing between objects and background.
Limitations
- Performance on highly complex or unrealistic text prompts remains limited in text alignment.
- Fine details of target concepts can be lost, constrained by the Stable Diffusion backbone.
- Mix-of-Show requires manual region guidance in the baseline comparison, and video comparison with DreamVideo is not direct since the method extends an I2V model.
Experimental setup
Datasets: CustomConcept101
Benchmarks: CustomConcept101
Metrics: CLIP text similarity CLIP image similarity DINO image similarity user study preference score
Baselines: Textual Inversion Custom Diffusion Perfusion Mix-of-Show ConceptWeaver DreamVideo
Builds on: CFG++ Custom Diffusion DreamBooth LoRA DDIM
Closest papers by meaning
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.88)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.87)
- Personalized Residuals for Concept-Driven Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis (ICLR 2025 · similarity 0.87)
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.87)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.