Research map / Image generation / Text-to-image diffusion
JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JeDi, a finetuning-free personalized text-to-image model that adapts Stable Diffusion v1.4 into a joint-image diffusion model with coupled self-attention, trained on a synthetic same-subject dataset (S3) plus video and multi-view data.
Problem
Personalized text-to-image generation usually requires costly per-subject finetuning, while existing finetuning-free encoder-based methods lose appearance detail and underperform finetuning approaches.
Approach
Learn the joint distribution of multiple same-subject text-image pairs by coupling self-attention across images in a set, then personalize at inference by treating reference images as known elements in an inpainting-style sampler with image guidance, without an image encoder.
Key results
- JeDi with 3 input images reaches DINO 0.6791 and MDINO 0.8037, beating DreamBooth (0.6341 / 0.7115) and Custom Diffusion (0.6343 / 0.7109).
- JeDi with 1 input image achieves MDINO 0.7510 and CLIP-T 0.3040, outperforming finetuning-free BLIP Diffusion (MDINO 0.6692) and ELITE (MDINO 0.6805).
- Removing image guidance drops DINO from 0.7501 to 0.4652 in the ablation, showing image guidance is crucial.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase S3 dataset size from 0.6M to 1.6M samples (ablation) | JeDi · DreamBooth test set | DINO: 0.7501 -> 0.7467 | No clear effect |
| increase S3 dataset size from 0.6M to 1.6M samples (ablation) | JeDi · DreamBooth test set | CLIP-T: 0.3020 -> 0.3015 | No clear effect |
| train without synthetic S3 data instead of using 0.6M S3 samples (ablation) | JeDi · DreamBooth test set | DINO: 0.7501 -> 0.8895 | Hurt |
| add CLIP image embedding as extra conditioning (ablation) | JeDi · DreamBooth test set | DINO: 0.7501 -> 0.7432 | No clear effect |
| remove image guidance (ablation) | JeDi · DreamBooth test set | DINO: 0.7501 -> 0.4652 | Hurt |
| use the joint-image model instead of a CLIP image encoder baseline | JeDi · DreamBooth test set | DINO: 0.3411 -> 0.7501 | Helped |
Surprising findings
- Training with only video and multi-view data (dataset size 0) gives the best image alignment (MDINO 0.8931) but worst text alignment (CLIP-T 0.2524), indicating the model learns to copy inputs and ignore text.
Limitations
- Synthetic dataset generated via SDXL and ChatGPT may carry generator biases and limited diversity compared with real collections.
- Results on the 3-image setting depend on multiple references; single-image performance is weaker on some CLIP-I/DINO metrics.
Experimental setup
Datasets: Synthetic Same-Subject (S3) WebVid10M Objaverse LAION aesthetic DreamBooth test set
Benchmarks: DreamBooth test set
Metrics: CLIP-T CLIP-I DINO MCLIP-I MDINO
Baselines: DreamBooth Custom Diffusion BLIP Diffusion ELITE
Builds on: Stable Diffusion classifier-free guidance DreamBooth BLIP-Diffusion
Closest papers by meaning
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.90)
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.89)
- Dis²Booth: Learning Image Distribution with Disentangled Features for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- Efficient Personalization of Quantized Diffusion Model without Backpropagation (CVPR 2025 · similarity 0.88)
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.87)
- Diffusion Self-Distillation for Zero-Shot Customized Image Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.