Research map / Image generation / Text-to-image diffusion
SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSR-Encoder, a finetuning-free image encoder that selects subject representations via text or mask queries using a token-to-patch aligner and multi-scale CLIP features, plugged into Stable Diffusion for single- and multi-subject generation.
Problem
Subject-driven image generation without test-time finetuning struggles to select and focus on the crucial subject representation in reference images (backgrounds and mixed subjects confuse encoders), and encoder methods are often tied to a specific base diffusion model.
Approach
A token-to-patch aligner (trainable linear projections over CLIP text tokens and image patches) produces attention maps that select subject regions by text or mask query; a detail-preserving encoder projects multi-layer CLIP features weighted by these maps into parallel subject cross-attention layers, trained with an embedding consistency regularization loss.
Key results
- On DreamBench, SSR-Encoder reaches CLIP-I 0.821 and aesthetic score 6.563, beating the fine-tuning-based Dreambooth (0.803, 6.330) and other finetuning-free methods.
- On the Multi-subject bench, SSR-Encoder achieves the lowest CLIP-ES (0.182, better exclusivity) among compared methods, with DINO-M 0.556 versus Dreambooth's 0.681.
- Ablation: removing the multi-scale features drops DINO-M from 0.556 to 0.510; removing the regularization loss lowers CLIP-T from 0.302 to 0.235 and raises CLIP-ES to 0.199.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SSR-Encoder instead of Dreambooth | Stable Diffusion V1-5 subject-driven generation · Multi-subject bench | CLIP-T: 0.298 -> 0.302 | Helped |
| use SSR-Encoder instead of Break-A-Scene | Stable Diffusion V1-5 subject-driven generation · Multi-subject bench | CLIP-ES: 0.187 -> 0.182 | Helped |
| use SSR-Encoder instead of Dreambooth | Stable Diffusion V1-5 subject-driven generation · Multi-subject bench | DINO-M: 0.681 -> 0.556 | Hurt |
| use SSR-Encoder instead of Dreambooth | Stable Diffusion V1-5 subject-driven generation · DreamBench | CLIP-T: 0.305† -> 0.308 | Helped |
| use SSR-Encoder instead of Dreambooth | Stable Diffusion V1-5 subject-driven generation · DreamBench | DINO: 0.668† -> 0.612 | Hurt |
| use SSR-Encoder instead of IP-adapter | Stable Diffusion V1-5 subject-driven generation · DreamBench | CLIP-I: 0.809 -> 0.821 | Helped |
| use SSR-Encoder instead of IP-adapter | Stable Diffusion V1-5 subject-driven generation · DreamBench | Aesthetic Score: 6.432 -> 6.563 | Helped |
| add multi-scale visual embeddings (ablation) | SSR-Encoder · Multi-subject bench | CLIP-T: 0.257 -> 0.302 | Helped |
| add multi-scale visual embeddings (ablation) | SSR-Encoder · Multi-subject bench | CLIP-ES: 0.185 -> 0.182 | Helped |
| add multi-scale visual embeddings (ablation) | SSR-Encoder · Multi-subject bench | DINO-M: 0.510 -> 0.556 | Helped |
| add embedding consistency regularization loss (ablation) | SSR-Encoder · Multi-subject bench | CLIP-T: 0.235 -> 0.302 | Helped |
| add embedding consistency regularization loss (ablation) | SSR-Encoder · Multi-subject bench | CLIP-ES: 0.199 -> 0.182 | Helped |
Surprising findings
- Dreambooth retains higher DINO-M (0.681) than SSR-Encoder (0.556) on the Multi-subject bench, so the method does not win on all subject-alignment metrics.
Limitations
- Multi-subject DINO-M is below Dreambooth, which the authors note is strong at detailed subject capture.
- Training uses 10M LAION pairs re-captioned by BLIP2; the self-built Multi-subject bench is small (100 images).
Experimental setup
Datasets: LAION-5B DreamBench
Benchmarks: Multi-subject bench DreamBench
Metrics: CLIP-T CLIP-ES DINO-M DINO CLIP-I Aesthetic Score
Baselines: Textual Inversion Dreambooth Break-a-Scene Reference Only ELITE IP-adapter BLIP-Diffusion
Builds on: Stable Diffusion CLIP ControlNet IP-adapter
Closest papers by meaning
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.85)
- DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation (ICLR 2024 · similarity 0.85)
- Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization (AAAI 2025 · similarity 0.84)
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.84)
- Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning (NeurIPS 2024 · similarity 0.84)
- Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.