Research map / Image generation / Text-to-image diffusion
DreamCache: Finetuning-Free Lightweight Personalized Image Generation via Feature Caching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamCache, a finetuning-free personalized image generator that caches multi-resolution reference features from a few U-Net layers at one timestep and injects them via 25M-parameter attention adapters trained on a synthetic triplet dataset.
Problem
Personalized text-to-image generation needs to preserve a reference subject while keeping text control, but existing methods require costly test-time finetuning, large encoders, or per-step reference processing that raises inference cost and memory.
Approach
Cache clean reference activations from selected denoiser layers at t=1 with a null text prompt, then train lightweight spatial cross-attention adapters (concatenated with self-attention outputs) on a frozen Stable Diffusion U-Net to inject them during sampling.
Key results
- On SD 2.1, DreamCache reaches DINO 0.767 and CLIP-T 0.301 on DreamBooth, with only 25M trainable parameters.
- On SD 1.5, DreamCache achieves DINO 0.713, the highest DINO among finetuning-free methods in the table, with CLIP-T 0.298.
- Inference takes 3.88 s per image on an A100 with a 42 MB extra-parameter footprint, versus 7.55 s and 1900 MB for BootPIG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use textual sum conditioning instead of spatial concatenation (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.788 | Hurt |
| Use textual sum conditioning instead of spatial concatenation (ablation) | DreamCache · DreamBooth | CLIP-T: 0.298 -> 0,282 | Hurt |
| Sum spatial self- and cross-attention outputs instead of concatenating them (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.812 | No clear effect |
| Use independent interleaved cross-attention blocks instead of spatial concatenation (ablation) | DreamCache · DreamBooth | CLIP-T: 0.298 -> 0.300 | Helped |
| Apply conditioning only in the encoder (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.721 | Hurt |
| Apply conditioning in the encoder and middle layer (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.749 | Hurt |
| Apply conditioning only in the middle layer (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.716 | Hurt |
| Apply conditioning only in the decoder (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.799 | Hurt |
| Add encoder conditioning to the middle-and-decoder configuration (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.813 | Helped |
| Add text conditioning while caching reference features (ablation) | DreamCache · DreamBooth | CLIP-T: 0.298 -> 0.295 | Helped |
| Train adapters on Synthetic-50K rather than Synthetic-400K (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.781 | Hurt |
| Train adapters on Synthetic-200K rather than Synthetic-400K (ablation) | DreamCache · DreamBooth | CLIP-I: 0.810 -> 0.797 | Hurt |
Surprising findings
- Caching reference features with a text prompt did not help; the text-free variant matched or slightly beat it on CLIP-T.
- CLIP-I of Kosmos-G is high but reflects background replication rather than subject preservation, per the authors.
Limitations
- May need adaptation for multi-subject generation where feature interference can occur.
- Edge cases such as highly abstract or stylistic reference images can challenge subject preservation.
Experimental setup
Datasets: DreamBooth dataset LAION-5M (ablation)
Benchmarks: DreamBooth dataset
Metrics: DINO CLIP-I CLIP-T inference time extra parameters
Baselines: DreamBooth Textual Inversion Custom Diffusion ELITE BLIP-Diffusion IP-Adapter Kosmos-G JeDi SuTI Subject-Diffusion BootPIG ToffeeNet
Builds on: Stable Diffusion BootPIG IP-Adapter ViCo
Closest papers by meaning
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.86)
- DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation (ICLR 2025 · similarity 0.86)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.86)
- DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models (NeurIPS 2024 · similarity 0.85)
- DeepCache: Accelerating Diffusion Models for Free (CVPR 2024 · similarity 0.85)
- DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.