Research map / Image generation / Text-to-image diffusion
Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hollowed Net, a LoRA personalization method for Stable Diffusion that trains on a layer-pruned U-Net with its middle deep blocks temporarily removed, using precomputed activations to cut training GPU memory to near-inference levels.
Problem
Subject-driven personalization of text-to-image diffusion models is memory-prohibitive on resource-constrained devices, because backpropagation through the full U-Net requires keeping the whole model in GPU memory, and prior efficiency work focuses on training steps or parameter counts rather than m
Approach
Identify U-Net blocks with small LoRA weight changes (central deep blocks), temporarily remove them to form a "hollowed" U-Net, pre-compute intermediate activations from the full model, and fine-tune LoRA on the hollowed network; the resulting LoRA weights transfer back to the original U-Net for inference without extra memory.
Key results
- Hollowed Net (39.2% hollowed) reaches 3.88GB peak training memory, only +11% over inference, versus 5.23GB (+50%) for LoRA r=128 and 16.62GB for full fine-tuning.
- On DreamBooth, Hollowed Net achieves DINO 0.660, CLIP-I 0.805, CLIP-T 0.300, comparable to LoRA r=128 (0.658, 0.806, 0.299) and full FT (0.663, 0.802, 0.302).
- Total training FLOPs are 2051.6 TFLOPs for Hollowed Net versus 2148 TFLOPs for LoRA FT, but inference costs about 0.204 TFLOPs more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace full U-Net fine-tuning with Hollowed Net fine-tuning | Stable Diffusion v2.1 · DreamBooth | DINO: 0.663 -> 0.660 | No clear effect |
| Replace LoRA FT (r=128) with Hollowed Net | Stable Diffusion v2.1 · DreamBooth | Peak training memory: 5.23GB -> 3.88GB | Helped |
| Replace LoRA FT (r=128) with Hollowed Net | Stable Diffusion v2.1 · DreamBooth | DINO: 0.658 -> 0.660 | No clear effect |
| Reduce LoRA rank from 128 to 1 (ablation) | LoRA FT · DreamBooth | DINO: 0.658 -> 0.516 | Hurt |
| Increase LoRA rank from 4 to 16 (ablation) | LoRA FT · DreamBooth | DINO: 0.564 $\pm$ 0.014 -> 0.618 $\pm$ 0.008 | Helped |
| Increase LoRA rank from 4 to 16 (ablation) | Hollowed Net · DreamBooth | DINO: 0.566 $\pm$ 0.009 -> 0.626 $\pm$ 0.009 | Helped |
| Remove 39.2% of U-Net parameters during fine-tuning (ablation) | Hollowed Net · DreamBooth | Peak GPU memory: 3.88GB | Helped |
| Increase hollowed fraction beyond 39.2% (ablation) | Hollowed Net · DreamBooth | Subject fidelity (DINO and CLIP-I) | Hurt |
| Use 200 rather than 1000 pre-computed samples (ablation) | Hollowed Net · DreamBooth | Personalization performance | No clear effect |
| Use layer-pruned BK-SDM instead of Hollowed Net | DreamBooth personalization · DreamBooth | DINO: 0.629 $\pm$ 0.012 (BK-SDM-Base); 0.602 $\pm$ 0.013 (BK-SDM-Small) | Hurt |
Surprising findings
- Pruning deep layers of pretrained SD (BK-SDM) causes significant degradation even after additional pre-training, unlike LLM layer pruning.
- Subject fidelity stays comparable to LoRA until about 39.2% of layers are removed, then drops sharply.
Limitations
- Performance is sensitive to coarse class tokens (e.g., 'toy' for poop emoji) because intermediate activations come from non-personalized prompts.
- Inference requires about 0.204 TFLOPs more than LoRA FT because part of early down-blocks is recomputed.
- Full fine-tuning slightly outperforms the method on DINO, though differences are within standard deviation.
Experimental setup
Datasets: DreamBooth CustomConcept101
Benchmarks: DreamBooth CustomConcept101
Metrics: DINO CLIP-I CLIP-T peak training GPU memory FLOPs human preference
Baselines: Full fine-tuning LoRA FT (r=128) LoRA FT (r=1) BK-SDM-Base BK-SDM-Small
Builds on: LoRA DreamBooth Ladder Side Tuning Stable Diffusion
Closest papers by meaning
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.88)
- Efficient Personalization of Quantized Diffusion Model without Backpropagation (CVPR 2025 · similarity 0.88)
- Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.87)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.86)
- Dis²Booth: Learning Image Distribution with Disentangled Features for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.