Research map / Image generation / Autoregressive and latent generation
Personalized Representation from Personalized Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns instance-specific ("personalized") vision representations by LoRA-contrastive fine-tuning a general backbone on DreamBooth/cut-and-paste synthetic images generated from only three real images, and introduces the PODS dataset.
Problem
General-purpose vision encoders adapt poorly to fine-grained, instance-level recognition when only a few real images of one object are available; the paper asks whether personalized synthetic data can yield a personalized representation.
Approach
Personalize a T2I model (DreamBooth) on a few real instance images, generate synthetic positives of that instance and negatives from the generic category prompt, then LoRA-finetune a pretrained encoder with an InfoNCE objective; also compares cheap cut-and-paste and mask-based variants.
Key results
- Personalized DINOv2 (DreamBooth) improves over pretrained DINOv2 on 32 of 36 task/backbone/dataset cases; e.g., CLIP classification improves about 89% on average relative to pretrained CLIP.
- Combined Masked DreamBooth + Cut/Paste with masks performs best across most datasets, e.g., PODS classification 61.9 vs 28.1 for pretrained DINOv2 (table of best-per-backbone results reports DINOv2-P PODS classification 48.0).
- Synthetic augmentation remains effective as real positives scale: 27% gain with 3 real images and 8% gain with 20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train DINOv2 with personalized DreamBooth synthetic data | DINOv2 representation · PODS, DF2, Dogs | classification, retrieval, detection, and segmentation scores | Mixed |
| Use LLM-generated prompts instead of no LLM prompts (ablation) | personalized representations trained with DreamBooth · PODS, DF2, Dogs | classification and retrieval | Helped |
| Use LLM-generated prompts instead of no LLM prompts (ablation) | personalized representations trained with DreamBooth · PODS, DF2, Dogs | detection and segmentation | No clear effect |
| Increase anchor-positive pairs from 300 to 3000 (ablation) | DINOv2-P on DF2 validation · DF2 validation | classification: 27.2 -> 38.1 | Helped |
| Increase anchor-positive pairs from 300 to 3000 (ablation) | DINOv2-P on DF2 validation · DF2 validation | retrieval: 51.2 -> 62.1 | Helped |
| Increase anchor-positive pairs from 3000 to 30000 (ablation) | DINOv2-P on DF2 validation · DF2 validation | classification: 38.1 -> 36.9 | Hurt |
| Increase generated positives from 3 to 450 (ablation) | DINOv2-P on DF2 validation · DF2 validation | classification: 12.8 -> 38.7 | Helped |
| Increase generated positives from 3 to 450 (ablation) | DINOv2-P on DF2 validation · DF2 validation | retrieval: 36.0 -> 61.9 | Helped |
| Increase generated positives from 450 to 600 (ablation) | DINOv2-P on DF2 validation · DF2 validation | retrieval: 61.9 -> 60.8 | Hurt |
| Use InfoNCE instead of Multi-Positive InfoNCE (ablation) | DINOv2-P on DF2 validation · DF2 validation | classification: 27.5 -> 36.5 | Helped |
| Use InfoNCE instead of Cross-Entropy (ablation) | DINOv2-P on DF2 validation · DF2 validation | retrieval: 37.0 -> 63.3 | Helped |
| Combine Masked DreamBooth and Cut/Paste data (ablation) | DINOv2-P · PODS, DF2, Dogs | classification, retrieval, detection, and segmentation | Helped |
Surprising findings
- Cut-and-Paste with real backgrounds performs comparably to generative Masked DreamBooth at a fraction of the compute (about 0.45 min per 450 images vs about 11-156 min).
- DreamBooth-trained and Cut/Paste-trained representations fail in complementary ways: DB excels at pose generalization but confuses similar instances, while CP is robust to distractors but struggles with unfamiliar poses.
Limitations
- High computational cost of T2I generation (DreamBooth fine-tuning and sampling) and filtering.
- Inherits biases and limitations of the T2I model; DreamBooth struggles to preserve fine-grained details.
- Evaluation uses training-free probing without task-specific heads, so it does not reach state-of-the-art segmentation/detection performance.
Experimental setup
Datasets: DeepFashion2 DogFaceNet PODS
Benchmarks: PODS DeepFashion2 (reformulated) DogFaceNet (reformulated)
Metrics: PR-AUC NDCG AP F1
Baselines: DINOv2 (pretrained) CLIP (pretrained) MAE (pretrained) Real-image-only contrastive fine-tuning Cut/Paste Masked DreamBooth perSAM
Builds on: DreamBooth DINOv2 InfoNCE perSAM Stable Diffusion 1.5
Closest papers by meaning
- Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.87)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.87)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.86)
- Contextualized Visual Personalization in Vision-Language Models (ICML 2026 · similarity 0.86)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.86)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.