Research map / Image generation / Editing and style transfer
DemoCaricature: Democratising Caricature Generation with a Rough Sketch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DemoCaricature, a single-image text-to-image personalisation method with Explicit Rank-1 Model Editing and Random Mask Reconstruction that turns a photo plus a rough sketch (and optional style image) into a personalised caricature.
Problem
Generating personalised caricatures that balance identity preservation with sketch-driven abstraction and style, from a single photo and a rough freehand sketch, without overfitting to one reference image.
Approach
Edits only the concept-token entry of the cross-attention Key/Value projections with a rank-1 cosine-gated update (Explicit ROME), adds random-mask latent reconstruction loss, and combines a frozen T2I-Adapter sketch conditioning on Stable Diffusion v1.5, enabling identity and style concepts to be merged without interference.
Key results
- Ours-full reaches ID 0.671, style 0.576 CLIP-Score at comparable shape similarity, outperforming TI (0.634/0.553) and Perfusion (0.536/0.549).
- Human study (15 users, 300 responses per method) gives overall score 4.1 for the proposed method versus 2.9 for TI and 2.7 for Perfusion.
- Explicit ROME uses about 30x fewer learnable parameters than Perfusion and fine-tunes in about 1 minute for identity on a single GTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Explicit ROME editing (ablation) | DemoCaricature · CLIP-Score ID | similarity: 0.664 -> 0.671 | Helped |
| add Explicit ROME editing (ablation) | DemoCaricature · CLIP-Score Style | similarity: 0.530 -> 0.576 | Helped |
| add Explicit ROME editing (ablation) | DemoCaricature · CLIP-Score Shape | similarity: 0.661 -> 0.654 | Hurt |
| add random mask reconstruction (ablation) | DemoCaricature · CLIP-Score ID | similarity: 0.659 -> 0.671 | Helped |
| add random mask reconstruction (ablation) | DemoCaricature · CLIP-Score Style | similarity: 0.567 -> 0.576 | Helped |
| add random mask reconstruction (ablation) | DemoCaricature · CLIP-Score Shape | similarity: 0.694 -> 0.654 | Hurt |
| replace Mahalanobis distance with cosine similarity (ablation) | DemoCaricature · CLIP-Score ID | similarity: 0.666 -> 0.671 | Helped |
| replace Mahalanobis distance with cosine similarity (ablation) | DemoCaricature · CLIP-Score Style | similarity: 0.574 -> 0.576 | Helped |
| replace Mahalanobis distance with cosine similarity (ablation) | DemoCaricature · CLIP-Score Shape | similarity: 0.663 -> 0.654 | Hurt |
| use DemoCaricature instead of Textual Inversion | caricature generation · CLIP-Score ID | similarity: 0.634 -> 0.671 | Helped |
| use DemoCaricature instead of Perfusion | caricature generation · CLIP-Score Style | similarity: 0.549 -> 0.576 | Helped |
| use DemoCaricature instead of Perfusion | caricature generation · CLIP-Score Shape | similarity: 0.676 -> 0.654 | Hurt |
Surprising findings
- Removing explicit editing drops style similarity sharply (to 0.530 from 0.576 in the table), indicating style blending is the main failure mode without it.
- Replacing the Mahalanobis distance with cosine similarity improved both visual quality and ID/style CLIP similarities slightly.
Limitations
- Evaluation relies on CLIP-Score proxies and a small human study rather than established identity metrics.
- Curated test set is small (20 identities, 4 styles) and not released.
Experimental setup
Datasets: WebCaricature
Benchmarks: WebCaricature test set (curated 20 identities, 4 styles, 12 shapes)
Metrics: CLIP-Score for ID CLIP-Score for style CLIP-Score for shape human study score (1-5)
Baselines: Textual Inversion Perfusion StyleCariGAN CariGANs WarpGAN
Builds on: ROME Perfusion Textual Inversion T2I-Adapter Stable Diffusion
Closest papers by meaning
- CaricatureBooth: Data-Free Interactive Caricature Generation in a Photo Booth (CVPR 2025 · similarity 0.88)
- MagiCapture: High-Resolution Multi-Concept Portrait Customization (AAAI 2024 · similarity 0.84)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.84)
- One-Shot Reference-based Structure-Aware Image to Sketch Synthesis (AAAI 2025 · similarity 0.83)
- DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation (CVPR 2024 · similarity 0.83)
- Zero-shot Image Editing with Reference Imitation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.