Research map / Image generation / Editing and style transfer
FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FontDiffuser, a diffusion-based one-shot Chinese font generator with multi-scale content aggregation, reference-structure interaction, and a contrastive style loss from a pre-trained style extractor.
Problem
One-shot font generation struggles with complex characters (many strokes) and large style variations between source and reference fonts, where GAN-based and prior-based methods fail with stroke loss, artifacts, and style inconsistency.
Approach
Model font imitation as a conditional noise-to-denoise diffusion process; inject multi-scale content features from a content encoder into the UNet (MCA), use cross-attention plus deformable convolution to handle structural deformation (RSI), and supervise the diffusion output with a style contrastive loss computed by a pre-trained VGG-based style extractor (SCR).
Key results
- On UFUC average, FontDiffuser reaches FID 8.5352, SSIM 0.4206, LPIPS 0.1496, L1 0.3870, the best on all metrics in the table.
- On UFSC, FontDiffuser achieves FID 7.6708 vs 7.7232 for CG-GAN and 12.0809 for Diff-Font.
- In a 25-participant user study, FontDiffuser was preferred in 47.15% of UFUC cases versus 20.97% for CG-GAN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FontDiffuser instead of CG-GAN | Chinese font generation · SFUC average | LPIPS: 0.1961 -> 0.1416 | Helped |
| use FontDiffuser instead of CF-Font | Chinese font generation · UFUC average | LPIPS: 0.2180 -> 0.1496 | Helped |
| use FontDiffuser instead of CG-GAN | Chinese font generation · UFSC | FID: 7.7232 -> 7.6708 | Helped |
| add MCA to the baseline (ablation) | FontDiffuser baseline · UFUC average | FID: 8.1153 -> 7.8419 | Helped |
| add RSI after MCA (ablation) | FontDiffuser with MCA · UFUC average | FID: 7.8419 -> 8.4427 | Hurt |
| add SCR after MCA and RSI (ablation) | FontDiffuser with MCA and RSI · UFUC average | SSIM: 0.4137 -> 0.4206 | Helped |
| add SCR augmentation (ablation) | FontDiffuser phase 2 · UFUC average | SSIM: 0.4172 -> 0.4206 | Helped |
| replace RSI CNN interaction with cross-attention (ablation) | FontDiffuser RSI · UFUC average | FID: 9.1659 -> 8.5352 | Helped |
| use 32 rather than 16 negative samples for SCR (ablation) | FontDiffuser SCR · UFUC average | FID: 8.5352 -> 8.0692 | Helped |
| use 8 rather than 16 negative samples for SCR (ablation) | FontDiffuser SCR · UFUC average | FID: 8.5352 -> 8.5900 | Hurt |
| use 48 rather than 16 negative samples for SCR (ablation) | FontDiffuser SCR · UFUC average | FID: 8.5352 -> 8.2454 | Hurt |
| use multiple VGG layer features instead of layer 3 only (ablation) | FontDiffuser SCR · UFUC average | FID: 9.2220 -> 8.2173 | Helped |
Surprising findings
- Ablations show the full model has a slightly worse FID than the MCA-only variant (8.54 vs 7.84), with the paper noting FID is not improved by each module.
- Guidance scale 1 yields the best FID (7.14) but worse SSIM, LPIPS, and L1 than scale 7.5.
Limitations
- Sampling still requires multiple denoising steps and is slower than GAN-based methods.
- Quantitative FID is not uniformly improved by each proposed module (ablation).
- Pixel-level metrics like SSIM and L1 may not reflect visual quality, as the authors note.
Experimental setup
Datasets: Chinese font dataset (424 fonts, 400 for training)
Benchmarks: SFUC UFUC UFSC
Metrics: FID SSIM LPIPS L1 user study preference
Baselines: FUNIT LF-Font DG-Font MX-Font Fs-Font CG-GAN CF-Font Diff-Font
Builds on: DDPM CG-GAN DPM-Solver++ classifier-free guidance deformable convolution
Closest papers by meaning
- Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models (CVPR 2024 · similarity 0.90)
- VecFusion: Vector Font Generation with Diffusion (CVPR 2024 · similarity 0.87)
- Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation (NeurIPS 2026 · similarity 0.86)
- Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models (ICLR 2025 · similarity 0.86)
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.86)
- Font-Agent: Enhancing Font Understanding with Large Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.