Research map / Image generation / Editing and style transfer
Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSD-Font, a latent diffusion model that splits the reverse process into structure construction, font transfer, and refinement stages, using a dual-network prediction for few-shot Chinese font generation.
Problem
Few-shot font generation (FFG) must produce stylized glyphs from few reference samples, but GAN-based style-content disentanglement methods often yield broken structures, inconsistent styles, and artifacts.
Approach
Incorporates a font transfer process into the latent diffusion forward process over a time interval (t1, t2], so the reverse denoising naturally separates into structure construction, font transfer, and refinement stages, with two separate prediction networks for the transfer and refinement stages.
Key results
- On UFUC, MSD-Font reaches RMSE 0.2475, PSNR 12.45, SSIM 0.7189, LPIPS 0.1527, beating NTF (0.2574 RMSE) and FsFont (0.2509 RMSE) on RMSE.
- On UFSC (unseen fonts, seen contents), MSD-Font achieves RMSE 0.2439 and PSNR 12.61, the best among compared methods.
- Dual-network design improves RMSE from 0.2601 (Single-Net) to 0.2475 on UFUC; the latent diffusion base model with a single net already improves over pixel-space baseline (0.2676 to 0.2601).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MSD-Font instead of FsFont | Few-shot font generation · UFSC | RMSE: 0.2443 -> 0.2439 | Helped |
| use MSD-Font instead of FsFont | Few-shot font generation · UFSC | PSNR: 12.56 -> 12.61 | Helped |
| use MSD-Font instead of FsFont | Few-shot font generation · UFSC | LPIPS: 0.1468 -> 0.1348 | Helped |
| use MSD-Font instead of FsFont | Few-shot font generation · UFUC | RMSE: 0.2509 -> 0.2475 | Helped |
| use MSD-Font instead of FsFont | Few-shot font generation · UFUC | PSNR: 12.29 -> 12.45 | Helped |
| use MSD-Font instead of FsFont | Few-shot font generation · UFUC | SSIM: 0.7185 -> 0.7189 | Helped |
| use MSD-Font instead of FsFont | Few-shot font generation · UFUC | LPIPS: 0.1642 -> 0.1527 | Helped |
| replace the traditional DDPM process with the proposed latent diffusion process (ablation) | Font generation on UFUC · UFUC | RMSE: 0.2676 -> 0.2475 | Helped |
| add a second stage-specific prediction network (ablation) | Font generation on UFUC · UFUC | RMSE: 0.2601 -> 0.2475 | Helped |
| add a second stage-specific prediction network (ablation) | Font generation on UFUC · UFUC | PSNR: 12.02 -> 12.45 | Helped |
| add a second stage-specific prediction network (ablation) | Font generation on UFUC · UFUC | SSIM: 0.7096 -> 0.7189 | Helped |
| add the proposed latent diffusion process (ablation) | Font generation on UFUC · UFUC | PSNR: 11.72 -> 12.45 | Helped |
Limitations
- Higher inference time and model size than GAN-based FFG methods.
- Hyperparameters t1 and t2 require tuning as trade-off choices.
Experimental setup
Datasets: 900 collected Chinese fonts (840 train / 60 test)
Benchmarks: Chinese font dataset UFUC (unseen fonts, unseen contents) UFSC (unseen fonts, seen contents)
Metrics: RMSE PSNR SSIM LPIPS
Baselines: LF-Font DG-Font MX-Font FsFont NTF Diff-Font
Builds on: Latent Diffusion Model DDPM VQ-VAE
Closest papers by meaning
- FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning (AAAI 2024 · similarity 0.90)
- IF-Font: Ideographic Description Sequence-Following Font Generation (NeurIPS 2024 · similarity 0.88)
- VecFusion: Vector Font Generation with Diffusion (CVPR 2024 · similarity 0.87)
- VQ-FONT: Few-Shot Font Generation with Structure-Aware Enhancement and Quantization (AAAI 2024 · similarity 0.87)
- DeepCalliFont: Few-Shot Chinese Calligraphy Font Synthesis by Integrating Dual-Modality Generative Models (AAAI 2024 · similarity 0.85)
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.