Research map / Image generation / Text-to-image diffusion
Zero-Shot Styled Text Image Generation, but Make It Autoregressive
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Emuru, a continuous-token autoregressive model (VAE plus T5-style Transformer) trained only on 2.2M synthetic text-line images to generate zero-shot styled handwritten and typewritten text images of any length without background artifacts.
Problem
Offline styled handwritten text generation struggles to generalize to unseen styles, is limited in output length, and reproduces background artifacts from style references.
Approach
Encode text-line images with a single-channel text-specific VAE into a continuous latent sequence, then autoregressively generate latent vectors with a T5-style encoder-decoder conditioned on style latents and text, stopping when padding embeddings appear; training on a massive synthetic font dataset with background removal enables zero-shot style transfer.
Key results
- Emuru VAE reaches average FID 19.22 and BFID 1.62 versus 29.39/7.36 for SD1.5 VAE, while using ~16% of its parameters.
- Zero-shot on CVL lines, Emuru achieves FID 14.39 vs 31.22 for the best competitor HWT; on RIMES lines FID 26.93 vs 89.79 for DiffPen.
- On IAM words, Emuru (zero-shot) FID 63.61 trails DiffPen (15.54) and on IAM lines it is comparable to DiffPen (13.89 vs 12.89), showing it matches IAM-trained competitors at line level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on synthetic text rendered in over 100,000 fonts rather than training on a single HTG dataset | Emuru · CVL Lines and RIMES Lines | FID | Helped |
| Use a one-channel text-image VAE trained to reconstruct text without the background | Emuru VAE · HTG datasets, mean | FID: 21.90 -> 19.22 | Helped |
| Use a one-channel text-image VAE trained to reconstruct text without the background | Emuru VAE · HTG datasets, mean | BFID: 3.61 -> 1.62 | Helped |
Surprising findings
- On IAM word-level, the zero-shot Emuru is worse than IAM-trained DiffPen on FID, showing weaker word-level performance despite strong line-level results.
- Emuru has higher ΔCER than several baselines, indicating it does not guarantee text correctness.
Limitations
- Word-level generation is weaker than specialized diffusion models on FID (IAM words).
- No explicit mechanism ensures text correctness, leading to higher ΔCER on some datasets.
Experimental setup
Datasets: synthetic text-line dataset (2.2M images, NLTK English words, 100k+ fonts) IAM CVL RIMES
Benchmarks: IAM CVL RIMES Karaoke
Metrics: FID BFID KID ΔCER HWD
Baselines: TS-GAN HiGAN+ HWT VATr VATr++ One-DM DiffPen
Builds on: GIVT T5 Stable Diffusion VAE VATr DiffPen
Closest papers by meaning
- Handwritten Text Generation From Visual Archetypes (CVPR 2023 · similarity 0.89)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.85)
- AnyText: Multilingual Visual Text Generation and Editing (ICLR 2024 · similarity 0.85)
- Decoupling Layout from Glyph in Online Chinese Handwriting Generation (ICLR 2025 · similarity 0.84)
- Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens (ICLR 2025 · similarity 0.84)
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.