Research map / Vision-language and multimodal / Human and embodied modeling
Towards Visual Text Design Transfer Across Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MuST-Bench, a multilingual film-poster benchmark for visual text style translation, and SIGIL, a Stable Diffusion-based method using VAE-latent glyph guidance and OCR-reward RL to generate style-consistent, legible text across languages.
Problem
Translating visual text design (style, typography) across writing systems is hard: text-to-image models rely on style descriptions that cannot convey visual design, and existing visual text generators struggle with non-Latin scripts and style fidelity.
Approach
Replace style text descriptions with style images: fine-tune a diffusion model on 3-5 style images of a source-language glyph, guide generation with a glyph loss computed in the pretrained VAE latent space, and refine legibility with an RL step that uses EasyOCR confidence as reward (with a character-count penalty against exploitation).
Key results
- SIGIL reaches OCR accuracy of 0.7163 (English), 0.7481 (Chinese), 0.6577 (Korean) vs. best baseline DS-Fusion 0.4808, AnyText 0.3548, and DALL-E 3 at 0 for Chinese/Korean.
- In GPT-4V Likert evaluation SIGIL scores 3.89 (English), 3.50 (Chinese), 4.00 (Korean) vs. best baseline DALL-E 3 at 3.00 for English.
- In the user study SIGIL gets ~0.40 style-fidelity ranking share across languages vs. at most 0.23 for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SIGIL instead of AnyText | Visual text generation · MuST-Bench English, Chinese, and Korean | OCR: 0.3333 -> 0.7163; 0.3548 -> 0.7481; 0.3245 -> 0.6577 | Helped |
| Use SIGIL instead of AnyText | Visual text generation · MuST-Bench English, Chinese, and Korean | CLIP-I: 0.8731 -> 0.8673; 0.8688 -> 0.8618; 0.8671 -> 0.8692 | Mixed |
| Add glyph latent loss (ablation) | SIGIL · MuST-Bench English, Chinese, and Korean | OCR: 0.001 -> 0.7163; 0.001 -> 0.7481; 0.002 -> 0.6577 | Helped |
| Add OCR-reward Corrector (ablation) | SIGIL · MuST-Bench English, Chinese, and Korean | OCR: 0.4529 -> 0.7163; 0.3962 -> 0.7481; 0.2188 -> 0.6577 | Helped |
| Add style-and-glyph Generator (ablation) | SIGIL · MuST-Bench English, Chinese, and Korean | OCR: 0.18 -> 0.7163; 0 -> 0.7481; 0 -> 0.6577 | Helped |
| Increase glyph loss scale λ from 0.1 to 0.5 (ablation) | SIGIL · MuST-Bench | OCR: 0.7163 -> 0.9423 | Helped |
| Increase glyph loss scale λ from 0.1 to 0.5 (ablation) | SIGIL · MuST-Bench | CLIP-I: 0.8673 -> 0.8565 | Hurt |
| Decrease glyph loss scale λ from 0.1 to 0.01 (ablation) | SIGIL · MuST-Bench | OCR: 0.7163 -> 0.0038 | Hurt |
| Decrease glyph loss scale λ from 0.1 to 0.01 (ablation) | SIGIL · MuST-Bench | CLIP-I: 0.8673 -> 0.8855 | Helped |
| Increase glyph influence factor τ from 50 to 100 (ablation) | SIGIL · MuST-Bench | OCR: 0.7163 -> 0.0663 | Hurt |
| Decrease glyph influence factor τ from 50 to 10 (ablation) | SIGIL · MuST-Bench | OCR: 0.7163 -> 0.9276 | Helped |
| Decrease glyph influence factor τ from 50 to 10 (ablation) | SIGIL · MuST-Bench | CLIP-I: 0.8673 -> 0.8603 | Hurt |
Surprising findings
- AnyText achieves the highest CLIP-I scores in some languages, which the authors attribute to background inclusion in the similarity computation, despite lower legibility.
- Removing the glyph loss drops OCR accuracy to near zero (0.001-0.002) while CLIP-I rises, showing a trade-off between glyph correctness and style similarity.
Limitations
- Small dataset (about 1,120 posters; 60-432 pairs per language) and low-resolution poster crops complicate VAE encoding.
- Complex strokes in Chinese and Korean glyphs are hard to generate accurately.
- EasyOCR reward has false negatives and limited vocabulary, which can mislead RL optimization.
Experimental setup
Datasets: MuST-Bench
Benchmarks: MuST-Bench
Metrics: OCR accuracy (PP-OCRv3) CLIP-I MLLM Likert style fidelity score user study ranking (style fidelity, legibility)
Baselines: DALL-E 3 AnyText DS-Fusion
Builds on: DreamBooth Stable Diffusion DDPO (Black et al.) PPO AnyText
Closest papers by meaning
- DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · similarity 0.88)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.87)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- Harmonizing Visual Text Comprehension and Generation (NeurIPS 2024 · similarity 0.86)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.86)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.