Research map / Vision-language and multimodal / Multimodal LLMs
Font-Agent: Enhancing Font Understanding with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 135K-sample font evaluation dataset (DFD) and a Qwen-VL-based font agent with an Edge-Aware Traces frequency module and a Dynamic DPO variant for interpretable font quality assessment and recognition.
Problem
Font quality assessment models give only basic visual scores without interpretable explanations, and existing MLLMs lack font-specific knowledge and fine stroke/curvature perception.
Approach
Combine a fine-tuned Qwen-VL with an Edge-Aware Traces module (FPS-sampled Bezier points as a saliency map, frequency-domain weighting with CDC on magnitude/phase) fused via cross-attention, and train with a dynamic DPO that weights characters by self-consistency voting, uncertainty, and font features.
Key results
- On DFD-UN, Font-Agent reaches 93.9% ACC and 96.8% AUC versus 89.3%/87.9% for the best baseline Xception (Table 1).
- On the DFD quality assessment split, Font-Agent achieves SRCC 0.830 and PLCC 0.823, above Q-Align (0.785/0.815) (Table 2).
- On the font recognition benchmark, Font-Agent averages 91.32% word accuracy versus 89.46% for MAERec (Table 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Font-Agent instead of Xception | Font quality classification · DFD Avg. | ACC (%): 87.9 -> 93.8(↑5.9) | Helped |
| Use Font-Agent instead of Q-Align | Font quality assessment · DFD | SRCC: 0.785 -> 0.830 | Helped |
| Use Font-Agent instead of Compare2Score | Image quality assessment · BID | prediction accuracy: 0.861 -> 0.839 | Hurt |
| Use Font-Agent instead of Compare2Score | Image quality assessment · LIVE, CSIQ, KADID-10k, CLIVE, and KonIQ-10k | prediction accuracy | Helped |
| Add instruction tuning to the pre-trained MLLM (ablation) | Qwen-VL font assessment · DFD-UN | ACC (%): 57.5 -> 79.4 | Helped |
| Add EAT to the instructed MLLM (ablation) | Qwen-VL font assessment · DFD-UN | ACC (%): 79.4 -> 83.5 | Helped |
| Add D-DPO to the instructed MLLM (ablation) | Qwen-VL font assessment · DFD-UN | ACC (%): 79.4 -> 86.4 | Helped |
| Add both EAT and D-DPO (ablation) | Qwen-VL font assessment · DFD-UN | ACC (%): 86.4 -> 93.6 | Helped |
| Use FPS point sampling with EAT instead of Random or KNN sampling (ablation) | Font-Agent · Font quality assessment | SRCC and PLCC | Helped |
| Use D-DPO instead of DPO, DPOP, IPO, or TDPO (ablation) | Qwen-VL+EAT · Font quality assessment | SRCC and PLCC | Helped |
Surprising findings
- Off-the-shelf GPT-4o performs worse than traditional CNN classifiers on DFD quality classification (62.5% average ACC vs ~86% for Xception).
Limitations
- Dataset annotations are partly generated by GPT-4o, Gemini-Pro, and QwenVL with manual correction, and the dataset is not stated to be released.
- Many hyperparameters (thresholds, epsilon, alpha, gamma, sigma) are described without reported sensitivity analysis.
- Reported results contain inconsistencies (e.g., Table 1 vs. text on Xception, varying baseline citations), limiting confidence in comparisons.
Experimental setup
Datasets: DFD LIVE CSIQ KADID-10k BID CLIVE KonIQ-10k IIIT5k SVT IC13 IC15 SVTP
Benchmarks: DFD LIVE CSIQ KADID-10k BID CLIVE KonIQ-10k IIIT5k SVT IC13 IC15 SVTP
Metrics: ACC AUC SRCC PLCC Word Accuracy
Baselines: Meso4 CNN-Aug Xception EfficientNet-B4 LLaVA-v1.6 mPLUG-Owl2 InternVL2 GPT-4o BLIP2 NIMA MLSP MUSIQ
Builds on: Qwen-VL Direct Preference Optimization Q-Align E2STR
Closest papers by meaning
- FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning (AAAI 2024 · similarity 0.86)
- VQ-FONT: Few-Shot Font Generation with Structure-Aware Enhancement and Quantization (AAAI 2024 · similarity 0.86)
- VecGlypher: Unified Vector Glyph Generation with Language Models (CVPR 2026 · similarity 0.84)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.84)
- Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models (CVPR 2024 · similarity 0.84)
- Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective. (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.