Research map / Image generation / Autoregressive and latent generation
Generating Handwritten Mathematical Expressions From Symbol Graphs: An End-to-End Pipeline
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end graph-to-layout-to-mask-to-image pipeline that generates handwritten math expression images from symbol graphs, using a Less-is-More training strategy and differentiable BBox-to-Mask refinement.
Problem
Generating high-quality, recognizable handwritten mathematical expression images from LaTeX/symbolic sequences, where no real masks are available for supervising layout refinement.
Approach
Formulates HME generation as graph-to-image generation with a GCN layout predictor trained on 1-degree sub-graphs (Less-is-More), a differentiable sequential BBox-to-Grid-to-Mask module that avoids real masks, and joint end-to-end optimization with GAN-based discriminators.
Key results
- On CROHME2019, the proposed method reaches SSIM 0.793, WER 0.326 and ExpRate 0.316, versus Sg2im at 0.787, 0.393 and 0.219.
- Synthetic augmentation with 6,000 generated samples improves CAN HMER results by about 5-11% absolute across CROHME14/16/19 per the paper.
- Removing LiM decreases mIoU from 0.364 to 0.335 in the ablation on CROHME2019.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use symbol graphs with LiM training and sequential B2M instead of Sg2im | HMEG · CROHME2019 | WER: 0.393 -> 0.326 | Helped |
| use symbol graphs with LiM training and sequential B2M instead of Sg2im | HMEG · CROHME2019 | ExpRate: 0.219 -> 0.316 | Helped |
| use symbol graphs with LiM training and sequential B2M instead of Sg2im | HMEG · CROHME2019 | FID: 10.08 -> 10.98 | Hurt |
| add LiM training for the layout predictor (ablation) | HMEG · CROHME2019 | mIoU: 0.335 -> 0.364 | Helped |
| add LiM training for the layout predictor (ablation) | HMEG · CROHME2019 | ExpRate: 0.315 -> 0.316 | No clear effect |
| add the B2M module (ablation) | HMEG · CROHME2019 | mIoU: 0.354 -> 0.364 | Helped |
| add the B2M module (ablation) | HMEG · CROHME2019 | WER: 0.332 -> 0.326 | Helped |
| add LiM and B2M (ablation) | HMEG · CROHME2019 | ExpRate: 0.279 -> 0.316 | Helped |
| add 6,000 generated samples for data augmentation | CAN · CROHME2014 | ExpRate: 44.7 -> 50.2 | Helped |
| add 6,000 generated samples for data augmentation | CAN · CROHME2016 | ExpRate: 42.8 -> 53.9 | Helped |
| add 6,000 generated samples for data augmentation | CAN · CROHME2019 | ExpRate: 39.3 -> 49.6 | Helped |
Surprising findings
- Sg2im achieves a lower FID (10.02) than the proposed method (10.98) on CROHME2019, so image realism is roughly on par despite better recognition scores.
Limitations
- Uses naive networks due to computing limits; generated symbols are occasionally unrecognizable for complex formulas.
- Performance on HMER augmentation is measured only with CAN and one HMER-recognizer, and the generated data is evaluated with the same recognizer used for scoring.
Experimental setup
Datasets: CROHME2014 CROHME2016 CROHME2019
Benchmarks: CROHME2014 CROHME2016 CROHME2019
Metrics: FID SSIM WER ExpRate mIoU StruRate
Baselines: CycleGAN FormulaGAN Sg2im ControlNet Stable Diffusion + LoRA DiffSketcher DiffAE
Builds on: Sg2im GCN (Gated Graph Neural Networks) CRN AC-GAN CAN
Closest papers by meaning
- SSAN: A Symbol Spatial-Aware Network for Handwritten Mathematical Expression Recognition (AAAI 2025 · similarity 0.90)
- Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition (NeurIPS 2025 · similarity 0.87)
- TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition (AAAI 2025 · similarity 0.85)
- Neuro-Symbolic Data Generation for Math Reasoning (NeurIPS 2024 · similarity 0.81)
- Generative Graph Pattern Machine (NeurIPS 2025 · similarity 0.81)
- A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.