Research map / Vision-language and multimodal / Human and embodied modeling
MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MarkushGrapher, a joint visual-text-layout and OCSR encoder-decoder that autoregressively outputs CXSMILES plus a substituent table for Markush structures, trained on a synthetic pipeline and evaluated on a new real-world benchmark M2S.
Problem
Recognizing multi-modal Markush structures (chemical structure templates with image backbones and textual variable-group definitions) in patent documents, which existing methods handle only partially and which lack real-world training data.
Approach
Jointly encode the Markush image, OCR text cells and their layout (via a UDOP-style VTL encoder) together with frozen MolScribe OCSR features, then decode an optimized CXSMILES graph plus a substituent table, trained on a synthetic Markush generation pipeline.
Key results
- On M2S, MarkushGrapher reaches 38% CXSMILES exact match versus 21% for MolScribe and 0% for GPT-4o and Pixtral-12B.
- On USPTO-Markush, MarkushGrapher reaches 32% CXSMILES exact match versus 7% for MolScribe.
- On MarkushGrapher-Synthetic, MarkushGrapher reaches 65% CXSMILES exact match versus 11% for MolScribe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MarkushGrapher instead of MolScribe | Markush structure recognition · M2S | CXSMILES EM: 21 -> 38 | Helped |
| use MarkushGrapher instead of MolScribe | Markush structure recognition · USPTO-Markush | CXSMILES EM: 7 -> 32 | Helped |
| use MarkushGrapher instead of MolScribe | Markush structure recognition · USPTO-Markush | CXSMILES T: 97 -> 74 | Hurt |
| add the OCSR encoder to the VTL encoder (ablation) | MarkushGrapher · USPTO-Markush | CXSMILES EM: 23 -> 32 | Helped |
| add the OCSR encoder to the VTL encoder (ablation) | MarkushGrapher · M2S | CXSMILES EM: 38 -> 38 | No clear effect |
| use late rather than early fusion of OCSR and VTL encodings (ablation) | MarkushGrapher · USPTO-Markush | CXSMILES EM: 28 -> 32 | Helped |
| add R-group compression to the optimized CXSMILES representation (ablation) | MarkushGrapher · M2S | CXSMILES EM: 30 -> 38 | Helped |
| add R-group compression to the optimized CXSMILES representation (ablation) | MarkushGrapher · USPTO-Markush | CXSMILES EM: 31 -> 32 | Helped |
| add atom indexing to the optimized CXSMILES representation (ablation) | MarkushGrapher · M2S | CXSMILES EM: 35 -> 38 | Helped |
| add atom indexing to the optimized CXSMILES representation (ablation) | MarkushGrapher · USPTO-Markush | CXSMILES EM: 24 -> 32 | Helped |
Surprising findings
- MolScribe achieves a higher CXSMILES Tanimoto score on USPTO-Markush (97 vs 74), indicating robustness on standard chemical structures despite lower exact match.
- General-purpose VLMs such as GPT-4o and Pixtral-12B essentially fail on Markush CXSMILES recognition, scoring 0% exact match on all three benchmarks.
Limitations
- Real-world evaluation set M2S is small (103 images) with selection bias toward claims.
- Performance on frequency variation indicators ('Sg' sections) is weak (31% on M2S, 18% on USPTO-Markush).
- Synthetic training lacks diversity in frequency variation indicators, as noted by the authors.
Experimental setup
Datasets: PubChem USPTO-Markush MarkushGrapher-Synthetic M2S SciAssess
Benchmarks: MarkushGrapher-Synthetic M2S USPTO-Markush SciAssess
Metrics: CXSMILES exact match Tanimoto similarity substituent table exact match table F1-score Markush structure exact match
Baselines: DECIMER MolScribe Pixtral-12B-2409 Llama-3.2-11B-Vision-Instruct GPT-4o Uni-SMART
Builds on: UDOP MolScribe T5 CDK CXSMILES
Closest papers by meaning
- Atom-Level Optical Chemical Structure Recognition with Limited Supervision (CVPR 2024 · similarity 0.89)
- RFL: Simplifying Chemical Structure Recognition with Ring-Free Language (AAAI 2025 · similarity 0.88)
- TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks (AAAI 2026 · similarity 0.86)
- ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area (AAAI 2025 · similarity 0.86)
- MolVision: Molecular Property Prediction with Vision Language Models (NeurIPS 2025 · similarity 0.86)
- Atomas: Hierarchical Adaptive Alignment on Molecule-Text for Unified Molecule Understanding and Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.