Research map / Vision-language and multimodal / Human and embodied modeling
PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PatentDesc-355K, a ~355K patent-figure dataset with brief and detailed descriptions, and PatentLMM, a multimodal LLM combining a patent-pretrained LayoutLMv3-based encoder (PatentMME) with a patent-adapted LLaMA-2 7B (PatentLLaMA) for generating patent figure descriptions.
Problem
Automatically generating accurate brief and detailed textual descriptions of patent figures, which are long, structured technical drawings unlike natural-image captions, lacks large-scale data and suitable models.
Approach
A vision-text encoder pretrained on patent figures with MLM, layout-aware masked image modeling (LAMIM), and patch classification (PC) losses using detector-derived element labels, paired with a patent-domain LLaMA-2 decoder via a projection MLP and two-stage LoRA training.
Key results
- PatentLMM reaches 44.59 Avg. BLEU and 56.44 METEOR on brief descriptions, versus 34.37 Avg. BLEU for fine-tuned LLaVA-1.5 (7.4B).
- On detailed descriptions PatentLMM gets 24.24 Avg. BLEU and 28.39 METEOR, above fine-tuned LLaVA-1.5 (19.81 and 24.71).
- Zero-shot GPT-4V is the strongest baseline overall (18.68 Avg. BLEU brief), but PatentLMM outperforms it on brief and detailed descriptions after fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PatentLMM instead of the strongest finetuned baseline, LLaVA-1.5 | Patent figure description generation · PatentDesc-355K test, brief | Avg. BLEU: 34.37 -> 44.59 | Helped |
| use PatentLMM instead of the strongest finetuned baseline, LLaVA-1.5 | Patent figure description generation · PatentDesc-355K test, detailed | Avg. BLEU: 19.81 -> 24.24 | Helped |
| add MLM and LAMIM pretraining objectives (ablation) | PatentMME with PatentLLaMA · PatentDesc-355K test, brief | Avg. BLEU: 40.86 -> 43.39 | Helped |
| add PC loss to MLM + LAMIM pretraining (ablation) | PatentMME with PatentLLaMA · PatentDesc-355K test, brief | Avg. BLEU: 43.39 -> 44.59 | Helped |
| add MLM, LAMIM, and PC pretraining objectives (ablation) | PatentMME with PatentLLaMA · PatentDesc-355K test, detailed | Avg. BLEU: 22.73 -> 24.24 | Helped |
| use OCR tokens throughout training and inference instead of omitting them (ablation) | PatentLMM · PatentDesc-355K test, brief | Avg. BLEU: 28.30 -> 44.59 | Helped |
| provide OCR tokens at inference after using them in training (ablation) | PatentLMM · PatentDesc-355K test, brief | Avg. BLEU: 9.83 -> 44.59 | Helped |
| add stage-2 task fine-tuning to the LLaMA-2 decoder (ablation) | PatentLMM with LLaMA-2 initialization · PatentDesc-355K test, brief | Avg. BLEU: 1.58 -> 41.66 | Helped |
| use PatentLLaMA instead of standard LLaMA-2 initialization with stage-2 fine-tuning (ablation) | PatentLMM · PatentDesc-355K test, brief | Avg. BLEU: 41.66 -> 44.59 | Helped |
Surprising findings
- Recent captioning models and multimodal LLMs performed poorly even after fine-tuning on the patent dataset, unlike their strong results on natural images.
- Using OCR tokens at training but not at inference drops brief-description Avg. BLEU from 44.59 to 9.83, showing heavy reliance on OCR text.
Limitations
- Hallucination of figure labels, node labels, and cross-figure references, often when OCR-detectable text is sparse.
- Downsampling to 384x384 distorts node labels and wiggly arrows, causing incorrect label associations.
- Each figure is described independently, so document-level cross-figure reasoning is absent.
Experimental setup
Datasets: PatentDesc-355K HUPD COCO TextCaps NoCaps
Benchmarks: PatentDesc-355K
Metrics: BLEU-2 BLEU-4 Avg. BLEU ROUGE-1 ROUGE-2 ROUGE-L METEOR coherence/relevance/accuracy (GPT-4V rubric, 0-2)
Baselines: BLIP-2 TextMonkey PEGASUS mPLUG-owl2 UReader LLaVA-1.5 GPT-4V GIT BLIP OFA MiniGPT-4
Builds on: LayoutLMv3 LLaMA-2 MiniGPT-4 LLaVA-1.5 BEiT
Closest papers by meaning
- IMPACT: A Large-scale Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents (NeurIPS 2024 · similarity 0.91)
- PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination (NeurIPS 2025 · similarity 0.84)
- MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding (ICML 2026 · similarity 0.82)
- BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks (ICLR 2025 · similarity 0.81)
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.81)
- CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.