Research map / Vision-language and multimodal / Visual question answering
IMPACT: A Large-scale Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IMPACT, a large multimodal dataset of 435k USPTO design patents (3.61M figures, 2007-2022) with LLaVA-generated descriptive captions, and evaluates classification and CLIP-based retrieval, showing captions and images help.
Problem
Design patents have only a title, one claim and brief figure descriptions, lacking detailed figure captions needed for multimodal patent tasks such as classification and retrieval; existing patent datasets are mostly utility-patent and text-only.
Approach
Build a large, organized multimodal design-patent dataset with metadata and elaborated VLM-generated captions (LLaVA chosen for cost/quality), then show that adding captions to images improves classification and retrieval, and propose PatentCLIP fine-tuned on image-caption pairs.
Key results
- Combined title+caption+image features give best classification accuracy (0.60 with SVM) on 835 design subclasses, up to 36% above caption+image features.
- Fine-tuned ViT-L-14 CLIP on image-caption pairs reaches R@10 of 53.00 (text-to-image) and 52.56 (image-to-text), far above zero-shot ViT-L-14 (10.40 and 7.32 on image-title).
- Image-caption pairs outperform image-title pairs in all retrieval settings; human evaluation found more than 60% of caption responses judged acceptable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine title, caption, and image features rather than use title and image (ablation) | SVM patent classification · 2021 patent classification | accuracy: 0.58 -> 0.60 | Helped |
| combine title, caption, and image features rather than use caption and image (ablation) | SVM patent classification · 2021 patent classification | accuracy: 0.44 -> 0.60 | Helped |
| replace title with generated caption while using text alone (ablation) | BERT patent classification · 2021 patent classification | accuracy: 0.55 -> 0.52 | Hurt |
| use image-caption pairs instead of image-title pairs (ablation) | zero-shot CLIP retrieval · multimodal retrieval | Text-Image R@1: 2.78 -> 4.46 | Helped |
| use image-caption pairs instead of image-title pairs (ablation) | zero-shot CLIP retrieval · multimodal retrieval | Image-Text R@1: 1.16 -> 3.42 | Helped |
| fine-tune CLIP on patent image-caption pairs (ablation) | ViT-L-14 retrieval · multimodal retrieval | Text-Image R@1: 4.46 -> 20.58 | Helped |
| replace generated captions with GPT-4o captions (ablation) | zero-shot multimodal retrieval · caption model comparison | Image-Text R@5: 8.4 -> 10.5 | Helped |
| fine-tune CLIP on IMPACT image-caption pairs | ViT-B image retrieval with ArcFace · image retrieval | mAP: 0.645 -> 0.657 | Helped |
Surprising findings
- Zero-shot retrieval with generated captions is low (best R@10 about 15%), indicating general CLIP models struggle with design patent figures.
- Qwen-VL captions often lack functional descriptions and sometimes include Chinese words, so LLaVA was chosen over the stronger GPT-4o on cost grounds.
Limitations
- Limited to US design patents from 2007-2022 in English, excluding other regions and earlier trends.
- High variance in number of figures per patent (2 to over 20), which may hurt tasks needing all views such as 3D construction.
- Captions are generated by a general VLM (LLaVA) and human evaluation found limitations in describing shape and function correctly.
Experimental setup
Datasets: IMPACT USPTO bulk data (USPTO BDSS) DeepPatent2
Benchmarks: IMPACT
Metrics: accuracy F1 score Recall@K mAP
Baselines: CLIP BERT SVM Logistic Regression MultiNB MLP GPT-4o Qwen-VL
Builds on: CLIP LLaVA DeepPatent2 RoBERTa
Closest papers by meaning
- PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures (AAAI 2025 · similarity 0.91)
- PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination (NeurIPS 2025 · similarity 0.85)
- Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content? (AAAI 2026 · similarity 0.80)
- Towards Open-Vocabulary Industrial Defect Understanding with a Large-Scale Multimodal Dataset (CVPR 2026 · similarity 0.80)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.80)
- Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective. (ICLR 2026 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.