Research map / Large language models / AI and society
PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PANORAMA, a dataset of 8,143 USPTO patent examination records with parsed Non-Final Rejections, and three claim-level LLM benchmarks (prior-art retrieval, paragraph identification, novelty/non-obviousness classification) with zero-shot/CoT baselines and SFT.
Problem
Patent examination (novelty under §102 and non-obviousness under §103) is usually framed as application-level grant prediction, which ignores the step-by-step examiner decision trails and rationales recorded in office actions.
Approach
Decomposes USPTO examination records into claim-level sequential benchmarks mirroring examiner workflow (retrieve prior art, locate cited paragraph, classify rejection basis), using LLM-parsed Non-Final Rejection documents as gold rationales.
Key results
- On NOC4PC, Claude 3.7 Sonnet reaches 45.40 with CoT versus a 32.33 random baseline, showing LLMs struggle with novelty and non-obviousness judgments.
- On PAR4PC, Gemma-3-12B-Instruct reaches 77.30 with CoT; LLMs are relatively effective at prior-art retrieval.
- SFT lifts PAR4PC for Qwen2.5-7B-Instruct from 66.11 to 81.06 and EXAONE-3.5-7.8B from 0.00 to 82.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CoT prompting instead of zero-shot prompting | Claude 3.7 Sonnet · NOC4PC | score: 35.84 -> 45.40 | Helped |
| Apply CoT prompting instead of zero-shot prompting | GPT-4o · PI4PC | score: 63.33 -> 62.62 | Hurt |
| Apply CoT prompting instead of zero-shot prompting | Qwen2.5-32B-Instruct · PAR4PC | score: 68.94 -> 55.05 | Hurt |
| Apply supervised fine-tuning on PANORAMA (ablation) | Llama-3.1-8B-Instruct · PAR4PC | score: 13.45 -> 77.98 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | Llama-3.1-8B-Instruct · PI4PC | score: 9.61 -> 69.95 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | Llama-3.1-8B-Instruct · NOC4PC | score: 15.71 -> 49.39 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | Qwen2.5-7B-Instruct · PAR4PC | score: 66.11 -> 81.06 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | Qwen2.5-7B-Instruct · PI4PC | score: 29.25 -> 68.87 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | Qwen2.5-7B-Instruct · NOC4PC | score: 28.92 -> 48.34 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | EXAONE-3.5-7.8B-Instruct · PAR4PC | score: 0.00 -> 82.92 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | EXAONE-3.5-7.8B-Instruct · PI4PC | score: 44.55 -> 65.84 | Helped |
| Apply supervised fine-tuning on PANORAMA (ablation) | EXAONE-3.5-7.8B-Instruct · NOC4PC | score: 15.00 -> 51.71 | Helped |
Surprising findings
- On PI4PC, zero-shot prompting outperforms CoT for most LLMs, unlike the other tasks.
- On NOC4PC several models collapse predictions to a single category, e.g., GPT-4o with CoT largely issues allowance decisions.
Limitations
- Benchmark abstracts examiner workflow and may diverge from real examination; expert difficulty and predictive validity unverified.
- Dataset limited to 8K applications from USPTO only; no reasons provided for allowances, so SFT with CoT was not possible.
- LLM-based parsing introduces errors (7 experts flagged 141 of 1,874 claims as erroneous).
Experimental setup
Datasets: PANORAMA
Benchmarks: PAR4PC PI4PC NOC4PC
Metrics: accuracy
Baselines: GPT-4o Claude 3.7 Sonnet Gemini 2.0 flash Llama-3.1-8B-Instruct Qwen2.5-7B-Instruct EXAONE-3.5-7.8B-Instruct Gemma-3-12B-Instruct QWQ-32B
Builds on: HUPD
Closest papers by meaning
- IMPACT: A Large-scale Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents (NeurIPS 2024 · similarity 0.85)
- PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures (AAAI 2025 · similarity 0.84)
- DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence (ICLR 2026 · similarity 0.81)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.80)
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.80)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.