Research map / Vision-language and multimodal / Human and embodied modeling
Nougat: Neural Optical Understanding for Academic Documents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Nougat is an end-to-end Swin-BART encoder-decoder that converts rasterized scientific document page images into markup (lightweight markup/LaTeX), trained on arXiv/PMC/IDL page-markup pairs, and released with code and models.
Problem
Scientific PDFs lose semantic structure, especially for mathematical expressions, so existing OCR and PDF-parsing tools (Tesseract, GROBID, pdf2htmlEX) cannot recover formulas and markup from academic documents.
Approach
Treat academic-page conversion as an image-to-markup OCR task solved by a Donut-style Swin encoder with an mBART-style decoder, trained on automatically paired page images and LaTeXML-derived markup, with a Gini-based page-splitting alignment pipeline and anti-repetition augmentation plus logit-variance repetition detection at inference.
Key results
- Nougat base (350M) reaches 0.071 edit distance, 89.1 BLEU and 93.1 F1 on the arXiv test set, versus 0.255 edit distance and 79.2 F1 for embedded PDF text.
- Nougat small (250M) performs on par with base, with 0.073 edit distance and 92.9 F1.
- Anti-repetition augmentation reduced failed page conversions by 32% on out-of-domain documents; repetitions occur on about 1.5% of test pages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Nougat base instead of embedded PDF text | Scientific document page conversion · arXiv test set, All | BLEU: 65.8 -> 89.1 | Helped |
| Use Nougat base instead of GROBID + LaTeX OCR | Scientific document page conversion · arXiv test set, All | BLEU: 55.6 -> 89.1 | Helped |
| Scale from Nougat small to Nougat base | Nougat · arXiv test set, All | BLEU: 88.9 -> 89.1 | Helped |
| Scale from Nougat small to Nougat base | Nougat · arXiv test set, Plain text | Edit distance: 0.058 -> 0.058 | No clear effect |
| Add random token replacement during training (ablation) | Nougat · Out-of-domain documents | failed page conversions: 32% decline | Helped |
| Add random token replacement during training (ablation) | Nougat · Test set | repetition frequency | Helped |
Surprising findings
- The 250M small model performs nearly on par with the 350M base model.
- Math modality scores are lower than plain text even for Nougat, due to LaTeX notation ambiguity.
Limitations
- Repetitive loop collapse with greedy decoding, worse on out-of-domain documents
- Slow generation compared with GROBID (10.6 PDF/s)
- Trained on English and Latin-script documents; non-Latin scripts cause repetitions
Experimental setup
Datasets: arXiv LaTeX sources PubMed Central XML Industry Documents Library OCR text
Benchmarks: arXiv test set
Metrics: normalized edit distance BLEU METEOR precision recall F1
Baselines: PDF embedded text GROBID with LaTeX-OCR
Builds on: Donut Swin Transformer mBART LaTeXML GROBID
Closest papers by meaning
- DocFormerv2: Local Features for Document Understanding (AAAI 2024 · similarity 0.82)
- UPOCR: Towards Unified Pixel-Level OCR Interface (ICML 2024 · similarity 0.82)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.82)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.81)
- HRVDA: High-Resolution Visual Document Assistant (CVPR 2024 · similarity 0.81)
- Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.