Research map / Vision-language and multimodal / Human and embodied modeling
Newswire: A Large-Scale Structured Database of a Century of Historical News
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs a de-duplicated archive of 2.7M US newswire articles (1878–1977) from 138M OCR'd local newspaper front-page articles, with georeferencing, topic tags, NER, and Wikidata person disambiguation.
Problem
No comprehensive archive exists of the newswire content (Associated Press, United Press) that filled U.S. local newspapers historically; existing scans suffer from OCR errors and no reproduction detection, limiting use for LLM pretraining and historical research.
Approach
Applies a customized deep learning pipeline (layout detection, custom OCR, contrastively trained Sentence-BERT MPNet bi-encoder for reproduced-article clustering, wire-classifier filtering, and a bi-encoder entity disambiguation model trained on Wikipedia disambiguation pages) to hundreds of terabytes of newspaper scans to build a structured newswire dataset.
Key results
- Dataset contains 2,719,607 unique wire articles (1878–1977) reproduced about 32M times, de-duplicated from 138M front-page articles.
- Reproduction detection achieves adjusted rand index 91.5 vs 73.7 for locality-sensitive hashing.
- Person disambiguation associates the correct entity in 72.9% of cases, with 96.8% precision when an association is made; georeferencing accuracy is 94.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a contrastively tuned Sentence-BERT MPNet bi-encoder with single-linkage clustering instead of locality-sensitive hashing | Reproduced article detection · Reproduced article clustering | adjusted rand index: 73.7 -> 91.5 | Helped |
| Use customized binary topic classifiers instead of Comparative Agendas categories | Newswire topic classification · Sports, fires, weather and death notices | classification performance | Helped |
| Add entity-marker special tokens around the target mention (ablation) | Wikipedia-trained entity coreference and disambiguation models · Entity linking | performance | Helped |
| Add in-context negatives that vary only the placement of entity-marker tokens (ablation) | Wikipedia-trained entity coreference and disambiguation models · Entity linking | model features | Helped |
| Replace the Comparative Agendas sports, fires, weather and death-notice labels with custom binary classifiers (ablation) | Newswire topic classification · Sports, fires, weather and death notices | classification performance | Helped |
Surprising findings
- Off-the-shelf OCR transcribed over half of characters wrong in the Associated Press Corporate Archives, motivating custom OCR.
- Libraries' page-level OCR does not detect individual articles, so n-gram viral-text methods are less effective than neural reproduction detection.
Limitations
- Corpus may not include all wire articles since many newspapers were not preserved.
- Texts are not filtered for toxicity or offensive content and OCR errors remain, so cleaned texts are not provided.
- Entity disambiguation misses many in-knowledge-base entities (recall limited to 72.9%) due to conservative no-match threshold.
Experimental setup
Datasets: Chronicling America newspaper scans Wikipedia XML dump (Nov 2022) Wikidata Comparative Agendas NYT Index GeoNames CoNLL03 AIDA-CONLL
Benchmarks: hand-labeled newswire topic test sets hand-labeled georeferencing test set hand-labeled NER test set hand-labeled person disambiguation set (157 articles)
Metrics: F1 accuracy adjusted rand index precision recall mean average precision
Baselines: Viral Texts n-gram approach locality-sensitive hashing AIDA-CONLL entity linking RoBERTa-Large fine-tuned on CoNLL03 Tesseract
Builds on: Sentence-BERT RoBERTa Viral Texts American Stories Comparative Agendas
Closest papers by meaning
- Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.81)
- Political Bias Prediction Models Focus on Source Cues, Not Semantics (AAAI 2025 · similarity 0.80)
- Sparse Autoencoders for Hypothesis Generation (ICML 2025 · similarity 0.79)
- Contradicted in Reliable, Replicated in Unreliable: Dual-Source Reference for Fake News Early Detection (AAAI 2025 · similarity 0.79)
- Merging AI Incidents Research with Political Misinformation Research: Introducing the Political Deepfakes Incidents Database (AAAI 2024 · similarity 0.79)
- Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM) (NeurIPS 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.