Research map / Label-efficient and robust learning / Recognition and long-tail
WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds WikiSQE, a dataset of about 3.4M English Wikipedia sentences labeled with 153 inline-template quality labels from full revision history, and benchmarks DeBERTa/BERT/RoBERTa classifiers plus human and GPT-4 comparisons.
Problem
Wikipedia contains many poor-quality sentences marked by editors with inline templates, but no large, comprehensive dataset exists for automatically detecting sentence-level quality problems across many aspects.
Approach
Extract sentences from the full edit history of English Wikipedia that carry curated inline cleanup template labels (153 labels grouped into five categories) and use them as a fine-grained sentence quality estimation dataset, with fine-tuned pretrained encoders as detectors.
Key results
- Fine-tuned models reach 70-85% F1 on category-level detection; the Citation, Syntactic/semantic revision, and Disputed claim categories are harder.
- BERT has the best overall score (72.3 F1 on All), though differences among DeBERTa, BERT and RoBERTa are not significant.
- On 11 labels in crowdsourced pairwise annotation, DeBERTa beats non-expert annotators on all but Citation needed; GPT-4 4-shot lost to DeBERTa on 10 of 11 labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Information addition labels rather than Citation labels | DeBERTa sentence classifier · WikiSQE categories | F1: 73.0 -> 85.2 | Helped |
| Use Information addition labels rather than Syntactic or semantic revision labels | DeBERTa sentence classifier · WikiSQE categories | F1: 73.0 -> 85.2 | Helped |
| Use Information addition labels rather than Disputed claim labels | DeBERTa sentence classifier · WikiSQE categories | F1: 74.3 -> 85.2 | Helped |
| Train on Common categories rather than All categories | DeBERTa sentence classifier · WikiSQE | F1: 70.4 -> 79.5 | Helped |
| Use BERT rather than DeBERTa | Sentence classifier · WikiSQE All | F1: 70.4 -> 72.3 | Helped |
| Use BERT rather than DeBERTa | Sentence classifier · WikiSQE Disputed claim | F1: 74.3 -> 73.2 | Hurt |
| Use RoBERTa rather than DeBERTa | Sentence classifier · WikiSQE Common | F1: 79.5 -> 80.0 | Helped |
| Use DeBERTa rather than crowdsourced annotators | WikiSQE quality-label identification · Pronunciation? | accuracy: 70.9 -> 98.7 | Helped |
| Use DeBERTa rather than crowdsourced annotators | WikiSQE quality-label identification · Citation needed | accuracy: 75.5 -> 69.5 | Hurt |
| Use DeBERTa rather than GPT-4 with 4-shot prompting | WikiSQE quality-label identification · Which? | accuracy: 62.0 -> 85.7 | Helped |
| Use GPT-4 with 4-shot prompting rather than DeBERTa | WikiSQE quality-label identification · Neutrality disputed | accuracy: 81.8 -> 83.0 | Helped |
| Use fine-tuned DeBERTa rather than GPT-4 with 4-shot prompting | WikiSQE quality-label identification · When? | accuracy: 64.0 -> 87.0 | Helped |
Surprising findings
- GPT-4 with 4-shot prompting lost to the fine-tuned DeBERTa on 10 of 11 labels, though it beat non-expert annotators on many labels.
- The combined 'All' category gave lower performance than the 'Common' cluster for all models, suggesting cross-category training hurts.
Limitations
- Evaluation uses balanced positive/negative test sets, not the realistic extremely imbalanced distribution of Wikipedia sentences.
- Labels depend on editor-applied inline templates, which are noisy and may not cover all quality problems; some require external source checking.
- English-only; multilingual versions not built.
Experimental setup
Datasets: WikiSQE English Wikipedia full edit history (pre-March 2019 HTML dump)
Benchmarks: WikiSQE
Metrics: F1 accuracy
Baselines: BERT RoBERTa GPT-4 4-shot crowdsourced human annotators
Builds on: DeBERTa BERT RoBERTa pySBD
Closest papers by meaning
- What's In My Big Data? (ICLR 2024 · similarity 0.84)
- Learning Pattern-Based Extractors from Natural Language and Knowledge Graphs: Applying Large Language Models to Wikipedia and Linked Open Data (AAAI 2024 · similarity 0.84)
- WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia (NeurIPS 2024 · similarity 0.84)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.84)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.83)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.