Research map / Large language models / Language, safety and interpretability
TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices
Gerrit Quaremba, Elizabeth Black, Denny Vrandecic, Elena Simperl
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate on task-specific MGT instead of generic MGT | off-the-shelf detectors · Wikipedia MGT (GPT-4o, English) | accuracy: >93% -> between 47% and 73% | Hurt |
| use supervised detectors instead of zero-shot | detector family · Task-specific MGT | accuracy: zero-shot max 64.7% vs supervised 79.7% to 91.8% | Improved |
| go from sentence-level to paragraph-level text | supervised detectors, TST · Text Style Transfer (English) | accuracy: +15.7% for supervised detectors | Improved |
| use paragraph-continuation (human-machine mixed) text instead of introductory paragraphs | zero-shot detectors · Paragraph Continuation | accuracy: Binoculars 57.8 -> 52.1 (English) | Hurt |
| move language from English to Vietnamese | Binoculars · Paragraph Continuation | accuracy: 52.1 -> 60 | Improved |
| train on task-specific data rather than generic data | mDeBERTa · Cross-domain test sets | accuracy: 73.3 -> 89.7 | Improved |
| train on generic data and test on task-specific data | mDeBERTa · TSM-Bench (English) | accuracy: 92.8-100 diagonal -> at most 73.3 | Hurt |
| train on one task and test on others | mDeBERTa · Cross-task (English) | accuracy: 72.1 (Summarisation) vs 60.5 (Introductory Paragraph), near chance otherwise | Hurt |
| add retrieval context (RAG) to paragraph prompt | GPT-4o mini · Introductory Paragraph prompt eval (English) | RougeL: 0.47 (+0.29) | Improved |
| add one-shot examples to summarisation prompt | GPT-4o mini · Summarisation prompt eval (English) | RougeL: 0.36 (+0.10) | Improved |
| add five-shot examples to TST prompt | GPT-4o mini · TST prompt eval (English) | BLEU: 0.55 (+0.21) | Improved |
Closest papers by meaning
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.86)
- Raidar: geneRative AI Detection viA Rewriting (ICLR 2024 · similarity 0.86)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.85)
- Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy (ICLR 2024 · similarity 0.85)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.85)
- WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
