Research map / Large language models / Language, safety and interpretability
Raidar: geneRative AI Detection viA Rewriting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Raidar detects machine-generated text by prompting GPT-3.5-Turbo to rewrite inputs and using edit-distance features (invariance, equivariance, uncertainty) with a simple classifier, outperforming zero-shot, GPTZero, DetectGPT and Ghostbuster on paragraph-level detection.
Problem
Detecting machine-generated paragraphs, especially short text, when token log-probabilities are unavailable (black-box LLMs such as GPT-3.5 and GPT-4) and high-dimensional detector features overfit.
Approach
LLM-generated text is edited less when an LLM is asked to rewrite it, so the character-level Levenshtein edit ratio of rewrites (under invariance, equivariance and output-uncertainty designs) serves as a symbolic, model-agnostic detection feature that needs only generated tokens, not logits.
Key results
- Raidar (invariance) reaches 95.38 F1 on Code and 87.75 on Yelp, outperforming Ghostbuster by up to 29 points on the main in-domain benchmarks.
- In the out-of-distribution setting following Ghostbuster, Raidar improves F1 by up to 32 points over Ghostbuster (e.g., 56.47 vs 34.01 on News).
- With multi-prompt training, adversarially rephrased GPT text is still detected at up to ~93 F1 on unseen evasive prompts (Code, Arxiv).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use invariance rewriting features rather than the strongest listed baseline | Raidar · Code | F1: 67.39 -> 95.38 | Helped |
| Use invariance rewriting features rather than the strongest listed baseline | Raidar · Yelp Reviews | F1: 71.47 -> 87.75 | Helped |
| Use equivariance instead of invariance (ablation) | Raidar · News | F1: 60.29 -> 58.00 | Hurt |
| Use equivariance instead of invariance (ablation) | Raidar · Creative Writing | F1: 62.88 -> 60.27 | Hurt |
| Use equivariance instead of invariance (ablation) | Raidar · Student Essay | F1: 64.81 -> 60.07 | Hurt |
| Use equivariance instead of invariance (ablation) | Raidar · Code | F1: 95.38 -> 80.55 | Hurt |
| Use equivariance instead of invariance (ablation) | Raidar · Yelp Reviews | F1: 87.75 -> 83.50 | Hurt |
| Use output uncertainty instead of invariance (ablation) | Raidar · News | F1: 60.29 -> 60.27 | No clear effect |
| Use output uncertainty instead of invariance (ablation) | Raidar · Code | F1: 95.38 -> 77.14 | Hurt |
| Train on multiple prompts instead of a single prompt (ablation) | Raidar · Code, adaptive prompt 1 | F1: 34.15 -> 86.95 | Helped |
| Train on multiple prompts instead of a single prompt (ablation) | Raidar · Yelp Reviews, no adaptive prompt | F1: 87.75 -> 58.04 | Hurt |
| Use GPT-3.5-Turbo rather than Ada for rewriting (ablation) | Raidar · Code | F1: 77.42 -> 95.38 | Helped |
Surprising findings
- Rewriting-based detection works on Yelp reviews as short as ten words, reaching about 74 F1, where many methods fail.
- Single-prompt detectors can be bypassed by adversarial rephrasing that makes GPT text get edited heavily (e.g., Code F1 drops to 25.64 for prompt 2).
Limitations
- Relies on proprietary GPT-3.5-Turbo API for rewriting, incurring API cost and dependence on a black-box model.
- Performance degrades on some datasets and under adversarial rephrasing; performance on Yelp drops with multi-prompt training.
- Human study with only three users and 20 examples per dataset is small.
Experimental setup
Datasets: Reuters 50-50 (News) WritingPrompts creative writing (Ghostbuster) British Academic Written English student essays (Ghostbuster) HumanEval (code) Yelp Reviews ICLR arXiv abstracts ASAP TOEFL essays
Benchmarks: News Creative Writing Student Essay Code Yelp Reviews Arxiv Abstract
Metrics: F1 score
Baselines: GPT Zero-Shot GPTZero DetectGPT Ghostbuster
Builds on: DetectGPT Ghostbuster Levenshtein distance
Closest papers by meaning
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.88)
- DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning (NeurIPS 2024 · similarity 0.88)
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning (NeurIPS 2025 · similarity 0.88)
- DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm (NeurIPS 2025 · similarity 0.87)
- Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection (AAAI 2025 · similarity 0.87)
- Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.