Research map / Large language models / Language, safety and interpretability
DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DNA-GPT, a training-free zero-shot detector that truncates a text, regenerates the remainder with an LLM several times, and scores divergence via n-gram overlap (black-box) or log-probability ratio (white-box).
Problem
Detecting whether a given text was generated by a known GPT-family LLM, especially closed API models where token probabilities are unavailable and trained classifiers fail to generalize to new models, domains, revisions, and non-English text.
Approach
Truncate the candidate text at a ratio, have the LLM regenerate the remainder K times from the prefix, and measure how similar the original remainder is to the regenerations: machine text reproduces itself closely (high n-gram overlap / likelihood), while human text diverges. The BScore (n-gram) works in black-box and WScore (log-likelihood ratio) in white-box, and the overlapping n-grams double a
Key results
- On Reddit-ELI5 with GPT-4-0314 black-box, DNA-GPT (K=10, gamma=0.5) reaches 99.34 AUROC and 91.00 TPR at 1% FPR, versus GPTZero 94.50/36.00 and OpenAI classifier 71.64/5.00.
- White-box WScore on text-davinci-003 reaches near-perfect results (e.g., Xsum AUROC 100.00, TPR 100.00 with K=10, gamma=0.5), far above DetectGPT (AUROC about 54-82).
- Under revised text (up to 50% of spans replaced), DNA-GPT AUROC only drops from 99.09 to 98.48 while baselines deteriorate sharply beyond 30% revision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DNA-GPT instead of the supervised GPTZero and OpenAI classifier | GPT-4-0314 black-box detection · Reddit-ELI5 | AUROC: 94.50 -> 99.63; 71.64 -> 99.63 | Helped |
| Use DNA-GPT instead of the supervised GPTZero and OpenAI classifier | GPT-3.5-turbo black-box detection · Xsum | AUROC: 90.79 -> 97.12; 94.74 -> 97.12 | Helped |
| Use DNA-GPT white-box instead of DetectGPT | text-davinci-003 detection · Reddit-ELI5 | AUROC: 54.21 -> 99.99 | Helped |
| Use DNA-GPT black-box rather than GPTZero or OpenAI classifier | LLaMa-13B detection · Reddit | AUROC: 84.34 -> 91.20; 66.74 -> 91.20 | Helped |
| Set γ=0.5 rather than γ=0.7 (ablation) | DNA-GPT, GPT-4-0314 black-box · Reddit-ELI5 | TPR at 1% FPR: 87.34 -> 91.00 | Helped |
| Set γ=0.5 rather than γ=0.7 (ablation) | DNA-GPT, GPT-3.5-turbo black-box · Scientific Abstracts | AUROC: 98.02 -> 99.65 | Helped |
| Remove the golden prompt (ablation) | DNA-GPT, GPT-4-0314 black-box · Xsum | TPR at 1% FPR: 30.13 -> 32.67 | Hurt |
| Remove the golden prompt (ablation) | DNA-GPT, GPT-3.5-turbo black-box · Xsum | TPR at 1% FPR: 52.48 -> 62.67 | Helped |
| Remove the golden prompt (ablation) | DNA-GPT, text-davinci-003 black-box · Xsum | TPR at 1% FPR: 51.00 -> 65.00 | Helped |
| Remove the golden prompt (ablation) | DNA-GPT, text-davinci-003 white-box · Xsum | TPR at 1% FPR: 100.00 -> 99.00 | Hurt |
| Use 10 re-generations instead of 20 (ablation) | DNA-GPT detection · Four datasets | AUROC | Mixed |
| Set the starting n-gram length n₀ to 1 or 6 rather than 4 (ablation) | DNA-GPT black-box · Reddit, PubMedQA, and Xsum | AUROC: 93.07; 92.60; 93.06 | Hurt |
Surprising findings
- Training-based detectors GPTZero and OpenAI's classifier are highly sensitive to decoding temperature, dropping significantly as temperature changes.
- Verbatim memorization of WikiText-103 by text-davinci-003 makes machine and human text indistinguishable in many cases, leading to low TPR for all detectors.
Limitations
- Only tested on GPT-style decoder-only models; does not claim to detect non-decoder-only generators.
- Tested only on texts of moderate length (about 1000-2000 characters); performance on very short or long texts is unclear.
- Low TPR on memorized WikiText-103 text, and a single-prefix truncation assumes the whole text is machine-generated.
Experimental setup
Datasets: Reddit ELI5 Scientific Abstracts PubMedQA XSum WMT16 English and German WikiText-103
Benchmarks: Reddit ELI5 Scientific Abstracts PubMedQA XSum WMT16 German WikiText-103
Metrics: AUROC TPR at 1% FPR F1 accuracy
Baselines: GPTZero OpenAI text classifier DetectGPT
Builds on: DetectGPT GPTZero OpenAI text classifier LeCam's lemma
Closest papers by meaning
- DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm (NeurIPS 2025 · similarity 0.90)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.89)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.89)
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.89)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.88)
- Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.