Research map / Large language models / Language, safety and interpretability
Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fast-DetectGPT, a zero-shot machine-generated text detector that replaces DetectGPT's perturbation step with conditional probability curvature computed via token-wise sampling from a sampling model, achieving ~340x speedup and ~75% relative AUROC gains.
Problem
Zero-shot detection of machine-generated text with DetectGPT is accurate but requires around 100 perturbation and scoring model calls per passage, making it computationally prohibitive.
Approach
Defines conditional probability curvature, comparing log p(x|x) for the passage against the expected log-probability of alternative tokens sampled conditionally per position from a sampling model; a single sampling pass and one scoring pass replace DetectGPT's many perturbation and scoring calls, and the expectation can be computed analytically.
Key results
- White-box average AUROC 0.9887 vs DetectGPT 0.9554 (relative gain 74.7%) across five source models.
- Black-box ChatGPT/GPT-4 detection averages AUROC 0.9615/0.9061 for Fast-DetectGPT vs 0.8223/0.6228 for DetectGPT.
- About 340x speedup over DetectGPT (233 s vs 79,113 s on XSum with an A100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DetectGPT perturbation-based scoring with conditional probability curvature | Zero-shot machine-generated text detection · 5-model generations, white-box | AUROC: 0.9554 -> 0.9887; relative↑ 74.7% | Helped |
| Replace DetectGPT perturbation-based scoring with conditional probability curvature | Zero-shot machine-generated text detection · 5-model generations, black-box | AUROC: 0.8736 -> 0.9677; relative↑ 74.5% | Helped |
| Replace DetectGPT with Fast-DetectGPT | Zero-shot machine-generated text detection · XSum news dataset, Tesla A100 | speedup: 340x | Helped |
| Use GPT-J rather than the source model as the sampling model (ablation) | Fast-DetectGPT in the white-box setting · XSum, SQuAD, and WritingPrompts; three source models | AUROC: +0.0020 AUROC; relative improvement of 27% | Helped |
| Use GPT-2 as the sampling model instead of the source model (ablation) | Fast-DetectGPT · XSum, SQuAD, and WritingPrompts; three source models | AUROC: 0.9925 -> 0.9743 | Hurt |
| Use Neo-2.7 as the sampling model instead of the source model (ablation) | Fast-DetectGPT · XSum, SQuAD, and WritingPrompts; three source models | AUROC: 0.9925 -> 0.9755 | Hurt |
| Normalize conditional probability curvature by the sample standard deviation (ablation) | Fast-DetectGPT · Detection experiments | AUROC: +0.0020; 10% relative improvement | Helped |
| Calculate curvature analytically rather than using sampling approximation (ablation) | Fast-DetectGPT · Detection experiments | AUROC | No clear effect |
| Calculate curvature analytically rather than using sampling approximation (ablation) | Fast-DetectGPT · Detection experiments | speedup: about 10% | Helped |
| Include the entropy term in the curvature numerator (ablation) | Fast-DetectGPT · Machine-generated text detection | AUROC: 65% | Helped |
Surprising findings
- Likelihood is the strongest baseline for ChatGPT/GPT-4 detection, unlike the open-source model results where DetectGPT and NPR lead.
- Entropy alone is near random (AUROC about 0.5) in the white-box setting, yet the combination of likelihood and entropy in the curvature numerator yields strong detection.
Limitations
- Black-box performance depends on surrogate scoring models and degrades when the surrogate's training corpus mismatches the source model (e.g., Llama-13 SQuAD 0.87 vs 0.93 average).
- Theoretical justification is left for future work; the paper is primarily empirical.
- Relies on pre-trained models that may be domain- or language-specific, which limits black-box coverage across domains and languages.
Experimental setup
Datasets: XSum SQuAD WritingPrompts WMT16 English WMT16 German PubMedQA
Benchmarks: XSum SQuAD WritingPrompts PubMedQA WMT16
Metrics: AUROC relative AUROC improvement inference speedup
Baselines: DetectGPT NPR DNA-GPT Likelihood Entropy LogRank LRR RoBERTa-base RoBERTa-large GPTZero
Builds on: DetectGPT NPR LRR DNA-GPT
Closest papers by meaning
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.89)
- Log-Likelihood, Simpson’s Paradox, and the Detection of Machine-Generated Text (NeurIPS 2026 · similarity 0.88)
- Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model (NeurIPS 2025 · similarity 0.88)
- Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy (NeurIPS 2026 · similarity 0.88)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.88)
- DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.