Research map / Large language models / Language, safety and interpretability
AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaDetectGPT, which learns a witness function over token log-probabilities (via B-spline features and a closed-form linear solve) to boost Fast-DetectGPT-style logits-based detection, with finite-sample FNR/TNR guarantees.
Problem
Logits-based detectors of LLM-generated text rely only on raw log-probabilities, which can be suboptimal for distinguishing human from machine text, and lack statistical guarantees on error rates.
Approach
Apply a learned one-dimensional witness function to token log-probabilities inside the Fast-DetectGPT statistic; learn it by maximizing a lower bound on TNR at fixed FNR, which reduces to a linear system over B-spline features; threshold set by martingale CLT normal quantile.
Key results
- In white-box settings AdaDetectGPT achieves the highest AUC on all dataset/source-model combinations, outperforming Fast-DetectGPT by 12.5%-37% in relative terms (e.g., Writing avg AUC 0.9233 vs 0.8886).
- In black-box setting with GPT-4o and Claude-3.5-Haiku, AdaDetectGPT improves average AUC over Fast-DetectGPT (0.9589 vs 0.9571 for GPT-4o; 0.9478 vs 0.9439 for Claude-3.5), with Essay gains up to 37.8%.
- Under paraphrasing and decoherence attacks, AdaDetectGPT is more resilient, with improvements up to 10% (paraphrasing) and 85% (decoherence).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn an adaptive witness function over log-probabilities instead of using the raw-logit statistic | Fast-DetectGPT · SQuAD, WritingPrompts, and XSum (white-box averages) | AUC: 0.8571 -> 0.8864; 0.8886 -> 0.9233; 0.8009 -> 0.8379 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · GPT-4o black-box, average across datasets | AUC: 0.9571 -> 0.9589 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Claude-3.5 black-box, average across datasets | AUC: 0.9439 -> 0.9478 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Paraphrasing, GPT-J/GPT-2, average across datasets | AUC: 0.8753 -> 0.8792 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Decoherence, GPT-J/GPT-2, average across datasets | AUC: 0.8449 -> 0.8882 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Paraphrasing, GPT-J/Neo-2.7, average across datasets | AUC: 0.8699 -> 0.8797 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Decoherence, GPT-J/Neo-2.7, average across datasets | AUC: 0.8630 -> 0.9019 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Paraphrasing, GPT-J/GPT-J, average across datasets | AUC: 0.8679 -> 0.8781 | Helped |
| Learn an adaptive witness function over log-probabilities | Fast-DetectGPT · Decoherence, GPT-J/GPT-J, average across datasets | AUC: 0.8752 -> 0.9070 | Helped |
| Use AdaDetectGPT rather than Fast-DetectGPT | Black-box detection of GPT-4o · Yelp | AUC: 0.9847 -> 0.9832 | Hurt |
| Use AdaDetectGPT rather than Fast-DetectGPT | Black-box detection of Claude-3.5 · Yelp | AUC: 0.9768 -> 0.9728 | Hurt |
| Use an adaptive witness function instead of the identity witness | Logits-based text detection · Hypothetical two-token Kingdom of Bit example | Detection power | Helped |
Surprising findings
- Identity-witness logits-based detectors can fail regardless of the distance between human and LLM distributions when q(1)=1/2 in the analytic example.
- Fast-DetectGPT occasionally underperforms Binoculars or RADAR in black-box settings.
Limitations
- Theoretical guarantees rely on asymptotic MCLT approximations and assumptions (margin, minimum eigenvalue, equal variance) that are only approximately validated empirically.
- Black-box performance depends on how well the open-source surrogate model matches the closed-source target LLM.
Experimental setup
Datasets: SQuAD WritingPrompts XSum Yelp Essay PubMed
Benchmarks: SQuAD WritingPrompts XSum Yelp Essay PubMed
Metrics: AUC TPR FPR TNR FNR
Baselines: Likelihood Entropy LogRank LRR NPR DetectGPT DNAGPT Fast-DetectGPT RoBERTaBase RoBERTaLarge Binoculars RADAR
Builds on: Fast-DetectGPT DetectGPT martingale central limit theorem
Closest papers by meaning
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.89)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.89)
- Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting (ICML 2025 · similarity 0.88)
- DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR 2024 · similarity 0.87)
- DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection (NeurIPS 2024 · similarity 0.86)
- BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.