Research map / Large language models / Language, safety and interpretability
Training-free LLM-generated Text Detection by Mining Token Probability Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lastde and Lastde++, training-free LLM-generated text detectors that combine a likelihood (global) statistic with multiscale diversity entropy of token probability sequences (local temporal dynamics), with Lastde++ adding fast sampling normalization.
Problem
Training-free detection of LLM-generated text relies on global statistics and performs poorly in black-box, cross-model, cross-lingual, and paraphrase-attack settings, while ignoring local temporal patterns of token probabilities.
Approach
Treat the token log-probability sequence as a time series and compute multiscale diversity entropy (local dynamics), then divide the log-likelihood by the aggregated MDE to form a score; Lastde++ applies fast-sampling discrepancy normalization using 100 samples.
Key results
- White-box: Lastde averages 95.89 AUROC across 12 source models vs 90.80 for DetectLRR; Lastde++ averages 98.85 vs 97.82 for Fast-DetectGPT.
- Black-box (GPT-J proxy): Lastde++ averages 91.47 AUROC vs 85.68 for Fast-DetectGPT across 12 source models, including GPT-4-Turbo.
- Under T5 paraphrasing attacks, Lastde++ shows smaller AUROC drops than Fast-DetectGPT on most dataset/setting combinations (e.g., XSum white-box 97.75 to 89.27 vs 96.04 to 85.48).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add local multiscale diversity entropy to likelihood scoring (Lastde versus DetectLRR) | White-box text detection · XSum/SQuAD/WritingPrompts, 12-model average | AUROC: 90.80 -> 95.89 | Helped |
| Add fast-sampling Lastde discrepancy (Lastde++ versus Fast-DetectGPT) | White-box text detection · XSum/SQuAD/WritingPrompts, 12-model average | AUROC: 97.82 -> 98.85 | Helped |
| Add local multiscale diversity entropy to likelihood scoring (Lastde versus DetectLRR) | Black-box text detection · XSum/WritingPrompts/Reddit, 12-model average | AUROC: 80.45 -> 86.38 | Helped |
| Add fast-sampling Lastde discrepancy (Lastde++ versus Fast-DetectGPT) | Black-box text detection · XSum/WritingPrompts/Reddit, 12-model average | AUROC: 85.68 -> 91.47 | Helped |
| Replace standard-deviation aggregation with 2-norm (ablation) | Lastde++ · GPT-4-Turbo, XSum/WritingPrompts/Reddit average | AUROC: 88.18 -> 85.97 | Hurt |
| Replace standard-deviation aggregation with range (ablation) | Lastde++ · GPT-4-Turbo, XSum/WritingPrompts/Reddit average | AUROC: 88.18 -> 86.92 | Hurt |
| Replace standard-deviation aggregation with 2-norm (ablation) | Lastde++ · GPT-4o, XSum/WritingPrompts/Reddit average | AUROC: 92.86 -> 92.07 | Hurt |
| Replace standard-deviation aggregation with range (ablation) | Lastde++ · GPT-4o, XSum/WritingPrompts/Reddit average | AUROC: 92.86 -> 91.29 | Hurt |
| Increase Lastde scale factor from 5 to 10 (ablation) | Lastde · Four analyzed datasets | AUROC | Helped |
| Increase the scale factor beyond 10 (ablation) | Lastde · Reddit (GPT-4-Turbo) versus other analyzed datasets | AUROC | Mixed |
| Increase the number of contrast samples (ablation) | Distribution-based detection methods · Gemma-7 white-box detection | AUROC | Helped |
| Use 10 contrast samples instead of 100 (ablation) | Lastde++ · Gemma-7 white-box detection | AUROC | Helped |
Surprising findings
- Fast-DetectGPT and Lastde++ performed worse on Llama3-8 than their base statistics (Likelihood and Lastde) in the black-box setting, which the authors attribute to saturation and sampling randomness.
- Detection performance was poor across all methods with GPT-4-Turbo when the proxy model was Llama3-8.
Limitations
- Detection depends strongly on the choice of proxy model; black-box performance drops notably with some proxy-source pairs.
- Short texts reduce detection performance because fewer scales can be computed.
- Hyperparameters (s, epsilon, tau') were tuned on the evaluation datasets, and paraphrasing results are reported only for three source models.
Experimental setup
Datasets: XSum SQuAD WritingPrompts Reddit ELI5 WMT16 Zhihu
Benchmarks: XSum SQuAD WritingPrompts Reddit ELI5 WMT16 Zhihu
Metrics: AUROC
Baselines: Likelihood LogRank Entropy DetectLRR DetectGPT DetectNPR DNA-GPT Fast-DetectGPT
Builds on: Fast-DetectGPT Diversity Entropy DetectLLM DetectGPT
Closest papers by meaning
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.91)
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.90)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.90)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.90)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.90)
- Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.