Research map / Large language models / Language, safety and interpretability
BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiScope, a training-free-LLM AI-text detector that combines forward cross-entropy (next-token prediction) and backward cross-entropy (memorization of the preceding token) from open-source surrogate LLMs, with a classifier on segment-wise loss statistics.
Problem
Detecting text generated by the latest commercial LLMs is hard because existing zero-shot methods rely only on next-token predictability from a surrogate model, which gives limited separation, especially on heterogeneous domains and paraphrased text.
Approach
Uses the output logits of a causal LM in teacher-forced guided completion to compute both forward CE (against the next token) and backward CE (against the immediately preceding input token), aggregates per-segment statistics across multiple surrogate LLMs, and trains a binary classifier; also adds a text-summary guidance prompt.
Key results
- BiScope achieves over 0.95 average F1 on the normal in-distribution setting, versus 0.70-0.85 for baselines, a gain of about 0.25 on average.
- BiScope runs in 0.14 s per sample without summary guidance, versus up to 27 s for baselines like Raidar.
- BiScope outperforms GPTZero in 72% of the cases across five datasets and five commercial LLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add backward cross-entropy (BCE) to forward cross-entropy (FCE) (ablation) | BISCOPE with Llama-2-7B · Arxiv, Yelp, Creative, Essay, Code | F1: 0.86 and 0.93 -> 0.94 | Helped |
| Add preceding-token BCE to next-token FCE (ablation) | BISCOPE motivation experiment with Llama-2-7B · Toy detection example | F1: 0.63 -> 0.85 | Helped |
| Add text summary guidance (ablation) | BISCOPE · Arxiv, Yelp, Creative, Essay, Code | F1 | Mixed |
| Use every 10% text split instead of every 50% text split (ablation) | BISCOPE · Code | F1: 0.7495 -> 0.8345 | Helped |
| Remove the completion prompt (ablation) | BISCOPE · Code | F1: 0.8345 -> 0.8444 | Mixed |
| Use the ensemble of six detection models rather than a single model (ablation) | BISCOPE · Five datasets | F1: 0.84 to 0.95 | Helped |
Surprising findings
- Backward cross-entropy (memorization of preceding token) alone was more discriminative than forward cross-entropy in 21 of 25 cases.
Limitations
- Cross-dataset out-of-distribution performance drops by more than 0.10 F1 relative to in-distribution results.
- Relies on a classifier trained on labeled human and AI samples, so it needs training data per domain.
- Summary guidance adds substantial latency (1.35 s per sample).
Experimental setup
Datasets: Arxiv Yelp Creative Essay Code (Python code dataset)
Benchmarks: Arxiv Yelp Creative Essay Code
Metrics: F1 score TPR at FPR time per sample
Baselines: Zero-shot Query LogRank LRR DetectGPT RADAR Raidar OpenAI Detector Binoculars GhostBuster GPTZero
Builds on: Binoculars LogRank LRR DetectGPT Raidar
Closest papers by meaning
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.88)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.87)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.87)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.86)
- AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guarantees (NeurIPS 2025 · similarity 0.86)
- Idiosyncrasies in Large Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.