Research map / Large language models / Language, safety and interpretability
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Binoculars, a zero-shot LLM-generated text detector that scores text by the ratio of log-perplexity under one model to cross-perplexity between two closely related models (Falcon-7B and Falcon-7B-Instruct), requiring no training data from the generator.
Problem
Detecting machine-generated text from modern LLMs post hoc, zero-shot and without model-specific training data, with low false-positive rates on diverse human text.
Approach
Normalize observer perplexity by cross-perplexity between two closely related LLMs (a performer and observer pair), which calibrates for prompt-induced high perplexity (the "capybara problem"); the resulting Binoculars score separates human and machine text without any training.
Key results
- Binoculars detects over 90% of ChatGPT-generated samples at 0.01% FPR across News, Creative Writing and Student Essay datasets, outperforming Ghostbuster, GPTZero and DetectGPT out-of-domain.
- Binoculars detects 92% of GPT-3 and 89.57% of GPT-4 Open-Orca samples using a global threshold.
- On EssayForum, Binoculars attains 99.67% accuracy on both original and grammar-corrected essays of non-native speakers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine perplexity and cross-perplexity as the Binoculars score instead of using Falcon perplexity alone (ablation) | Falcon-based text detection · Writing Prompts | TPR @ 0.01% FPR: 0.86 -> 0.93 | Helped |
| Combine perplexity and cross-perplexity as the Binoculars score instead of using Falcon cross-perplexity alone (ablation) | Falcon-based text detection · Writing Prompts | TPR @ 0.01% FPR: 0.56 -> 0.93 | Helped |
| Combine perplexity and cross-perplexity as the Binoculars score instead of using Falcon perplexity alone (ablation) | Falcon-based text detection · News | TPR @ 0.01% FPR: 0.65 -> 0.95 | Helped |
| Combine perplexity and cross-perplexity as the Binoculars score instead of using Falcon cross-perplexity alone (ablation) | Falcon-based text detection · News | TPR @ 0.01% FPR: 0.04 -> 0.95 | Helped |
| Combine perplexity and cross-perplexity as the Binoculars score instead of using Falcon perplexity alone (ablation) | Falcon-based text detection · Essay | TPR @ 0.01% FPR: 0.78 -> 0.98 | Helped |
| Combine perplexity and cross-perplexity as the Binoculars score instead of using Falcon cross-perplexity alone (ablation) | Falcon-based text detection · Essay | TPR @ 0.01% FPR: 0.25 -> 0.98 | Helped |
| Use an instruction-tuned performer model rather than the identical base scorer (ablation) | Binoculars · Reference datasets | detection performance | Helped |
| Use pirate-style system prompting (ablation) | LLaMA-2-13B-chat generations scored by Binoculars · Modified prompting strategies | sensitivity: 1% | Hurt |
Surprising findings
- Perplexity alone and cross-perplexity alone are poor detectors, while their ratio is highly accurate.
- Binoculars detects only 41.86% of GPT-4 samples from March 2024 (58% false negative rate), while detecting 96.89% of Gemini-1.0-Pro samples.
Limitations
- Not evaluated on larger (30B+) open-source models due to GPU memory limits.
- Does not address adversarial attempts to evade detection; non-conversational domains such as source code not studied.
- Poor recall on low-resource languages and weak performance on some newer API models (GPT-4 false negative rate 58%).
Experimental setup
Datasets: Ghostbuster News Ghostbuster Creative Writing Ghostbuster Student Essay Writing Prompts CC News CNN PubMed Open-Orca M4 EssayForum Alpaca
Benchmarks: Ghostbuster News Ghostbuster Creative Writing Ghostbuster Student Essay M4 Open-Orca EssayForum PubMed CC News CNN
Metrics: TPR at 0.01% FPR AUC F1-score precision recall false negative rate
Baselines: Ghostbuster GPTZero DetectGPT Fast-DetectGPT DNA-GPT RoBERTa classifier LR GLTR Stylistic NELA Falcon PPL LLaMA PPL
Builds on: DetectGPT Fast-DetectGPT Ghostbuster contrastive decoding speculative decoding
Closest papers by meaning
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.87)
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.86)
- AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guarantees (NeurIPS 2025 · similarity 0.85)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.85)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.85)
- Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.