Research map / Large language models / Language, safety and interpretability
Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Glimpse, which estimates a full next-token distribution from the top-K logprobs returned by proprietary completion APIs, so white-box zero-shot detectors (Entropy, Rank, LogRank, Fast-DetectGPT) can run on GPT-3.5/GPT-4 scoring models to detect LLM-generated text.
Problem
Zero-shot detection of LLM-generated text with white-box methods is limited to weaker open-source scoring models, since proprietary APIs expose only partial (top-K) token probabilities rather than full predictive distributions or embeddings.
Approach
Estimate the unobserved tail of the rank-probability distribution from the top-K probabilities using constrained parametric (Geometric, Zipfian) or learned (MLP) estimators, then compute white-box metrics such as conditional probability curvature on the estimated distribution.
Key results
- Glimpse Fast-Detect with GPT-3.5 as scoring model achieves average Mix3 AUROC of about 0.95 across five source models (0.9537 with Geometric, 0.9583 with MLP using GPT-4), versus 0.9057 for open-source Fast-Detect (GPT-J/Neo-2.7).
- Glimpse LogRank and Rank with GPT-3.5 improve over Neo-2.7 counterparts by about 21% and 41% relative, respectively.
- Naive top-K zero-padding reduces Fast-Detect (GPT-3.5) average AUROC from 0.9630 to 0.9311 relative to Geometric estimation, showing estimation is necessary.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Glimpse with Geometric estimation and Fast-Detect instead of Fast-Detect with Neo-2.7 | Fast-Detect · Mix3, five source models | AUROC: 0.9057 -> 0.9537 | Helped |
| Use Glimpse with Geometric estimation and Fast-Detect instead of Likelihood (GPT-3.5) | GPT-3.5 detector · Mix3, five source models | AUROC: 0.8727 -> 0.9537 | Helped |
| Use Geometric instead of Zipfian or MLP distribution estimation (ablation) | Glimpse Fast-Detect · Five source models | AUROC | No clear effect |
| Replace Geometric estimation with Naive zero probability beyond top-K (ablation) | Fast-Detect (GPT-3.5) · Five source models | AUROC: 0.9630 -> 0.9311 | Hurt |
| Increase top-K; gains generally occur with Geometric, while some Zipfian and MLP combinations peak at top-1 (ablation) | Glimpse Rank, LogRank, and Fast-Detect · GPT-4 generations, datasets | AUROC | Mixed |
| Increase rank-list size (ablation) | Glimpse with Geometric or Zipfian estimation · GPT-4 generations, three datasets | AUROC | Mixed |
| Change the prompt used for GPT-4 scoring (ablation) | Glimpse detectors using GPT-4 · GPT-4 generations, three datasets | AUROC | Mixed |
Surprising findings
- Larger proprietary scoring models (GPT-3.5, GPT-4) act as better universal detectors than smaller ones, contrary to a recent study suggesting smaller LLMs are more efficient detectors.
- Limiting the estimator to top-1 probability sometimes gives the highest accuracy for Zipfian and MLP estimators, indicating the top-K ablation is non-monotonic.
Limitations
- Does not support white-box methods requiring inner embeddings (e.g., PHD).
- Only proprietary models exposing Completion API with logprobs can serve as scoring models.
- Estimation algorithms show dataset-dependent preferences and GPT-4 is sensitive to prompt choice.
Experimental setup
Datasets: XSum Writing PubMed M4
Benchmarks: XSum Writing PubMed M4 Mix3 Mix6
Metrics: AUROC accuracy KL divergence TPR at low FPR
Baselines: Likelihood Entropy Rank LogRank DNA-GPT DetectGPT Fast-DetectGPT GPTZero
Builds on: Fast-DetectGPT DetectGPT DNA-GPT Log-Rank
Closest papers by meaning
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.92)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.90)
- Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model (NeurIPS 2025 · similarity 0.90)
- Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (ICLR 2024 · similarity 0.89)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.89)
- DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.