Research map / Large language models / Language, safety and interpretability

Log-Likelihood, Simpson’s Paradox, and the Detection of Machine-Generated Text

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Zero-shot machine-text detection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows likelihood-based machine-text detectors degrade on frontier-model text and proposes a Bayes-motivated learned local calibration of token scores conditioned on OPT-125M hidden-space position, which substantially improves AUROC across detectors.

Problem

Likelihood-based detectors of machine-generated text are losing accuracy as instruction-tuned frontier models reduce overconfidence, so token-level signals are weakened when averaged naively across hidden space.

Approach

Argues naive averaging of token scores across heterogeneous hidden-space regions causes a Simpson's-paradox-like loss of signal; inserts a learned local calibrator (per-generator MLP predicting score distributions given PCA-reduced hidden state and top-k probabilities) and aggregates per-token log-likelihood ratios instead of raw scores.

Key results

• Calibrated log-surprisal raises GPT-5.4 AUROC from 0.562 to 0.918 on Modern RAID; calibrated DMAP reaches 0.939 on GPT-5.4 vs baseline 0.689.
• Calibrated DMAP achieves highest AUROC in 5 of 6 settings in Table of SOTA comparisons, within 0.02 of DALD in the remaining case.
• Calibration improved AUROC in all 30 scorer/generator/dataset comparisons; calibrated Token-FastDetectGPT on GPT-5.4 reaches 0.854 vs 0.628 baseline.

Surprising findings

• Binoculars scores 0.289 AUROC on Peer-Review GPT-4o, below chance.
• Calibration improves AUROC on every scorer and dataset, yet the calibrated score is not a new detector; zero-shot detectors degrade sharply on frontier-model text (Fast-DetectGPT 0.96 on GPT-3.5 to 0.63 on GPT-5.4).

Limitations

• Requires training separate calibrators per target generator, so it does not generalise zero-shot to arbitrary sources.
• Uses naive Gaussian/MLP calibrators and conditional independence assumptions; no hyperparameter tuning.
• English-only evaluation and limited comparability between the two datasets.

Experimental setup

Datasets: RAID Modern RAID Peer-Review benchmark

Benchmarks: Modern RAID Peer-Review RAID

Metrics: AUROC TPR@0.1% FPR TPR@1% FPR

Baselines: Fast-DetectGPT Binoculars NPR DALD log-surprisal log-rank DMAP

Builds on: Fast-DetectGPT DMAP DALD Binoculars NPR

Closest papers by meaning

• DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.89)
• Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (ICLR 2024 · similarity 0.88)
• Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.88)
• Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.88)
• Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection (NeurIPS 2025 · similarity 0.88)
• Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.88)