Research map / Large language models / Language, safety and interpretability
Few-Shot Detection of Machine-Generated Text using Style Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes few-shot detection of machine-generated text by comparing writing-style embeddings (UAR, trained on human Reddit authorship, no LLM samples at training) of a handful of support documents against query documents, evaluated on LWD and M4 benchmarks.
Problem
Detecting text generated by LLMs (including newer, unseen models) is brittle for supervised classifiers under distribution shift, and existing methods need access to the generator's distribution or many of its samples.
Approach
Use pretrained authorship-style representations learned contrastively from human-authored text only; a few support documents from an LLM of concern define a style prototype, and queries are scored by cosine similarity to it, so no LLM samples are needed at training time.
Key results
- Single-target detection with N=10 documents: UAR (Reddit, Twitter, StackExchange) reaches pAUC 0.9676 versus 0.7157 for SBERT and 0.587 for the off-the-shelf AI Detector.
- Unknown-LLM detection with N=1: CISR reaches pAUC 0.707 and UAR variants 0.64–0.684, while SBERT gets 0.552.
- Under paraphrasing attacks, the multi-target UAR variant with paraphrased support samples mitigates the drop in performance relative to single-target UAR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train UAR with Reddit, Twitter, and StackExchange rather than Reddit alone (ablation) | UAR single-target detection · Unseen domains, topics, and LLM, N=5 | pAUC: 0.868 (0.001) -> 0.886 (0.001) | Helped |
| Train UAR with Reddit, Twitter, and StackExchange rather than Reddit alone (ablation) | UAR single-target detection · Unseen domains, topics, and LLM, N=10 | pAUC: 0.958 (0.001) -> 0.9676 (0.0008) | Helped |
| Fine-tune UAR on AAC and Reddit politics rather than human-only Reddit (ablation) | UAR single-target detection · Unseen domains, topics, and LLM, N=5 | pAUC: 0.868 (0.001) -> 0.877 (0.001) | Helped |
| Fine-tune UAR on AAC and Reddit politics rather than human-only Reddit (ablation) | UAR single-target detection · Unseen domains, topics, and LLM, N=10 | pAUC: 0.958 (0.001) -> 0.9400 (0.0013) | Hurt |
| Add length control while controlling for topic in ProtoNet training data (ablation) | ProtoNet · Amazon | pAUC: 0.769 (0.002) -> 0.844 (0.009) | Helped |
| Add topic control to ProtoNet training data when length is uncontrolled (ablation) | ProtoNet · Amazon | pAUC: 0.765 (0.002) -> 0.769 (0.002) | No clear effect |
| Paraphrase an increasing proportion of target-model query episodes (ablation) | UAR and ProtoNet single-target detection · Paraphrasing attack, N=5 | pAUC | Hurt |
| Include a paraphrase of the support sample as an additional match (ablation) | UAR multi-target detection · Paraphrasing attack, N=5 | pAUC | Helped |
| Use a 32-token truncation instead of the usual 128-token truncation (ablation) | Few-shot detectors · Single-target detection, N=10 | pAUC | Mixed |
| Use multi-domain UAR rather than Reddit-only UAR (ablation) | Unknown-LLM detection · N=1 | pAUC: 0.682 (0.002) -> 0.684 (0.002) | No clear effect |
| Use multi-domain UAR rather than Reddit-only UAR (ablation) | Unknown-LLM detection · N=2 | pAUC: 0.759 (0.002) -> 0.706 (0.002) | Hurt |
Surprising findings
- Style representations trained only on human Reddit authorship outperform zero-shot likelihood detectors and supervised-then-transferred detectors, without any LLM samples at training time.
- Supervised RoBERTa-based detectors reach nearly perfect in-domain performance but drop sharply to chance on unseen LLM, topic, or domain data.
Limitations
- Assumes access to a handful of in-distribution documents from each LLM of concern.
- Experiments are English-only, with style representations relying on high-resource author-labeled text.
- Detection performance is weakest on M4 PeerRead, which has fewer unique unigrams and bigrams.
Experimental setup
Datasets: Reddit comments (5M authors) Twitter StackExchange Amazon reviews AAC (GPT-2/OPT generations on r/politics) LWD M4
Benchmarks: LWD M4
Metrics: pAUC (FPR up to 1%) AUC FPR@95TPR
Baselines: Prototypical Networks MAML SBERT CISR OpenAI AI Detector Rank LogRank Entropy
Builds on: UAR CISR Prototypical Networks MAML DIPPER
Closest papers by meaning
- Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection (AAAI 2025 · similarity 0.90)
- Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection (NeurIPS 2025 · similarity 0.89)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.88)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.88)
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.87)
- DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.