Research map / Large language models / Language, safety and interpretability
Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HUMPA, a decoding-time proxy attack that uses a DPO-fine-tuned humanized small model's logit offset to shift large source LLMs toward human-like text and evade AI-text detectors.
Problem
Machine-generated text detectors are assumed robust, but attacking large source LLMs via fine-tuning to evade them is costly; the paper seeks a cheaper way to make LLM outputs evade detection while preserving generation quality.
Approach
Fine-tune a small language model with DPO using preference pairs labeled by a detector's human-ness score, then at decoding time add the scaled logit difference (fine-tuned minus reference small model) to the large source model's logits; theoretically this matches DPO fine-tuning the large model with beta divided by alpha.
Key results
- Across white- and black-box detectors, HUMPA reduces AUROC by an average of 70.4% across datasets, with a maximum drop of 95.0% on one dataset (Binoculars on Mixtral-8x7B OpenWebText).
- In cross-discipline evaluation on HSS, DNA-GPT shows a 24.4% relative decrease for Llama2-13B and 59.4% for Mixtral-8x7B in the white-box setting.
- Cross-language black-box evasion reaches relative drops of 91.3% (Likelihood) for Llama2-13B attacked by HUMPA (Llama2-7B), while utility stays within the stated BERTScore and ROUGE-1 budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HUMPA with a humanized proxy SLM during decoding | Llama2-13B, Llama3-70B, and Mixtral-8x7B · Multiple datasets, white-box and black-box detection | AUROC: average AUROC drop of 70.4% | Helped |
| Apply HUMPA with a humanized proxy SLM during decoding | LLMs evaluated on a single dataset · Multiple datasets | AUROC: maximum drop of 95.0% | Helped |
| Use a humanized proxy SLM fine-tuned on another discipline | Source LLMs · Cross-discipline detection | AUROC: relative decrease of up to 90.9% | Helped |
| Use an English-fine-tuned humanized proxy SLM | Source LLMs generating German text · Cross-language detection | AUROC: drop reaches 91.3% | Helped |
| Apply HUMPA rather than use the unattacked source model | Source LLMs · OpenWebText, WritingPrompts, PubMedQA | BERTScore: 0.8278 -> 0.8206; 0.8276 -> 0.8126; 0.8342 -> 0.8281 | Hurt |
| Increase attack ratio α (ablation) | HUMPA on Llama2-13B · WritingPrompts | AUROC | Helped |
| Decrease DPO regularization parameter β (ablation) | Humanized proxy SLM · Black-box detection | AUROC | Helped |
Surprising findings
- Detectors can be misled by a humanized SLM fine-tuned on a different discipline or language, not only on the target domain.
- Fast-DetectGPT and other zero-shot detectors may rank machine text above human text after attack, suggesting they may favor machine-generated content.
Limitations
- Attack ratio alpha and DPO beta must be tuned to trade off evasion against generation quality.
- Requires shared vocabulary between the small and large models.
- Evaluated only on three open-source source models and a limited set of detectors and datasets.
Experimental setup
Datasets: OpenWebText WritingPrompts PubMedQA GPABench2 WMT-2016
Benchmarks: OpenWebText WritingPrompts PubMedQA GPABench2 (HSS, PHX) WMT-2016 (German)
Metrics: AUROC AUPRC BERTScore ROUGE-1 ROUGE-2 ROUGE-L TPR at 1% FPR
Baselines: DPO fine-tuning of source LLM DetectGPT DNA-GPT Likelihood LogRank LRR NPR RoBERTa-base RoBERTa-large Binoculars
Builds on: DPO Fast-DetectGPT proxy-tuning Bradley-Terry model
Closest papers by meaning
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.89)
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.88)
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.87)
- Persistent Pre-training Poisoning of LLMs (ICLR 2025 · similarity 0.87)
- Assessing Vulnerabilities in State-of-the-Art Large Language Models Through Hex Injection (Student Abstract) (AAAI 2025 · similarity 0.87)
- Exploiting LLM Quantization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.