Research map / Large language models / Language, safety and interpretability
Can LLM-Generated Misinformation Be Detected?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper builds a taxonomy of LLM-generated misinformation, creates the LLMFake dataset, and shows that LLM-generated misinformation with the same semantics as human-written misinformation is harder for humans and zero-shot LLM detectors to catch.
Problem
Whether LLM-generated misinformation is more harmful than human-written misinformation, approached by comparing how hard it is to detect by humans and detectors.
Approach
Categorizes real-world LLM misinformation generation approaches (hallucination, arbitrary, controllable generation) and compares detection success rates for humans and zero-shot LLM detectors on LLM-rewritten versus human-written misinformation that preserves semantics.
Key results
- Human evaluators detect only 9.6% of hallucinated news on average, versus 40.7% of human-written misinformation; Rewriting (p = 9.15e-5) and Open-ended generation (p = 1.01e-6) are significantly harder to detect than human-written misinformation.
- GPT-4 zero-shot detects 10.0% of hallucinated news, and GPT-4 outperforms humans on detecting LLM-generated misinformation in the reported setting.
- Llama2-7B-chat with CoT shows a 19.6% drop in detection on ChatGPT Rewriting-generated Politifact misinformation compared with human-written misinformation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate misinformation with rewriting instead of using human-written misinformation, preserving the semantics | ChatGPT-generated misinformation evaluated by human readers · Politifact-derived misinformation | Success Rate (%): 40.7 -> 24.2 | Hurt |
| Generate misinformation with open-ended generation instead of using human-written misinformation, preserving the semantics | ChatGPT-generated misinformation evaluated by human readers · Politifact-derived misinformation | Success Rate (%): 40.7 -> 21.4 | Hurt |
| Generate misinformation with paraphrasing instead of using human-written misinformation, preserving the semantics | ChatGPT-generated misinformation evaluated by human readers · Politifact-derived misinformation | Success Rate (%): 40.7 -> 38.4 | Mixed |
| Generate misinformation with rewriting instead of using human-written misinformation, preserving the semantics | GPT-4 zero-shot detector · Politifact | Success Rate (%): 62.6 -> 53.6 | Hurt |
| Generate misinformation with open-ended generation instead of using human-written misinformation, preserving the semantics | GPT-4 zero-shot detector · Politifact | Success Rate (%): 62.6 -> 41.6 | Hurt |
| Generate misinformation with rewriting instead of using human-written misinformation, preserving the semantics | Llama2-7B zero-shot detector with CoT · Politifact | Success Rate (%): 47.4 -> 27.8 | Hurt |
| Use CoT prompting rather than No CoT | ChatGPT-3.5 zero-shot misinformation detector · Politifact, human-written misinformation | Success Rate (%): 15.7 -> 39.9 | Helped |
| Use CoT prompting rather than No CoT | GPT-4 zero-shot misinformation detector · Politifact, human-written misinformation | Success Rate (%): 48.6 -> 62.6 | Helped |
| Prompt ChatGPT to generate each listed misinformation type | ChatGPT misinformation generation · Misinformation generation jailbreak attempts | ASR | Mixed |
| Use Hallucinated News Generation rather than arbitrary generation prompts | ChatGPT misinformation generation · Misinformation generation jailbreak attempts | ASR: 5\% -> 100\% | Helped |
Surprising findings
- Information Manipulation prompts containing the word 'misinformation' had a high attack success rate (87%) despite the explicit term, while totally arbitrary generation was rejected 95% of the time.
- Detectors using chain-of-thought sometimes performed worse on LLM-generated misinformation than no-CoT, and CoT gains on human-written data did not transfer.
Limitations
- Human evaluation used only 10 MTurk workers and sampled subsets of 50 to 100 items per method.
- Detectors are evaluated zero-shot only; supervised detectors trained on LLM-generated data are not studied.
- Detection difficulty is measured by success rate, which conflates style with differences in dataset composition and label balance.
Experimental setup
Datasets: Politifact Gossipcop CoAID
Benchmarks: Politifact Gossipcop CoAID LLMFake
Metrics: Success Rate (%) paired t-test p-value attack success rate
Baselines: BERT
Builds on: chain-of-thought prompting jailbreak attacks
Closest papers by meaning
- Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting (ICML 2025 · similarity 0.86)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.86)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.86)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.85)
- Truth is Universal: Robust Detection of Lies in LLMs (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.