Research map / Large language models / Language, safety and interpretability
JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks
Masahiro Kaneko, Ayana Niwa, Timothy Baldwin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Formality sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.19 when removed | Improved |
| add the Adherence sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.17 when removed | Improved |
| add the Faithfulness sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.16 when removed | Improved |
| add the Agitativeness sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.15 when removed | Improved |
| add the Verifiability sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.11 when removed | Improved |
| add the Scope sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.10 when removed | Improved |
| add the Subjectivity sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.10 when removed | Improved |
| add the Scale sub-metric to the judge (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: -0.09 when removed | Improved |
| use eight rubric sub-metrics instead of a single overall score (ablation) | LLM-as-a-Judge · Judge meta-evaluation | Spearman rank correlation: 0.52 -> 0.68 | Improved |
| probe hidden layers instead of reading token outputs | self-detection of own fake news · Self-detection (Llama3-8B) | F1: 56.3 -> 66.0 | Improved |
| translate news and instructions into English | LLM defense in non-English regions · JailNewsBench | average sub-metric score | No clear effect |
| use the paper's jailbreak prompt instead of HG prompts | existing fake-news generation prompts · JailNewsBench (GPT-5) | ASR: -32.4 for HG relative to ours | Improved |
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.89)
- Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection (AAAI 2024 · similarity 0.88)
- Weak-to-Strong Jailbreaking on Large Language Models (ICML 2025 · similarity 0.87)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.87)
- An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.87)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
