Research map / Large language models / Language, safety and interpretability
JULI: Jailbreak Large Language Models by Self-Introspection
Jesson Wang, Zhanhao Hu, David Wagner
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use JULI instead of FLIP on Gemini-2.5-Pro via top-5 API | Gemini-2.5-Pro · AdvBench | Harmful Info Score: 1.38 -> 3.19 | Improved |
| use JULI instead of FLIP on Gemini-2.5-Flash | Gemini-2.5-Flash · AdvBench | Harmful Score: FLIP 3.13 vs JULI 2.52 | Hurt |
| restrict access from full vocabulary log probs to top-5 log probs (ablation) | JULI · AdvBench | Harmful Info Score: 3.05 -> 2.21 | Hurt |
| top-10 vs top-5 log probs on MaliciousInstruct (non-monotonic) (ablation) | JULI black-box · MaliciousInstruct | Harmful Info Score: 1.23 (top-10) vs 1.67 (top-5) | No clear effect |
| add the target LLM's own log probabilities to BiasNet output (vs BiasNet alone) (ablation) | JULI · AdvBench | Harmful Info Score: 0.78 -> 3.44 (Llama3-8B) | Improved |
| apply JULI against circuit-breaker defended model | Llama3-8B-CB · AdvBench | Harmful Info Score: 0.41 -> 2.35 | Improved |
| use API-compatible JULI instead of FLIP on circuit-breaker model | Llama3-8B-CB · AdvBench | Harmful Info Score: 0.50 -> 0.75 | Improved |
| use longer prefix (Naive+) instead of Naive (ablation) | prefix baseline · AdvBench (Llama2-7B-Chat) | Harmful Info Score: 1.74 -> 1.56 | Hurt |
| transfer BiasNet across Llama3 models of different size (ablation) | JULI · AdvBench | Harmful Info Score: 3.05 -> 1.00 (8B target, 3B source); 2.18 -> 1.19 (3B target, 8B source) | Hurt |
| random seeds for projection layer initialization (ablation) | JULI black-box · AdvBench | Harmful Info Score: 2.30 to 2.82 across 5 seeds | No clear effect |
| use Harmful Info Score instead of Harmful Score for judging | evaluation metric · Human-labeled 120 responses | Cohen's kappa: 0.20 -> 0.53 | Improved |
| JULI vs ED on Llama3-3B/8B AdvBench Harmful Score (ED best on BERT Score for 8B) | JULI · AdvBench (Llama3-8B-Instruct) | BERT Score: JULI 4.33 vs ED 4.39 | Mixed |
Closest papers by meaning
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (ICLR 2024 · similarity 0.91)
- h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.90)
- An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.90)
- Weak-to-Strong Jailbreaking on Large Language Models (ICML 2025 · similarity 0.90)
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.90)
- AdvPrefix: An Objective for Nuanced LLM Jailbreaks (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
