Research map / Adversarial robustness and security / Attacks on text and graphs

Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Adversarial prompts for language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LADE, a model-agnostic jailbreak/harmful-query filter that uses first-token probabilities of safety-discriminative "dark knowledge" tokens, mapped across tokenizers, with kNN distance to harmful references.

Problem

Decoding-stage jailbreak defenses trade safety against over-refusal and rely on architecture-specific hidden states, limiting cross-model generalization.

Approach

Select top-k first-token vocabulary tokens whose mean probability differs most between harmful and benign queries on a reference LLM, map them across tokenizers with duplicate-ratio weighting, and classify queries by kNN distance to harmful reference features, filtering before generation using only output probabilities.

Key results

• On the jailbreak table, LADE averages 2.00 compliant responses on Qwen2-7B-Instruct versus 20.60 with no defense, and 2.20 on Mistral-7B-Instruct-v0.3 versus 36.60 with no defense.
• Held-out average of 1.40 on Llama-3-8B-Instruct versus 3.20 for RDS (best baseline there).
• Against guard models on a single model, LADE reaches 0.969 average accuracy versus 0.947 for WildGuard-7B; a white-box adaptive attack drops LADE accuracy from 1.000 to 0.480.

Surprising findings

• Under a white-box adaptive GCG-style attack designed to evade LADE, accuracy falls to 0.480, showing LADE is not robust to adaptive attacks.
• Using only the top-1 token gives accuracy of 0.4505, and refusal tokens alone are insufficient on some models (e.g., Mistral, Qwen2-7B).

Limitations

• Not robust to adaptive white-box attacks (accuracy 0.480 on Llama-3-8B-Instruct).
• Reference LLM choice is not principled; hyperparameters tuned on Hex-Phi/XSTest used offline.
• Harmful reference set is kNN-based and needs re-tuning per reference set; higher false positives on some models for realistic benign queries (e.g., Mistral up to 28% on JailbreakBench).

Experimental setup

Datasets: Hex-Phi XSTest AdvBench StrongReject MMLU Alpaca GSM8K

Benchmarks: AdvBench Hex-Phi StrongReject MMLU Alpaca GSM8K XSTest AutoDAN DeepInception GCG PAIR LIAR

Metrics: compliance rate on harmful queries refusal rate on benign queries classification accuracy latency memory

Baselines: Self-Reminder SafeDecoding SafeInfer RDS Prompt-Guard-2-86M Llama-Guard-4-12B WildGuard-7B

Builds on: SafeDecoding RDS SafeInfer Self-Reminder

Closest papers by meaning

• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.90)
• Safety Layers in Aligned Large Language Models: The Key to LLM Security (ICLR 2025 · similarity 0.90)
• Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.90)
• Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.90)
• Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.90)
• Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.89)