Research map / Large language models / Language, safety and interpretability
LLMScan: Causal Scan for LLM Misbehavior Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLMScan, which builds causal maps from token-level attention perturbations and layer-skip interventions and trains lightweight MLP detectors to flag lies, jailbreaks, toxicity, and backdoor-triggered responses in LLMs.
Problem
Detecting diverse LLM misbehaviors (untruthful, jailbreak-harmful, toxic, and backdoor-triggered responses) in a unified way, without relying on analyzing generated outputs.
Approach
Treat the LLM as a structured causal model and use causal mediation analysis to compute per-token (attention-score distance under token masking) and per-layer (logit change when a layer is skipped) causal effects; train an MLP on these causal maps to classify normal vs. misbehavior, detectable at the first generated token.
Key results
- LLMScan reaches AUC 1.00 on 10 of 20 lie-detection detector settings and averages AUC above 0.99 on jailbreak and toxicity detection.
- On backdoor detection LLMScan achieves an average AUC of 0.97 and about 50% higher accuracy than the ONION baseline.
- Layer-level causal effects outperform token-level effects in 39 of 52 comparisons in the ablation study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLMScan instead of the lie-detection baseline | Lie detection on Questions1000 with Llama-2-7b · Questions1000 | accuracy: 0.97 vs 0.97 | No clear effect |
| Use LLMScan instead of the lie-detection baseline | Lie detection on SciQ with Llama-2-7b · SciQ | accuracy: 0.92 vs 0.97 | Hurt |
| Use LLMScan instead of RepE | Jailbreak detection on AutoDAN with Llama-2-7b · AutoDAN | accuracy: 0.98 vs 1.00 | Hurt |
| Use LLMScan instead of RepE | Toxicity detection on SocialChem with Llama-2-7b · SocialChem | accuracy: 0.95 vs 0.95 | No clear effect |
| Use LLMScan instead of ONION | Backdoor detection on Badnet with Llama-2-7b · Badnet | accuracy: 0.48 -> 0.96 | Helped |
| Add layer-level causal effects to token-level causal effects (ablation) | LLMScan, Lie detection on Questions1000 with Llama-2-7b · Questions1000 | accuracy: 0.94 vs 0.94 | No clear effect |
| Add layer-level causal effects to token-level causal effects (ablation) | LLMScan, toxicity detection on SocialChem with Llama-2-7b · SocialChem | accuracy: 0.61 -> 0.95 | Helped |
| Add layer-level causal effects to token-level causal effects (ablation) | LLMScan, backdoor detection on Badnet with Llama-2-7b · Badnet | accuracy: 0.69 -> 0.95 | Helped |
| Add token-level causal effects to layer-level causal effects (ablation) | LLMScan, jailbreak detection on GCG with Llama-2-7b · GCG | accuracy: 0.95 -> 0.99 | Helped |
| Use LLMScan instead of Lie Detector | Detection time on Llama-2-7b · LLMScan detection time | seconds per input: 6.79 -> 3.82 | Helped |
Surprising findings
- Token-level causal effects perform poorly for toxicity detection (e.g., 0.61 AUC on Llama-2-7b), which the authors attribute to toxic knowledge being embedded in model parameters.
- The baseline Pacchiardi lie detector achieves higher accuracy than LLMScan on a few lie-detection settings (e.g., 0.97 vs 0.94 on Mistral Questions1000 and 0.97 vs 0.83 for other cases).
Limitations
- Overhead is an order of magnitude above RepE and ONION, and roughly doubles latency for 13B models without parallelization.
- Robustness against white-box adaptive attackers is argued from non-differentiability rather than empirically tested.
- Evaluated only on four 7-13B open-weight models with lie-instruction or single-turn settings; multi-turn results are limited to a small added experiment.
Experimental setup
Datasets: Questions1000 WikiData SciQ CommonSenseQA 2.0 MathQA AutoDAN GCG PAP SocialChem Badnet CTBA MTBA
Benchmarks: Questions1000 WikiData SciQ CommonSenseQA 2.0 MathQA AutoDAN GCG PAP SocialChem Badnet CTBA MTBA
Metrics: AUC accuracy
Baselines: Pacchiardi et al. lie detector TTPD RepE ONION
Builds on: Causal Mediation Analysis Pacchiardi lie detector TTPD RepE
Closest papers by meaning
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.87)
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.87)
- Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoring (ICML 2026 · similarity 0.87)
- Statistical Hypothesis Testing for Auditing Robustness in Language Models (ICML 2025 · similarity 0.86)
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.86)
- Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.