Research map / Large language models / Language, safety and interpretability
Detecting High-Stakes Interactions with Activation Probes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains activation probes (Attention, Softmax, Mean, etc.) on GPT-4o-generated synthetic high-stakes data and tests them out-of-distribution against prompted and finetuned LLM monitors, showing comparable AUROC at far lower compute and benefits from a probe-then-LLM cascade.
Problem
Detecting high-stakes LLM interactions (where text indicates possible significant real-world harm) cheaply enough for continuous monitoring, and generalizing beyond synthetic training data to real-world distributions.
Approach
Reuse the residual-stream activations of the monitored LLM (Llama-3.3-70B) to train lightweight probes with several aggregation architectures (notably Attention and Softmax), and combine them in a median-uncertainty cascade with expensive LLM classifiers.
Key results
- Attention and Softmax probes on Llama-3.3-70B activations reach mean AUROC above 0.91 on OOD test datasets, comparable to medium-sized (8B-12B) finetuned and prompted baselines.
- Probes are around six orders of magnitude cheaper in compute than the baseline LLM monitors, since they reuse the monitored model's activations.
- Adding deployment dev samples (as few as 32) to probe training significantly improves performance, with probes exceeding the prompted Llama-3.3-70B baseline when trained on dev samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Attention probe instead of Softmax or Mean aggregation (ablation) | Activation probes on Llama-3.3-70B · OOD evaluation datasets | mean AUROC | Helped |
| Use Last Token aggregation instead of an aggregation over all tokens (ablation) | Activation probes on Llama-3.3-70B · OOD evaluation datasets | AUROC | Hurt |
| Use Max or Max of Rolling Means aggregation (ablation) | Activation probes on Llama-3.3-70B · MTS | AUROC | Mixed |
| Add development-split samples to synthetic training data | Mean aggregation probe · Evaluation datasets with development splits | mean AUROC: 32 dev split samples | Helped |
| Train on synthetic data plus deployment-domain samples rather than deployment samples alone | Attention and Softmax probes · Evaluation datasets with development splits | AUROC: 256 deployment samples | Helped |
| Train directly on combined synthetic and development data instead of finetuning a synthetic-trained probe (ablation) | Activation probes · Evaluation datasets with development splits | AUROC | Hurt |
| Route uncertain samples to an LLM baseline and average its score with the probe score | Probe-plus-baseline cascade · OOD test datasets | mean AUROC | Helped |
| Combine the probe with prompted Gemma-3-1B (ablation) | Probe-plus-baseline cascade · Evaluation datasets | AUROC | Hurt |
| Use synthetic training data instead of deployment-domain training data (ablation) | Activation probes · Evaluation datasets with development splits | AUROC | Hurt |
| Use a TF-IDF and SVM word-statistics classifier instead of the Attention probe (ablation) | High-stakes interaction detection · Evaluation datasets | AUROC | Hurt |
| Use activation probes instead of larger prompted or finetuned monitors | High-stakes interaction monitoring · OOD evaluation datasets | mean AUROC | Mixed |
Surprising findings
- Best probe detects only about 43% of high-stakes cases at 1% FPR versus about 55% for the best LLM baseline.
- Humor, emergency-type signaling words, and ethical-coded language can cause probes to misclassify high-stakes or low-stakes samples.
Limitations
- Labels come from GPT-4o, so probes may replicate GPT-4o's interpretation of high-stakes rather than a broader notion.
- Low TPR at 1% FPR means probes cannot serve as standalone monitors; usefulness depends on higher FPR tolerance.
- Linear probes are vulnerable to targeted adversarial attacks, and the synthetic training data contained confounding words.
Experimental setup
Datasets: synthetic high-stakes dataset generated by GPT-4o Anthropic HH-RLHF ToolACE MT Samples MTS Dialog Mental Health Aya Redteaming
Benchmarks: Anthropic HH-RLHF ToolACE MT Samples MTS Dialog Mental Health Aya Redteaming
Metrics: AUROC TPR at 1% FPR computational FLOPs
Baselines: Llama-3.3-70B prompted Gemma-3-27B prompted Llama-3.1-8B finetuned Gemma-3-12B finetuned TF-IDF + SVM
Builds on: linear probes AdamW
Closest papers by meaning
- Language Models Can Predict Their Own Behavior (NeurIPS 2025 · similarity 0.87)
- Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR 2025 · similarity 0.86)
- Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.86)
- Constitutional Black-Box Monitoring for Scheming in LLM Agents (ICML 2026 · similarity 0.86)
- Monitoring Decomposition Attacks with Lightweight Sequential Monitors (ICLR 2026 · similarity 0.86)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.