Research map / Large language models / Language, safety and interpretability
Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, Samuel Marks
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a generalist Activation Oracle instead of best white-box baseline | Activation Oracle · Taboo (Gemma-2-9B-IT) | accuracy: 0.041 -> 0.205 | Improved |
| use Activation Oracle instead of best white-box baseline | Activation Oracle · Secret Side Constraint (Llama-3.3-70B) | accuracy: 0.522 -> 0.590 | Improved |
| use Activation Oracle instead of best black-box prefill method | Activation Oracle · Secret Side Constraint (Llama-3.3-70B) | accuracy: 0.968 -> 0.590 | Hurt |
| replace ADL with Activation Oracle for model diffing | emergent misalignment audit · emergent misalignment audit | grade (1-5): 2.03 -> 2.00 | No clear effect |
| add classification and context prediction tasks to the AO training mixture (ablation) | Activation Oracle · 12 model/evaluation combinations | accuracy: 10 of 12 best with full mixture | Improved |
| train on diverse mixture instead of SPQA only (ablation) | Activation Oracle · PersonaQA and held-out classification | accuracy: better in 7 of 8 settings | Improved |
| inject activations at layer 1 instead of layer 0 (ablation) | Activation Oracle · Secret Side Constraint | accuracy: +11% | Improved |
| use activations from layer 75% instead of 50% at evaluation (ablation) | Activation Oracle · Taboo (Gemma-2-9B-IT) | accuracy: 0.205 -> 0.056 | Hurt |
| use full-sequence activations instead of single-token input (ablation) | Activation Oracle · secret-keeping evaluations | accuracy | Improved |
| filter predictions by majority-vote consensus >=0.9 | Activation Oracle · Taboo (Qwen3-8B) | accuracy: 47% -> 97% | Improved |
| scale training mixture diversity and quantity (ablation) | Activation Oracle (Qwen3-8B) · controlled data ablation | accuracy | Improved |
| use layers 0% and 10% as input depth (ablation) | Activation Oracle · classification | accuracy | Hurt |
Closest papers by meaning
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
- Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data (NeurIPS 2024 · similarity 0.89)
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.89)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
- Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
