Research map / Large language models / Language, safety and interpretability
CALM: Curiosity-Driven Auditing for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALM, which fine-tunes a GPT-2 audit LLM with PPO plus a token-level policy-cover intrinsic bonus to find input-output pairs where black-box target LLMs produce toxic completions about celebrities or specific senator names.
Problem
Auditing black-box LLMs for illegal, immoral, or unsafe behaviors is hard because harmful outputs are sparse, the prompt space is discrete and large, and hand-crafted prompts have limited coverage.
Approach
Treat auditing as black-box optimization and use curiosity-driven RL: an audit LLM is rewarded both by an extrinsic auditing objective (name hit or toxic output) and by a token-level policy-cover intrinsic bonus computed from random-network prediction errors on token embeddings of the audit LLM, with no access to the target's parameters.
Key results
- CALM exceeds the baselines RL and CRT by over 35% in GPT-2 and over 50% in Llama-3-8B on the senator-related toxic completion task.
- CALM reaches over 80% auditing objective on Llama-3-8B with about 1.5x10^4 queries and attains 50% with about 1x10^4 queries.
- CALM shows consistently lower variance than RL on NameSet coverage, particularly for Llama-3-8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use curiosity-driven CALM instead of RL | Black-box auditing of target LLMs · Inverse suffix generation across GPT-2, Dolly-2-7B, Llama-2-7B, and Llama-3-8B | L0 norm of NameSet coverage | Helped |
| Use curiosity-driven CALM instead of RL or CRT | Black-box auditing of target LLMs · Senator-related toxic completion across tested models | auditing objective: over 35% in the GPT-2 and LLAMA3 models, respectively, 50% | Helped |
| Use curiosity-driven CALM instead of RL and CRT | Black-box auditing of Llama-3-8B · Senator-related toxic completion | queries to reach auditing objective: approximately $1.5\times10^4$ queries | Helped |
| Use curiosity-driven CALM instead of baseline methods | Black-box auditing of Llama-3-8B · Senator-related toxic completion | queries to reach 50% auditing objective: 50% accuracy rate with just $1\times10^4$ queries | Helped |
| Add intrinsic rewards (ablation) | CALM auditing the Llama-3-8B target · Inverse suffix generation, intrinsic coefficient $\lambda=100$ | auditing objective | Helped |
| Add intrinsic rewards (ablation) | CALM auditing the Llama-3-8B target · Inverse suffix generation, intrinsic coefficient $\lambda=100$ | L0 norm of set coverage | Helped |
| Add sentence-level diversity score | Vanilla PPO · Senator-related toxic completion | auditing objective | Hurt |
Surprising findings
- A small GPT-2 audit model (with only its last two transformer blocks fine-tuned) can uncover undesired behaviors of much larger target LLMs such as Llama-3-8B.
Limitations
- Uses only a lightweight GPT-2 audit backbone; a more powerful auditor may improve results (not tested).
- Toxicity is detected with a simple NSFW word-list classifier, which misses toxicity without listed terms.
- Only two auditing objectives (inverse suffix generation and toxic completion) and two baselines are evaluated.
Experimental setup
Datasets: senator names and affiliations (inverse suffix task) English NSFW word list (LDNOOBW GitHub)
Benchmarks: inverse suffix generation (senator names) senator-related toxic completion
Metrics: auditing objective success rate L0 norm of NameSet coverage entropy of set coverage
Baselines: RL (LLM-assisted red teaming with PPO) CRT
Builds on: PPO policy cover random network distillation generalized advantage estimation
Closest papers by meaning
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.87)
- Eliciting Language Model Behaviors with Investigator Agents (ICML 2025 · similarity 0.87)
- HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.87)
- Eliciting Secret Knowledge from Language Models (NeurIPS 2026 · similarity 0.86)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.86)
- Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.