Research map / Large language models / Language, safety and interpretability

Graph-Regularized Sparse Autoencoders for LLM Safety Steering

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Mechanistic analysis of attention and representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GSAE, an SAE trained with a graph Laplacian penalty over a neuron co-activation graph, paired with a scored direction bank and two-gate runtime controller, to steer LLMs toward refusal on harmful prompts.

Problem

Standard SAEs treat latent features as independent, which may fit poorly the distributed structure of safety behaviors like refusal, making SAE-derived steering directions unstable for safety steering.

Approach

Add a Dirichlet-energy (normalized graph Laplacian) penalty on decoder columns over a neuron co-activation graph so decoder directions vary smoothly across co-activating neurons; combine with a supervised probe, a geometric-mean-scored steering bank, and input/continuation gates with hysteresis.

Key results

• On Llama-3-8B, GSAE reaches 78.9% Delta_s on JailbreakBench vs 58.8% for the matched SAE + Bank + Two-Gate pipeline (+20.1 points).
• GSAE reaches 74.1% Delta_s on HarmBench vs 57.3% for the matched SAE ablation and 77.3% on XSTest vs 58.5%.
• Under gray-box GCG attack, GSAE keeps HRR at 86.9% vs 55.3% for Llama-3.1-8B.

Surprising findings

• The steering-aware white-box stress test drops GSAE's HRR to 51.9%, showing robustness mainly holds under black-box and gray-box access.
• Without gating, GSAE raises over-refusal (13.9% SRR on XSTest), so graph-regularized directions alone are not selective.

Limitations

• Requires hidden-state access, so applies only to open-weight models.
• Co-activation graph is fixed after training and may not adapt to threat distribution shifts or fine-tuning.
• Steering applied only at source layers; cross-layer interventions unexplored.

Experimental setup

Datasets: WildJailbreak TriviaQA TruthfulQA GSM8K

Benchmarks: JailbreakBench HarmBench XSTest TriviaQA TruthfulQA GSM8K

Metrics: Delta_s (HRR - SRR) HRR SRR accuracy truthful x informative TPS

Baselines: SAE + Linear Probe SAE + Bank + Two-Gate Llama Guard in-only Llama Guard in+out CAA SafeSwitch Gradient Cuff Llama-3.1-8B-Instruct Prompt guardrails

Builds on: sparse autoencoders Laplacian regularization Contrastive Activation Addition SafeSwitch

Closest papers by meaning

• LLMs Encode Harmfulness and Refusal Separately (NeurIPS 2025 · similarity 0.85)
• Local Sparsity Enables Unsupervised LLM Safety Detection (NeurIPS 2026 · similarity 0.84)
• Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.84)
• Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (NeurIPS 2026 · similarity 0.84)
• Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.84)
• SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering (AAAI 2025 · similarity 0.84)