Research map / Large language models / AI and society
Safe Evolution with Circuit Anchors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Circuit-Anchored Evolution (CAE), which identifies a small safety circuit of transcoder features via circuit tracing and adds a KL anchor on those features during self-evolution (EVOL-RL, Abs-ZERO) to preserve refusal behavior while capability evolves.
Problem
Self-evolution algorithms for LLMs optimize only for capability, causing safety (refusal) to degrade over training ("misevolution"). The paper asks how to enable capability evolution while preventing safety degradation.
Approach
Identify a tiny set of transcoder features (<2% of features) that causally mediate refusal, then anchor their activation distributions with a KL penalty during evolution while leaving other features free; this implicit internal constraint is compared against explicit reward-model-based safety constraints.
Key results
- Unconstrained EVOL-RL drops Qwen3-4B-Instruct safety from 99.04% to 69.85%, Gemma-2-2B-IT from 98.65% to 75.38%, and Llama-3.2-1B-Instruct from 96.22% to 72.13%.
- CAE keeps safety above 95% throughout evolution while capability gains match unconstrained evolution, per the paper's claims.
- On Gemma-2-9B-IT over 500 steps, CAE reaches 98.47% safety vs 69.85% unconstrained with capability 64.58% vs 64.72%.
Surprising findings
- Amplifying safety-circuit activations beyond natural levels produces only modest further refusal gains, suggesting the circuit is near saturation under natural conditions.
- Explicit reward-model safety constraints slow capability gains, while circuit anchoring does not.
Limitations
- Theoretical safety bound is conditional on a bounded anchor loss and a Lipschitz assumption that the paper itself only loosely justifies.
- Evaluation covers only refusal on AdvBench and two evolution algorithms on math-focused tasks with small models mostly.
- Reliance on available pre-trained transcoders restricts which model families can be studied.
Experimental setup
Datasets: Do-Not-Answer DAN Malicious Instruct AdvBench Alpaca
Benchmarks: AdvBench Alpaca over-refusal
Metrics: refusal rate safety score capability score (average accuracy) over-refusal rate
Baselines: unconstrained evolution Beaver-Cost reward-model constraint
Builds on: GRPO Abs-ZERO EVOL-RL circuit tracing transcoders
Closest papers by meaning
- Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents (ICLR 2026 · similarity 0.84)
- FunctionEvolve: Structure-Guided Symbolic Regression with LLMs (NeurIPS 2026 · similarity 0.82)
- Knowledge-aware Reinforced Language Models for Protein Directed Evolution (ICML 2024 · similarity 0.82)
- Convergent Functions, Divergent Forms (NeurIPS 2025 · similarity 0.82)
- Contrastive losses as generalized models of global epistasis (NeurIPS 2024 · similarity 0.81)
- Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.