Research map / Large language models / Language, safety and interpretability

Mechanistic Circuit Identification for Controllable Data Generation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a circuit-grounded data generation framework that discovers EAP-IG circuits tied to AUM, EL2N, and GradAlign training-utility metrics on SciQ, steers them to generate utility-targeted MCQA data, and schedules the pools with a stage-aware curriculum (SAMS).

Problem

Prompt-based, black-box data synthesis gives limited insight into how individual samples shape a model's learning dynamics and offers only coarse control over training-relevant data properties.

Approach

Uses mechanistic circuits discovered by EAP-IG (with Gaussian-noise counterfactuals and a logit-margin target) for training-utility metrics as causal steering interfaces (activation addition and attention steering) to generate utility-targeted synthetic data, then schedules these pools across fine-tuning stages.

Key results

• SAMS reaches 85.8% accuracy on in-domain SciQ at 60% source ratio with ECE 0.055, versus 83.4% for source-only training.
• Circuit ablation collapses AUM and EL2N high/low score separation (GapRed 98-99.6%) while random-edge ablation gives 16-27% gap reduction.
• On ARC-Easy at 70% source ratio, SAMS achieves 74.9% accuracy versus 72.9% for prompt-based uniform mixing.

Surprising findings

• Filtering by the original GradAlign score severely degrades accuracy (66.2%) while the contrastive circuit score yields 87.7%.
• Ablating the GradAlign circuit causes negative gap reduction (-23.7% for High), indicating the separation does not collapse under zero-style ablation.

Limitations

• Evaluation limited to a single task (SciQ MCQA) and one model family (Qwen2.5).
• Steering and selection are tuned with a handful of hyperparameters and the paper shows mixed results (e.g., SAMS not best at 70% SciQ accuracy).
• Circuit discovery requires fine-grained interpretability tooling and per-metric training runs, limiting scalability.

Experimental setup

Datasets: SciQ ARC-Easy

Benchmarks: SciQ ARC-Easy

Metrics: accuracy ECE NLL Abs-CPR gap reduction ratio Vendi G-Vendi perplexity recall

Baselines: Prompt-based generation (Uniform-Mix) Stage-Aware Scheduling with prompt-generated data Source Only (Full) Random filtering OriS filtering

Builds on: EAP-IG Representation Engineering (RepE) AUM EL2N GradAlign

Closest papers by meaning

• Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.84)
• Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery (NeurIPS 2026 · similarity 0.84)
• Compact Proofs of Model Performance via Mechanistic Interpretability (NeurIPS 2024 · similarity 0.84)
• Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference (AAAI 2025 · similarity 0.84)
• Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis (ICML 2025 · similarity 0.84)
• Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization (NeurIPS 2026 · similarity 0.83)