Research map / Large language models / Language, safety and interpretability

Finding Interpretable Prompt-Specific Circuits in Language Models

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ACC++, a refined single-forward-pass circuit-tracing method built on attention-causal communication that yields smaller, interpretable, prompt-specific circuits, and applies it to IOI prompt sensitivity and multilingual IOI.

Problem

Circuit tracing in language models is noisy and typically relies on replacement models or activation patching, making per-prompt explanations of why attention heads attend where they do hard to obtain and interpret.

Approach

Generates candidate signals as the cross-product of all upstream component outputs with all QK-SVD singular directions, scores them by Integrated Gradients on the true post-Softmax attention weight, and greedily removes top candidates until attention falls below a threshold; also unifies bias and RoPE into one QK matrix per head.

Key results

• ACC++ circuits are 2x to 10x smaller than ACC circuits (e.g., Gemma-2 2B IOI: 688 to 87 nodes, 7086 to 215 edges).
• 63% of GPT-2 Small, 50% of Pythia-160M, and 31% of Gemma-2 2B ACC++ signals are labeled interpretable under the FDR-controlled criterion.
• Cross-language mGPT circuit distances correlate with URIEL+ syntactic (r=0.83) and genetic (r=0.88) distances.

Surprising findings

• GPT-2 circuits are sensitive to role order (ABBA vs BABA) while Pythia circuits are more sensitive to low-level surface wording.
• Name-mover head (9,9) receives markedly different (negatively correlated) incoming signals across ABBA and BABA templates despite the same token.

Limitations

• Threshold tau relies on a heuristic per model/dataset rather than a principled rule.
• FFN-caused attention-causal signals are not traced upstream.
• Interpretations rely on an LLM judge and multilingual correlation is based on only six language pairs.

Experimental setup

Datasets: IOI prompts (3000 balanced templated prompts) multilingual IOI dataset (English, Spanish, French, Portuguese) The Pile (via pile-10k)

Benchmarks: IOI Greater-Than Gender Pronoun

Metrics: precision recall F1 node and edge counts fraction of interpretable signals (FDR<=5%) autointerpretation accuracy/precision/recall Pearson correlation with URIEL+ distances

Baselines: ACC Path Patching EP ACDC EAP

Builds on: attention-causal communication (ACC) integrated gradients EAP-IG path patching

Closest papers by meaning

• Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024 · similarity 0.90)
• Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.88)
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · similarity 0.88)
• Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers (NeurIPS 2025 · similarity 0.88)
• Interpreting the Repeated Token Phenomenon in Large Language Models (ICML 2025 · similarity 0.87)
• Transcoders find interpretable LLM feature circuits (NeurIPS 2024 · similarity 0.87)