Research map / Large language models / Instruction and fine-tuning

Training Language Models to Explain Their Own Computations

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes LMs to verbalize three kinds of interpretability ground truth (SAE feature descriptions, activation patching outcomes, input ablation outcomes) and finds models explain themselves better than other explainers, with data efficiency gains.

Problem

Verbalized LM explanations often fail to reflect internal computations; it is unclear whether models can be trained to faithfully describe their own internal features, causal interventions, and decision rules.

Approach

Use interpretability procedures (auto-interp, activation patching, input ablation) as supervision and train explainer LMs to output the resulting explanations, comparing self-explainers (same model as target) against other explainers to test a privileged access hypothesis.

Key results

• Llama-3.1-8B self-explainer reaches 76.2 LM judge and 45.1 simulator score on held-out SAE features, matching gold labels' simulator score (43.3) and beating nearest neighbors (58.5 LM judge).
• At 0.8% of training data (1024 samples per layer), self-explanation reaches 71-89% of end performance, while Qwen3-8B reaches 24-66% and nearest neighbors 55-75%.
• For input ablation on Qwen3-8B, the self-explainer achieves 83.4 exact match vs 58.1 for Llama-3.1-8B trained as explainer and 8.9 for untrained Qwen3-8B.

Surprising findings

• Self-trained explainers beat gold Neuronpedia SAE labels under simulator scoring, attributed to noise in the gold labels.
• Untrained Qwen3-8B reports a has-changed value of True only 8.6% of the time, indicating pre-trained LMs do not verbalize hint use without fine-tuning.

Limitations

• Target model is fixed; only the explainer is fine-tuned, so this is not strict self-consistency.
• LM judge and gold SAE labels are noisy; 27.7% of judge-zero cases were genuine explainer errors.
• Privileged access inferred from comparisons across a limited set of explainer models and target models.

Experimental setup

Datasets: Llama Scope SAE features with Neuronpedia labels FineWeb CounterFact MMLU

Benchmarks: Llama Scope SAE held-out features FineWeb full activations CounterFact activation differences MMLU hint input ablation

Metrics: LM judge score simulator correlation exact match has-changed F1 content match

Baselines: Nearest Neighbors SelfIE Gold SAE Labels Qwen3-8B (Untrained) Llama-3.1-70B (LoRA)

Builds on: SelfIE Latent Interpretation Tuning Neuronpedia auto-interp Llama Scope

Closest papers by meaning

• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (ICML 2024 · similarity 0.89)
• Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.89)
• Unraveling the Influence of Training Data and Internal Structures in Large Language Models for Enhanced Explainability (Student Abstract) (AAAI 2025 · similarity 0.89)
• Monitoring Latent World States in Language Models with Propositional Probes (ICLR 2025 · similarity 0.88)
• Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.88)
• Looking Inward: Language Models Can Learn About Themselves by Introspection (ICLR 2025 · similarity 0.88)