Research map / Large language models / Instruction and fine-tuning

Comparing Explanations is not Enough,Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper proposing Comparative XAI, which treats the behavioral shift between a reference and an updated LLM checkpoint as the object of explanation, with ten desiderata and an illustrative Qwen2.5-0.5B fine-tuning case study.

Problem

Single-checkpoint XAI cannot attribute behavioral shifts in LLMs to interventions, and existing comparative approaches compare independent explanations without a principled standard, which creates gaps for governance and audit documentation under AI regulations.

Approach

Define Comparative XAI as an explainer mapping (reference model, updated model, intervention) to an explanation of the change itself, and specify ten desiderata (comparability, validity, actionability, monitoring) that such explanations should meet, illustrated with delta-attribution, CKA, activation patching, steering, SAEs and data attribution on a narrow medical fine-tune.

Key results

• Linear CKA localizes the largest divergence between Qwen2.5-0.5B pre- and post-fine-tune models at the third-to-last layer (layer 22), with layer-wise profiles correlating at r=0.99 across prompt scaling and seeds.
• Activation patching of layer 22 raises LLM-judge safety score from 2.07 to 3.67 (reference 4.07); off-target layer 3 patching yields a 2.4x smaller effect.
• Steering with a probe direction (alpha=15) shifts post-fine-tune outputs toward safer advice without changing weights; the probe reaches test accuracy above 0.95.

Surprising findings

• Off-target delta-attribution analysis is dominated by a single token ('mild') rather than the salient 'headache' seen in single-checkpoint attribution.
• Procedural non-medical in-context learning increased disclaimer-style medical markers by 0.49 per response on average, a shift without medical demonstrations.

Limitations

• Evidence is illustrative, from a single small model and a 15-prompt evaluation set, with no quantitative benchmark for the proposed desiderata.
• Several analyses rely on an LLM judge (GPT-5.3) and manual labelling of SAE features.
• Desiderata are not operationalized into measurable tests; the paper argues for standards rather than a validated method.

Experimental setup

Datasets: narrow medical fine-tuning dataset GPT-5.2-expanded medical-advice prompts (400)

Metrics: linear CKA Pearson correlation of layer-wise divergence cosine similarity sentence embedding similarity LLM-as-a-judge safety score (1-5) probe test accuracy SAE reconstruction MSE urgent-care recommendation rate overconfidence score disclaimer score

Baselines: single-checkpoint XAI endpoint-comparison of explanations

Builds on: Integrated Gradients linear CKA LoRA sparse autoencoders emergent misalignment

Closest papers by meaning

• Thought Branches: Interpreting LLM Reasoning Requires Resampling (ICLR 2026 · similarity 0.86)
• iLLuMinaTE: An LLM-XAI Framework Leveraging Social Science Explanation Theories Towards Actionable Student Performance Feedback (AAAI 2025 · similarity 0.86)
• Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.86)
• Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.86)
• Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes (AAAI 2024 · similarity 0.86)
• Can large language models explore in-context? (NeurIPS 2024 · similarity 0.85)