Research map / Large language models / Language, safety and interpretability
Unraveling the Influence of Training Data and Internal Structures in Large Language Models for Enhanced Explainability (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes using FAISS nearest-neighbor pre-selection to make influence-function estimation of training-data influence on GPT-2 Small tractable, with preliminary results on a movie review task.
Problem
Computing influence functions over full LLM training sets is computationally prohibitive (Hessian cost), and it is unclear how training data and internal components drive LLM predictions, hallucination, bias, and generalization.
Approach
Use FAISS semantic similarity search to restrict influence-function computation to the top-100 nearest training samples per test sample, and compare against random and TF-IDF selection; plan to pair this with edge-attribution mechanistic interpretability for internal components.
Key results
- Removing 1% of random training data raised GPT-2 Small accuracy from 82% to 84% on Movie Review.
- Removing TF-IDF-selected neighbors left accuracy unchanged at 82%.
- Removing FAISS-selected neighbors dropped accuracy from 82% to 80%, suggesting semantically related points matter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove 1% of training data selected randomly | GPT-2 Small · Movie Review | accuracy: 82% -> 84% | Helped |
| Remove 1% of training data selected by TF-IDF word-level similarity | GPT-2 Small · Movie Review | accuracy: 82% -> 82% | No clear effect |
| Remove 1% of training data selected by FAISS semantic similarity | GPT-2 Small · Movie Review | accuracy: 82% -> 80% | Hurt |
Surprising findings
- Random removal of 1% of training data increased accuracy by 2%, which the authors attribute to removing noisy samples.
Limitations
- Very small preliminary evaluation on a single dataset and single model, with differences of 2% and no variance reported.
- Simpler surrogate models approximating LLMs are proposed but not evaluated.
- RQ2 and RQ3 (usage modes and mechanistic interpretability) are not yet empirically addressed.
Experimental setup
Datasets: Movie Review
Benchmarks: Movie Review
Metrics: accuracy before and after influential-sample removal
Baselines: random selection TF-IDF nearest neighbor
Builds on: influence functions (Koh and Liang 2017) edge attribution patching FAISS
Closest papers by meaning
- Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.89)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.87)
- Large Language Models are Interpretable Learners (ICLR 2025 · similarity 0.87)
- Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.87)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.86)
- Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.