Research map / Large language models / Language, safety and interpretability
First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation
Dmytro Vitel, Anshuman Chhabra
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Vote instead of Mean aggregation of layer scores | DataInf · Noise detection vs accuracy correlation (Mistral 7B) | Spearman correlation of NDR: 0.5 -> 0.9 | Improved |
| use Vote instead of Mean aggregation | TracIn · Mistral 7B | layer rank of classification head: 12 -> 1 | Improved |
| use Vote instead of Mean aggregation | TracIn · Roberta-Large | layer rank of classification head: 10 -> 2 | Improved |
| use Vote instead of Mean aggregation | DataInf layers 00-07 · Llama/GLUE filtering | win rate: 0.84 | Improved |
| use Vote aggregation instead of Mean | Cosine · Mistral 7B GLUE | accuracy: degraded | Hurt |
| use Vote instead of Mean aggregation | DataInf layers 18-23 · Roberta-Large COLA | test accuracy: 83.3 -> 79.1 | Hurt |
| use Vote instead of Mean aggregation | DataInf layers 18-23 · Roberta-Large QNLI | test accuracy: 87.4 -> 87.7 | Improved |
| filter with influence scores versus random | DataInf Mean layers 18-23 · Roberta-Large RTE | test accuracy: 57.8 -> 60.5 | Improved |
| select cancellation-based low-C layers as influence estimator | gradient cancellation C · Roberta-Large/Llama/Mistral | Spearman correlation with accuracy: 0.2 | No clear effect |
| use attention layers rather than classification head | TracIn · Mistral 7B | win rate: .20 -> .61 | Improved |
Closest papers by meaning
- LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions (NeurIPS 2025 · similarity 0.90)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.89)
- Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation (NeurIPS 2026 · similarity 0.89)
- Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data (NeurIPS 2025 · similarity 0.89)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.89)
- Small-to-Large Generalization: Training Data Influences Models Consistently Across Scale (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
