Research map / Large language models / Language, safety and interpretability
LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LayerIF, which uses layer-wise influence function scores (DataInf) as a data-driven layer quality estimate to allocate LoRA-MoE experts and set layer-wise sparsity for LLM pruning.
Problem
Existing layer quality estimates for LLMs rely on model-only heuristics (spectral statistics, outliers, uniform allocation) and ignore training data, limiting layer-wise expert allocation in LoRA-MoE and layer-wise sparsity for pruning.
Approach
Compute per-layer influence function scores (with DataInf Hessian approximation) between training and validation samples, sum positive influences per Transformer block, and map these task-specific scores to expert counts or layer sparsity ratios.
Key results
- LayerIF for LoRA-MoE expert allocation reaches 82.27 average accuracy (Top 25%) vs 80.97 for best MoLA baseline on Mistral-7B, a reported 1.61% relative gain.
- For 50% pruning on Mistral-7B with SparseGPT, LayerIF gets 60.61 vs 60.07 for AlphaPruning and 59.99 for OWL.
- Reversed allocation at 70% sparsity lowers average accuracy from 35.69 to 34.52, supporting that low-IF layers should be preserved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LayerIF Top 25% influence samples for expert allocation instead of the strongest baseline | LoRA-MoE expert allocation · Average accuracy across five datasets | accuracy: 82.27 vs 80.97 | Helped |
| use LayerIF +ve for layer-wise sparsity allocation instead of OWL | Mistral-7b-v0.1 pruning · 50% sparsity, average across pruning methods | average accuracy: 58.81 vs 58.30 | Helped |
| use LayerIF +ve for sparsity allocation instead of AlphaPruning | Gemma-7b pruning · 50% sparsity | average accuracy: 45.62 vs 45.45 | Helped |
| retain only the top 25% most influential positive samples rather than all positive samples (ablation) | LayerIF expert allocation · Average accuracy across five datasets | accuracy: 81.76 -> 82.27 | Helped |
| include samples of either influence sign instead of only positive-influence samples (ablation) | LayerIF expert allocation · Average accuracy across five datasets | accuracy: 81.76 -> 81.85 | No clear effect |
| reverse the allocation so lower-sensitivity layers receive higher pruning ratios (ablation) | LayerIF pruning · Mistral-7b-v0.1 at 70% sparsity | average accuracy: 35.69 -> 34.52 | Hurt |
| compute influence scores per Transformer block rather than per matrix (ablation) | LayerIF pruning · LLaMA2-7B at 70% sparsity across three pruning methods | average accuracy: 2.14% average improvement | Helped |
| apply Savitzky–Golay smoothing to normalized influence scores (ablation) | LayerIF pruning · Mistral-7b-v0.1 at 70% sparsity, epsilon 0.1–0.5 | zero-shot accuracy: (+) 1.49; (+) 0.94; (+) 0.97; (+) 0.52; (+) 1.15; (+) 0.46; (+) 0.03; (+) 0.05; (+) 0.41; (+) 1.81; (-) 0.69; (+) 0.48; | Mixed |
| use Hessian-free TracIn instead of DataInf for influence estimation (ablation) | LayerIF expert allocation · MRPC, CoLA, Openbook Q, Text Science Q, Common Q | Spearman correlation: -0.22; 0.44; 0.09; -0.69; 0.12 | Mixed |
| use a Gauss-Newton Hessian (EK-FAC) instead of the DataInf approximation (ablation) | LayerIF influence estimation · Influence computation | runtime: 160× slower | Hurt |
| vary total expert count across 80, 160, and 224 (ablation) | LayerIF expert allocation on Mistral-7b-v0.1 · Average accuracy across expert-count settings | accuracy: 3.64% | Helped |
Surprising findings
- Hessian-free TracIn yields near-zero or negative Spearman correlation with DataInf-based layer allocations (e.g., -0.69 on Text Science Q), so it fails to capture layer quality.
- Gauss-Newton (EK-FAC) Hessian was about 160x slower than DataInf in the authors' experiments.
Limitations
- Influence functions rely on an implicit convexity assumption that may not hold for LLMs.
- Hessian-based IF computation is computationally expensive, especially for large datasets.
- IF estimates depend on representativeness of the train and validation subsets used.
Experimental setup
Datasets: MRPC CoLA ScienceQA CommonsenseQA OpenBookQA BoolQ HellaSwag Winogrande ARC Easy ARC Challenge RTE
Benchmarks: MRPC CoLA ScienceQA CommonsenseQA OpenBookQA BoolQ HellaSwag Winogrande ARC Easy ARC Challenge RTE
Metrics: zero-shot accuracy average accuracy Spearman correlation p-value (t-test)
Baselines: AlphaLoRA MoLA-2468 MoLA-5555 MoLA-8642 Uniform AlphaPruning OWL
Builds on: DataInf MoLA AlphaLoRA AlphaPruning OWL
Closest papers by meaning
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.90)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.90)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.90)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.90)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.89)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.