Research map / Large language models / Instruction and fine-tuning
Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JoLA, which jointly learns which attention heads to edit and whether to apply additive, multiplicative, or both interventions via Hard-Concrete gates with expected-L0 regularization, for low-resource fine-tuning of LLMs.
Problem
Activation editing methods for LLM adaptation are sensitive to which modules are edited and to the intervention type, giving inconsistent performance across datasets in low-resource settings of a few hundred examples.
Approach
Jointly learn head localization and intervention type end-to-end: each attention head gets a hybrid multiplicative-plus-additive edit gated by two Hard-Concrete gates trained with expected-L0 sparsity, replacing LoFIT's two-stage manual head selection.
Key results
- On Llama-3.1-8B-Instruct, JoLA averages 70.55% accuracy on commonsense reasoning versus 66.58% for LoRA, the best baseline.
- On Qwen2.5-7B-Instruct, JoLA reaches 82.40% reasoning accuracy and 51.57% understanding accuracy, above LoRA's 78.28% and 46.22%.
- JoLA reports the best BLEU, ROUGE-L and BERTScore on generation tasks for both models, e.g. Qwen ROUGE-L 50.23 vs 45.34 for LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use JoLA instead of LoRA | Llama-3.1-8B-Instruct · Commonsense reasoning average | accuracy: 66.58 -> 70.55 | Helped |
| use JoLA instead of LoRA | Llama-3.1-8B-Instruct · Understanding average | accuracy: 42.07 -> 47.00 | Helped |
| use JoLA instead of LoRA | Llama-3.1-8B-Instruct · Generation average | BLEU: 13.27 -> 17.07 | Helped |
| use JoLA instead of LoRA | Qwen2.5-7B-Instruct · Commonsense reasoning average | accuracy: 78.28 -> 82.40 | Helped |
| use JoLA instead of LoRA | Qwen2.5-7B-Instruct · Understanding average | accuracy: 46.22 -> 51.57 | Helped |
| use JoLA instead of LoRA | Qwen2.5-7B-Instruct · Generation average | BLEU: 19.46 -> 24.00 | Helped |
| add gating to attention interventions (ablation) | JoLA attention intervention · SIQA, WinoGrande, Law, Physics, E2E_NLG, WEB_NLG | reported task score: 55.94 -> 66.22; 55.33 -> 58.33; 36.00 -> 40.00; 7.00 -> 46.00; 14.77 -> 15.54; 18.12 -> 24.39 | Helped |
| add gating to MLP interventions (ablation) | JoLA MLP intervention · SIQA, WinoGrande, Law, Physics, E2E_NLG, WEB_NLG | reported task score: 50.10 -> 52.46; 51.62 -> 52.43; 34.00 -> 36.00; 20.00 -> 23.00; 10.31 -> 11.23; 14.45 -> 16.25 | Helped |
| add gating to combined attention and MLP interventions (ablation) | JoLA combined intervention · SIQA, WinoGrande, Law, Physics, E2E_NLG, WEB_NLG | reported task score: 52.17 -> 53.28; 48.74 -> 52.07; 23.00 -> 27.00; 13.00 -> 16.00; 8.23 -> 10.42; 12.36 -> 14.83 | Helped |
| use separate gates for additive and multiplicative vectors instead of a shared gate (ablation) | JoLA · Reported ablation tasks | performance | Helped |
| use additive offsets rather than scaling interventions (ablation) | activation editing · Reported component and operation comparisons | performance | Helped |
| intervene on attention heads rather than MLP layers or hidden states (ablation) | activation editing · Reported component comparisons | performance | Helped |
Surprising findings
- Combining interventions across multiple components (attention plus MLP) degrades performance, which the authors interpret as overfitting.
- Bias offsets contribute more to performance than multiplicative scaling in the component ablation.
Limitations
- Baseline hyperparameter search is limited to five configurations per task, and results depend on early stopping and a chosen learning-rate schedule.
- Performance gap to LoRA narrows or reverses at larger data sizes (20,000 to 100,000 examples).
Experimental setup
Datasets: ARC-c ARC-e BoolQ HellaSwag OBQA PIQA SIQA WinoGrande MMLU-Pro CommonGen E2E NLG WebNLG
Benchmarks: ARC-c ARC-e BoolQ HellaSwag OBQA PIQA SIQA WinoGrande MMLU-Pro CommonGen E2E NLG WebNLG
Metrics: accuracy BLEU ROUGE-L BERTScore
Baselines: LoRA BitFit RED RePE ReFT LoFIT SMP DSP PASS
Builds on: LoFIT ReFT BitFit LoRA expected-L0 regularization
Closest papers by meaning
- LoFiT: Localized Fine-tuning on LLM Representations (NeurIPS 2024 · similarity 0.89)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.88)
- Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024 · similarity 0.88)
- Efficient Model Editing with Task-Localized Sparse Fine-tuning (ICLR 2025 · similarity 0.87)
- HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · similarity 0.87)
- Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.