Research map / Large language models / Language, safety and interpretability
ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, Lyle Ungar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ZeroTuning to Llama-3.1-8B-Instruct | Llama-3.1-8B-Instruct · Text classification (7 datasets) | accuracy: 59.59 -> 71.44 | Improved |
| add ZeroTuning to Qwen-2-7B | Qwen-2-7B · Text classification (7 datasets) | accuracy: 55.10 -> 68.19 | Improved |
| add ZeroTuning to DeepSeek-R1-14B | DeepSeek-R1-14B · Text classification (7 datasets) | accuracy: 67.67 -> 71.87 | Improved |
| add ZeroTuning to Llama-3.1-8B-Instruct | Llama-3.1-8B-Instruct · Multiple-choice QA (7 datasets) | accuracy: 58.84 -> 61.48 | Improved |
| add ZeroTuning to Qwen-2-7B | Qwen-2-7B · Multiple-choice QA (7 datasets) | accuracy: 63.10 -> 64.84 | Improved |
| add ZeroTuning to DeepSeek-R1-14B | DeepSeek-R1-14B · Multiple-choice QA (7 datasets) | accuracy: 60.05 -> 62.20 | Improved |
| add ZeroTuning to Llama-3.1-8B-Instruct | Llama-3.1-8B-Instruct · MT-Bench | score: 7.804 -> 7.966 | Improved |
| add ZeroTuning to Llama-2-13B-Chat | Llama-2-13B-Chat · MT-Bench | score: 6.650 -> 6.916 | Improved |
| ZeroTuning vs Auto-PASTA on Llama-3.1-8B-Instruct classification | ZeroTuning · Text classification (7 datasets) | accuracy: 63.73 -> 71.44 | Improved |
| use head-specific scaling (Up or Down heads) instead of uniform scaling of all heads (ablation) | ZeroTuning · SST-2 and MMLU | accuracy | Improved |
| add hybrid Up+Down head scaling on top of Up or Down alone (ablation) | ZeroTuning · SST-2 and MMLU | accuracy | No clear effect |
| use unsupervised entropy-minimization calibration instead of labeled calibration (ablation) | ZeroTuning · 5 datasets (Llama-3.1-8B-Instruct) | accuracy: 67.52 -> 67.02 | No clear effect |
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.91)
- Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.89)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.89)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.89)
- Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
