Research map / Architectures and efficiency / Pruning and efficient tuning
On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a mixture-of-experts theoretical account of LLaMA-Adapter's zero-initialized attention, proving least-square estimation rates for linear and non-linear (MLP-reparameterized) prompts and gating factor, and shows non-linear prompts slightly beat linear ones on Open LLM benchmarks with Alpaca fine-tuning.
Problem
The theoretical foundations of zero-initialized attention (LLaMA-Adapter) are unexplored, including how well its prompts and gating factor can be estimated and whether non-linear prompt parameterizations retain statistical guarantees.
Approach
Views zero-initialized attention as a prefix mixture-of-experts model in which pre-trained tokens are fixed experts and learnable prompts are extra experts gated by softmax, with tanh(alpha) as the gate; uses least-squares mixing-measure estimation with a Voronoi loss to derive convergence rates for linear and non-linear prompts.
Key results
- LLaMA-13B with non-linear zero-init prompts reaches 61.67 average accuracy versus 59.70 for linear prompts and 59.12 for random-init.
- On LLaMA-7B, non-linear prompts average 53.28 versus 52.83 for linear and 49.06 for random-init.
- Theory gives Voronoi-loss and gating-factor estimation rates of O_P(sqrt(log n / n)) up to fourth-root slowdowns, identical for linear and non-linear prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-7B · ARC Acc (aver) | accuracy: 52.73 -> 54.45 | Helped |
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-7B · MMLU | accuracy: 36.28 -> 36.95 | Helped |
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-7B · Hellaswag | accuracy: 76.79 -> 76.67 | Hurt |
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-7B · TruthfulQA | accuracy: 45.53 -> 45.04 | Hurt |
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-7B · Average | accuracy: 52.83 -> 53.28 | Helped |
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-13B · MMLU | accuracy: 49.64 -> 51.32 | Helped |
| Replace the linear prompt with a nonlinear prompt (ablation) | Zero-initialized attention with LLaMA-13B · TruthfulQA | accuracy: 34.88 -> 38.92 | Helped |
| Replace random initialization with zero-initialized attention | Linear prompt tuning with LLaMA-7B · Average | accuracy: 49.06 -> 52.83 | Helped |
| Replace random initialization with zero-initialized attention | Linear prompt tuning with LLaMA-13B · Average | accuracy: 59.12 -> 59.70 | Helped |
| Use a nonlinear rather than linear prompt (ablation) | Prompt tuning on LLaMA-13B · Average | accuracy: 59.70 -> 61.67 | Helped |
| Use zero-initialized linear prompts instead of random-initialized prompts | LLaMA-7B prompt tuning · Hellaswag | accuracy: 72.62 -> 76.79 | Helped |
| Use zero-initialized linear prompts instead of random-initialized prompts | LLaMA-7B prompt tuning · TruthfulQA | accuracy: 37.82 -> 45.53 | Helped |
Surprising findings
- With LLaMA-13B on 1% of Alpaca data, linear prompts outperform non-linear prompts by 1.99 points.
- On LLaMA-13B, random-init prompts perform close to zero-init, with the authors attributing this to larger model capacity.
Limitations
- Evaluated only on unimodal LLMs, not multimodal LLMs.
- Did not systematically explore adapter integration or the optimal number of prompt embeddings across layers.
- Focuses on performance metrics; interpretability and stability of non-linear prompts are not analyzed.
Experimental setup
Datasets: Alpaca ARC HellaSwag MMLU TruthfulQA
Benchmarks: ARC MMLU HellaSwag TruthfulQA
Metrics: accuracy
Baselines: LoRA Full fine-tuning Prompt Tuning IA3 VeRA Random-Init prompt
Builds on: LLaMA-Adapter prefix tuning LoRA
Closest papers by meaning
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.86)
- OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning (ICML 2025 · similarity 0.85)
- Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (ICLR 2025 · similarity 0.85)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.85)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.85)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.